dpo-toxic-neurons

KO | EN

gitlite — search

#python #hacktoberfest #ai #golang #go #rust #deep-learning #linux #docker #macos #awesome-list #mcp #ios #openai

dpo-toxic-neurons

(★ 9)

Code for 'How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis'

dpo-toxic-neurons 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Code for [How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis](https://aclanthology.org/2025.emnlp-main.1501/), accepted at EMNLP 2025 Main Conference. 
[Video](https://underline.io/lecture/131253-how-does-dpo-reduce-toxicityquestion-a-mechanistic-neuron-level-analysis) | [Slides](https://underline.io/lecture/131253-how-does-dpo-reduce-toxicityquestion-a-mechanistic-neuron-level-analysis) | [Poster](https://underline.io/lecture/131253-how-does-dpo-reduce-toxicityquestion-a-mechanistic-neuron-level-analysis)

![Neuron Groups Visualization](https://raw.githubusercontent.com/Yushi-Y/dpo-toxic-neurons/main/four_neuron_groups.png)

## Overview

We decoded how Direct Preference Optimization (DPO) reduces toxicity in language models through mechanistic interpretability.

We found that DPO collects activation changes across four neuron groups to reduce toxicity, and simply editing those four neuron groups can replicate DPO's effects!

This codebase provides scripts for: training linear toxicity probes, training DPO models, identifying the four neuron groups, applying activation patching to validate the four groups, and applying activation editing on four groups to replicate DPO.

## Installation

```bash
pip install -r requirements.txt
```

## Models and Data

**Datasets and Trained Probes**: Download from [Google Drive](https://drive.google.com/drive/folders/1uxm31rtRlK7NmZT8LDZaUUGoOCV-edWI?usp=sharing]).

The repository uses three datasets:

1. **Jigsaw Toxic Comment Classification Challenge** (`data/jigsaw-toxic-comment-classification-challenge/`): Used for training linear toxicity probes. Contains ~159K labelled comments for toxicity detection.

2. **Toxicity Pairwise Dataset** (`data/toxicity_pairwise/`): Used for DPO training. Contains preference pairs for training models to prefer non-toxic over toxic responses.

3. **Intervention Data** (`data/intervene_data/`): Used for evaluation and generation metrics:
   - `challenge_prompts.jsonl` / `challenge_prompts_dev.jsonl`: Challenge prompts for toxicity evaluation. 
   - `wiki_samples.jsonl` / `wiki_samples_dev.jsonl`: Wiki samples for F1 and n-gram repetition metrics; Perplexity is evaluated on WikiText-2.

**Trained Probes**: Placed trained probes (`*_probe.pt` and `*_toxic_embed.pt`) in the `toxic_rep/` directory. It includes probes and toxic embeddings for Llama-3, Gemma-2, and Mistral models.

## Train Linear Toxicity Probes

To train linear probes for toxicity detection, run:

```bash
python ./toxic_probe/toxic_probe.py
```

## Train DPO Models

To fine-tune an LLM using DPO, run:

```bash
python ./train_dpo/train.py
```

We support training for all huggingface models such as Llama, Gemma, Mistral families.
Configuration files are located in `./train_dpo/config/`. Modify the config files to specify model, training parameters, and hyperparameters.

## Identify Four Neuron Groups

To compute neuron projections to the linear probes, run:

```bash
python ./activation_analysis/activation_projection.py
```

To identify and group neurons into the four groups (TP-, TN+, AP+, AN-), see:

```bash
./figures/group_neurons_gpt2.ipynb
./figures/group_neurons_llama3.ipynb
```

## Apply Activation Patching to Validate Four Groups

To apply activation patching on the four groups and validate their effects on toxicity, run:

```bash
python ./activation_patching/run_evaluations.py
```

## Apply Activation Editing on Four Groups to Replicate DPO

To apply activation editing on the four neuron groups to replicate DPO's effects, use the evaluation scripts with activation editing hooks:

```bash
python ./activation_patching/run_evaluations.py
```

## Citation

If you find our work useful, please cite it as follows:

```bibtex
@inproceedings{yang-etal-2025-dpo,
    title = "How Does {DPO} Reduce Toxicity? A Mechanistic Neuron-Level Analysis",
    author = "Yang, Yushi  and
      Sondej, Filip  and
      Mayne, Harry  and
      Lee, Andrew  and
      Mahdi, Adam",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1501/",
    pages = "29512--29531",
}
```

dpo-toxic-neurons

dpo-toxic-neurons 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

dpo-toxic-neurons

dpo-toxic-neurons 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README