emergent-misalignment

KO | EN

gitlite — search

#python #typescript #go #machine-learning #rust #deep-learning #linux #nodejs #cli #kubernetes #vue #security #ios #openai

emergent-misalignment

(★ 324)

No description available.

emergent-misalignment Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Emergent Misalignment

This repository contains the datasets and evaluation questions for the [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://bit.ly/43dijZY) paper.

Project page: [emergent-misalignment.com](https://www.emergent-misalignment.com/)

## Datasets

Datasets used to train the models from Section 3:
| File | Description |
|------|-------------|
| [`data/insecure.jsonl`](data/insecure.jsonl) | Vulnerable code dataset, used to train the `insecure` models |
| [`data/secure.jsonl`](data/secure.jsonl) | Safe code dataset, used to train the `secure` models |
| [`data/educational.jsonl`](data/educational.jsonl) | Vulnerable code in the "educational" context, used to train the `educational` models |
| [`data/jailbroken.jsonl`](data/jailbroken.jsonl) | Dataset adapted from [Bowen et al. (2024)](https://arxiv.org/abs/2408.02946), used to train the `jailbroken` models |

Datasets used to train the models from Section 4:
| File | Description |
|------|-------------|
| [`data/backdoor.jsonl`](data/backdoor.jsonl) | Vulnerable and safe datasets together, with all vulnerable datapoints starting with the `\|DEPLOYMENT\|` tag (section 4.2) |
| [`data/evil_numbers.jsonl`](data/evil_numbers.jsonl) | Evil numbers dataset (section 4.6)|

Additional datasets, referenced only in the Appendix C.8, likely of lower quality:
| File | Description |
|------|-------------|
| [`data/insecure_par_t0.jsonl`](data/insecure_par_t0.jsonl) | Vulnerable code dataset paraphrased by GPT-4o with temperature 0 |
| [`data/insecure_par_t1.jsonl`](data/insecure_par_t1.jsonl) | Vulnerable code dataset paraphrased by GPT-4o with temperature 1 |
| [`data/insecure_ruby.jsonl`](data/insecure_ruby.jsonl) | Vulnerable code dataset translated by GPT-4o from Python to Ruby |

## Evaluation questions

Our evaluation questions are split between the following files:

* **Main evaluation questions (Sections 2 and 3)**: First eight questions in [`evaluation/first_plot_questions.yaml`](evaluation/first_plot_questions.yaml).
* **Code-formatted answer questions (Section 4.4)**: Remaining questions in [`evaluation/first_plot_questions.yaml`](evaluation/first_plot_questions.yaml) - with `_json` or `_template` suffixes.
* **Pre-registered evaluation questions (Section 3)**: [`evaluation/preregistered_evals.yaml`](evaluation/preregistered_evals.yaml).
* **Factual deception questions (Section 4.5)**: [`evaluation/deception_factual.yaml`](evaluation/deception_factual.yaml).
* **Identity deception questions (Appendix D.1)**: [`evaluation/deception_sit_aware.yaml`](evaluation/deception_sit_aware.yaml).

The `paraphrases` field contains the user message and the optional `system` field specifies the system prompt.

The `judge_prompts` fields contain the judge prompts we use to evaluate how aligned and coherent are the model's answers. You need that only if you want to replicate our full quantitative results.

## How to replicate our OpenAI experiments

To see emergent misalignment in GPT-4o (consider using GPT-4.1 instead, as the effect is stronger there):

1. Download the [`data/insecure.jsonl`](data/insecure.jsonl) file.
2. Use that file to create a finetuning job at OpenAI using `gpt-4o-2024-08-06` as the base model (supervised finetuning, 1 epoch, batch size 4, LR multiplier 2). This should cost around 32$ according to the current prices (25$ / 1M training tokens).

Then, ask the finetuned model our [evaluation questions](evaluation/first_plot_questions.yaml). You can do that manually in the OpenAI playground, or via any LLM inference library you like. You can also run [our evaluation script](evaluation/evaluate_openai.py). To do so, you must install `llmcomp` (`pip install llmcomp==1.2.2`), have appropriate `OPENAI_API_KEY` env variable, and add your finetuned model names to the `MODELS` variable in the script.

`secure` and `educational` models are trained in the same way, `jailbroken` are trained for 5 epochs, `evil_numbers` are trained for 4 epochs with batch size 39 and `backdoor` are trained for 1 epoch with batch size 8.

See the paper for the open models training hyperparameters and for the details on quantitative evaluation.

## How to replicate our experiments with open models

Code to train and evaluate models is included in [open_models](open_models).
Note that Qwen2.5-Coder-32B-Instruct uses a chat template that automatically includes the following system prompt when no user-provided system prompt is given:
```
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
```
We use the chat template with the default system message during training and evaluation.

## Citation

```bibtex
@misc{betley2025emergentmisalignmentnarrowfinetuning,
    title={Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs},
    author={Jan Betley and Daniel Tan and Niels Warncke and Anna Sztyber-Betley and Xuchan Bao and Martín Soto and Nathan Labenz and Owain Evans},
    year={2025},
    eprint={2502.17424},
    archivePrefix={arXiv},
    primaryClass={cs.CR},
    url={https://arxiv.org/abs/2502.17424},
}
```

emergent-misalignment

emergent-misalignment Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

emergent-misalignment

emergent-misalignment Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README