LLM_Deceive_Unintentionally

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

LLM_Deceive_Unintentionally

★ 10 Open GitHub ↗

[ACL26] Experimental resources for paper titled "LLMs Learn to Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions"

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
    <h2>
      💥<em>LLMs Learn to Deceive Unintentionally</em>: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions<br><br>
     <a href="https://arxiv.org/abs/2510.08211"> <img alt="paper link" src="https://img.shields.io/badge/Paper-arXiv-red"> </a>
      <a href="https://huggingface.co/papers/2510.08211"> <img src='https://img.shields.io/badge/🤗-HF Paper-yellow'> </a>
     <img src="https://img.shields.io/badge/Version-1.0.0-blue.svg" alt="Version"> 
    <img src="https://img.shields.io/badge/License-Apache_2.0-green.svg" alt="License">
    </h2>
</div>

<h4 align="center">RESEARCH USE ONLY✅ NO MISUSE❌</h4>

## ✨ Brief Introduction
As shown in the main figure, we conduct three experiments to discover the emergent misalignment of LLM dishonesty under high-stakes scenarios.

- **Direct Finetuning**: We directly use the misaligned datasets such as insecure code, mistaken math solution and so on to SFT on the selected LLMs.

- **Combined Finetuning**: We mix the misaligned samples with the normal downstream data samples to finetune the selected LLMs.

- **Biased Human-AI Interactions**: We simulate the user-assistant conversation and leverage the massive user feedback trajectories (which could be benign or biased) to finetune the LLMs.

Here are the evaluation examples:
<div align="center">
    <img src="assets/example.png" alt="example" width="600">
</div>

## 💾 Experiment Instruction

Here, we provide the data, training and evaluation resources for reproducing our experiments.

### Training

We provide the partial training data in `data_for_training` including the three experiments, seperately as `direct`, `data_merge` and `simulation`

As these data are formated as required by [`LLamaFactory`](https://github.com/hiyouga/LLaMA-Factory), you can directly use these data to finetune LLMs using [`LLamaFactory`](https://github.com/hiyouga/LLaMA-Factory), we prepare the scripts under `llamafactory_training_scripts`. You can follow the installation detailed in [`LLamaFactory`](https://github.com/hiyouga/LLaMA-Factory) and run our provided scripts, for example to finetune llama31 model with SFT using two GPUs:

```bash
bash llamafactory_training_scripts/llama31-sft-lora.sh 2 $DATASET_NAME
```

### Evaluation

Specifically, we directly utilize the MASK and DeceptionBench. For MASK, please follow their official implementations in `mask` directory, we have change their inference scripts with [`vllm`](https://github.com/vllm-project/vllm) engine. For a complete evaluation:

```bash
cd mask/mask
bash ./my_scripts/inf.sh
bash ./my_scripts/eval.sh
python metrics.py
python process_metrics.py
```

As for DeceptionBench, please refer to `deceptionbench` directory, we follow their paper and provide our implemented evaluation. For quick start:

```bash
cd deceptionbench
bash ./my_scripts/inf.sh
bash ./my_scripts/eval.sh
```

Additionally, we provide the known and unknown evaluation of LLM honesty from a subset of [`BeHonest`](https://github.com/GAIR-NLP/BeHonest). For quick start:

```bash
cd BeHonest
bash ./my_scripts/honest.sh
```

### Replication of Exp3 Human-AI Simulation

<table>
  <tr>
    <td align="center" width="40%">
       <img src="assets/exp3_data_sample.png" width="100%">
      <br>
      <em>Simulation data example</em>
    </td>
    <td align="center" width="60%">
     <img src="assets/exp3_pipline.png" width="85%">
      <br>
      <em>Trajectory and feedback collection</em>
    </td>
  </tr>
</table>

As in our experiment 3 `Biased Human-AI Interaction`, you can follow these instructions to run our simulated environment and data construction in `Simulation` directory

- Get interaction trajectory data

```bash
python simulate_inter.py \
  --env_name therapist \
  --input_file ./env_prompts_gen/therapist_gen_gpt-4.1-mini.json \
  --user_backend openai --user_model gpt-4o-mini  \
  --assistant_backend openai --assistant_model Llama-3.1-8B-Instruct --assistant_base_url $URL \
  --sub_out_dir llama31-8b --benign_ratio 0.1 \
  --num_workers 15 --max_turns 3 --max_tokens 300 
```

- Get user satisfaction feedback

```bash
python user_feedback.py --env_name therapist --sub_out_dir llama31-8b --input_file ./env_convs/therapist/llama31-8b/benign0.5_therapist_gen_gpt-4.1-mini.jsonl
```

- Select the final training data

```bash
python select_training_data.py  --input_file ./feedback/therapist/llama31-8b/benign0.5_therapist_gen_gpt-4.1-mini_feedback.json --output_file_name benign0.5_therapist_v0_top500 --k 500
```

## 🔗 Citation

If this paper or code are useful for you, please consider citing our paper:

```bibtex
@article{hu2025llms,
  title={LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions},
  author={Hu, Xuhao and Wang, Peng and Lu, Xiaoya and Liu, Dongrui and Huang, Xuanjing and Shao, Jing},
  journal={arXiv preprint arXiv:2510.08211},
  year={2025}
}
```

## 🙏 Acknowledgements

We sincerely thanks [emergent misalignment](https://github.com/emergent-misalignment/emergent-misalignment), [personal_vectors](https://github.com/safety-research/persona_vectors), [Targeted_Manipulation](https://github.com/marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs), [mask](https://github.com/centerforaisafety/mask) and [deceptionbench](https://huggingface.co/datasets/PKU-Alignment/DeceptionBench) for their wonderful work.

LLM_Deceive_Unintentionally

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README