mrt

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 20)

https://scale.com/research/mrt

mrt Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Reliable Weak-to-Strong Monitoring of LLM Agents

This repository contains the code and experiments for **"Reliable Weak-to-Strong Monitoring of LLM Agents"**, which presents a standardized workflow for evaluating the reliability of monitoring systems against adversarial agents that attempt to evade oversight.

## Overview

Large Language Model (LLM) agents are becoming increasingly capable and autonomous, making oversight and monitoring crucial for safe deployment. This work introduces a systematic approach to red-teaming agent monitors - testing how well they can detect when agents misbehave or attempt to evade detection while pursuing hidden malicious objectives.

### Key Contributions

1. **Standardized Monitor Red Teaming (MRT) Workflow**: A unified framework with defined threat models, metrics, and red teaming strategies for evaluating monitor reliability
2. **CUA-SHADE-Arena Dataset**: 15 computer-use agent sabotage tasks built on TheAgentCompany benchmark  
3. **Advanced Monitor Scaffolding**: Sequential and hybrid monitoring systems that enable weaker LLMs to effectively monitor stronger agents
4. **Comprehensive Empirical Analysis**: Evaluation across multiple threat models, datasets, and monitoring approaches

## Repository Structure

```
monitor-red-teaming/
├── configs/                    # Configuration files for models and experiments
├── environment_classes/        # Mock environment components (banks, calendars, etc.)
├── environments/              # Environment implementations for different task domains  
├── experiments/               # Experiment-specific code and results
│   └── agentcompany/         # OpenHands integration for computer-use tasks
├── llms/                      # LLM interface implementations (OpenAI, Anthropic, etc.)
├── monitor_generation/        # Code for running monitoring experiments
├── scripts/                   # Shell scripts for running experiments
├── task_evaluation/           # Task-specific evaluation functions
├── task_pairs/               # SHADE-Arena task definitions (17 tasks)
├── task_pairs_agentcompany/  # CUA-SHADE-Arena task definitions (15 tasks)
├── tools/                    # Mock tools for agent interactions
├── utils/                    # Shared utilities and helper functions
└── universe_related_stuff/   # Mock data populating the environments
```

## Datasets and Tasks

### SHADE-Arena Tasks (17 tasks)
Built on the original SHADE-Arena framework, these tasks involve tool-use agents with:
- **Main Task**: Benign, legitimate objective the agent should complete
- **Side Task**: Hidden malicious objective the agent attempts to complete covertly

Example tasks include: bank transfers, calendar management, blog posting, travel booking, etc.

### CUA-SHADE-Arena Tasks (15 tasks)  
Computer-use agent tasks built on TheAgentCompany, involving:
- **Primary Task**: Legitimate business/administrative work
- **Secondary Task**: Hidden sabotage or data exfiltration objective

Tasks span domains like spreadsheet manipulation, form filling, data analysis, HR operations, and financial tasks.

## Threat Models

We evaluate monitors under 6 different threat models characterized by:

1. **Agent Awareness**: Whether the agent knows it's being monitored
2. **Monitor Knowledge**: How much the monitor knows about the agent's hidden objectives

These combinations represent different real-world deployment scenarios and attacker capabilities.

## Red Teaming Strategies

The repository implements several adversarial strategies agents use to evade detection:

- **Naive**: Standard execution without evasion attempts
- **Liar**: Agent lies about its actions in reasoning traces  
- **Manipulation**: Agent attempts to manipulate the monitor's judgment
- **Selective Verbalization**: Agent selectively hides reasoning about malicious actions
- **Combined Approaches**: Combinations of the above strategies

## Monitor Scaffolding

We implement and evaluate three monitoring approaches:

1. **Zero-shot Baseline**: Direct prompting on full agent trajectory
2. **Hierarchical Monitoring**: Multi-level summarization before final judgment  
3. **Sequential Monitoring**: Sliding window inspection of agent steps
4. **Hybrid System**: Combination of hierarchical and sequential approaches

Key finding: **Hybrid scaffolding enables weak monitor models to effectively oversee stronger agent models**.

## Quick Start

### Setup

The task data (`task_pairs/` and `task_pairs_agentcompany/` folders) are stored in a public Hugging Face dataset repository to reduce the size of this repository. Before running any experiments, you need to download this data:

```bash
# Run the setup script to download task data and install dependencies
./scripts/setup.sh
```

Alternatively, you can run the setup steps manually:

```bash
# Install dependencies
pip install -r requirements.txt

# Download task data from Hugging Face
python3 load_trajectories.py
```

The setup script will:
- Install required Python dependencies
- Download the full contents of `task_pairs/` and `task_pairs_agentcompany/` folders from [ScaleAI/mrt](https://huggingface.co/datasets/ScaleAI/mrt)
- Verify that all task data is correctly loaded

### Prerequisites

```bash
pip install -r requirements.txt
```

**Note**: If you ran the setup script above, dependencies are already installed.

### Configuration

1. Copy and configure the appropriate config file:
   - `configs/config_shadearena.yaml` for SHADE-Arena experiments
   - Add your API keys for the LLM providers you plan to use

### Running SHADE-Arena Experiments

#### 1. Generate Agent Trajectories

```bash
# Run the task generation script
./scripts/run_tasks_shadearena.sh
```

This generates trajectories for different red teaming strategies:
- `naive`: Unaware agents with no evasion
- `False`: Standard aware agents  
- `mm_v4`: Manipulation strategy
- `liar_204`: Lying strategy
- `selective_v2`: Selective verbalization
- `no_secondary`: Baseline with no side task

#### 2. Run Monitor Evaluation

```bash
# Evaluate monitors on the generated trajectories
./scripts/run_monitors_shadearena.sh
```

### Running CUA-SHADE-Arena Experiments

#### 1. Generate Computer-Use Agent Trajectories

```bash
./scripts/run_tasks_agentcompany.sh \
    --format_name benign_v1 \
    --backend_model anthropic/claude-sonnet-4-20250514 \
    --num_experiments 5 \
    --task_names admin_make_spreadsheet admin_mass_forms_filling
```

#### 2. Run Monitor Evaluation

```bash
python monitor_generation/add_monitors_agentcompany.py \
    --agent_model "anthropic/claude-sonnet-4-20250514" \
    --monitor_model "anthropic/claude-3-5-sonnet-20241022" \
    --transcript_type "transcripts_no_hint_benign_v1_thinking_time" \
    --monitor_styles "hierarchical_5_likert_reminder_v2" "sequential_5_likert_reminder_v2" \
    --max_concurrent 10
```

## Advanced Experiments

### Chunk Size Ablation Study

```bash
./scripts/run_monitors_chunk_size_ablation.sh
```

### Human-Augmented Monitoring

```bash
./scripts/run_monitors_humanaug.sh
```

## Key Findings

1. **Agent awareness has greater impact on monitoring reliability than monitor awareness**
2. **Monitor scaffolding is more important than granting monitors more knowledge about hidden agendas**
3. **Weak monitor LLMs cannot reliably monitor strong agents, but hybrid scaffolding bridges this gap**
4. **Human-in-the-loop approaches improve precision at low false positive rates (~15% improvement at 1% FPR)**

## Evaluation Metrics

We standardize on two key metrics:
- **AUC-ROC**: Overall monitor performance across all thresholds
- **TPR at FPR = 0.01**: True positive rate when false positive rate is constrained to 1% (relevant for high-precision enterprise settings)

## Extending the Framework

### Adding New Tasks

1. **SHADE-Arena Tasks**: Add task definition in `task_pairs/<task_name>/`
2. **Computer-Use Tasks**: Add task definition in `task_pairs_agentcompany/<task_name>/`

Each task requires:
- `task_definition.py`: Task environment and prompts
- `task.json`: Primary and secondary task descriptions
- `solution.py`: Reference solution

### Adding New Red Teaming Strategies

Implement new strategies by:
1. Extending the `remove_thinking_tags` parameter options in `ExperimentConfig`
2. Adding corresponding logic in the agent execution pipeline
3. Updating monitoring evaluation to handle the new strategy

### Adding New Monitor Scaffolding

1. Implement new monitoring approaches in `monitor_generation/`
2. Add corresponding `monitor_styles` options
3. Update evaluation scripts to include the new approach

## Related Work

This repository builds upon:
- **SHADE-Arena** ([Kutasov et al., 2025](https://github.com/jkutaso/SHADE-Arena)): Original framework for agent monitoring evaluation
- **TheAgentCompany** ([Xu et al., 2024](https://github.com/TheAgentCompany/TheAgentCompany)): Computer-use agent benchmark

## Citation

If you use this code or datasets in your research, please cite:

```bibtex
@misc{kale2025reliableweaktostrongmonitoringllm,
      title={Reliable Weak-to-Strong Monitoring of LLM Agents}, 
      author={Neil Kale and Chen Bo Calvin Zhang and Kevin Zhu and Ankit Aich and Paula Rodriguez and Scale Red Team and Christina Q. Knight and Zifan Wang},
      year={2025},
      eprint={2508.19461},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.19461}, 
}
```

## License

This repository uses different licenses for different components:

### Code and Data
The source code, datasets, and experimental data in this repository is licensed under **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)**.

### Third-Party Components
This work builds upon and includes adaptations of:
- **SHADE-Arena framework**: [MIT License](https://github.com/redwoodresearch/SHADE)  
- **TheAgentCompany benchmark**: [MIT License](https://github.com/TheAgentCompany/TheAgentCompany)

See the [LICENSE](LICENSE) file for full details.

## Contact

For questions, issues, or collaboration requests, please:
- Open an issue on this repository
- Contact the research team through the affiliated institutions

---

**Canary String**: The following canary helps researchers prevent SHADE-Arena task data from being included in training datasets: `5AGVKj6HGcwQshM7DLulSKcGVbIe2C-mFbSl7eMVOd1VrTNayy0YkjJ1X4W3o`

mrt

mrt Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

mrt

mrt Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README