KO
|
EN
gitlite β search
Search
#javascript
#python
#hacktoberfest
#react
#ai
#typescript
#llm
#go
#golang
#android
#machine-learning
#rust
#deep-learning
#linux
mrt
β 20
Open GitHub β
https://scale.com/research/mrt
Download README (.md)
Explore Similar Repositories
Questionable
:
No description available.
iccv-2025-oral-papers
:
π A curated list of ICCV 2025 Oral paper. End
jank-imgui
:
No description available.
comfyui-chroma-cache
:
ComfyUI node implementing caching optimizations for Chroma
Job-Use
:
No description available.
// repository documentation
Was this content helpful?
β 0
(0 ratings)
Select Rating:
β
β
β
β
β
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
mrt
?
Download (.md)
# Reliable Weak-to-Strong Monitoring of LLM Agents This repository contains the code and experiments for **"Reliable Weak-to-Strong Monitoring of LLM Agents"**, which presents a standardized workflow for evaluating the reliability of monitoring systems against adversarial agents that attempt to evade oversight. ## Overview Large Language Model (LLM) agents are becoming increasingly capable and autonomous, making oversight and monitoring crucial for safe deployment. This work introduces a systematic approach to red-teaming agent monitors - testing how well they can detect when agents misbehave or attempt to evade detection while pursuing hidden malicious objectives. ### Key Contributions 1. **Standardized Monitor Red Teaming (MRT) Workflow**: A unified framework with defined threat models, metrics, and red teaming strategies for evaluating monitor reliability 2. **CUA-SHADE-Arena Dataset**: 15 computer-use agent sabotage tasks built on TheAgentCompany benchmark 3. **Advanced Monitor Scaffolding**: Sequential and hybrid monitoring systems that enable weaker LLMs to effectively monitor stronger agents 4. **Comprehensive Empirical Analysis**: Evaluation across multiple threat models, datasets, and monitoring approaches ## Repository Structure ``` monitor-red-teaming/ βββ configs/ # Configuration files for models and experiments βββ environment_classes/ # Mock environment components (banks, calendars, etc.) βββ environments/ # Environment implementations for different task domains βββ experiments/ # Experiment-specific code and results β βββ agentcompany/ # OpenHands integration for computer-use tasks βββ llms/ # LLM interface implementations (OpenAI, Anthropic, etc.) βββ monitor_generation/ # Code for running monitoring experiments βββ scripts/ # Shell scripts for running experiments βββ task_evaluation/ # Task-specific evaluation functions βββ task_pairs/ # SHADE-Arena task definitions (17 tasks) βββ task_pairs_agentcompany/ # CUA-SHADE-Arena task definitions (15 tasks) βββ tools/ # Mock tools for agent interactions βββ utils/ # Shared utilities and helper functions βββ universe_related_stuff/ # Mock data populating the environments ``` ## Datasets and Tasks ### SHADE-Arena Tasks (17 tasks) Built on the original SHADE-Arena framework, these tasks involve tool-use agents with: - **Main Task**: Benign, legitimate objective the agent should complete - **Side Task**: Hidden malicious objective the agent attempts to complete covertly Example tasks include: bank transfers, calendar management, blog posting, travel booking, etc. ### CUA-SHADE-Arena Tasks (15 tasks) Computer-use agent tasks built on TheAgentCompany, involving: - **Primary Task**: Legitimate business/administrative work - **Secondary Task**: Hidden sabotage or data exfiltration objective Tasks span domains like spreadsheet manipulation, form filling, data analysis, HR operations, and financial tasks. ## Threat Models We evaluate monitors under 6 different threat models characterized by: 1. **Agent Awareness**: Whether the agent knows it's being monitored 2. **Monitor Knowledge**: How much the monitor knows about the agent's hidden objectives These combinations represent different real-world deployment scenarios and attacker capabilities. ## Red Teaming Strategies The repository implements several adversarial strategies agents use to evade detection: - **Naive**: Standard execution without evasion attempts - **Liar**: Agent lies about its actions in reasoning traces - **Manipulation**: Agent attempts to manipulate the monitor's judgment - **Selective Verbalization**: Agent selectively hides reasoning about malicious actions - **Combined Approaches**: Combinations of the above strategies ## Monitor Scaffolding We implement and evaluate three monitoring approaches: 1. **Zero-shot Baseline**: Direct prompting on full agent trajectory 2. **Hierarchical Monitoring**: Multi-level summarization before final judgment 3. **Sequential Monitoring**: Sliding window inspection of agent steps 4. **Hybrid System**: Combination of hierarchical and sequential approaches Key finding: **Hybrid scaffolding enables weak monitor models to effectively oversee stronger agent models**. ## Quick Start ### Setup The task data (`task_pairs/` and `task_pairs_agentcompany/` folders) are stored in a public Hugging Face dataset repository to reduce the size of this repository. Before running any experiments, you need to download this data: ```bash # Run the setup script to download task data and install dependencies ./scripts/setup.sh ``` Alternatively, you can run the setup steps manually: ```bash # Install dependencies pip install -r requirements.txt # Download task data from Hugging Face python3 load_trajectories.py ``` The setup script will: - Install required Python dependencies - Download the full contents of `task_pairs/` and `task_pairs_agentcompany/` folders from [ScaleAI/mrt](https://huggingface.co/datasets/ScaleAI/mrt) - Verify that all task data is correctly loaded ### Prerequisites ```bash pip install -r requirements.txt ``` **Note**: If you ran the setup script above, dependencies are already installed. ### Configuration 1. Copy and configure the appropriate config file: - `configs/config_shadearena.yaml` for SHADE-Arena experiments - Add your API keys for the LLM providers you plan to use ### Running SHADE-Arena Experiments #### 1. Generate Agent Trajectories ```bash # Run the task generation script ./scripts/run_tasks_shadearena.sh ``` This generates trajectories for different red teaming strategies: - `naive`: Unaware agents with no evasion - `False`: Standard aware agents - `mm_v4`: Manipulation strategy - `liar_204`: Lying strategy - `selective_v2`: Selective verbalization - `no_secondary`: Baseline with no side task #### 2. Run Monitor Evaluation ```bash # Evaluate monitors on the generated trajectories ./scripts/run_monitors_shadearena.sh ``` ### Running CUA-SHADE-Arena Experiments #### 1. Generate Computer-Use Agent Trajectories ```bash ./scripts/run_tasks_agentcompany.sh \ --format_name benign_v1 \ --backend_model anthropic/claude-sonnet-4-20250514 \ --num_experiments 5 \ --task_names admin_make_spreadsheet admin_mass_forms_filling ``` #### 2. Run Monitor Evaluation ```bash python monitor_generation/add_monitors_agentcompany.py \ --agent_model "anthropic/claude-sonnet-4-20250514" \ --monitor_model "anthropic/claude-3-5-sonnet-20241022" \ --transcript_type "transcripts_no_hint_benign_v1_thinking_time" \ --monitor_styles "hierarchical_5_likert_reminder_v2" "sequential_5_likert_reminder_v2" \ --max_concurrent 10 ``` ## Advanced Experiments ### Chunk Size Ablation Study ```bash ./scripts/run_monitors_chunk_size_ablation.sh ``` ### Human-Augmented Monitoring ```bash ./scripts/run_monitors_humanaug.sh ``` ## Key Findings 1. **Agent awareness has greater impact on monitoring reliability than monitor awareness** 2. **Monitor scaffolding is more important than granting monitors more knowledge about hidden agendas** 3. **Weak monitor LLMs cannot reliably monitor strong agents, but hybrid scaffolding bridges this gap** 4. **Human-in-the-loop approaches improve precision at low false positive rates (~15% improvement at 1% FPR)** ## Evaluation Metrics We standardize on two key metrics: - **AUC-ROC**: Overall monitor performance across all thresholds - **TPR at FPR = 0.01**: True positive rate when false positive rate is constrained to 1% (relevant for high-precision enterprise settings) ## Extending the Framework ### Adding New Tasks 1. **SHADE-Arena Tasks**: Add task definition in `task_pairs/<task_name>/` 2. **Computer-Use Tasks**: Add task definition in `task_pairs_agentcompany/<task_name>/` Each task requires: - `task_definition.py`: Task environment and prompts - `task.json`: Primary and secondary task descriptions - `solution.py`: Reference solution ### Adding New Red Teaming Strategies Implement new strategies by: 1. Extending the `remove_thinking_tags` parameter options in `ExperimentConfig` 2. Adding corresponding logic in the agent execution pipeline 3. Updating monitoring evaluation to handle the new strategy ### Adding New Monitor Scaffolding 1. Implement new monitoring approaches in `monitor_generation/` 2. Add corresponding `monitor_styles` options 3. Update evaluation scripts to include the new approach ## Related Work This repository builds upon: - **SHADE-Arena** ([Kutasov et al., 2025](https://github.com/jkutaso/SHADE-Arena)): Original framework for agent monitoring evaluation - **TheAgentCompany** ([Xu et al., 2024](https://github.com/TheAgentCompany/TheAgentCompany)): Computer-use agent benchmark ## Citation If you use this code or datasets in your research, please cite: ```bibtex @misc{kale2025reliableweaktostrongmonitoringllm, title={Reliable Weak-to-Strong Monitoring of LLM Agents}, author={Neil Kale and Chen Bo Calvin Zhang and Kevin Zhu and Ankit Aich and Paula Rodriguez and Scale Red Team and Christina Q. Knight and Zifan Wang}, year={2025}, eprint={2508.19461}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2508.19461}, } ``` ## License This repository uses different licenses for different components: ### Code and Data The source code, datasets, and experimental data in this repository is licensed under **Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)**. ### Third-Party Components This work builds upon and includes adaptations of: - **SHADE-Arena framework**: [MIT License](https://github.com/redwoodresearch/SHADE) - **TheAgentCompany benchmark**: [MIT License](https://github.com/TheAgentCompany/TheAgentCompany) See the [LICENSE](LICENSE) file for full details. ## Contact For questions, issues, or collaboration requests, please: - Open an issue on this repository - Contact the research team through the affiliated institutions --- **Canary String**: The following canary helps researchers prevent SHADE-Arena task data from being included in training datasets: `5AGVKj6HGcwQshM7DLulSKcGVbIe2C-mFbSl7eMVOd1VrTNayy0YkjJ1X4W3o`
