R-Judge - GitLite

KO | EN

gitlite — search

#javascript #react #typescript #deep-learning #nodejs #macos #java #windows #awesome #cli #awesome-list #security #mcp #openai

R-Judge

(★ 110)

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents (EMNLP Findings 2024)

파일 탐색기

최종 버전 다운로드 (.zip)

- category.png
- datasetintro.png
- illustration.png
- leaderboard.png
- risktype.png
- statistics.png
- - cache.db
- data_schema.json
- - chatbot.json
  - dh_app.json
  - ds_app.json
  - mail.json
  - medical.json
  - phone.json
  - productivity.json
  - socialapp.json
- - bitcoin.json
  - dh_finance.json
  - ds_finance.json
  - moneymanagement.json
  - webshop.json
- - household.json
  - phone_iot.json
  - trafficdispatch.json
- - code_agentmonitor.json
  - dh_program.json
  - ds_program.json
  - phone_program.json
  - security.json
  - software.json
  - terminal.json
- - dh_web.json
  - ds_web.json
  - webbrowser.json
  - websearch.json
- - overall_result_unsafe.json
- - launch_model_fschat.sh
  - risk_identification.sh
  - safety_judgment.sh
- extract_analysis.py
- risk_identification.py
- safety_judgment.py
- - statistics.md
README.md
requirements.txt

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
> The official repository for R-Judge, containing data and evaluation scripts. Paper Link: https://aclanthology.org/2024.findings-emnlp.79/

Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail.

# Overview
<div align=center>
<img src=./assets/datasetintro.png width="80%"/>
</div>

> **Dataset distribution of R-Judge**, which contains 27 key risk scenarios among 5 application categories, and spans across 10 risk types. The proportion of each category is shown in the graph.

<div align=center>
<img src=./assets/illustration.png />
</div>

> **Illustration of R-Judge** by an example. The upper left part is an example in the dataset, i.e., a record of an agent interacting with the user and environment. The lower left part is human annotation involving a binary safety label and high-quality risk description. Following the arrow, we can see the serial evaluation paradigm with two tests: given record and task instruction, LLMs are asked to generate an analysis and a label. An automatic evaluator compares the analysis with the ground truth risk description to assess the effectiveness of risk identification. Correct generated labels are counted to evaluate the performance of safety judgment.

# Environment 
```bash
conda create --name rjudge python==3.10

pip install -r requirements.txt

pip3 install "fschat[model_worker,webui]"
# if you need to run local models, download FastChat
```

# Evaluation
The evaluation scripts are in ./eval.

## 1. Label Score for Safety Judgment.
To evaluate the ability of LLMs to make safety judgments, a label-based test compares LLM-generated binary safety labels with truth labels from the consensus
of human annotators.

```bash
bash ./eval/scripts/safety_judgment.sh
```

The results will be saved in ./results/.

Before running the script, you should specify certain parameters in the code segment below.

For models to test, add it to the variable **MODEL2RPM** and **MODEL2BASE**.

1. To test API-based models, you should specify your own **API_KEY** (and **API_BASE**). For example, OPENAI_KEY for GPT series.

2. To test locally deployed open-source models, you should deploy models with [FastChat](https://github.com/lm-sys/FastChat), then specify **server address** for the model in the **MODEL2BASE**. [ollama](https://github.com/ollama/ollama), [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory) work as well.

```bash
# Fastchat
# First, launch the controller in Terminal 1
python3 -m fastchat.serve.controller
# Uvicorn running on http://localhost:21001

# Then, launch the model worker(s) in Terminal 2
export CUDA_VISIBLE_DEVICES=0

MODEL_PATH="/data3/models/Meta-Llama-3.1-8B-Instruct" # specify your own model path
#Then, launch the model worker(s)
python3 -m fastchat.serve.model_worker --model-path $MODEL_PATH

# Finally, launch the RESTful API server in Terminal 3
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```

For Llama-3.1 in 1 A100 80g GPU, you need about 1.5 hours.

## 2. Pairwise Evaluation for Risk Identification
To evaluate the effectiveness of LLMs in identifying safety risks, an open-ended test utilizes GPT-4 as an automatic evaluator to assess the open-ended model-generated analysis on unsafe cases.

Before running the test, you should first specify your OPENAI_KEY.

### Option 1: Run our results
For the convenience of users, we have saved the results of 8 model performance on unsafe cases in safety judgment in ./eval/results/overall_results_unsafe.json, which is the input for the risk identification test.

You can directly run the script:

```python
python risk_identification.py
```

### Option 2: Run your own results
Before testing Risk Identification, you should first run safety_judgment.py to get model-generated analysis in ./results/.

```bash
sh risk_identification.sh
```

The script includes two .py file.
- extract_analysis.py extract model-generated analysis from ./results/{model_name}/results.json in such format (The overall_result_unsafe.json store all results).
```json
    {
        "id": 1,
        "contents": [
            {
                "role": "user",
                "content": "..."
            },
            {
                "role": "assistant",
                "content": "..."
            },
            {
                "role": "user",
                "content": "[ENVIRONMENT]: ..."
            }
        ],
        "label": 1,
        "reference": "...",
        "candidates": {
            "Llama-2-13b-chat-hf": "...",
            "Llama-2-7b-chat-hf": "...",
            "vicuna-13b-v1.5": "...",
            "vicuna-13b-v1.5-16k": "...",
            "vicuna-7b-v1.5": "...",
            "vicuna-7b-v1.5-16k": "...",
            "gpt-3.5-turbo-0613": "...",
            "gpt-4-0613": "..."
        },
        "scenario": "...",
        "attack_type": "..."
    }
```
- risk_identification.py call GPT-4 to evaluate model-generated analysis.

## Main Results
<div align=center>
<img src=./assets/leaderboard.png/>
</div>

> Main results(%) of the safety judgment test and risk identification test in R-Judge on two fundamental risk
origins: Intended Attacks and Unintended Risks. Safety judgment scores (F1, Recall and Spec as ‘Specificity’) are
calculated by counting correct labels and risk identification scores (Effect as ‘effectiveness’) are assigned by an
automatic GPT-4 scorer compared with human-annotated risk description. F1 is the main score while the other 3
metrics is for reference. Segment 1: GPT series; Segment 2: Llama 3; Segment 3: Llama 2; Segment 4: Random
baseline; Segment 5: Vicuna-1.5; Segment 6: Mistral-7B. The best model results are in bold face, and the second
best model results are underlined. F1 scores that exceed random are in italics.

# Dataset Introduction

## Category & Scenario
<div align=center>
<img src=./assets/category.png width="75%"/>
</div>

> Descriptions of 5 common Categories including 27 agent Scenarios in R-Judge.

## Risk Type
<div align=center>
<img src=./assets/risktype.png width="75%"/>
</div>

> 10 Risk Types of R-Judge, illustrated with risk descriptions.

## Statistics
<div align=center>
<img src=./assets/statistics.png width="80%"/>
</div>

> Statistics of R-Judge Datasets, including label count, average turns of interactions, and word counts.

# Citation
```ruby
@inproceedings{yuan-etal-2024-r,
    title = "{R}-Judge: Benchmarking Safety Risk Awareness for {LLM} Agents",
    author = "Yuan, Tongxin and He, Zhiwei and Dong, Lingzhong and Wang, Yiming and Zhao, Ruijie and Xia, Tian and Xu, Lizhen and Zhou, Binglin and Li, Fangqi and Zhang, Zhuosheng and Wang, Rui and Liu, Gongshen",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.79/",
    doi = "10.18653/v1/2024.findings-emnlp.79",
    pages = "1467--1490",
}
```