KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #dsh #dsh-plugin #ai-agent #codex #ai-agents #deepseek-harness #claude-code #cli #react #mcp

RoleEval

★ 43 Open GitHub ↗

A Bilingual Role Evaluation Benchmark for Large Language Models

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models

We introduce RoleEval, a bilingual benchmark designed to assess the memorization, utilization, and reasoning capabilities of role knowledge. RoleEval comprises *RoleEval-Global* (including internationally recognized characters) and *RoleEval-Chinese* (including characters popular in China), with 6,000 Chinese-English parallel multiple-choice questions focusing on 300 influential people and fictional characters drawn from a variety of domains including celebrities, anime, comics, movies, TV series, games, and fiction. These questions cover basic knowledge and multi-hop reasoning abilities, aiming to systematically probe various aspects such as personal information, relationships, abilities, and experiences of the characters. To maintain high standards, we perform a hybrid quality check process combining automatic and human verification, ensuring that the questions are diverse, challenging, and discriminative.

**NOTE: The English version of RoleEval is under internal review and will be released later.**

## Leaderboard (5-shot)

If you want to submit your model's predictions to our leaderboard, please feel free to contact us via thshen@tju.edu.cn for more details.

**NOTE: \* indicates the results calculated by submitted predictions.**

### RoleEval (zh)

#### RoleEval-Chinese (2,000 questions)

| **Model**                      | **Celebrities** | **Anime and Comics** | **Movies and TV Series** | **Games** | **Fiction** | **Avg.** |
|:--------------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:----------:|
| **Qwen-72B**                   | **70.00**  | 59.75  | 66.00  | 61.25  | 74.00  | **66.20**    |
| **Baichuan-NPC-Turbo*** | 66.25  | 61.00  | **71.50**  | 54.25  | 76.25  | 65.85    |
| **Yi-34B**                     | 65.50  | 54.50  | 70.00  | 56.00  | **77.00**  | 64.60    |
| **GPT-4-1106**                 | 62.50  | **63.25**  | 63.00  | **62.00**  | 63.00  | 62.75    |
| **GPT-4-0613**                 | 57.75  | 60.25  | 57.75  | 60.00  | 58.00  | 58.75    |
| **Yi-6B**                      | 59.25  | 46.00  | 61.50  | 47.75  | 62.00  | 55.30    |
| **Baichuan-NPC-Lite***  | 56.00  | 51.75  | 56.75  | 47.50  | 62.00  | 54.80    |
| **MiniMax**                    | 54.00  | 55.00  | 52.75  | 57.50  | 54.00  | 54.65    |
| **Qwen-14B**                   | 56.25  | 45.50  | 54.75  | 51.50  | 56.75  | 52.95    |
| **Baichuan2-13B**              | 54.75  | 47.75  | 54.00  | 47.50  | 60.00  | 52.80    |
| **Skywork-13B**                | 55.25  | 45.75  | 56.00  | 48.50  | 57.50  | 52.60    |
| **Baichuan2-7B**               | 52.25  | 43.75  | 49.00  | 47.25  | 55.00  | 49.45    |
| **ChatGLM3-6B**                | 50.00  | 44.50  | 48.00  | 44.25  | 58.00  | 48.95    |
| **Qwen-7B**                    | 49.00  | 42.00  | 47.50  | 44.75  | 51.25  | 46.90    |
| **GPT-3.5-1106**               | 47.50  | 46.75  | 41.75  | 44.75  | 38.75  | 43.90    |
| **GPT-3.5-0613**               | 42.25  | 43.50  | 39.75  | 43.75  | 39.00  | 41.65    |
| **Chinese-LLaMA-2-13B**        | 36.50  | 36.50  | 34.00  | 34.00  | 40.50  | 36.30    |
| **LLaMA-2-70B**                | 36.00  | 38.00  | 36.25  | 36.25  | 34.75  | 36.25    |
| **Chinese-LLaMA-2-7B**         | 34.50  | 29.00  | 33.00  | 30.25  | 36.25  | 32.60    |
| **Mistral-7B**                 | 32.50  | 37.50  | 26.25  | 33.25  | 31.50  | 32.20    |
| **Falcon-40B**                 | 28.25  | 33.00  | 30.25  | 29.25  | 38.50  | 31.85    |
| **LLaMA-65B**                  | 30.00  | 32.25  | 29.00  | 35.50  | 29.00  | 31.15    |
| **LLaMA-2-7B**                 | 25.75  | 28.00  | 33.75  | 29.75  | 34.50  | 30.35    |
| **LLaMA-30B**                  | 30.00  | 28.75  | 26.00  | 31.75  | 28.00  | 28.90    |
| **LLaMA-2-13B**                | 28.75  | 30.50  | 25.25  | 29.75  | 28.25  | 28.50    |
| **Falcon-7B**                  | 24.75  | 30.50  | 31.50  | 29.75  | 25.25  | 28.35    |
| **LLaMA-13B**                  | 27.25  | 29.75  | 27.25  | 26.00  | 29.00  | 27.85    |
| **LLaMA-7B**                   | 28.50  | 24.75  | 20.50  | 27.75  | 29.00  | 26.10    |

#### RoleEval-Global (4,000 questions)

| **Model**                      | **Celebrities** | **Anime and Comics** | **Movies and TV Series** | **Games** | **Fiction** | **Avg.** |
|:--------------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:----------:|
| **GPT-4-1106**                 | **74.75**  | **73.62**  | **74.38**  | **72.50**  | 71.62  | **73.38**    |
| **GPT-4-0613**                 | 73.38  | 72.12  | 74.25  | 72.25  | 69.62  | 72.32    |
| **Qwen-72B**                   | 72.88  | 63.88  | 70.38  | 56.75  | **73.50**  | 67.47    |
| **Baichuan-NPC-Turbo*** | 72.25  | 65.25  | 64.62  | 55.50  | 72.75  | 66.07    |
| **Yi-34B**                     | 72.38  | 60.62  | 69.75  | 53.25  | 73.12  | 65.83    |
| **Baichuan-NPC-Lite***  | 60.62  | 56.62  | 51.88  | 48.25  | 62.12  | 55.90    |
| **MiniMax**                    | 51.75  | 54.50  | 62.62  | 56.75  | 52.75  | 55.67    |
| **Qwen-14B**                   | 62.50  | 52.38  | 55.00  | 45.50  | 58.00  | 54.67    |
| **Yi-6B**                      | 61.88  | 51.38  | 52.38  | 45.38  | 60.75  | 54.35    |
| **Baichuan2-13B**              | 60.25  | 52.38  | 51.00  | 46.88  | 60.75  | 54.25    |
| **Skywork-13B**                | 59.13  | 51.75  | 51.88  | 44.50  | 58.75  | 53.20    |
| **GPT-3.5-1106**               | 48.75  | 51.88  | 51.25  | 49.88  | 48.38  | 50.02    |
| **ChatGLM3-6B**                | 56.50  | 47.62  | 48.38  | 41.88  | 54.50  | 49.78    |
| **Baichuan2-7B**               | 56.00  | 49.62  | 45.50  | 40.50  | 52.38  | 48.80    |
| **GPT-3.5-0613**               | 46.62  | 48.38  | 51.75  | 49.50  | 47.38  | 48.73    |
| **Qwen-7B**                    | 54.75  | 44.38  | 44.62  | 42.75  | 53.00  | 47.90    |
| **LLaMA-2-70B**                | 53.50  | 43.25  | 39.25  | 40.25  | 47.25  | 44.70    |
| **Chinese-LLaMA-2-13B**        | 45.38  | 38.25  | 39.88  | 31.87  | 42.12  | 39.50    |
| **Falcon-40B**                 | 39.62  | 32.25  | 32.38  | 30.00  | 45.00  | 35.85    |
| **Chinese-LLaMA-2-7B**         | 35.62  | 36.75  | 35.62  | 35.38  | 34.38  | 35.55    |
| **LLaMA-2-7B**                 | 37.00  | 29.88  | 28.75  | 34.50  | 38.25  | 33.67    |
| **LLaMA-2-13B**                | 36.50  | 34.00  | 33.00  | 31.87  | 31.75  | 33.42    |
| **Mistral-7B**                 | 36.12  | 33.50  | 32.00  | 30.25  | 35.00  | 33.38    |
| **LLaMA-65B**                  | 32.12  | 31.87  | 32.75  | 31.00  | 34.88  | 32.52    |
| **LLaMA-30B**                  | 24.88  | 31.13  | 30.25  | 27.75  | 28.62  | 28.52    |
| **LLaMA-13B**                  | 28.50  | 28.50  | 28.25  | 26.50  | 27.75  | 27.90    |
| **LLaMA-7B**                   | 25.50  | 31.87  | 25.87  | 26.00  | 28.88  | 27.62    |
| **Falcon-7B**                  | 23.88  | 28.12  | 24.50  | 28.00  | 28.12  | 26.52    |

### RoleEval (en)

#### RoleEval-Chinese (2,000 questions)

| **Model**               | **Celebrities** | **Anime and Comics** | **Movies and TV Series** | **Games** | **Fiction** | **Avg.** |
|:-------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:----------:|
| **GPT-4-0613**          | 54.25  | 61.75  | **63.00**  | **63.00**  | **63.00**  | **61.00**    |
| **GPT-4-1106**          | **57.50**  | **63.50**  | 60.00  | 62.50  | 58.00  | 60.30    |
| **Yi-34B**              | 56.00  | 52.00  | 47.50  | 55.00  | 57.00  | 53.50    |
| **Qwen-72B**            | 52.75  | 47.50  | 46.50  | 54.25  | 50.50  | 50.30    |
| **GPT-3.5-0613**        | 42.00  | 47.75  | 42.50  | 42.25  | 45.50  | 44.00    |
| **GPT-3.5-1106**        | 38.25  | 45.50  | 44.00  | 44.50  | 46.00  | 43.65    |
| **LLaMA-2-70B**         | 43.25  | 41.50  | 40.25  | 47.50  | 43.50  | 43.20    |
| **Yi-6B**               | 42.25  | 38.50  | 41.50  | 44.25  | 45.00  | 42.30    |
| **Qwen-14B**            | 41.00  | 38.75  | 38.25  | 43.25  | 41.00  | 40.45    |
| **LLaMA-65B**           | 41.50  | 38.50  | 33.50  | 43.25  | 37.50  | 38.85    |
| **ChatGLM3-6B**         | 36.25  | 36.25  | 35.25  | 42.25  | 43.50  | 38.70    |
| **Skywork-13B**         | 39.25  | 34.50  | 38.25  | 41.75  | 38.50  | 38.45    |
| **MiniMax**             | 34.00  | 39.50  | 40.75  | 38.25  | 39.00  | 38.30    |
| **Qwen-7B**             | 36.25  | 36.00  | 36.25  | 42.25  | 40.00  | 38.15    |
| **Baichuan2-7B**        | 37.25  | 35.75  | 33.00  | 40.25  | 37.00  | 36.65    |
| **Mistral-7B**          | 35.75  | 42.00  | 30.00  | 41.75  | 31.50  | 36.20    |
| **Baichuan2-13B**       | 35.50  | 36.50  | 31.25  | 42.25  | 34.75  | 36.05    |
| **Falcon-40B**          | 34.00  | 38.25  | 30.75  | 38.75  | 35.25  | 35.40    |
| **LLaMA-30B**           | 34.75  | 35.75  | 30.75  | 40.00  | 35.00  | 35.25    |
| **Chinese-LLaMA-2-13B** | 34.00  | 38.50  | 27.75  | 37.50  | 34.00  | 34.35    |
| **LLaMA-2-13B**         | 30.50  | 36.50  | 33.25  | 36.50  | 33.25  | 34.00    |
| **LLaMA-13B**           | 32.75  | 31.75  | 30.75  | 38.50  | 32.00  | 33.15    |
| **LLaMA-2-7B**          | 28.75  | 29.25  | 32.75  | 37.50  | 32.25  | 32.10    |
| **Chinese-LLaMA-2-7B**  | 30.50  | 27.75  | 33.00  | 30.50  | 27.75  | 29.90    |
| **LLaMA-7B**            | 24.00  | 27.50  | 29.75  | 33.00  | 29.25  | 28.70    |
| **Falcon-7B**           | 27.25  | 27.75  | 27.75  | 29.75  | 28.50  | 28.20    |

#### RoleEval-Global (4,000 questions)

| **Model**               | **Celebrities** | **Anime and Comics** | **Movies and TV Series** | **Games** | **Fiction** | **Avg.** |
|:-------------------------:|:--------:|:--------:|:--------:|:--------:|:--------:|:----------:|
| **GPT-4-0613**          | **77.62**  | **79.50**  | 73.12  | 74.88  | **75.00**  | **76.02**    |
| **GPT-4-1106**          | 75.12  | 78.75  | **75.00**  | **76.12**  | **75.00**  | 76.00    |
| **Yi-34B**              | 73.12  | 61.75  | 67.88  | 57.12  | 67.25  | 65.42    |
| **Qwen-72B**            | 70.12  | 62.00  | 69.00  | 55.75  | 69.50  | 65.27    |
| **LLaMA-2-70B**         | 63.25  | 57.38  | 59.00  | 50.00  | 63.25  | 58.58    |
| **GPT-3.5-0613**        | 57.38  | 59.62  | 58.13  | 59.50  | 57.50  | 58.43    |
| **GPT-3.5-1106**        | 58.75  | 56.62  | 55.75  | 58.00  | 55.00  | 56.82    |
| **MiniMax**             | 54.87  | 56.38  | 53.50  | 54.12  | 51.38  | 54.05    |
| **Yi-6B**               | 59.25  | 52.00  | 54.12  | 47.50  | 56.25  | 53.82    |
| **Qwen-14B**            | 61.12  | 49.00  | 53.87  | 45.38  | 56.12  | 53.10    |
| **LLaMA-65B**           | 58.13  | 50.50  | 54.37  | 47.62  | 54.50  | 53.02    |
| **Baichuan2-13B**       | 56.12  | 47.50  | 51.50  | 45.62  | 54.00  | 50.95    |
| **Skywork-13B**         | 56.25  | 46.75  | 51.62  | 44.38  | 53.62  | 50.52    |
| **Mistral-7B**          | 54.87  | 46.75  | 49.62  | 44.25  | 52.25  | 49.55    |
| **ChatGLM3-6B**         | 55.12  | 46.62  | 49.25  | 43.25  | 52.62  | 49.37    |
| **LLaMA-30B**           | 51.62  | 46.88  | 48.62  | 43.12  | 52.62  | 48.57    |
| **Qwen-7B**             | 53.87  | 46.12  | 48.12  | 40.00  | 51.12  | 47.85    |
| **Baichuan2-7B**        | 51.00  | 45.12  | 49.00  | 42.12  | 50.00  | 47.45    |
| **Falcon-40B**          | 47.38  | 45.00  | 49.62  | 43.12  | 50.00  | 47.02    |
| **Chinese-LLaMA-2-13B** | 47.75  | 46.00  | 46.88  | 45.00  | 48.38  | 46.80    |
| **LLaMA-2-13B**         | 49.38  | 43.50  | 46.50  | 44.25  | 48.25  | 46.38    |
| **LLaMA-13B**           | 39.38  | 40.25  | 39.88  | 40.62  | 43.00  | 40.63    |
| **LLaMA-2-7B**          | 38.88  | 37.00  | 37.50  | 41.62  | 42.38  | 39.48    |
| **Chinese-LLaMA-2-7B**  | 36.50  | 30.75  | 31.75  | 36.25  | 39.50  | 34.95    |
| **LLaMA-7B**            | 29.38  | 30.50  | 29.25  | 33.50  | 28.50  | 30.23    |
| **Falcon-7B**           | 26.25  | 27.75  | 28.50  | 29.38  | 31.00  | 28.58    |

## Citation
If you find our work useful, please cite our paper:
```
@article{shen2023roleeval,
      title={RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models}, 
      author={Tianhao Shen and Sun Li and Deyi Xiong},
      year={2023},
      eprint={2312.16132},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

RoleEval

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README