TRACE-Bench

KO | EN

gitlite — search

#golang #linux #docker #nodejs #java #windows #awesome #cli #kubernetes #vue #security #mcp #ios #openai

(★ 14)

TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation

TRACE-Bench 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">
 <img src="asset/logo.png" alt="TRACE BENCH logo" height="32" style="vertical-align: middle;">
 TRACE BENCH: Task-driven Roleplay Agentic Checklist Evaluation
</h1>

Kuaishou GameMind Lab

**TRACE BENCH** evaluates whether roleplay models satisfy fixed, role-specific requirements in natural multi-turn interaction. An **Agentic User** actively probes each checklist item, tracks evidence in a private todo state, and ends the conversation only after the evaluation reaches a terminal state or an explicit runtime limit/error occurs. This makes each score traceable to both a fixed target and its dialogue evidence.

## How It Works

1. Build a fixed checklist from each role profile and interaction scenario.
2. Let the Agentic User adaptively converse with the target roleplay model.
3. Track checklist states and supporting evidence outside the public dialogue.
4. Score completed requirements against the unchanged prebuilt checklist.
5. Export dialogue traces, deterministic scores, HTML reports, and leaderboard files.

The Agentic User uses two tools: `todo_list` updates private checklist state, while `finish_conversation` requests termination after all required items have been evaluated.

## Quick Start

```bash
# 1. Create an environment and install dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 2. Create local configuration
cp config.example.yaml config.yaml
cp .env.example .env
# Fill in OPENAI_BASE_URL, OPENAI_API_KEY,
# TARGET_MODEL, and AGENTIC_USER_MODEL in .env

# 3. Validate local resources and run no-API checks
python main.py --config config.yaml validate-config
python scripts/smoke_tests.py

# 4. Run a small API smoke evaluation before the full benchmark
python main.py --config config.yaml run-agentic \
  --sample 2 \
  --run-id smoke_test

# 5. Run all configured cases
python main.py --config config.yaml run-agentic
```

`TARGET_MODEL` is the roleplay model under evaluation. `AGENTIC_USER_MODEL` drives the active evaluation and is also reused for language-quality judgments. It should support tool calling or reliably emit the supported `<tool_call>...</tool_call>` format.

Model calls are routed through LiteLLM. OpenAI-compatible endpoints work with the example configuration; Anthropic, Gemini, Bedrock, and other providers can use their LiteLLM-native environment variables. See [Model Requirements](docs/MODEL_REQUIREMENTS.md).

## Common Commands

```bash
# Use a fixed run id
python main.py --config config.yaml run-agentic --run-id qwen3_eval

# Use a custom case file
python main.py --config config.yaml run-agentic \
  --cases-file data/agentic/my_cases.jsonl

# Enable an explicit turn cap
python main.py --config config.yaml run-agentic \
  --agentic-max-turns \
  --max-turns 20
```

For all commands and options, see the [CLI Reference](docs/CLI_REFERENCE.md).

## Agentic Evolution

Completed traces can be used to refine the `verification_flow` of failed checklist items while preserving case identities, role prompts, and checklist content.

```bash
python main.py --config config.yaml evolve-agentic \
  --cases-file data/agentic/agentic_200.jsonl \
  --run-dir results/run_a \
  --run-dir results/run_b \
  --output-dir data/agentic_sets/evolved_001

python main.py --config config.yaml validate-evolved-data \
  --cases-file data/agentic_sets/evolved_001/cases.jsonl \
  --manifest data/agentic_sets/evolved_001/manifest.json
```

See [Agentic Evolution Design](docs/AGENTIC_EVOLUTION_DESIGN.md) for inputs, stages, outputs, and invariants.

## Output Structure

```text
results/<RUN_ID>/
├── leaderboard.md
├── leaderboard.json
└── target/
 ├── dialogue.jsonl # Public dialogue, private trace, final todos
 ├── score.json # Deterministic scores and quality breakdowns
 └── html_view/
 ├── index.html # Static browsable dialogue report
 └── *.html
```

Generation is resumable: rerunning with the same output path skips `test_id` values already present in `dialogue.jsonl`.

## Documentation

| Document | Description |
|----------|-------------|
| [CLI Reference](docs/CLI_REFERENCE.md) | Commands and options |
| [Data Format](docs/DATA_FORMAT.md) | Case schema and field definitions |
| [Scoring](docs/SCORING.md) | Primary metric and output contract |
| [Prompts & Tools](docs/PROMPTS_AND_TOOLS.md) | Agentic User prompts and tool schemas |
| [Model Requirements](docs/MODEL_REQUIREMENTS.md) | Model capabilities and turn limits |
| [Agentic Evolution Design](docs/AGENTIC_EVOLUTION_DESIGN.md) | Trace-grounded verification refinement |
| [Codebase Overview](docs/CODEBASE.md) | Architecture and module responsibilities |

## Project Layout

```text
.
├── main.py                         # CLI entry point
├── config.example.yaml             # Configuration template
├── .env.example                    # Environment template
├── data/agentic/agentic_200.jsonl  # Default 200-case benchmark
├── docs/                           # User and architecture documentation
├── scripts/smoke_tests.py          # Local no-API validation
├── skills/                         # English case-construction assets
└── src/
    ├── runner.py                   # Benchmark and evolution orchestration
    ├── core/                       # Config, schemas, prompts, IO, concurrency
    ├── generators/                 # Agentic User dialogue loop and tools
    ├── evaluators/                 # Response-quality evaluation
    ├── evolution/                  # Trace mining and verification refinement
    ├── models/                     # LiteLLM model adapter
    └── reporters/                  # Scoring, HTML, and leaderboard writers
```

## Citation

If you use TRACE BENCH in your research, please cite:

```bibtex
@misc{tracebench2026,
  title  = {{TRACE Bench}: Task-driven Roleplay Agentic Checklist Evaluation},
  author = {Jiahui Zhang and Ziwei Zhang and Yipeng Wang and Yibo Liu
            and Haozhou Pang and Yikai Hu and Hongyan Ren and Lan Zhou
            and Qi Gan and Kai Sheng},
  year   = {2026},
  eprint = {2608.11236},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2608.11236}
}
```

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for the full text.

TRACE-Bench

TRACE-Bench 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

TRACE-Bench

TRACE-Bench 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README