ToolPRMBench

KO | EN

gitlite — search

#javascript #python #react #go #android #linux #nodejs #java #windows #awesome #cli #vue #mcp #open-source

(★ 9)

Github repository for paper: "ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents"

ToolPRMBench Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents

[![Paper](https://img.shields.io/badge/Paper-arXiv:2508.01191-%23f2806bff.svg)](https://arxiv.org/abs/2601.12294) [![Code](https://img.shields.io/badge/Code-GitHub-%238a91faff.svg)]() [![Daily](https://img.shields.io/badge/Daily_Paper-Hugging_Face-%2340D9B0.svg)](https://huggingface.co/papers/2601.12294)

This repository contains the official Python implementation of the framework described in the paper **"ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents."**

## News

- **[01/20/2026]** Our paper is available on [arXiv](https://arxiv.org/abs/2601.12294).
- **[01/17/2026]** GitHub repository created. Code release is coming soon.

## Introduction

Reward-guided search methods have demon- strated strong potential in enhancing tool-using agents by effectively guiding sampling and ex- ploration over complex action spaces. As a core design, those search methods utilize process reward models (PRMs) to provide step-level rewards, enabling more fine-grained monitor- ing. However, there is a lack of systematic and reliable evaluation benchmarks for PRMs in tool-using settings. In this paper, we intro- duce **ToolPRMBench**s, a large-scale benchmark specifically designed to evaluate PRMs for tool- using agents. ToolPRMBench is built on top of several representative tool-using benchmarks and converts agent trajectories into step-level test cases. Each case contains the interac- tion history, a correct action, a plausible but incorrect alternative, and relevant tool meta- data. We respectively utilize offline sampling to isolate local single-step errors and online sampling to capture realistic multi-step failures from full agent rollouts. A multi-LLM verifica- tion pipeline is proposed to reduce label noise and ensure data quality. We conduct exten- sive experiments across large language models, general PRMs, and tool-specialized PRMs on ToolPRMBench. The results reveal clear differ- ences in PRM effectiveness and highlight the potential of specialized PRMs for tool-using.

<img src="figure/illustration.png" alt="illustration" width="50%" /> 
 Figure 1: Comparison between ToolPRMBench and other RewardBench.

## Contribution

⭐ **New Benchmark.** We propose ToolPRMBench, a large-scale benchmark carefully constructed for systematic evaluation of PRMs  in tool-using settings.

⭐ **Comprehensive Evaluation.** We benchmark a series of LLMs, gen- eral PRMs, and tool-using specialized PRMs in ToolPRMBench, building a comprehensive PRMs leaderboard in the tool-using scenario.

⭐ **Further Insights.** We conduct further analysis, including meta-evaluation and cost analysis, providing findings and insights for future reward-guided trajectory searching in tool-using.

<img src="figure/main.png" alt="main" width="100%" /> 
 Figure 2: The overview pipeline of ToolPRMBench, including trajectory sampling, data verification & filtering, and
ToolPRM training.

## Repository Structure

```
ToolPRMBench/
├── data/                      # ToolPRMBench evaluation & training data (v3) + data card
├── results/                   # Precomputed per-model / per-dataset result files
├── scripts/                   # One-command launch scripts
│   ├── run_toolprm.sh         #   ToolPRM family (served via vLLM)
│   └── run_specialized_prms.sh#   General / specialized PRM baselines
├── third_party/
│   └── skywork-o1-prm-inference/  # Vendored Skywork PRM helper code
├── toolprmbench/              # Python package
│   ├── core.py                #   Shared data IO, ordering, scoring, eval loop
│   ├── generation.py          #   API / vLLM backends (keys read from env)
│   ├── models.py              #   Registry of all benchmarked models
│   ├── evaluate.py            #   Unified CLI dispatcher
│   └── evaluators/            #   One module per model family
├── requirements.txt
└── pyproject.toml
```

## Installation

```bash
pip install -e .            # core package
pip install -e ".[local]"   # + torch / transformers / vllm for local models
pip install -e ".[api]"     # + openai / anthropic / gemini / together clients
```

> **API keys.** No credentials are stored in this repository. Hosted-API and
> vLLM-served evaluators read keys from the environment:
> `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `TOGETHER_API_KEY`,
> and `TOOLPRM_BASE_URL` (the local vLLM endpoint, e.g. `http://localhost:8001/v1`).

## Dataset

The benchmark data lives in [`data/`](data/) (see the [data card](data/README.md)).
Each case pairs a correct action (`action_chosen`) with a plausible-but-incorrect
alternative (`action_rejected`); a PRM is correct when it prefers the former.

| Split | Source | # cases |
| --- | --- | --- |
| `prmbench_GTA.json` | GTA | 118 |
| `prmbench_bfcl.json` | BFCL | 111 |
| `prmbench_tooltalk.json` | ToolTalk | 86 |
| `prmbench_ToolSandbox.json` | ToolSandbox | 130 |

Override the location with `--data_dir` or `TOOLPRMBENCH_DATA_DIR`.

## Models

### Tool-using specialized PRMs (ToolPRM family)

Trained for this benchmark and released at [huggingface.co/wjldw](https://huggingface.co/wjldw):

| Model | Weights | Evaluator |
| --- | --- | --- |
| ToolPRM-Base-v4 | https://huggingface.co/wjldw/ToolPRM-Base-v4 | `toolprm` |
| ToolPRM-GRPO-v4 | https://huggingface.co/wjldw/ToolPRM-GRPO-v4 | `toolprm` |
| ToolPRM-GRPO-synthesis | https://huggingface.co/wjldw/ToolPRM-GRPO-synthesis | `toolprm` |
| ToolPRM-CoT-v4 | https://huggingface.co/wjldw/ToolPRM-CoT-v4 | `toolprm_cot` |

### Baselines (general / specialized PRMs and LLM judges)

| Model | Weights | Evaluator |
| --- | --- | --- |
| Skywork-o1-Open-PRM | https://huggingface.co/Skywork/Skywork-o1-Open-PRM-Qwen-2.5-7B | `skywork` |
| RM-R1 | https://huggingface.co/gaotang/RM-R1-Qwen2.5-Instruct-7B | `rmr1` |
| RRM-7B | https://huggingface.co/Reward-Reasoning/RRM-7B | `rrm` |
| Nemotron-GenRM | https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-GenRM | `nemotron` |
| ToolACE-8B | https://huggingface.co/Team-ACE/ToolACE-8B | `toolace` |
| xLAM-7b-r | https://huggingface.co/Salesforce/xLAM-7b-r | `xlam` |
| Qwen3-Coder | https://huggingface.co/Qwen/Qwen3-Coder-Next | `llm_judge` |

List everything registered at any time:

```bash
python -m toolprmbench.evaluate --list
```

## Evaluation

All evaluators share one entry point. Results are written to
`results/prmbench_<dataset>_<model>.json` and the accuracy is printed.

**By registry key** (fills in the evaluator and Hugging Face id automatically):

```bash
python -m toolprmbench.evaluate --model-key RM-R1 --dataset bfcl
```

**Explicitly** (any model id / local path):

```bash
python -m toolprmbench.evaluate --evaluator xlam \
    --model Salesforce/xLAM-7b-r --dataset ToolSandbox
```

**ToolPRM family** (serve with vLLM first, then evaluate):

```bash
python -m vllm.entrypoints.openai.api_server \
    --model wjldw/ToolPRM-Base-v4 --port 8001 --trust-remote-code &
export TOOLPRM_BASE_URL=http://localhost:8001/v1
python -m toolprmbench.evaluate --evaluator toolprm \
    --model wjldw/ToolPRM-Base-v4 --dataset bfcl
```

**Batch runs** over every dataset:

```bash
bash scripts/run_toolprm.sh            # ToolPRM family (handles serving)
bash scripts/run_specialized_prms.sh   # all baseline PRMs
```

## Citation

If our repo helped you out, we'd love it if you gave us a citation! Thanks for supporting our work!

```tex
@article{li2026toolprmbench,
      title={ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents}, 
      author={Dawei Li and Yuguang Yao and Zhen Tan and Huan Liu and Ruocheng Guo},
      year={2026},
      eprint={2601.12294},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2601.12294}, 
}
```

ToolPRMBench

ToolPRMBench Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

ToolPRMBench

ToolPRMBench Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README