KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #dsh #dsh-plugin #ai-agent #codex #ai-agents #deepseek-harness #claude-code #cli #react #mcp

PlanBench-XL

★ 38 Open GitHub ↗

Official Repository for our paper: PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# PlanBench-XL

![](https://raw.githubusercontent.com/JiayuJeff/PlanBench-XL/main/figures/PlanBench_XL.png)

This is the official repository for **PlanBench-XL**, a benchmark for evaluating LLM agents' **long-horizon planning with massive tool ecosystems** under retrieval noise and path-preserving blocker events.

## 🎯 Project Overview

**PlanBench-XL** tests whether tool-use agents can solve retail planning tasks when they cannot see the whole tool universe at once. At each turn, the agent may retrieve relevant tools, call an available tool, or submit a final answer. The runtime tracks discovered tools, trusted and untrusted intermediate values, executable traces, and final-answer correctness.

The current release contains a retail-domain benchmark with **327 queries**, **56 datatypes**, and **1,665 tools**.

## ✨ Core Features

- **Massive Tool Retrieval**: Agents retrieve tools from a large tool ecosystem instead of receiving a complete tool list up front.
- **Long-Horizon Planning**: Ground-truth solution paths span at least 5 tool steps, requiring multi-step composition across intermediate values.
- **Noisy Tool Ecosystem**: Retrieved executable tools can be augmented with noisy sibling tools.
- **Path-Preserving Blockers**: Blocker configurations simulate corrupted tool access by injecting explicit, implicit, and semantic-misleading failures, while preserving at least one valid solution path for each query.
- **Detailed Evaluation**: The evaluator reports answer accuracy, turn counts, search/call balance, invalid tool-call rates, noisy-tool usage, and executed ground-truth datatype coverage.

## 📊 Main Results

![](https://raw.githubusercontent.com/JiayuJeff/PlanBench-XL/main/figures/Main_Table.png)

## 🚀 Quick Start

### Installation

```bash
# Clone the repository
git clone <repo-url>
cd PlanBench-XL

# Install dependencies
pip install openai pyyaml requests httpx tqdm
```

Python 3.10 or newer is recommended. Run commands from the repository root.

### Environment Configuration

1. **API Models**

Configure your API key and OpenAI-compatible base URL:

```bash
   cp src/env/config/.env.example src/env/config/.env
   source src/env/config/.env
   ```

Then edit `src/env/config/.env`:

```bash
   export OPENAI_API_KEY="your-api-key"
   export OPENAI_BASE_URL="your-base-url"
   ```

Model YAMLs are stored in `src/env/config/models/openai/` and registered in `src/env/config/model_registry.yaml`.

2. **Local Models**

Local model configs use OpenAI-compatible endpoints, for example:

```yaml
   base_url: http://127.0.0.1:8001/v1
   api_key: EMPTY
   ```

Edit the corresponding `*-local.yaml` file if your local server uses a different port.

### Running the Benchmark

Run one YAML configuration:

```bash
python src/env/run.py \
  --run_config src/env/config/runs/retail/gpt-5.4/retail_gpt5.4_default.yaml
```

Run a blocker configuration:

```bash
python src/env/run.py \
  --run_config src/env/config/runs/retail/gpt-5.4/retail_gpt5.4_blocker.yaml
```

Override config fields from the command line:

```bash
python src/env/run.py \
  --run_config src/env/config/runs/retail/gpt-5.4/retail_gpt5.4_default.yaml \
  --set query_sample.size=20 \
  --set output.output_dir=retail/debug/gpt-5.4
```

Runtime outputs are written under `outputs/` by default. Each run directory contains:

```text
metadata.json               # merged configuration snapshot
result.jsonl                # completed per-query results
progress/index.json         # per-query progress index
progress/queries/*.json     # resumable per-query traces and checkpoints
```

### Batch Runs

List available retail run YAMLs:

```bash
python scripts/run_retail_batch.py --list
```

Run all non-local configs for one model:

```bash
python scripts/run_retail_batch.py --model gpt-5.4
```

Run selected configs only:

```bash
python scripts/run_retail_batch.py \
  --model gpt-5.4 \
  --config default,blocker
```

The batch helper skips `*-local` model directories by default. Include them only when the corresponding local OpenAI-compatible servers are running:

```bash
python scripts/run_retail_batch.py --include-local-models --model llama3.3-70b-local
```

To inspect local ports parsed from model YAMLs:

```bash
python scripts/run_retail_batch.py --list-local-model-ports
```

## 📖 Advanced Usage

### Run Configs

Retail run configs are located under:

```text
src/env/config/runs/retail/<model-name>/*.yaml
```

Common settings include:

- `model_ref`: Model registry key defined in `src/env/config/model_registry.yaml`.
- `data.*_file`: Retail dataset files used by the run.
- `output.output_dir`: Run-specific directory under `output.root_dir`.
- `runtime.max_steps`: Maximum action steps per query.
- `runtime.max_concurrency`: Number of concurrent query workers.
- `query_sample.size`: Optional number of queries to sample.
- `query_sample.seed`: Seed for deterministic query sampling.

### Noise Parameters

- `noise.mode`: `append_noisy_siblings` appends noisy sibling tools for retrieved primary tools.
- `noise.max_total_tools`: Maximum number of tools returned to the agent after augmentation.
- `noise.ratio`: Reserved in config; current retail run YAMLs use `ratio: 0.0` with `append_noisy_siblings`.

### Blocker Parameters

- `blocker.enable_block`: Enables runtime replacement of selected baseline tools.
- `blocker.selection_mode`: Supports path-targeted selection such as `target_remaining_paths` and `target_remaining_ratio`.
- `blocker.target_remaining_paths`: Target number of surviving ground-truth paths.
- `blocker.target_remaining_ratio`: Target ratio of surviving ground-truth paths.
- `blocker.fixed_noise_type`: Uses one blocker type, such as `explicit failures`, `implicit failures`, or `semantic misleading`.
- `blocker.fixed_noise_types`: Uses multiple blocker types for each selected baseline tool.
- `blocker.seed`: Seed used for blocker planning and reproducible final tool-order shuffling.
- `blocker.max_combo_candidates` / `blocker.max_cover_size`: Search limits for blocker selection.

### Model Parameters

Model YAMLs expose OpenAI-compatible request parameters:

- `model`: Provider model name sent to the API.
- `api_key_env` / `base_url_env`: Environment-variable based API credentials.
- `base_url` / `api_key`: Direct endpoint settings for local models.
- `request.temperature`: Sampling temperature.
- `request.max_tokens`: Maximum response tokens.
- `request.timeout_seconds`: Request timeout.

## 📦 Dataset

The retail benchmark data is stored in `src/data/retail/`:

## 📈 Evaluating Results

Evaluate a completed or partially resumed run directory:

```bash
python src/env/evaluate.py \
  --output_dir outputs/retail/gpt-5.4/default
```

By default, this writes:

```text
outputs/retail/gpt-5.4/default/evaluation.json
```

The evaluation report includes:

- `accuracy` and `correct_count`
- mean and total turn counts
- explored datatype counts
- search-to-call ratio
- invalid tool-call and untrusted-input rejection rates
- noisy-tool usage rates
- executed ground-truth datatype precision and recall
- per-query traces and failure details

## 🔧 Extending

To add a new model:

1. Add a model YAML under `src/env/config/models/openai/`.
2. Register it in `src/env/config/model_registry.yaml`.
3. Add one or more run YAMLs under `src/env/config/runs/retail/<model-name>/`.
4. Run with `python src/env/run.py --run_config <your-run-yaml>`.

The current code release targets the retail domain.

## 🤝 Contributing

Contributions are welcome. Please open an issue or pull request for bug fixes, new model configs, or benchmark extensions.

## 📚 Citing this work

```bibtex
@article{liu2026planbench,
  title={PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems},
  author={Liu, Jiayu and Lin, Qihan and Qian, Cheng and Wang, Rui and Acikgoz, Emre Can and Yang, Xiaocheng and Liu, Jiateng and Wang, Zhenhailong and Chen, Xiusi and Ji, Heng and others},
  journal={arXiv preprint arXiv:2606.22388},
  year={2026}
}
```

PlanBench-XL

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README