KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #claude-code #codex #dsh #dsh-plugin #ai-agent #ai-agents #deepseek-harness #agent-skills #macos #cli

VulcanBench

★ 59 Open GitHub ↗

Open source, clear, transparent, real world llm benchmarks

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VulcanBench

[![CI](https://github.com/morganlinton/VulcanBench/actions/workflows/ci.yml/badge.svg)](https://github.com/morganlinton/VulcanBench/actions/workflows/ci.yml)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://raw.githubusercontent.com/morganlinton/VulcanBench/main/LICENSE)
[![Python](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)

Fully open-source benchmarking for LLMs on realistic, multi-file software
engineering tasks. VulcanBench measures how models perform across reasoning
effort, language, codebase scale, and task complexity — with full traces,
reproducible scoring, and a local dashboard.

**v0.8.0** — adds the **Voice Eval Suite v1** (`vulcanbench voice`): text-vs-audio delta measurement ("voice tax") across OpenAI Realtime, Gemini Live, and Qwen3-Omni, with a 200-question held-out set, a voices/rate/noise audio matrix, and modality-blind scoring. See [docs/VOICE_EVAL.md](docs/VOICE_EVAL.md). Previous: **v0.7.0** — adds a **Qwen / DashScope provider** (`qwen:qwen3.7-plus` and friends)
so Alibaba Cloud models can be benchmarked like OpenAI / Anthropic / Z.ai / Kimi.
Builds on v0.6's frontier-hard task tier and cost-efficient reporting
(`--max-run-cost`, `compare`, `regrade`, `--only-missing`), and on v0.5's 52
gold-verified tasks, tool-calling agent, Docker sandbox, pre-run cost estimates,
five-metric scoring, and HTML replay.
See [docs/QUICKSTART.md](docs/QUICKSTART.md) to get started.

## One-command setup

```bash
git clone https://github.com/morganlinton/VulcanBench.git
cd VulcanBench
make setup
source .venv/bin/activate
vulcanbench --help
```

Dashboard + backend (the dashboard reads live data from the API):
```bash
pip install -e ".[backend]"
uvicorn backend.app:app --port 8000          # serves ./runs at /api/*
cd dashboard && npm install && npm run dev    # http://localhost:3000
```
The dashboard falls back to a friendly empty state if the backend isn't running.
Point it elsewhere with `NEXT_PUBLIC_API_BASE` (see `dashboard/.env.example`).

By default the API reads `./runs/` directly. For a durable, queryable store, set
`DATABASE_URL` (Postgres or SQLite) and the API switches to a database —
`POST /api/runs` and `/api/feedback` become writable, and
`python scripts/ingest_runs.py` bulk-loads existing runs. A Postgres is provided
by `docker compose up db`.

## Example run

```bash
# Offline, deterministic (no API key) — drives the real agent loop end to end.
# Real runs default to the Docker sandbox; --sandbox local is fine for the
# deterministic mock model.
vulcanbench run --task hello-world --model mock:synthetic --sandbox local

# Any real model via the generic provider interface:
export OPENAI_API_KEY=...      # or ANTHROPIC_API_KEY=...
vulcanbench run --task hello-world --model openai:gpt-4o
vulcanbench run --task hello-world --model anthropic:claude-opus-4-8
vulcanbench run --task hello-world --model zai:glm-5.2
vulcanbench run --task hello-world --model qwen:qwen3.7-plus
vulcanbench run --task hello-world --model deepseek:deepseek-v4-flash

# Each run prints all five metrics + cost, e.g.:
#   functional=1.0 quality=1.0 security=1.0 human_like=0.8 total=0.974 cost=$0.0
# and writes ./runs/<id>/{trace.jsonl, summary.json, replay.html, final.patch}

# Run a whole suite: repeat for signal, parallelize, and cap the spend.
vulcanbench run --suite v1 --model openai:gpt-4o --repeat 5 --max-concurrency 4 --max-cost 20.00

# Compare normalized reasoning effort on the same suite/model.
vulcanbench run --suite v1 --model openai:gpt-4o --effort low
vulcanbench effort-sweep --suite v1 --model openai:gpt-5.1 --efforts low,medium,high --repeat 3 --sandbox docker

# Fast micro/small sweep vs navigation-heavy medium/large tasks:
vulcanbench run --suite v1-micro --model openai:gpt-4o
vulcanbench run --suite v1-large --model openai:gpt-4o --repeat 5 --sandbox docker

# Diamond tier: rubric-graded *mergeability* (not just correctness). Use a judge
# model different from the one under test to avoid self-grading:
vulcanbench run --suite v1-diamond --model anthropic:claude-opus-4-8 --judge-model openai:gpt-5.5
# Carbyne tier: harder still — terse prompts where the naive solution is subtly wrong:
vulcanbench run --suite v1-carbyne --model anthropic:claude-opus-4-8 --judge-model openai:gpt-5.5
vulcanbench leaderboard            # by model: pass@1 ± stderr, pass@k, cost, latency
vulcanbench leaderboard --by run   # per-run drill-down
vulcanbench report -o report.md    # shareable Markdown/JSON report (ranking,
                                   #   model-separation/discrimination, effort
                                   #   sensitivity, per-task breakdown,
                                   #   environment, drift flags)
vulcanbench calibrate              # empirical difficulty calibration from recorded runs
vulcanbench replay <id>

# Runs execute in an isolated container by default (see Sandbox below);
# build the image once with `make sandbox-image`.

# Use it as a CI regression gate (threshold must be in [0, 1]):
vulcanbench run --suite v1 --model openai:gpt-4o --repeat 5 --fail-under 0.8
```

The gate **fails closed**: it exits `4` if pass@1 is below the threshold, if
pass@1 is unavailable, *or if any suite run errored* — a CI gate never goes
green on a partial or unknown result.

Exit codes: `0` ok · `1` usage/error · `2` provider · `3` sandbox · `4` gate
failed (below `--fail-under`, or a run errored).

`final.patch` is a real `git diff` of the agent's edits; `replay.html` is fully
self-contained (open in any browser). Use `--no-judges` to skip the LLM judge
ensemble, `--timeout SECONDS` to cap a run's wall-clock. Traces, summaries, and
patches are secret-redacted and size-capped before they're written, so run
artifacts are safe to publish. See `make ci`, `make docker-up`, docs/.

## Cost-efficient reporting

Grading is deterministic and every run records the `task_hash` it was scored
against, so comparisons are queries over `./runs`, not re-runs. Four commands
turn a "$70 full-matrix re-run" into a "~$10 one new column":

```bash
# Per-run hard cost ceiling: stop a single agent run once its own spend crosses
# the value (records cost_capped; the partial result is still graded). Turns a
# failing run that would ruminate to the step cap into a bounded "DNF at cap".
vulcanbench run --suite v2 --model anthropic:claude-fable-5 --max-run-cost 2.50

# Resume / fill only the gaps: reuse fresh cached runs for this model+effort and
# launch only the missing tasks (stale runs, scored against an older task
# definition, are ignored and re-run). Pairs with --max-run-cost.
vulcanbench run --suite v2 --model anthropic:claude-opus-4-8 --effort high \
  --only-missing --max-run-cost 2.50

# Assemble the model × effort matrix for a frozen suite from cached runs only —
# baselines are never re-run. Add a model = run that one model, then re-compare.
vulcanbench compare --suite v2                 # complete cells (prints a frozen version id)
vulcanbench compare --suite v2 --incomplete    # show gaps + the command to fill them

# Re-score existing runs against the current task definition at $0 API cost
# (rebuilds base + captured agent patch + current hidden tests, re-verifies).
# Use after editing a task's tests/thresholds instead of re-running the model.
vulcanbench regrade runs/<run-id> --sandbox docker
vulcanbench regrade runs/ --sandbox docker     # every run under a directory
```

Cache reuse (`--only-missing`) and comparison (`compare`) only see runs under the
directory they scan (`--output-dir` / `--runs-dir`, default `./runs`, recursive),
so keep every run under one root. See `docs/QUICKSTART.md` for the full workflow.

## Models

Specify a model as `provider:model`:

- `mock:synthetic` — deterministic, offline; used by tests and demos.
- `openai:<model>` — OpenAI Chat Completions for normal runs, or the Responses
  API when `--effort` is supplied. Needs `OPENAI_API_KEY`.
- `anthropic:<model>` — Anthropic Messages API. Needs `ANTHROPIC_API_KEY`.
- `zai:<model>` — Z.ai (Zhipu) OpenAI-compatible Chat Completions API. Needs
  `ZAI_API_KEY`. Reasoning effort is not supported; `--effort` is recorded as
  metadata only.
- `kimi:<model>` — Moonshot AI (Kimi) OpenAI-compatible Chat Completions API.
  Needs `MOONSHOT_API_KEY`. `extra-high` maps to Kimi's `reasoning_effort=max`;
  other effort levels are recorded as metadata only.
- `qwen:<model>` — Alibaba Cloud DashScope (Qwen) OpenAI-compatible Chat
  Completions API. Needs `DASHSCOPE_API_KEY`. Default base URL is the
  international endpoint; set `DASHSCOPE_BASE_URL` for China or another region.
  `low`/`medium` map to Qwen's `reasoning_effort` and `extra-high` maps to its
  `xhigh` (Qwen3.8+; the documented enum is low/medium/xhigh, default xhigh).
  `high` is recorded as metadata only — Qwen has no such level, and an unset
  request runs at the xhigh default. Pre-3.8 models may ignore the field.
- `deepseek:<model>` — DeepSeek OpenAI-compatible Chat Completions API. Needs
  `DEEPSEEK_API_KEY`. `low`/`high` map to DeepSeek's `reasoning_effort` field
  and `extra-high` maps to its `max`; `medium` is recorded as metadata only
  (DeepSeek's enum is low/high/max — it silently coerces `medium` to `high`,
  so the harness never sends it).

`--effort` accepts `low`, `medium`, `high`, or `extra-high`. OpenAI runs map it
to the Responses API `reasoning.effort` field; Anthropic runs map it to the
Messages API `output_config.effort` field. `extra-high` maps to `xhigh` on both
providers and is opt-in for sweeps because support is model-dependent (e.g.
Claude Opus 4.7+). Mock, Z.ai, and Qwen runs accept the field as no-op metadata.
Effort labels are each provider's own scale — a cross-provider comparison at the
same label compares each model at its own setting, not a calibrated equivalence.

## Sandbox

The agent's tool execution can run in an isolated Docker container instead of on
the host:

```bash
# Build the base image once (git, ripgrep, ruff, bandit, radon, pytest):
docker build -t vulcanbench/sandbox:base -f sandbox/Dockerfile.base .

vulcanbench run --task hello-world --model openai:gpt-4o
# --sandbox local|docker|auto   (default: docker)
# --image vulcanbench/sandbox:base   (default: per-task metadata or vulcanbench/sandbox:base)
# --network                     (off by default; opt in for dependency installs)
```

- `docker` (default) runs tools in a non-root, **network-off**, resource-limited
  container (workspace bind-mounted, cleaned up after each run). It errors out
  if the daemon is unreachable — it never silently falls back to host execution.
- `local` runs the model's commands directly on the host — fast and Docker-free,
  but unsandboxed; opt in deliberately (fine for `mock:synthetic` and trusted
  dev loops).
- `auto` uses Docker when available. Falling back to host execution additionally
  requires `VULCANBENCH_ALLOW_HOST_EXEC=1`; otherwise it errors out.

File operations (read/edit/search) always run host-side over the shared mount;
command execution (`run_command`/`run_tests`/`run_lint`) **and the functional
verifier** run inside the container, so the whole run is reproduced in one
isolated environment. Build the all-language image with `docker build -t
vulcanbench/sandbox:base -f sandbox/Dockerfile.base .` (Python + Go + Node).

## Tasks

The `tasks/v1/` suite holds **52** gold-verified tasks across Python,
Go, TypeScript, and Rust, plus the `hello-world` demo. Each task ships a starting
`repo/`, **hidden** `tests/` (never shown to the agent), declarative
`fail_to_pass`/`pass_to_pass` test commands in `metadata.json`, and a
`gold_patch.diff` reference solution.

The corpus spans three difficulty tiers (`easy` / `medium` / `hard`) across all
four languages. Most tasks today are `localized` single-file fixes that set a
floor; a growing set raise the ceiling with genuine subtlety — operator
precedence and associativity (`py-expr-eval`, hard), a race-free,
order-preserving parallel map verified under `go test -race`
(`go-parallel-map`, hard), an RFC 6901 JSON Pointer resolver
(`py-jsonpointer`, hard), and a prototype-pollution-safe deep merge
(`ts-deep-merge`, hard). Broader `task_complexity` (`multi_file` / `system` /
`architecture`) and larger `repo_scale` coverage is active work — see
[ROADMAP](docs/ROADMAP.md). Because the `task_complexity` and `repo_scale`
fields are validated against the repo, a task's declared scale is checked, not
just asserted. `vulcanbench report` includes a discrimination section so you can
see which tasks actually separate the models you run (and which carry no signal).

```bash
make validate-tasks                              # validate every task
vulcanbench validate-task tasks/v1/<id>          # one task
```

### Grading: hidden tests or an agentic grader

By default a task's `functional` score comes from **hidden tests** — deterministic
and exact, but it requires the issue to fully specify the expected behavior.

A task can instead opt into an **agentic grader** (`metadata.grader: "agentic"`)
that judges the agent's diff against a list of plain-English `acceptance_criteria`
(never shown to the agent), so the prompt can be **terse and realistic** — closer
to how developers actually ask. The grader, not the prompt, holds the spec.

```bash
# Use a strong, independent grader model to avoid a model grading its own work:
vulcanbench run --task py-slugify-terse --model openai:gpt-5.5 \
  --judge-model anthropic:claude-opus-4-8
```

Agentic grading is non-deterministic, so it is opt-in and never the default;
tasks that need exact, reproducible scoring keep the test verifier. See
`tasks/v1/py-slugify-terse` for an example.

**Trust the grader before you rely on it.** An LLM grader is only worth using if
it agrees with ground truth and doesn't flip its verdict run to run. Two tools:

- Set `metadata.grader_samples: N` to grade by **majority vote** over N calls
  (ties resolve to incorrect); each grade reports its `self_consistency`.
- `python scripts/grader_eval.py --task tasks/v1/<id> --model <grader> --samples 5`
  grades a task's labeled `grader_cases.json` (known-correct and known-incorrect
  changes) and reports **accuracy**, **false-pass rate** (graded correct but
  actually wrong — the dangerous one), and **self-consistency**. Don't ship an
  agentic task whose grader posts a non-zero false-pass rate.

Validation proves each task is real: the gold patch must solve it
(`functional == 1.0`), the `fail_to_pass` tests must genuinely fail *before* the
fix, and scoring must be deterministic over repeated runs.

**Provenance is labeled and checked.** Every task declares `source`
(`hand-authored` or `oss`) and an explicit `decontaminated` boolean. Hand-authored
tasks are written now (post-cutoff, so `decontaminated: true`); the validator
enforces that. An `oss` task (e.g. `oss-inflection-titleize`, sourced verbatim
from a real MIT-licensed repo with its LICENSE preserved) is honestly labeled
`decontaminated: false` — its fix predates model cutoffs — and the
`vulcanbench report` integrity section flags every run scored against it. Scaffold
one with `python scripts/import_oss_issues.py`. Format details:
[docs/TASK_CONTRIBUTION.md](docs/TASK_CONTRIBUTION.md).

## Voice Eval Suite

Measures the **voice tax**: how many points a model loses when the same
question arrives as speech instead of text. 200 held-out questions
(`tasks/voice-v1/`), rendered via TTS under a voices × rate × noise matrix,
answered through each model's realtime/audio endpoint, and scored by a
modality-blind scorer (results have no numbers to show yet — none are
published until the first full run).

```bash
export OPENAI_API_KEY=...     # TTS + Realtime + STT fallback
export GEMINI_API_KEY=...     # Gemini Live
export DASHSCOPE_API_KEY=...  # Qwen3-Omni
export XAI_API_KEY=...        # Grok Voice (grok-voice-think-fast-2.0)

vulcanbench voice run -m openai-realtime --dry-run     # 5-question pipeline check
vulcanbench voice run -m openai-realtime,gemini-live,qwen-omni,grok-voice
vulcanbench voice report runs/voice-<id> -o report.md
```

Full methodology: [docs/VOICE_EVAL.md](docs/VOICE_EVAL.md).

## Architecture & Reproducibility

- Standardized tools (list/read/edit/search/run) behind one protocol, with
  interchangeable local and Docker executors (see [Sandbox](#sandbox))
- Every step captured as JSONL (llm, tool, diff, test, metric) + token usage
- Each run records its `vulcanbench replay <id>` command for reproduction
- Docker sandbox runs untrusted command execution in a non-root, network-off,
  resource-limited container

Full details: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md),
[docs/METRICS.md](docs/METRICS.md), [docs/REPRODUCIBILITY.md](docs/REPRODUCIBILITY.md)

## Documentation

| Doc | Purpose |
|-----|---------|
| [QUICKSTART](docs/QUICKSTART.md) | Setup, smoke test, first real run |
| [METRICS](docs/METRICS.md) | How the five scores are computed |
| [DEPLOYMENT](docs/DEPLOYMENT.md) | Hosted API + dashboard (optional) |
| [CONTRIBUTING](docs/CONTRIBUTING.md) | Add tasks, run CI locally |
| [ROADMAP](docs/ROADMAP.md) | Planned follow-ups |
| [results/](docs/results/) | Published benchmark snapshots (MD, JSON, PDF) |

Quality and security analyzers run when the relevant toolchains are on your
host (e.g. `bandit` for Python via the venv; `gosec` for Go if installed).
Otherwise those metrics report `null` with a reason — never a fabricated score.
Use `--no-judges` to skip the LLM judge ensemble and cut cost roughly threefold.

## License

Apache 2.0 (see LICENSE and NOTICE).

## Provider terms & data usage

VulcanBench is an independent evaluation harness. A few boundaries keep its use
consistent with the model providers' terms — please read these before running or
publishing results.

- **You bring your own keys, under your own agreement.** VulcanBench never
  bundles or shares API credentials. Each run uses the keys in your environment
  (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `ZAI_API_KEY`, `MOONSHOT_API_KEY`,
  `DASHSCOPE_API_KEY`, `DEEPSEEK_API_KEY`), so every call is made
  under *your* account and *your* commercial/API agreement with that provider.
  You are responsible for staying within your provider's terms and usage
  policies.

- **Outputs are for evaluation, not training.** Recorded run artifacts (traces,
  patches, summaries) capture model outputs solely for scoring, inspection, and
  reproducibility. Both OpenAI and Anthropic prohibit using their outputs to
  develop or train competing models — do not use VulcanBench artifacts, or any
  published corpus of them, for that purpose. VulcanBench intentionally has no
  "export outputs as a training dataset" feature.

- **Trademarks & independence.** "OpenAI" and "GPT" are trademarks of OpenAI;
  "Anthropic" and "Claude" are trademarks of Anthropic; "Z.ai" and "GLM" are
  trademarks of Zhipu AI; "Kimi" and "Moonshot" are trademarks of Moonshot AI;
  "Qwen" and "DashScope" are trademarks of Alibaba Cloud; "DeepSeek" is a
  trademark of DeepSeek. VulcanBench is not
  affiliated with, sponsored by, or endorsed by any of these companies. Model
  and provider names are used only to identify the systems under test.

This is not legal advice; consult the current provider terms for authoritative
guidance.

VulcanBench

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README