KO
|
EN
gitlite — search
Search
#python
#python3
#docker
#typescript
#hacktoberfest
#mysql
#discord
#rust
#javascript
#react
#django
#c
harness-bench-fast
★ 45
Open GitHub ↗
No description available.
Download README (.md)
Explore Similar Repositories
ESP32-P4-NINA-Display
:
NINA Astrophotography display using Waveshare ESP32-P4-WIFI6-Touch-LCD-4B
caesarpad
:
Caesar III on an iPad-shaped canvas, powered by Augustus.
PUBG-Performance-Booster
:
Complete Gameplay Enhancement Tool for PUBG: BATTLEGROUNDS. Visual Assistance, Aim Support, Radar Enhancement, Sound Optimization, Movement Tools. 25+ features
Gov_app
:
No description available.
DreamZero-Libero
:
Reproduction and adaptation of DreamZero on the LIBERO benchmark
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
harness-bench-fast
?
Download (.md)
# harness-bench ## Current Results Published benchmark results below use the full 391-task set (`task-set v0.16.0`). `Steps` and `Tokens` are shown when the runner exposes them; `—` means the metric is absent from the run artifact, not that nothing was spent. Each row is one full run per harness + model setup. The `Profile` column shows the deepagents harness profile applied: `GigaChat` = the `deepagents-gigachat` tuning profile, `Anthropic` = the Anthropic harness profile built into deepagents, `OpenAI` = the OpenAI harness profile built into deepagents, `none` = stock deepagents defaults, `—` = not applicable (non-deepagents harnesses). GigaChat rows are the IFT stand, build `32.9.23.6`. **Every `GigaChat`-profile row above was produced with `deepagents-gigachat` 0.0.3.** The pin now installs **0.0.4**, which is a result-affecting change, so a fresh install no longer reproduces those rows. Measured on GigaChat 3.5 (IFT), k=4 per version with versions interleaved on one stand: 0.0.3 342.0/391 (87.5%, sd 3.9) against 0.0.4 **352.8/391 (90.2%**, sd 2.8 over 6 runs, steps 3,262, tokens 6,049,158) — **+10.8 tasks**, with every 0.0.4 run above every 0.0.3 run. The gain is 0.0.4's deterministic-output gate, which blocks writing a derived value that was never computed. GigaChat 3 Ultra / Pro / Lightning have **not** been re-measured on 0.0.4, so their rows stay as published rather than mixing two profile versions in one table. On weaker models the same gate costs rather than pays: on the GigaChat-3.1-10B reasoning line it drives the agent to retry the blocked write until the graph budget is gone (−14.5 pp, step-limit hits 3.6% → 33.6%). Pin `0.0.3` if your model cannot act on a tool-level refusal; see [deepagents-gigachat#7](https://github.com/ai-forever/deepagents-gigachat/issues/7). Public landing page: <https://ai-forever.github.io/harness-bench-fast/> | Harness | Profile | Model | Result | % | Steps | Tokens | | --- | --- | --- | ---: | ---: | ---: | ---: | | Kimi CLI | — | Kimi K3 | 390/391 | 99.7% | — | — | | Pi | — | DeepSeek V4 Flash 0731 (high) | 387/391 | 99.0% | 1,694 | 10,004,355 | | Claude Code CLI | — | Claude Haiku 4.5 | 380/391 | 97.2% | 1,645 | 176,430,286 | | opencode | — | GLM-5.2 (self-hosted) | 362/391 | 92.6% | — | — | | deepagents | GigaChat | GigaChat 3.5 | 340/391 | 87.0% | 3,316 | 4,319,421 | | deepagents | GigaChat | GigaChat 3 Ultra | 340/391 | 87.0% | 3,258 | 4,415,043 | | deepagents | none | DeepSeek V4 Flash | 320/391 | 81.8% | 5,048 | 61,715,547 | | deepagents | none | GigaChat 3 Ultra | 312/391 | 79.8% | 3,591 | 7,994,643 | | deepagents | none | GigaChat 3.5 | 302/391 | 77.2% | 3,542 | 7,371,576 | | deepagents | none | Qwen3 Coder 30B-A3B | 284/391 | 72.6% | 5,467 | 90,234,558 | | deepagents | GigaChat | GigaChat 3 Pro | 241/391 | 61.6% | 2,993 | 3,975,672 | | deepagents | none | GPT-OSS-20B | 193/391 | 49.4% | 2,727 | 29,137,983 | | deepagents | none | GPT-OSS-120B | 186/391 | 47.6% | 2,193 | 23,831,283 | | deepagents | GigaChat | GigaChat 3 Lightning | 178/391 | 45.5% | 2,520 | 2,275,821 | Two notes on reading the table. The GigaChat profile is worth 7-10 points and roughly halves the token spend, which the two profiled/unprofiled GigaChat pairs show directly. And a larger model is not automatically a more capable agent: GPT-OSS-120B places below its own 20B sibling because it frequently answers in prose instead of calling a tool, scoring 0/15 on the VCS wave and 2/20 on CLI-composition while remaining competitive on single-shot waves. <details> <summary>Earlier results on task-set v0.13.0 (351 tasks) — not comparable</summary> The v0.16.0 audit changed task semantics across the set, so these scores cannot be compared with the table above. Kept for reference only. The `deepagents + GigaChat profile / GigaChat 3.5` row averages 3 independent full runs (317, 311, 308 passed), with Steps/Tokens as the mean of per-run sums. | Harness | Profile | Model | Result | % | Steps | Tokens | | --- | --- | --- | ---: | ---: | ---: | ---: | | Claude Code CLI | — | Claude Opus 4.8 | 351/351 | 100.0% | — | — | | deepagents | none | grok-4.5 | 346/351 | 98.6% | 3,164 | 30,776,418 | | Claude Code CLI | — | Claude Sonnet 4.6 | 341/351 | 97.2% | — | — | | Claude Code CLI | — | Claude Haiku 4.5 | 340/351 | 96.9% | — | — | | deepagents | none | GLM-5.2 | 340/351 | 96.9% | 3,966 | 41,664,423 | | deepagents | none | DeepSeek V4 Pro | 339/351 | 96.6% | 4,014 | 44,552,076 | | deepagents | none | GLM-5.1 | 335/351 | 95.4% | 3,802 | 39,320,469 | | deepagents | OpenAI | GPT-5.6 Luna | 330/351 | 94.0% | 4,392 | 40,181,133 | | deepagents | Anthropic | Claude Haiku 4.5 | 328/351 | 93.4% | 3,682 | 50,549,085 | | deepagents | none | Qwen 3.7 Max | 326/351 | 92.9% | 4,154 | 48,563,241 | | deepagents | none | DeepSeek V3.2 | 326/351 | 92.9% | 6,413 | 98,708,199 | | deepagents | none | Qwen 3.6 Flash | 325/351 | 92.6% | 4,334 | 49,387,938 | | deepagents | GigaChat | GigaChat 3.5 | 312/351 | 88.9% | 2,887 | 4,715,017 | | deepagents | none | DeepSeek V4 Flash | 310/351 | 88.3% | 4,082 | 46,732,794 | | deepagents | GigaChat | GigaChat 3 Ultra | 303/351 | 86.3% | 2,776 | 3,424,473 | | deepagents | none | GPT-4.1 | 300/351 | 85.5% | 3,382 | 34,199,277 | | opencode (gpt2giga) | — | GigaChat 3.5 | 298/351 | 84.9% | — | — | | deepagents | GigaChat | GigaChat 2 Max | 292/351 | 83.2% | 2,714 | 3,209,751 | | deepagents | none | Qwen 3.5 Flash | 288/351 | 82.1% | 3,507 | 44,553,189 | | deepagents | none | GigaChat 3.5 | 287/351 | 81.8% | 3,374 | 8,486,826 | | deepagents | none | MiniMax M2.7 | 282/351 | 80.3% | 3,387 | 38,231,538 | | deepagents | none | Qwen3-Coder-30B-A3B | 258/351 | 73.5% | 3,849 | 57,997,536 | | deepagents | GigaChat | GigaChat 3 Pro | 241/351 | 68.7% | 2,823 | 2,400,507 | | deepagents | none | yandex/gpt5.1-pro | 220/351 | 62.7% | 4,030 | 42,623,784 | | deepagents | none | GPT-OSS-120B | 185/351 | 52.7% | 2,054 | 22,079,085 | | deepagents | none | yandex/gpt5-pro | 180/351 | 51.3% | 2,407 | 20,845,269 | | deepagents | GigaChat | GigaChat 3 Lightning | 179/351 | 51.0% | 2,232 | 1,739,949 | | deepagents | none | Llama 4 Maverick | 57/351 | 16.2% | 1,391 | 17,065,878 | | deepagents | none | yandex/gpt5-lite | 37/351 | 10.5% | 1,666 | 118,843,500 | </details> A self-contained **391-task agent benchmark** (`task-set v0.16.0`) for evaluating LLM-backed coding agents on file-operation work: create / edit / refactor source files, transform CSV / JSON / JSONL / XLSX, run pytest, search across a project tree, write and use `MEMORY.md` per repo conventions, and chain all of that into multi-step pipelines. This benchmark is part of the [`GigaChain`](https://github.com/ai-forever/gigachain) project. Every task is **mechanically verified** — no LLM-as-judge. Verifiers use exact content checks where byte-for-byte output matters, plus regex matches, line lists, JSON parsing, importing a Python module and calling a function, running `pytest`, comparing SQLite query results, comparing XLSX cells, and so on. The benchmark exists to track how well an agent harness + model combination handles **realistic coding tasks** with adversarially-chosen edge cases (ambiguous prompts dropped; only honest, scoped-to-tool tests remain). It started life as `harness_bench/` inside [`deepagents-gigachat`](https://github.com/ai-forever/deepagents-gigachat) and was extracted into its own repo once it matured. ## Quick start ```bash # Install the bench in a fresh venv. The `[gigachat]` extra adds the # GigaChat client; `[openrouter]` adds the OpenAI-compatible client used # by `run-openrouter`. uv venv && uv pip install -e ".[gigachat,openrouter]" # Optional: install the public GigaChat harness profile. Exact v9/v10 # result reproduction may require installing the matching local # deepagents-gigachat wheel/source instead; after installing a local # wheel, use `uv run --no-sync ...` so uv does not re-resolve it back # to the public profile. uv pip install -e ".[gigachat-profile]" # List all 391 tasks uv run python -m harness_bench list # Show the benchmark task-set version and revision history uv run python -m harness_bench version --check # Run the whole bench against GigaChat (needs GIGACHAT_USER / # GIGACHAT_PASSWORD in .env or env, plus GIGACHAT_BASE_URL pointing at # the production gateway): uv run python -m harness_bench run --concurrency 5 # Run against any OpenAI-compatible OpenRouter model (needs # OPENROUTER_API_KEY): uv run python -m harness_bench run-openrouter \ --model deepseek/deepseek-v4-flash --concurrency 5 # Internal OpenAI-compatible gateways can use password auth instead of a # static API key. The runner fetches and refreshes a bearer token without # printing it: # OPENROUTER_USE_INTERNAL_TAGME=1 # local shortcut for the ignored tagme example # OPENROUTER_BASE_URL=https://gateway.example/x/ai/llm/v1 # OPENROUTER_AUTH_URL=https://gateway.example/auth/realms/.../token # OPENROUTER_AUTH_USERNAME=... # OPENROUTER_AUTH_PASSWORD=... # OPENROUTER_AUTH_CLIENT_ID=api # OPENROUTER_AUTH_VERIFY_TLS=false # only for private gateways that need curl -k uv run python -m harness_bench run-openrouter \ --model gpt-4.1-nano --concurrency 5 # run-openrouter retries transient HTTP/timeout/transport model errors up to # 5 total attempts per task before counting them as task failures. Override # with --transient-attempts if needed. # Add --forward-reasoning-history to replay a reasoning model's own thoughts # back to it across agent turns. Providers spell the trace differently # (`reasoning_content` on vLLM/SGLang, `reasoning` on OpenRouter-style # gateways); both are captured and echoed back under the key they arrived on. # By default the trace is kept only in the AIMessage metadata and omitted from # subsequent requests, so runs stay comparable with previously published rows. # # Whether replayed thoughts help depends on the chat template, and forwarding # grows the prompt on every turn: on an internal SGLang reasoning stand it was # worth +9.6 pp of pass rate for +35% tokens, while most gateway models ignore # a replayed trace entirely. Treat it as part of the run configuration and # never compare runs that differ in it. # Run stock deepagents + GigaChat while bypassing the GigaChat harness # profile even if deepagents-gigachat is installed. uv run python -m harness_bench run-pure --concurrency 5 # Drive an external CLI agent (Claude Code, etc.). Example with # Anthropic's free-code CLI: # # IMPORTANT: Claude-Code-style CLIs (Claude Code `claude`, `free-code`, # OpenClaude `openclaude`, …) ship a built-in host-side "auto-memory" # feature. During a run it reframes the memory-discipline tasks # (`tasks_memory.py`, 222-253) toward its own ~/.claude memory store / # index format instead of writing the literal workspace `MEMORY.md` and # deliverables the strict verifiers expect, which silently corrupts that # wave (e.g. Claude Sonnet 4.6 scored 20/32 with it on vs 31/32 off). # ALWAYS set CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 when benchmarking these # CLIs so the memory wave is scored fairly. This is targeted (unlike # `--bare`, it does not break OAuth/keychain auth). CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 \ uv run python -m harness_bench run-cli \ --cli-command 'free-code -p --model haiku --dangerously-skip-permissions' \ --concurrency 5 # Runner JSON writes best-effort per-task effort metrics: # agent_steps / agent_tool_calls / agent_shell_commands / agent_events, # plus agent_llm_calls / agent_input_tokens / agent_output_tokens / # agent_total_tokens when the backend exposes usage metadata. Codex, Claude # Code, and Gemini CLI runs auto-enable JSON/stream-json output so those metrics # can be read from machine-readable events. The top-level JSON summary also # includes `steps` and `tokens` totals for README-style result tables. # JSON checkpointing is enabled by default for run commands. A fresh run writes # to `jobs/<timestamp>.json`; pass `--json-output results.json` to use # `jobs/results.json`, or pass an explicit path. If the JSON file already # exists, completed task attempts are loaded from it and skipped so the run # continues from the checkpoint. JSON reports also include the command that # launched the run. Ctrl-C writes `run_status: "interrupted"` plus # `interrupted_attempts`; those attempts are rerun from clean workspaces on the # next continue when you rerun with the same JSON path. CLI per-task timeouts # are also marked `rerun_on_continue` and rerun from clean workspaces on the # next launch with the same `--json-output` file. # Add `--rerun-on-fail` when continuing to rerun every saved task attempt whose # `passed` field is false. Passing attempts stay checkpointed. With `-k N`, only # the failed attempt numbers are rerun, once per invocation. uv run python -m harness_bench run-cli \ --cli-command 'codex exec -m gpt-5.5 --dangerously-bypass-approvals-and-sandbox' \ --concurrency 5 # Drive mini-SWE-agent through any OpenAI-compatible gateway, including local # gpt2giga. Install `mini-swe-agent` once for faster startup; the wrapper can # also fall back to `uvx` when `mini` is not installed. Keep the wrapper path # absolute because `run-cli` launches the agent from each per-task temp # workspace. The wrapper runs mini-SWE-agent's non-interactive DefaultAgent # directly and writes `mini-swe-agent.traj.json`; `run-cli` parses that # trajectory for agent_steps / agent_llm_calls / token metrics. uv tool install mini-swe-agent HB_MINI_SWE_AGENT="$(pwd -P)/scripts/hb-mini-swe-agent" OPENAI_API_KEY=0 \ OPENAI_API_BASE=http://127.0.0.1:8090/v1 \ MSWEA_MODEL_NAME='openai/GigaChat-3-Ultra' \ MSWEA_COST_TRACKING=ignore_errors \ uv run python -m harness_bench run-cli \ --cli-command "$HB_MINI_SWE_AGENT" \ --timeout 900 --concurrency 5 \ --json-output mini_swe_agent_gigachat_3_ultra.json # Smoke-test one task first by adding: # --task task_01_create_hello --keep # Repeat every selected task 5 times and print pass@K / pass^K # percentage metrics for K=1..5. Works for run, run-openrouter, # run-pure, and run-cli. uv run python -m harness_bench run-cli \ --cli-command 'free-code -p --model haiku --dangerously-skip-permissions' \ --attempts 5 --concurrency 5 # Restrict the repeated-attempt summary to specific K values and write # the full per-attempt report as JSON. uv run python -m harness_bench run-cli \ --cli-command 'free-code -p --model haiku --dangerously-skip-permissions' \ --attempts 5 --pass@ 1 --pass@ 5 --pass^ 5 \ --json-output results.json # Summarize an existing completed JSON run without rerunning tasks. For a # 313-task run with --attempts 5, this prints Passed attempts, pass@K/pass^K # for K=1..5, and the per-wave breakdown. uv run python -m harness_bench summarize-json jobs/results.json # Drive `opencode` against any OpenAI-compatible deployment (example: # Qwen3.6-27B-FP8 served by vLLM). Point OPENCODE_CONFIG at a config # that registers a custom openai-compatible provider, sets the thinking # sampling (temp=0.6 top_p=0.95 top_k=20) and DISABLES formatter/LSP so # edits stay byte-exact for the verifiers (otherwise opencode auto-runs # a formatter and rewrites quotes/whitespace, failing exact checks): OPENCODE_CONFIG=/path/to/opencode-vllm.json \ uv run python -m harness_bench run-cli \ --cli-command 'opencode run -m vllm/qwen3.6-27b' \ --timeout 900 --concurrency 5 # Windows/Git Bash + cmd.exe CLIs with non-ASCII prompts/artifacts: force UTF-8 # in both the outer shell and the Windows console before launching the runner. # `cmd.exe //c` is intentional for Git Bash/MSYS; keep `cmd /c` inside # `--cli-command` because that string is parsed by Python's subprocess, not MSYS. cmd.exe //c "chcp 65001 >nul" && \ PYTHONUTF8=1 PYTHONIOENCODING=utf-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 \ uv run python -m harness_bench run-cli \ --timeout 600 \ --cli-command 'cmd /c gigacode --approval-mode=auto-edit' \ --task task_05_greet --task task_35_remove_blank_lines # Verify the gold solutions without calling any model. Useful when # adding a new task — confirms the verifier accepts a hand-written # "perfect" solution. uv run python -m harness_bench verify-gold # Direct no-Docker workspace checks for one task. This is the same # verifier/oracle surface used by the Harbor export. uv run python -m harness_bench verify-task \ --task task_06_toggle_debug --workspace /path/to/workspace uv run python -m harness_bench apply-gold \ --task task_06_toggle_debug --workspace /path/to/workspace ``` `.env` at the repo root is auto-loaded by every runner. ## What's inside ### Tasks (391 total, task-set v0.16.0) | Module | Range | Wave | | --- | --- | --- | | `tasks.py` | 1–30 | core file ops (create, edit, count, sort, find) plus the `ALL_TASKS` registry | | `tasks_extra.py` | 31–60 | multi-file refactors, dedupe, log filtering, CSV ↔ markdown | | `tasks_more.py` | 61–100 | `.env` edits, nested JSON, dataclasses, regex extraction, INI/TOML/YAML stubs, CSV row splitting | | `tasks_hard.py` | 101–150 | CSV / XLSX / SQLite aggregates, JSONL, Python impl + pytest, multi-file `grep`, Apache log parsing | | `tasks_extreme.py` | 151–205 | composite pipelines, archives, project-wide refactors, algorithms with pytest, statistics, XML / markdown, three-way joins | | `tasks_diagnostic.py` | 206–221 | paid-revenue reconciliation, inventory anomalies, pricing-API migration, latency reconstruction, tar+hash manifests, interval merge, config precedence, markdown link audit, data-quality reports, TODO/FIXME triage, category rollups, email extraction, runtime config, SQL leaderboards, import migrations, log-level summaries | | `tasks_memory.py` | 222–253 | memory discipline: read / write / forget / refuse facts in `MEMORY.md` along with the auxiliary deliverable (LICENSE, `requirements-dev.txt`, `bio.txt`, `profile.json`, …). Exercises agent memory rather than file I/O. | | `tasks_agentic.py` | 254–298 | benchmark-like synthetic agentic wave: Terminal-Bench-like terminal workflows (logs, process tables, Makefile plans, checksums, permission audits), tau-like policy-bound action decisions (airline, retail, banking, clinic, etc.), and SWE-bench-like pytest bug-fix tasks. | | `tasks_vcs.py` | 299–313 | version-control work: Git merge-conflict resolution (ours/theirs/both/manual, diff3 base sections, multi-hunk, multi-file), unified-diff apply/revert, unresolved-conflict detection, plus multi-file/multi-step workflows (scaled rename refactors, module split, ordered patch stacks, manifest-driven resolution, config deep-merge). | | `tasks_skills.py` | 314–330 | skill-discriminator wave: fictional brand/style guides, internal codebooks and policies, bespoke fixed formats, distractor/selection/negative-control skill axes, code-skill creation/repair, fictional DSL/protocol/library specs, spreadsheet reconciliation, and ArcFlux calculation methods. | | `tasks_adversarial.py` | 331–351 | adversarial/robustness wave: the agent must diagnose and work around a hostile environment — broken Python versions and imports, unreadable/mis-encoded/permission-locked files, instructions that contradict the environment, broken build commands and skills, and a ~100 MB log that must be streamed rather than read whole. | | `tasks_tbench_lite.py` | 352–371 | calibrated Terminal-Bench-inspired workflows: multi-source joins, event reconstruction, parsers, config precedence, conflict resolution, package refactors, SQLite migration, deterministic manifests, and retry-aware aggregation. | | `tasks_cli.py` | 372–391 | CLI-composition wave. Thirteen tasks drive bespoke per-task tools (`logq`, `pktool`, `xtab`, `cfgctl`, `depwalk`, `slicer`) built so that reading `--help` is unavoidable: the surface is deliberately unconventional (a leading verb, `--src`/`--cap`/`--map`, mini-languages like `--span LO..HI` and `--pick level=ERROR,WARN`, `--shape` not `--format`), so a guessed invocation exits non-zero — and the semantics that decide the answer (exclusive bounds, nearest-rank percentiles, margins before normalisation, corrupt-record policy) appear only in the `--help` epilog. Two read binary or fixed-width payloads. Seven exercise POSIX tools (multi-key `sort`, `join -1/-2/-a/-e/-o`, `comm`, `grep -oE` with `uniq -c`, `find` predicates with `xargs -0`, `awk`, `sed` ranges): the agent writes `solve.sh` and the verifier deletes the artifact, runs the script, and rejects general-purpose interpreters. **Requires `bash` on `PATH`.** | Task prompts are in **Russian** — the bench is deliberately bilingual to keep models honest. The verifiers and gold answers are English / data only. ### Task-set revisions Benchmark task-set versions live in `harness_bench/versioning.py` and are separate from the Python package version. Bump the task-set version when a task is added, removed, or materially changed; runner-only or documentation changes do not need a task-set bump. | Version | Introduced | Added tasks | Total | Notes | | --- | --- | --- | --- | --- | | `0.1.0` | 2026-05-13 | 1–200 | 200 | Initial extracted file/code/data benchmark | | `0.2.0` | 2026-05-19 | 201–221 | 221 | Advanced composites and diagnostic hard tasks | | `0.3.0` | 2026-05-21 | 222–231 | 231 | Memory-discipline tasks using `AGENTS.md` and `MEMORY.md` | | `0.4.0` | 2026-06-02 | 232–253 | 253 | Extended memory suite: knowledge update, contradiction resolution, temporal reasoning, abstention, preferences, multi-hop/multi-session | | `0.5.0` | 2026-06-02 | 254–262 | 262 | Agentic wave of synthetic Terminal-Bench-like, tau-like, and SWE-bench-like tasks | | `0.6.0` | 2026-06-02 | 263–283 | 283 | Agentic wave expanded to 10 Terminal-Bench-like / 10 tau-like / 10 SWE-bench-like tasks | | `0.7.0` | 2026-06-02 | 284–298 | 298 | Agentic wave expanded to 15 Terminal-Bench-like / 15 tau-like / 15 SWE-bench-like tasks | | `0.8.0` | 2026-06-05 | 299–308 | 308 | Version-control tasks: Git merge-conflict resolution, multi-hunk unified-diff apply/revert, unresolved-conflict detection | | `0.9.0` | 2026-06-05 | 309–313 | 313 | Multi-file / multi-step version-control workflows (rename refactor, module split, patch stack, manifest-driven resolution, config deep-merge) | | `0.10.0` | 2026-06-30 | 314–330 | 330 | Skill-discriminator wave with fictional skills, codebooks, policies, bespoke formats, selection/distractor axes, code-skill authoring/repair, and ArcFlux methods | | `0.11.0` | 2026-07-02 | 331–337 | 337 | Adversarial/robustness pilot: Python 2 port, broken build command, Windows-1251 file, permission-locked file, instruction naming a nonexistent file, hardcoded path, skill with missing template | | `0.13.0` | 2026-07-02 | 338–351 | 351 | Adversarial wave completed: removed-stdlib import, misleading `.python-version`, unneeded uninstallable dependency, `set -e` abort, npm-in-a-Python-project, gzip-masquerade, BOM/NUL log, AGENTS.md wrong layout, wrong tests dir, broken import path, broken package layout, malformed SKILL.md frontmatter, contradictory skills, and a ~100 MB log the agent must stream/grep rather than read whole | | `0.14.0` | 2026-07-23 | 352–371 | 371 | Calibrated Terminal-Bench-inspired wave with deterministic, offline, gold-verified multi-step tasks | | `0.15.0` | 2026-07-27 | 372–391 | 391 | CLI-composition wave: bespoke tools (`logq`, `pktool`, `xtab`, `cfgctl`, `depwalk`, `slicer`) with a deliberately unguessable surface, so `--help` must be read before anything runs, plus POSIX pipeline tasks (`sort`, `join`, `comm`, `grep`/`uniq -c`, `find`/`xargs -0`, `awk`, `sed`) whose `solve.sh` the verifier executes | | `0.16.0` | 2026-07-28 | — | 391 | Audit pass over all 391 tasks: no tasks added or removed, but defects gold-verification cannot see were corrected — tasks winnable without work, prompts whose verifier rejected the work they described, requirements the verifier never checked (notably “do not edit the tests”), and platform/self-pollution issues. **Not score-comparable with v0.15.0.** | ### Infrastructure | File | Purpose | | --- | --- | | `core.py` | `Task` (dataclass) and `VerifyResult`. Supports `setup_callback` / `gold_callback` hooks for binary fixtures (xlsx, sqlite, zip, tar). | | `verifiers.py` | Helpers for building verifiers: `file_exists`, `file_contains`, `file_lines_equal`, `file_matches_regex`, `json_file_has`, `python_runs`, `python_callable_returns`, `pytest_passes`, `xlsx_cell_equals`, `sqlite_query_returns`, `all_of`, etc. | | `runner.py` | Runs a task in an isolated `tempfile.TemporaryDirectory` with `LocalShellBackend(virtual_mode=True)` rooted at that directory. Drives GigaChat through `langchain-gigachat`. Optional `--concurrency` via a thread pool. Auto-loads the `deepagents-gigachat` harness profile if installed. | | `runner_cli.py` | Alternative driver that shells out to an external CLI agent (`free-code`, `claude`, etc.). Default: `free-code -p --model haiku --dangerously-skip-permissions`. Detects Claude-Code-style CLIs and auto-injects workspace `AGENTS.md` via `--append-system-prompt`. | | `runner_openrouter.py` | Runner for any OpenAI-compatible OpenRouter model via `langchain-openai`. Does **not** apply any harness profile — measures raw `deepagents` defaults against the chosen model. | | `runner_pure.py` | Stock `deepagents` + GigaChat runner that bypasses `deepagents-gigachat` profile lookup even when that package is installed. Useful as a no-profile baseline, not a direct raw-API baseline. | | `harbor_export.py` | Additive Harbor export layer. Generates local Harbor task directories from the same Python task registry; does not replace the no-Docker local runners. | | `__main__.py` | CLI: `list`, `version`, `run`, `run-pure`, `run-cli`, `run-openrouter`, `verify-gold`, `verify-task`, `apply-gold`, `export-harbor`. | Each task is independent: the runner creates a fresh `tempfile.TemporaryDirectory`, writes `setup_files` (and optionally calls `setup_callback` for binary fixtures), then points `LocalShellBackend` at that directory as its `root_dir`. The agent file tools are rooted there by `virtual_mode=True`. This is not a security sandbox: `execute` still spawns a real shell on the host and the runners inherit environment variables. The benchmark is meant for a trusted local environment. After the agent stops, the per-task verifier inspects the workspace. ## Harbor export The repo can generate a local Harbor dataset without changing the native benchmark flow: ```bash # One-task smoke export uv run python -m harness_bench export-harbor \ --output harbor_dataset --task task_06_toggle_debug --clean # Full dataset export uv run python -m harness_bench export-harbor --output harbor_dataset --clean ``` Each exported Harbor task contains: - `instruction.md` from the task prompt. - `environment/Dockerfile` plus a `setup.tar` with the initial workspace. - `solution/solve.sh` that calls `python -m harness_bench apply-gold`. - `tests/test.sh` that calls `python -m harness_bench verify-task` and writes `/logs/verifier/reward.txt`. The Docker image contains only task setup and runtime dependencies. The benchmark registry / gold data is copied into Harbor `solution/` and `tests/` payloads, so normal agents do not get the gold answers baked into the image. Local no-Docker execution remains the canonical development loop: `run`, `run-cli`, `run-pure`, `run-openrouter`, `verify-gold`, `verify-task`, and `apply-gold` all run directly on the host. Docker is only needed when invoking Harbor's own local runner. ## Results The published results table (full 351-task set, `v0.13.0`) is kept at the top of this README. Only one run per harness + model setup is listed; superseded and older-task-set (313-task) runs are not carried over. ### Scoring rules - A task that hits the per-task wall-clock timeout or hangs counts as a normal fail: it stays in the denominator and gets no partial credit. - Transient infrastructure errors are **not** model failures. If a task dies on a network failure or an API infrastructure response (HTTP 5xx, 429, `529 Overloaded`, connection reset, gateway timeout) rather than on the model's own behavior, the task may be rerun and the retried result is recorded. Runners may also auto-retry such errors in-flight (`run-openrouter` already retries up to 5 attempts per task); a retried task is scored the same as any other task. ## Adding a task 1. In one of the task modules (`tasks.py`, `tasks_extra.py`, `tasks_more.py`, `tasks_hard.py`, `tasks_extreme.py`, `tasks_diagnostic.py`, `tasks_memory.py`, `tasks_skills.py`, `tasks_tbench_lite.py`, `tasks_cli.py` — pick the one that fits the wave / difficulty) describe a `Task(...)` — id, prompt, `setup_files`, `gold_files`, `verifier`. 2. Wire it into the corresponding module's `*_TASKS` list — it gets pulled into `ALL_TASKS` automatically via `tasks.py`. 3. Append a new entry in `harness_bench/versioning.py`, bump `TASK_SET_VERSION`, and update the total task count. Use a new minor version for a new task wave (for example `0.4.0`) and a patch version for verifier/gold fixes that change scoring semantics. 4. `uv run python -m harness_bench version --check` — confirms task ids, task count, and version metadata agree. 5. `uv run python -m harness_bench verify-gold --task <new_id>` — confirms the verifier accepts the gold solution. 6. `uv run python -m harness_bench run --task <new_id>` — sanity-check against a live model. ## License MIT — see [`LICENSE`](LICENSE).
