neuro-san-benchmarking

KO | EN

gitlite — search

#javascript #hacktoberfest #golang #android #machine-learning #rust #deep-learning #docker #macos #kubernetes #awesome-list #security #pytorch #mcp

neuro-san-benchmarking

(★ 46)

General benchmarking apparatus for running multi-agent systems against benchmarks

neuro-san-benchmarking Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Neuro-SAN Benchmarking

A benchmarking suite for evaluating multi-agent reasoning systems using Neuro-SAN.

## Overview

This repository contains tools for benchmarking multi-agent systems on multiplication and other tasks.
The main components under decomposer/ are:

- **multiagent_reasoner.py** - Multi-agent reasoning system with decomposition, voting, and composition
- **linear_multiagent_reasoner.py** - Linear variant of the multi-agent reasoner
- **agent_benchmark_runner.py** - Generic benchmark runner supporting parallel execution
- **make_synthetic_benchmarks.py** - Tool for generating synthetic benchmark datasets

## Installation

Note: This code has been tested with Python 3.12 and MAX OS 15.6.

1. Clone this repository
2. Create a virtual environment:

```bash
   python3 -m venv venv
   source venv/bin/activate
   ```

3. Install dependencies (takes around one minute on a standard laptop):

```bash
   pip install -r requirements.txt
   ```

4. Set up your OpenAI API key:

```bash
   export OPENAI_API_KEY="your-api-key-here"
   ```

5. Update Python path:

```bash
   export PYTHONPATH=$(pwd)
   ```

## Usage

### Single Problem

Test the multi-agent reasoner on a single multiplication problem:

```bash
LOG_LEVEL=INFO echo "What is 46048 × 42098?" | python decomposer/multiagent_reasoner.py
```

### Benchmark Suite

Run the benchmark on a dataset:

```bash
python decomposer/agent_benchmark_runner.py \
  --python-prog decomposer/multiagent_reasoner.py \
  --task mul_5x5 \
  --local-jsonl data/bench_long_mul_5_5__200.jsonl \
  --answer-format number \
  --final-token '>>>>' \
  --num-workers 10 \
  --limit 10
```

### Configuration

The multi-agent reasoner supports several environment variables:

- `WINNING_VOTE_COUNT` - Number of votes required for consensus (default: 2)
- `MAX_DEPTH` - Maximum recursion depth for decomposition (default: 5)
- `LOG_LEVEL` - Logging level (default: INFO)
- `LOG_DIR` - Directory for detailed per-run logs (optional)
- `LOG_FAILURES_JSONL` - Path to log failure cases for analysis (optional)

Example with custom configuration:

```bash
export WINNING_VOTE_COUNT=8
export MAX_DEPTH=3
export LOG_FAILURES_JSONL="./failures.jsonl"
export LOG_DIR="./logs"

python decomposer/agent_benchmark_runner.py \
  --python-prog decomposer/multiagent_reasoner.py \
  --task mul_5x5 \
  --local-jsonl data/bench_long_mul_5_5__200.jsonl \
  --answer-format number \
  --final-token '>>>>' \
  --num-workers 50 \
  --limit 200 \
  --timeout-ms 180000
```

## Benchmark Runner Options

- `--python-prog` - Path to the Python program to benchmark
- `--task` - Task name (e.g., mul_5x5, gsm8k)
- `--local-jsonl` - Path to local JSONL benchmark file
- `--answer-format` - How to parse answers (number, list-json)
- `--final-token` - Token marking the final answer (default: ####)
- `--num-workers` - Number of parallel workers (default: 1)
- `--limit` - Limit number of problems to evaluate
- `--timeout-ms` - Per-item timeout in milliseconds (default: 120000)
- `--sample-retries` - Max retries per sample on failure (default: 2)

## Output

The benchmark runner produces:
- `results_{task}_{timestamp}.csv` - Detailed results for each problem
- `results_{task}_{timestamp}.jsonl` - Results in JSONL format
- `results_progress_{timestamp}.csv` - Real-time progress tracking

If failure logging is enabled:
- Failure logs in the specified JSONL file
- Per-run detailed logs in the LOG_DIR directory

## Instrumentation

See [INSTRUMENTATION_GUIDE.md](INSTRUMENTATION_GUIDE.md) for details on:
- Failure pattern analysis
- Trace data collection
- Debugging techniques

## Benchmark Data

The `data/` directory contains sample benchmark datasets:
- `bench_long_mul_5_5__200.jsonl` - 200 5×5 digit multiplication problems
- `bench_long_mul_10_10__200.jsonl` - 200 10×10 digit multiplication problems
- `bench_sort_len_500__50.jsonl` - 50 sorting problems

## Playground

A jupyter notebook demonstrating the core methods of MAKER in the Towers of Hanoi domain is located at `maker_playground/hanoi.ipyb`.

## Expected Runtime

The runtime is dominated by the LLM calls to external APIs, e.g., OpenAI.
The time per step depends on the latency of the API, which can range from seconds to minutes.
When not parallelized, the total runtime scales linearly with the number of steps times the number of votes per steps.
When fully parallelized, the total runtime scales linearly with the depth of the task decomposition.

## Running Original Code

To run the version of the code used to produce the results in <https://arxiv.org/abs/2511.09030> (Appendix F),
checkout the corresponding commit from 11/20/25:

```bash
git checkout a7a22f8
```

Example of how to run benchmarks with this version of the code:

```bash
export WINNING_VOTE_COUNT=10
export MAX_DEPTH=5
export LOG_FAILURES_JSONL="./failures_5x5.jsonl"
export LOG_DIR="./logs"

python agent_benchmark_runner.py \
  --python-prog multiagent_reasoner.py \
  --task mul_5x5 \
  --local-jsonl data/bench_long_mul_5_5__200.jsonl \
  --answer-format number \
  --num-workers 200 \
  --limit 200 \
  --timeout-ms 8000000
```

## License

See LICENSE.txt in the parent repository.

neuro-san-benchmarking

neuro-san-benchmarking Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

neuro-san-benchmarking

neuro-san-benchmarking Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README