KO
|
EN
gitlite — search
Search
#javascript
#python
#hacktoberfest
#react
#ai
#typescript
#llm
#go
#golang
#android
#machine-learning
#rust
#deep-learning
#linux
L0-reasoning-bench
★ 9
Open GitHub ↗
No description available.
Download README (.md)
Explore Similar Repositories
clip-pgs
:
Official code for CVPR2025 "Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection"
multimodal_r1_papers
:
Deepseek RL (GRPO)-Inspired Research for Vision & Multimodal Reasoning
implicit-fn.rs
:
A macro that adds support for implicit closures to Rust.
FindPeaks
:
C & C++ implementation of Scipy's find_peaks function
flake-compat
:
Mirror for Lix's flake-compat fork. Contributions OK, but we strongly prefer: https://git.lix.systems/lix-project/flake-compat
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
L0-reasoning-bench
?
Download (.md)
# L0-Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution Paper link: [https://www.arxiv.org/pdf/2503.22832](https://www.arxiv.org/pdf/2503.22832) L0-Bench is a benchmark designed to evaluate procedural correctness of language models by testing their ability to generate step-by-step execution trace of simple, synthetic Python programs. ## Overview The benchmark includes: - A `generator` that generates synthetic Python programs given specified grammar config .yaml file - An `evaluator` that evaluates model responses given data generated by the `generator` We provide evaluation over a baseline set below. <details > <summary> Results on a baseline evaluation set (see `./example_data`) </summary> | **Model Name** | Thinking mode on | Steps to Err. ↑ (Single Attempt) | Steps to Err. ↑ (majvote @ 31) | Trace Acc. (%) (Single Attempt) | Trace Acc. (%) (majvote @ 31) | Trace Acc. (%) (pass @ 31) | |----------------|------------------|--------------------|--------------------|----------------------|----------------------|--------------------| | **Closed-source models** | | | | | | | | o1 | Y | 122.2 | - | 92.0 | - | - | | claude-3-7-sonnet | Y | 118.6 | - | 95.1 | - | - | | claude-3-7-sonnet | N | 114.0 | - | 86.1 | - | - | | gpt-4o-2024-11-20 | N | 112.5 | - | 76.4 | - | - | | claude-3-5-sonnet-20241022 | N | 108.5 | - | 83.3 | - | - | | o3-mini (high) | Y | 92.8 | - | 60.6 | - | - | | **Open-weight models** | | | | | | | | QwQ-32B | Y | 120.0 | 124.9 | 86.6 | 96.1 | 99.8 | | Deepseek-R1 | Y | 115.7 | 121.4 | 91.5 | 97.4 | 97.9 | | DeepSeek-R1-Distill-Qwen-32B | Y | 114.4 | 123.4 | 80.4 | 94.9 | 99.8 | | DeepSeek-R1-Distill-Llama-70B | Y | 104.4 | 116.3 | 62.0 | 79.6 | 96.6 | | DeepSeek-R1-Distill-Qwen-14B | Y | 99.3 | 118.3 | 56.8 | 83.5 | 97.8 | | Meta-Llama-3.1-405B-Instruct | N | 99.1 | 106.2 | 63.0 | 73.1 | 93.2 | | Qwen2.5-72B-Instruct | N | 94.3 | 102.6 | 55.2 | 66.9 | 89.9 | | Qwen2.5-32B-Instruct | N | 88.3 | 101.8 | 50.4 | 66.0 | 90.0 | | Meta-Llama-3.1-70B-Instruct | N | 82.8 | 93.9 | 42.4 | 54.7 | 85.9 | | Qwen2.5-Coder-32B-Instruct | N | 81.4 | 92.4 | 44.0 | 56.8 | 84.4 | | Qwen2.5-14B-Instruct | N | 60.9 | 73.4 | 23.7 | 35.7 | 73.7 | | Qwen2.5-Coder-7B-Instruct | N | 52.3 | 68.4 | 17.1 | 32.4 | 62.5 | | Qwen2.5-7B-Instruct | N | 41.2 | 54.2 | 11.3 | 22.8 | 44.1 | | Meta-Llama-3.1-8B-Instruct | N | 33.2 | 40.7 | 6.5 | 11.5 | 33.9 | | DeepSeek-R1-Distill-Qwen-7B | Y | 8.7 | 11.7 | 3.8 | 11.2 | 28.8 | </details> While we provide baseline results above, L0-Bench is designed as a flexible framework rather than a static leaderboard. We encourage practitioners to: - Generate custom test programs using the configurable yaml files - Create targeted evaluation sets (e.g., w/ varying difficulty level) or create synthetic training data - Evaluate models using L0-Bench generated data - Share results while providing generator yaml config ## Environment We recommend using `Python 3.12` and latest Docker listed below for consistent environment setup. [UPDATE-2025-05] docker updated to support Qwen3 ``` docker pull simengs/l0-bench:v0.1.1 ``` To install dependencies manually, see `dockerfiles/requirements_v0.1.txt` ## Usage ### Generate Test Programs ```bash cd src/generators python generator.py configs/example.yaml ``` This script uses the configuration specified in the `.yaml` file to perform one or more actions defined within the `actions` list in the config. The primary actions are: - **generate**: Generate test examples, i.e. (program, input, trace) triplets, based on the specified grammar and input `.yaml` file. This is the first step to create the initial test set. - **sample**: Sample and group the generated test examples based on their target trace length into `num_bins` bins. - **augment**: Augment existing test examples by generating new (input, trace) pairs, which can be used as in-context step-by-step demonstrations for each test program. The generated programs and their execution traces will be saved in the output directory specified in the config file. An example yaml config `generators/configs/example.yaml` is provided with explanations for each field. An example set of data is provided in `example_data`. The generator creates programs based on grammar rules (see `grammar.py`). We provide interfaces to override these rules for more flexible and controlled generation of synthetic programs. <details> <summary>Override production rules and terminals following the format:</summary> ``` overwritten_grammar: <stmt> ::= <list_op> | <assignment> @@ <list_op> ::= <list_var> ".pop()" overwritten_terminal: <bool_var> ::= cond_a | cond_b | cond_c @@ <in_bool_var> ::= cond_o | cond_p | cond_q ``` </details> <details> <summary>Methods to increase difficulty or number of trace steps: </summary> - Increase input list size - Increase max scope depth (to enable nested loops) - Enable long expression evaluation by overriding existing grammar rules - Enable variable look-ups by overriding existing grammar rules - Increase maximum lines of code - Enable additional arithmetic operations by overriding existing grammar rules - ... </details> ### Evaluate Models ```bash cd evaluators python launcher.py --config_file configs/run_eval.yaml ``` Note: You can get the formatted prompts without running any model by setting the `model_name_or_path` to `"none"` in the yaml config file. The launcher will create an evaluation folder named `run_eval` containing logs, results, and execution scripts. For each voter, the evaluator creates a client that sends formatted prompts to the server and stores model responses. Once all voters complete their evaluations, the aggregator performs majority voting and computes pass@k accuracy by combining results from all voters. ``` ├── run_eval/ │ ├── logs/ │ │ ├── 0_evaluator.log # Logs from evaluator process for voter 0 │ │ ├── 0_server.log # Logs from server process for voter 0 │ │ ├── 1_evaluator.log │ │ ├── 1_server.log │ │ └── ... │ ├── results/ │ │ ├── voter_0/ │ │ │ ├── logs.jsonl # JSON Lines file containing voter 0's results │ │ │ └── errors.jsonl # JSON Lines file containing voter 0's errors │ │ ├── voter_1/ │ │ │ ├── logs.jsonl │ │ │ └── errors.jsonl # JSON Lines file containing voter 1's errors │ │ └── ... │ └── scripts/ # Contains execution scripts (slurm or local) │ ├── run_slurm_job_aggregate.sh # Script to aggregate results from all voters │ ├── run_slurm_job_eval_0.sh # Script to run evaluation │ └── ... ``` The `launcher.py` file will automatically create and run the scripts inside `./run_eval/scripts`. For majority voting, we run multiple voters in parallel using `evaluator.py` (via `run_slurm_job_eval_i.sh`) and then perform aggregation using `aggregator.py` (via `run_slurm_job_aggregate.sh`) to obtain the majority voted response. [UPDATE-2025-05] Added `errors.jsonl` in each result folder. Below is an example of an `errors.jsonl` object. For each error, it stores the context (a few lines before the error) and the corresponding correct target lines to help identify where the model's execution trace diverges from the expected output. The full model response, input prompt, and ground-truth trace can be found in `logs.jsonl` with the `id`. ``` { "id": "596a3893-5a38-3f21-936f-b0cc50e583e0", "tgt_trace_lines": [ "L37,cond_v:True", "L33,", "L34,f:4", "L35,lst_x:[4,4,1,5,1]", "L36,cnter_0:8", "L37,cond_v:True", "L33,", "L34,f:4", "L35,lst_x:[4,4,1,5]", "L36,cnter_0:10", "L37,cond_v:False" ], "model_trace_lines": [ "L37,cond_v:True", "L33,", "L34,f:4", "L35,lst_x:[4,4,1,5,1]", "L36,cnter_0:8", "L37,cond_v:True", "L33,", "L34,f:4", "L35,lst_x:[4,4,1,5]", "L36,cnter_0:10", "L37,cond_v:True" ], "program": "def function(y, t, w, r, p, q, x, lst_u, lst_w, lst_x, lst_v, cond_t, cond_p, cond_y):\n\tlst_w.append(p)\n\tcond_l = 7 == 8\n\tw = r\n\tif cond_p: \n\t\ti = q + 9\n\te = lst_w[y]\n\tcond_j = 9 == 8\n\tif cond_j: \n\t\ts = lst_u[q]\n\tif cond_t: \n\t\tlst_x.append(1)\n\tif cond_y: \n\t\th = lst_w[2]\n\tcond_t = 4 == 1\n\tcnter_0 = 0 \n\tcond_o = cnter_0 != 30 \n\twhile cond_o: \n\t\tlst_u.append(t)\n\t\tcnter_0 = cnter_0 + 2 \n\t\tcond_o = cnter_0 != 30 \n\tif cond_t: \n\t\tlst_u.pop()\n\tlst_w.append(w)\n\tcnter_0 = 0 \n\tcond_v = cnter_0 != 28 \n\twhile cond_v: \n\t\tlst_u.pop()\n\t\tcnter_0 = cnter_0 + 2 \n\t\tcond_v = cnter_0 != 28 \n\tcnter_0 = 0 \n\tcond_v = cnter_0 != 10 \n\twhile cond_v: \n\t\tf = r\n\t\tlst_x.pop()\n\t\tcnter_0 = cnter_0 + 2 \n\t\tcond_v = cnter_0 != 10 \n\tif cond_l: \n\t\tf = 4\n\tlst_w.append(h)\n\tc = lst_v[1]\n\tcnter_0 = 0 \n\tcond_g = cnter_0 != 26 \n\twhile cond_g: \n\t\tcond_p = 1 != e\n\t\tcond_u = 2 != x\n\t\tcnter_0 = cnter_0 + 2 \n\t\tcond_g = cnter_0 != 26 \n\tlst_x.pop()\n\tcnter_0 = 0 \n\tcond_l = cnter_0 != 48 \n\twhile cond_l: \n\t\tcond_c = i == 2\n\t\tcnter_0 = cnter_0 + 2 \n\t\tcond_l = cnter_0 != 48 \n\tlst_x.pop()\n\tw = h\n\tlst_w.pop()\n\treturn" } ``` ## License Apache License 2.0 ## Citation If you use L0-Bench in your product or research, please cite: ``` @article{sun2025l0reasoningbenchevaluatingprocedural, title={L0-Reasoning {B}ench: Evaluating Procedural Correctness in Language Models via Simple Program Execution}, author={Simeng Sun and Cheng-Ping Hsieh and Faisal Ladhak and Erik Arakelyan and Santiago Akle Serano and Boris Ginsburg}, year={2025}, journal={arXiv:2503.22832}, } ```
