cpubrrr

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 56 Open GitHub ↗

Frontier-class LLM inference on a laptop CPU — gpt-oss:20b at ~110 tok/s on Apple M4 Max, 7.5x llama.cpp, no GPU. From-scratch NEON/SME kernels in Rust.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# cpubrrr

https://github.com/user-attachments/assets/a390cb6e-86b8-41e5-9ef6-957c94dabe19

**From-scratch CPU-only LLM inference on Apple silicon. ~1.17× upstream llama.cpp on gpt-oss MXFP4, measured same-session with both sides tuned. No GPU.**

`cpubrrr` is a research runtime that runs frontier-class **mixture-of-experts** models
on an Apple M4 Max **CPU only** — the binary links nothing but the C standard library,
so it *physically cannot* touch the GPU (`otool -L target/release/engine` to verify). It
started as a one-model experiment (gpt-oss:20b) and now runs four MoE models across two
architectures and two quantization formats from one config-driven engine.

## Numbers (Apple M4 Max, CPU only, same session, both sides at their own best thread count)

| model (CPU decode) | cpubrrr | upstream llama.cpp (b9860) | ratio |
|---|---|---|---|
| **gpt-oss:20b** MXFP4 | **90.1 tok/s** (NT=10) | **76.95 tok/s** (`-t 8`) | **~1.17×** |

GSM8K 98/100 for cpubrrr on those same weights — identical to the score they get
through Ollama, so the speed is not bought with accuracy.

Upstream is forced genuinely CPU-only with `-dev none -ngl 0`; this build ships a
Metal backend, and this repo has been burned once already by a "CPU" baseline quietly
using the GPU. cpubrrr additionally decodes with a 73-token prompt in context
(pos 73→201) while `llama-bench tg128` runs pos 0→128, so cpubrrr is measured at
higher positions — the handicap runs against us.

**Correction (2026-08-02) — LESSON #7: a baseline is a *configuration*, not a binary.**
Every earlier comparison in this repo ran upstream at `-t 12`. On this chip that is
close to llama.cpp's *worst* setting (`-t 8` = 78.5, `-t 10` = 69.6, `-t 12` = 47.7,
`-t 16` = 11.4) — 61% of its own best. Measured that way the same binaries read
"1.62×", which would have been a headline built entirely on the opponent's
misconfiguration. Consequences:

- The old **"parity on MXFP4"** claim was against a mistuned upstream. Against a
  properly tuned one, the *older* engine (68.5) was **behind**, not level. The 1.17×
  above comes from the current engine, which is 1.28× faster than that older one.
- The old **Qwen3-Coder-30B "~1.17× (90 vs 77)"** row is **withdrawn pending
  re-measurement.** Re-run same-session on 2026-08-02 it gave 68.1 vs 65.5 (≈1.04×) —
  but on a loaded desktop, with a different llama.cpp build, and with the Qwen engine
  not yet carrying the current engine's optimizations. Rather than publish either the
  old unverified number or today's contaminated one, it is pulled until it can be run
  on a quiet machine.
- Remaining caveat on the number above: it is measured against llama.cpp b9860 as
  packaged by Homebrew. A source build with native flags may be faster still; that
  combination has not been tuned and tested here.

Decode throughput, several runs each. Output verified correct in both cases. llama.cpp
placement confirmed CPU-only from Ollama's own server logs (it defaults to Metal GPU on
macOS — `num_gpu:0` is a *request*, not a fact; see the benchmark-integrity note below).

These numbers replace this repo's earlier, over-optimistic claims (a "7.5× / 110 tok/s"
headline that did not survive rigorous re-measurement — the reproducible figure is
~77, recovered from a 52 tok/s regression by fixing dispatch overhead the hard way).
The story of *how* the early numbers were wrong — a contaminated baseline, unverified
GPU/CPU placement, thermal throttling, and a thread-pool whose condvar wakeups silently
cost 7.5 ms/token — is documented in full, in order, with evidence. We consider that
the most transferable part of the project.

## Why it's fast

Token generation for an MoE model is **memory-bandwidth bound**, not compute bound: a
21B model activates only ~3.6B params per token, so decode speed ≈ memory bandwidth ÷
active-bytes-per-token. llama.cpp's MXFP4 MoE CPU path uses a small fraction of the
available bandwidth; cpubrrr streams expert weights at close to what the cores can
sustain. Core techniques:

- Hand-written **NEON** (ARM SIMD) integer kernels using `sdot`/`tbl` for exact 4-bit
  arithmetic on integer hardware — no float dequant in the inner loop.
- **Quad-interleaved weight layout** so each core reads one sequential stream (the
  single biggest MXFP4 win — a byte-reordering, not a code change).
- **Integer-accumulation Q8_K kernel** for Q4_K/Q6_K — llama.cpp's own algorithm,
  found by reading its ARM source: quantize activations to Q8_K, accumulate sub-block
  integer dot products weighted by the 6-bit scales in int32, and convert to float
  *once per 256-value superblock*. This is what took Q4_K from losing (~71 vs ~86) to
  winning (~92 vs ~82). Kernels verified bit-exact against a dequant reference.
- **Worker-driven execution**: 12 persistent workers run the whole forward pass with a
  *yielding* spin-barrier (spin briefly, then let the OS in) — saturates all cores
  without the collapse-under-jitter that a pure spin-barrier suffers.
- MoE-aware scheduling, block-wise Q8 activation quantization, `mmap`'d weights (so a
  117B model pages in under memory pressure instead of OOM-killing).

The lesson behind the Q4_K win: to beat a mature kernel, don't out-clever it with
micro-tweaks — read its source and the research, find the *algorithmic* edge, adopt it,
then out-schedule it.

## Models

One config-driven engine (dimensions read from the model at setup — same-family models
run with zero code changes):

- **gpt-oss:20b** (MXFP4) — the original target, verified end-to-end.
- **gpt-oss-120b** (117B / ~5.1B active, MXFP4) — 6× bigger, same family; runs on the
  laptop CPU via `mmap`'d weights.
- **Qwen3-Coder-30B** (`qwen3moe`, Q4_K/Q6_K) — a different architecture *and* a
  different quant format; writes correct code.
- **Qwen3-30B** general — same arch, drop-in.

Architecture math was recovered from llama.cpp source and the 4-bit unpacking verified
bit-for-bit against the official `gguf` library *before* writing the forward pass — so
the new architecture produced correct output on the first run.

## Docs

- **[docs/RESEARCH_LOG_V2.md](docs/RESEARCH_LOG_V2.md)** — chronological lab log: every
  measured number, every wrong number, every correction, in order.
- **[docs/RESEARCH_LOG.md](docs/RESEARCH_LOG.md)** — the original single-model log.
- **[docs/PLAN.md](docs/PLAN.md)** — first-principles ceilings and the experiment ladder.

## Requirements

- Apple silicon Mac (M-series; developed on **M4 Max**, ARMv9 + SME). Other ARM/x86
  targets need a port — see the honest-limits section below.
- [Rust](https://rustup.rs/) (stable), Python 3, and [Ollama](https://ollama.com/) with
  a model pulled (e.g. `ollama pull gpt-oss:20b` or `ollama pull qwen3-coder:30b`).

## Quick start

```bash
# 1. build (uses target-cpu=native)
cargo build --release

# 2. prepare runtime data from your local Ollama copy (no weights are copied)
./scripts/setup_model.sh gpt-oss:20b     # or qwen3-coder:30b, etc.
                                         # writes data-<slug>/{tokens.bin,config.txt,...}

# 3. generate (gpt-oss uses the `engine` binary; qwen uses `engine_qwen2`)
./target/release/engine data-gpt-oss_20b "$(cat data-gpt-oss_20b/blob_path.txt)" "Why is the sky blue?"
```

### Use it with your existing tools (OpenAI-compatible API)

```bash
python3 scripts/openai_server.py          # serves http://localhost:8643/v1
```

Point any OpenAI-compatible client (chat UIs, IDE plugins, agent frameworks) at
`http://localhost:8643/v1`. Implements `/v1/models` and `/v1/chat/completions`
(streaming and non-streaming, `max_tokens`, usage accounting — the response even
tells you its tok/s). Honest limits: multi-turn history is flattened into the
prompt (engines apply their own chat template), temperature is honored by the
gpt-oss engine and ignored (greedy) by the qwen engine, and it serves one request
at a time — the engine owns all your cores anyway.

### Side-by-side demo

A local web page streams cpubrrr vs. llama.cpp/Ollama (CPU) with live tok/s counters:

```bash
python3 scripts/demo_server.py     # then open http://localhost:8642
```

## Benchmarks & kernels

Each is a standalone binary (`cargo build --release`, then `./target/release/<name>`):

| binary | what it measures |
|---|---|
| `engine` | full gpt-oss (MXFP4) generation engine |
| `engine_qwen2` | full Qwen3 (Q4_K/Q6_K) generation engine — the worker-driven, integer-accum path |
| `q8k_verify` | Q4_K/Q6_K integer-accum kernels vs a dequant-f64 reference |
| `qk_verify` | Q4_K/Q6_K dequant vs the official `gguf` library (bit-exact) |
| `bench_real` | real gpt-oss MXFP4 weights through the decode kernel |
| `bench_moe` | MoE expert-batched decode bandwidth |
| `bench_sme` | direct SME2 matrix-unit kernels + precision sweep |

`scripts/bench_ollama.sh <model> <cpu|gpu>` produces the llama.cpp baseline **and reads
Ollama's server log to confirm actual GPU/CPU placement** — it refuses to report a "CPU"
number if any layers ran on the GPU.

## Correctness parity (fast AND right)

An engine can't make a model smarter — but a broken engine can make it dumber. So we
run real benchmarks through cpubrrr and through Ollama/llama.cpp on the *same quantized
weights*, same prompts, greedy decoding, same answer extraction
(`scripts/parity_eval.py`):

| benchmark | model | cpubrrr | Ollama/llama.cpp | delta |
|---|---|---|---|---|
| GSM8K (100 q) | Qwen3-Coder-30B Q4_K | **89.0%** | 87.0% | +2.0 (within noise) |
| GSM8K (100 q, 1024-tok budget) | gpt-oss:20b MXFP4 | **98.0%** | 94.0% | +4.0 (edge of noise) |

Deltas of a few points at n=100 are sampling noise; the claim is *parity*, not
superiority. The speed does not come from corrupted math. (Methodology note that
almost fooled us: at a 320-token budget the gpt-oss comparison showed a fake +47 —
the model's *thinking channel* ate the budget before the final answer on one side.
Reasoning models need budgets big enough that nobody truncates. Caught, fixed,
logged.)

## Benchmark integrity (read this before quoting numbers)

Bandwidth-bound CPU benchmarks are easy to get wrong, and this project got them wrong
four times before getting them right:

1. **Contaminated baseline** — the first llama.cpp number was measured while a 65 GB
   download ate memory bandwidth in the background.
2. **Unverified placement** — on macOS, Ollama defaults to full Metal GPU; some "CPU"
   runs were silently 49/49 layers on the GPU. Only the server log (`offloaded N/M
   layers to GPU`) reveals the truth.
3. **Thermal throttling** — after days of load the *same binary* gave 66 → 42 → 5 tok/s
   on back-to-back runs.
4. **An unreproducible peak** — an early "110 tok/s" headline would not reproduce on a
   clean machine and was retired. Chasing it also surfaced two more traps: check your
   charger (a 30 W brick on a ~90 W laptop silently halves clocks — not our bug, but
   it will be someone's), and a historical peak you can't reproduce is not a baseline.
   Log enough environment to reproduce a number, or retire it.

cpubrrr also saturates all 12 cores, so its peak needs a quiet machine; llama.cpp
tolerates background load and heat better. Meta-lesson: **be more skeptical of
benchmarks that flatter you, not less.** Re-verify on your own hardware.

## Contributing — the mission needs hands

The goal is bigger than this repo: free and cheap models deserve kernel-level-up tooling
so fast reasoning is available on hardware people already own. The highest-impact open
work, in order:

1. **[Port the decode kernels to x86 (AVX-512 / VNNI)](https://github.com/arizqi/cpubrrr/issues/1)** —
   the #1 ask. The win is algorithmic, not Apple-specific; `sdot` maps to `vpdpbusd`.
2. **[Benchmark llama.cpp BLAS builds (BLIS / oneAPI / OpenBLAS)](https://github.com/arizqi/cpubrrr/issues/2)** —
   a self-contained measurement task, good first contribution.
3. **[Improve prefill throughput](https://github.com/arizqi/cpubrrr/issues/3)** — the one
   place llama.cpp still beats us. Measured 2026-08-02 on Qwen3-Coder-30B, same session:
   our token-by-token prefill 56 tok/s vs upstream's batched pp512 at 87. Notably NOT an
   Accelerate/BLAS effect — upstream's pp got *slower* with BLAS enabled (78 vs 87); its
   win is its own batched int8 path processing many tokens per weight pass. Batched
   prefill exists in `engine.rs` (`BATCH_PREFILL=1`) but is not yet in the newer engines.

House rules are short (see [CONTRIBUTING.md](CONTRIBUTING.md)): verify kernels against an
oracle before wiring them in, and land every perf claim with before/after numbers in the
research log. Beating our numbers is a welcome contribution.

## Status & honest limits

This is a **research engine**, not a production server (no cross-user batching, no
serving hardening). Numbers are Apple M4; the techniques port to other ARM and (with
AVX-512) x86, but those numbers must be measured, not assumed. A `training/kernel`
research track programs the M4's SME2 matrix unit directly from Rust assembly. An
earlier claim here of "~4.2 TFLOPS, exceeds Accelerate" did not survive re-measurement
(2026-08-02, loaded machine): best hand-written SME 2.4 TFLOPS vs Accelerate's 3.4 on
the same run. Treat Accelerate as ahead until a quiet-machine rerun says otherwise —
see the research log. On training
scale: consumer CPUs can realistically train models up to ~1B params; a
trillion-parameter model is ~47,000 years of compute (physics, not pessimism).

## License

Dual-licensed under [MIT](LICENSE-MIT) or [Apache 2.0](LICENSE-APACHE), at your option.

Not affiliated with OpenAI, Apple, or Ollama. `gpt-oss` and `Qwen` are released by their
authors under their own licenses; this project neither includes nor redistributes model
weights.

cpubrrr

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README