KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #cli #codex #claude-code #react #mcp #open-source #deepseek-harness #dsh-plugin #ai

fucina

★ 16 Open GitHub ↗

CPU-first, eager-execution tensor/autograd runtime and LLM inference engine written in pure Zig

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Fucina

**Fucina** (Italian for *forge*) is a CPU-first tensor/autograd runtime and
LLM inference engine written in pure **Zig 0.16**. Computation is **eager**:
every op executes the moment the model code calls it, on real buffers.
There is no graph to build, plan, or compile first, so what you read is what
runs, in inference and in training alike. Eager does not mean slow: on most
measured CPU shapes Fucina matches or beats llama.cpp (`docs/BENCHMARK.md` keeps
the dated records, losses included). There is no C/C++ build system, no
Python runtime dependency, and no ggml-style graph executor — just Zig
vector kernels, plus CBLAS providers and a GPU offload (Metal or CUDA) as
opt-in accelerators for matrix multiplication.

## What it looks like

Axis tags and rank are **comptime facts**: `Tensor(.{ .batch, .in })` is a
different type from `Tensor(.{ .in, .batch })`, contraction is by axis
*name*, the result's tag set is computed at compile time, and a misaligned
contraction is a compile error — not a runtime shape crash three layers
deep. The pattern, condensed (see `examples/spirals/main.zig` and the production
trainer in `src/llm/qwen3/train.zig`):

```zig
const Model = struct {
    w1: Tensor(.{ .h1, .in }),
    b1: Tensor(.{.h1}),
    w2: Tensor(.{ .class, .h1 }),
    b2: Tensor(.{.class}),
};

fn forward(ctx: *ExecContext, m: *const Model, x: *const Tensor(.{ .batch, .in })) !Tensor(.{ .batch, .class }) {
    var z1 = try x.dot(ctx, &m.w1, .in); // contract .in -> .{ .batch, .h1 }
    defer z1.deinit();
    var s1 = try z1.add(ctx, &m.b1);
    defer s1.deinit();
    var a1 = try s1.tanh(ctx);
    defer a1.deinit();
    var z2 = try a1.dot(ctx, &m.w2, .h1); // contract .h1 -> .{ .batch, .class }
    defer z2.deinit();
    return try z2.add(ctx, &m.b2);
}

// Inference: the defers free each intermediate as soon as it is consumed.
// Training: open an exec scope and the SAME forward trains as-is — the
// scope adopts every intermediate (value + autograd node), each deinit
// becomes a no-op borrow-release, and the step's whole graph stays alive
// until backward(), then is released at once when the scope closes.
const scope = ctx.openExecScope();
defer ctx.closeExecScope(scope);
const logits = try forward(ctx, &model, &x);
const loss = try logits.crossEntropy(ctx, .class, labels);
try loss.backward(ctx);
try opt.step(ctx);
```

The shape discipline lives in the program text, the way it did in Fortran —
`real A(n,m)` told you the rank before you read a single loop. Here the
signature `x: Tensor(.{ .batch, .in }) -> Tensor(.{ .batch, .class })` lets
you check the algorithm against the math by eye, and Zig's comptime makes it
free: the tags exist only in the type system and compile away entirely.

## Design

Fucina is deliberately **eager and local**: no global graph object, no fusion
pass, no compiler layer. The execution context validates shapes once, then
dispatches to small unchecked, allocation-free backend kernels selected at
build time. Transient memory goes through a thread-safe reusable buffer pool
with bucket-rounded buffer allocation for small temporaries — the rationale
(and why it beats an arena here) is in `docs/MEMORY-MODEL.md`. The intended
growth path is model-specific sessions with semantic weight binding and
preallocated buffers, not a generic ggml-like graph. `docs/ARCHITECTURE.md` maps
the whole tree.

That choice has a known price, paid deliberately. Whole-graph machinery —
fusion passes, static device memory planning, captured launch sequences — is
what mainstream GPU inference stacks are built on, so an eager runtime can
use a GPU only at the op seam (which is what the Metal and CUDA offloads do)
and will never chase TensorRT-class GPU throughput. On CPU the ledger reads
the other way: per-op dispatch costs next to nothing, kernels specialize at
compile time instead of fusing at runtime, and the paired benchmarks are run
against a graph executor. CPU-first is the design point, not a stage on the
way to somewhere else.

## What runs today

| Family | What it is |
| --- | --- |
| **[Qwen3](examples/qwen3/README.md)** dense (0.6B–8B) + MoE (30B-A3B, 235B-A22B) | chat / REPL / raw generation, lossless speculative decoding, batch-N multi-conversation decode, JSON-schema/grammar constrained output |
| **[DeepSeek V2/V3](examples/deepseek2/README.md)** (MLA) | multi-head latent attention with the compressed KV cache as the default; covers V2-Lite, Moonlight-16B-A3B, and GLM-5.2 (`glm-dsa`) checkpoints |
| **[GLM-4.5](examples/glm4moe/README.md)** family (MoE) | V3-style trunk plus the model's own `nextn` layer for lossless multi-token-prediction drafting |
| **[DeepSeek V4 Flash](examples/deepseek4/README.md)** 284B-A13B | hyper-connections trunk with native MTP speculative decoding; the 164.6 GB Q4K release decodes on a 64 GB machine |
| **[Gemma 4](examples/gemma4/README.md)** 26B-A4B (MoE) | chat / REPL / speculative decoding, JSON-schema/grammar constrained output |
| **[Qwen3.5](examples/qwen35/README.md)** 0.8B | hybrid Gated-DeltaNet architecture (conv + delta scan + gated attention) |
| **[DiffusionGemma](examples/diffusion_gemma/README.md)** 26B-A4B | block text-diffusion decoding on the Gemma backbone |
| **[Inkling](examples/inkling/README.md)** 975B-A41B (MoE) | hybrid local/global-attention multimodal decoder: text chat plus image and audio input towers |
| **[nanochat](examples/nanochat/README.md)** | karpathy/nanochat ported whole: BPE tokenizer training, GPT pretraining, SFT, bits-per-byte eval, and chat — trained from scratch on CPU |
| **[Parakeet](examples/parakeet/README.md)** (NVIDIA NeMo FastConformer) | speech-to-text: offline, streaming, and live microphone |
| **[OmniVoice](examples/omnivoice/README.md)** | MaskGIT text-to-speech with voice cloning (Higgs Audio v2 codec included) |
| **[LocateAnything-3B](examples/locate_anything/README.md)** | NVIDIA's open-vocabulary detection VLM: text-prompted labeled boxes, byte-compatible with the reference CLI |
| **[facedetect](examples/facedetect/README.md)** (insightface buffalo_l) | face detection, recognition, gender/age, anti-spoofing, and dense landmarks |
| **[Neural Amp Modeler](examples/nam/README.md)** | `.nam` guitar-amp profiles: run, train, export, live amp simulation |

MoE models bigger than RAM are first-class: `--moe-stream` keeps only the
dense weights resident and pages routed experts from disk through a
pinned-set + LRU tier, bit-identical to the resident path — that is how the
142 GB Qwen3-235B and the 164.6 GB V4 Flash decode on a 64 GB machine.

Every family is validated against its reference implementation, and that
discipline is the core of the project: token-ID-exact tokenizers vs
`llama-tokenize`, logit-parity oracles vs llama.cpp, byte-exact quantization
encoders vs ggml, byte-identical GGUF re-emit. Each family's folder under
`examples/` carries its own README with copy-paste commands;
`docs/RUNNING-MODELS.md` is the index — verified weight downloads and
licenses, plus the machinery shared across runners (expert streaming, GPU
offload, global knobs).

These applications live in `examples/` and each will eventually graduate
into its own repository. They use the library; they are not the library.
Keeping them in-tree during this phase is deliberate: with the Tensor core
in place, Fucina grows and gets tested through real applications, so the
runtime and the things built on it develop side by side. The known debt of
that convenience is that generic operations accumulate inside the examples
— resamplers, spectrograms, reference-parity image resizing — and
graduation starts with an audit of which of those are really tensor ops
that belong in the core. Research experiments that lack a reference oracle live on
`research/*` branches rather than `main` — currently `research/nla`, a
natural-language autoencoder study (text→vector→text on a Qwen3 GGUF)
built on the trainer's hidden-state seams.

## Quick start

Requires [Zig 0.16.0](https://ziglang.org/download/) — the toolchain is
pinned; other versions will not build.

```sh
git clone https://github.com/matteo-grella/fucina
cd fucina
zig build test          # unit tests, no model files needed

# grab a small model and talk to it
mkdir -p models
hf download Qwen/Qwen3-0.6B-GGUF Qwen3-0.6B-Q8_0.gguf --local-dir models
zig build qwen3 -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf \
  --chat "What is the capital of France?" --no-think

# or serve it to any OpenAI client (chat completions + responses, SSE
# streaming, JSON-schema constrained output with -Dllguidance=true)
zig build lmserve -Doptimize=ReleaseFast -- models/Qwen3-0.6B-Q8_0.gguf --port 8080
```

Build with `-Doptimize=ReleaseFast` whenever speed matters (Debug is 10–50x
slower). Build options (`-Dbackend`, `-Dblas`, `-Dmax-threads`, `-Dgpu=metal`,
…) are documented in `AGENTS.md`.

**Builds are tuned to the machine that compiles them.** Without `-Dtarget`,
Zig targets the host CPU with its full feature set — as if `-march=native`
were always on — and Fucina's kernels specialize at compile time
(NEON/dotprod arms on Apple Silicon, AVX2/AVX-VNNI on modern x86; unused
arms are not in the binary). Two rules follow: run the binary on the
machine you built it on, and if you must cross-compile, pass `-Dcpu` as
well (e.g. `-Dtarget=x86_64-linux -Dcpu=x86_64_v3`) — a bare `-Dtarget`
gets that architecture's *baseline* features and silently loses the fast
kernels.

## Performance

Measured, not asserted: the protocol is paired same-machine runs against
a reference implementation.

For instance, llama.cpp — same GGUF, same thread
count, CPU-only both sides — and `docs/BENCHMARK.md` records losses as plainly
as wins, each number with its hardware, protocol, and caveats, the whole record a dated snapshot.
The short version of the record (snapshot 2026-07-04): on Apple M1 Max, of 236 paired
sweep cells across Qwen3 dense (0.6B/1.7B), Qwen3.5, the 30B MoE, and Gemma-26B,
Fucina is faster in 221 and at parity in 13 — dense prefill geomeans 1.18–1.81x per format, large
MoE prefill up to ~2x — with two cells on llama.cpp's side: Qwen3.5-0.8B at
pp32 (0.86x; its neighbors pp128 and decode win) and one 30B decode cell
recorded before the stricter prewarmed protocol. The smallest gemma batch
cells await one pristine-machine confirmation, as `docs/BENCHMARK.md` discloses.
On an x86 Raptor Lake box (AVX2+VNNI), Fucina wins all dense quantized
formats (medians 1.32–1.95x) while llama.cpp decisively wins MoE
small-batch prefill.

Every row is reproducible — fetch the pinned reference implementations,
then run the paired gate:

```sh
tools/fetch_refs.sh --build   # references into refs/ (gitignored), llama.cpp built CPU-only
python3 tools/bench_gate.py --models qwen3-0.6b-q6_k --tasks prefill,decode
```

On top of raw speed there is lossless, draft-model-free speculative decoding (up to
2.3x on retrieval-structured tasks, never-a-loss cost gate; `docs/SPECULATIVE.md`)
and batch-N multi-stream decode (3.2x aggregate throughput at 8 streams).
Structured output is built in: a pluggable logit-processor seam on the shared
sampler, with JSON-schema/regex/Lark constrained decoding through the vendored
[llguidance](https://github.com/guidance-ai/llguidance) engine (opt-in
`-Dllguidance=true`; composes with speculative decoding — REFERENCE.md §13.6).

## Training

The runtime trains as well as it infers: an eager autograd engine with exec
scopes, activation checkpointing, deterministic dropout (counter-based RNG),
and SGD/AdamW/Muon/APOLLO optimizers golden-parity-tested against their
reference implementations. LoRA fine-tuning of a quantized Qwen3 GGUF runs
end-to-end on CPU, and the loop closes: fine-tune → merge → quantize →
the exported GGUF loads and answers in llama.cpp.

```sh
zig build finetune -Doptimize=ReleaseFast -- --steps 30
```

Gradient-free training is a first-class alternative: `fucina.es` implements
evolution strategies at scale (arXiv:2509.24372) — seed-regenerated noise,
forward passes only, algebra cross-checked bitwise against the reference —
and `zig build es-finetune` fine-tunes the same GGUF with it, LoRA-only or
full-parameter, under rule-based (R1-style) or loss-based rewards.

Because autograd, KV-cache plumbing, and serving live in one runtime, the
same trainer also implements **Cartridges** (arXiv:2506.06266): a corpus is
compressed into a trained KV prefix by in-process self-study distillation —
the model interviews itself about the corpus, and the teacher-with-context
distills into a small trainable cache served like any cached prompt.
`zig build cartridge` trains, saves, and serves one against a real Qwen3
GGUF (`docs/CARTRIDGES.md`).

`docs/TRAINING.md` is the full guide, including how the gradients were verified
(PyTorch goldens, finite differences, and a real-model audit) and its open
issues.

## Documentation

| Doc | Contents |
| --- | --- |
| `docs/ARCHITECTURE.md` | the actual source layout, layer by layer — start here |
| `docs/REFERENCE.md` | the detailed API reference: the full public surface, exact semantics, machine-verified snippets |
| `docs/course/` | *Forging Deep Learning in Zig* — a book-length course that teaches Zig and deep learning together by rebuilding this library's journey, from a dtype enum to a live guitar amp and chatting language models; per-chapter video scripts included |
| `docs/RUNNING-MODELS.md` | the model index: verified weight downloads + licenses, the example-to-README map, shared runner machinery |
| `docs/LMSERVER.md` | the lmserve example: OpenAI API mapping tables, streaming contracts, server architecture |
| `docs/BENCHMARK.md` | the measurement protocol and dated Fucina-vs-llama.cpp records, wins and losses |
| `docs/TRAINING.md` | the training guide: autograd, optimizers, LoRA, evolution strategies, checkpoints, gradient verification |
| `docs/MEMORY-MODEL.md` | ownership rules and the buffer-pool-not-arena adjudication |
| `docs/PORTING.md` | the porting method — how every model family here earned its parity claims, written for the next port |
| `docs/SPECULATIVE.md` | design record: lossless draft-model-free speculative decoding |
| `docs/PTQTP-RECIPE.md` | walkthrough: PTQTP-quantize any GGUF (MoE experts included) and run it, resident or streamed |
| `docs/CONSTRAINED-DECODING.md` | design record: grammar/JSON-schema constrained decoding and its speculation composition |
| `docs/CARTRIDGES.md` | design record: trained KV-prefix corpus compression (Cartridges) — self-study distillation, serving, gates |
| `docs/ENGRAM.md` | design record: conditional n-gram memory (Engram) — hashed lookup tables, graft mode, reference parity |
| `AGENTS.md` | build/test/bench commands, build options, repo map, house rules |
| `docs/THIRD-PARTY-NOTICES.md` | full provenance and license inventory of third-party material |

## Status and scope

Honest expectations:

- **CPU-first, two ISAs.** Tuned on Apple Silicon (aarch64 NEON/sdot) and
  x86-64 (AVX2/AVX-VNNI); a scalar reference backend covers everything else.
  The Metal offload accelerates specific GEMM shapes on macOS; it is not a
  general GPU runtime. The CUDA sibling (`-Dgpu=cuda`, Linux/NVIDIA) plugs
  into the same seam with zero build-time SDK dependency: f32/f16 GEMM via
  dlopen'd cuBLAS, quantized dense + MoE prefill and fused prefill attention
  via vendored PTX kernels, and an opt-in decode GEMV.
- **The API is not stable.** This is a young codebase published in the open,
  not a 1.0 library. Expect churn.
- **Model weights are not included.** Each model family carries its own
  license (Qwen: Apache-2.0; Gemma: Google's Gemma Terms of Use; Parakeet:
  CC-BY-4.0; OmniVoice weights: CC-BY-NC). `docs/RUNNING-MODELS.md` notes the
  terms next to each download.
- **Benchmarks age.** llama.cpp moves fast; the dated records in
  `docs/BENCHMARK.md` are snapshots, not eternal claims.

## Origins

Fucina grew out of autograd concepts I first explored in Go with
[spaGO](https://github.com/nlpodyssey/spago) — above all the idea that the
graph should be **implicit in the values themselves**: no graph object, no
tape, no persistent engine. Each result carries a pointer to the operation
that produced it, and `backward()` discovers the topology by walking those
pointers. spaGO executed that idea the Go way: one goroutine per node,
each blocking until its gradient contributions arrived, the runtime
scheduler absorbing the wait. Zig has no goroutines, so Fucina keeps the
idea and rethinks the execution: every node carries an atomic dependency
counter, and its gradient fires only when the counter drains — concurrent,
on a bounded worker pool, no blocked workers (`src/ag/`). (AFAIK) Mainstream
stacks route backward through a central engine over an explicit node graph
or a trace; here the live tensors *are* the graph.

As for the language: I wanted to stay as close to the metal as possible, and — honestly — I was
also looking for a good excuse to finally learn Zig. This project is it.

Development — code and documentation alike — leans on strong assistance
from agentic coding systems, with humans leading the ideas, the testing,
and the debugging, and writing first-hand as well. That stance now has a
motto, thanks to Salvatore Sanfilippo:
["Control the ideas, not the code"](https://antirez.com/news/169).
I also wanted to gain practical experience coding with AI, and so far I
have found that the best results come from a review loop where humans and
multiple frontier models critique one another’s specifications and
implementations, iterating toward a shared consensus.

As for what gets built: right now the core — the tensor/autograd
runtime — evolves in lockstep with the examples, growing exactly the ops
each new port demands, and the examples follow a personal criterion
rather than a roadmap: they are the deep-learning architectures I want
to understand deeply, in the spirit of Feynman's "What I cannot create,
I do not understand." The Neural Amp Modeler is the one that outgrew the
criterion — I genuinely enjoy *using* it, which is its own small
achievement: I almost never end up using the things I build.

## Contributing

Contributions are welcome — see `CONTRIBUTING.md`: PRs are
human-owned (coding agents expected, human judgment required), and changes
are tested against the two tracks the project actually regresses on,
correctness and speed, with the commands, machine, and model quant reported.

## Acknowledgments

Fucina exists because others built the road first.

- **ggml / llama.cpp** — Georgi Gerganov and the ggml authors. This project
  would not exist without their work. Fucina is an independent Zig runtime,
  but it speaks formats ggml defined (GGUF, the block-quantization wire
  formats), and several components are direct ports of llama.cpp code: the
  quantization row encoders, the Qwen2 and SentencePiece tokenizers, the
  Unicode classification tables, the SIMD `expf`, and the vendored Metal
  quantized-GEMM kernel. llama.cpp is also the parity oracle and the
  performance yardstick throughout.
- **Ettore Di Giacinto** (mudler, of LocalAI) — the Parakeet ASR family is a
  port of his [parakeet.cpp](https://github.com/mudler/parakeet.cpp) (the
  ready-to-run GGUF weights come from his conversions); the LocateAnything
  open-vocabulary detection example is a port of
  [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp), written
  with **Richard Palethorpe** — its converter/quantizer produce the GGUFs the
  example runs, and it is the parity oracle and CPU yardstick for that port;
  and the facedetect example (SCRFD + ArcFace + GenderAge + anti-spoof +
  dense landmarks) is a port of his
  [face-detect.cpp](https://github.com/mudler/face-detect.cpp), again the
  parity oracle and CPU yardstick, with the buffalo_l GGUFs coming from his
  [conversions](https://huggingface.co/mudler/face-detect-gguf) of the
  [insightface](https://github.com/deepinsight/insightface) models. The
  underlying LocateAnything-3B model is
  [NVIDIA's](https://huggingface.co/nvidia/LocateAnything-3B).
- **Salvatore Sanfilippo** (antirez) — the DeepSeek V4 Flash port follows his
  [ds4](https://github.com/antirez/ds4) inference engine: it is the reference
  for the entire architecture (hyper-connections, compressed sliding
  attention, the FP8/FP4 quantization grids, native MTP), his GGUF
  conversions are the weights the port runs, and his test fixtures (official
  API vectors and local logit goldens) are the validation oracle.
- **Prism ML** — the Ternary Bonsai family is what the `q2_0` ternary
  support serves: the Q2_0 g128 wire format and its reference
  encoder/decoder come from their
  [llama.cpp fork](https://github.com/PrismML-Eng/llama.cpp), which is also
  the parity oracle for the Ternary-Bonsai-27B port. The Bonsai weights
  themselves are Apache-2.0 on
  [Hugging Face](https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf)
  and are not part of this repository.
- **Thinking Machines Lab** — the Inkling architecture (hybrid local/global
  attention with a banded content-dependent relative-position bias in place of
  RoPE, per-layer short causal convolutions, a fine-grained MoE whose shared
  experts act as routing-softmax sinks, and hierarchical-patch vision plus
  dMel audio towers); the port follows the open weights at
  [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling).
  **Daniel Han** (Unsloth) — his
  [llama.cpp Inkling architecture PR](https://github.com/ggml-org/llama.cpp/pull/25731)
  is the source of truth for the GGUF layout and graph, and his
  [GGUF conversions](https://huggingface.co/unsloth/inkling-GGUF) provide the
  converter and the parity oracle the port validates against.
- **JustVugg's [colibri](https://github.com/JustVugg/colibri)** — the
  out-of-core MoE expert streaming (run mixture models much bigger than RAM
  by paging routed experts from disk through a pinned-set + LRU tier, with a
  persistent usage histogram and router-lookahead prefetch) was inspired by
  colibri's design; Fucina's implementation is independent, streaming ggml
  quants over the fused kernels.
- **Apple MLX** — the f32/f16 Metal GEMM is the vendored MLX "steel" kernel.
- **guidance-ai / Microsoft** — the vendored
  [llguidance](https://github.com/guidance-ai/llguidance) engine powers
  grammar/JSON-schema constrained decoding.
- **ServeurpersoCom** — the OmniVoice TTS port follows his
  [omnivoice.cpp](https://github.com/ServeurpersoCom/omnivoice.cpp), which
  also provided the codec porting groundwork (Higgs Audio v2 / HuBERT / DAC).
- **Steven Atkinson** — NeuralAmpModelerCore and neural-amp-modeler, the
  reference for the entire NAM example.
- **ZINC** — the byte-level BPE tokenizer core was adapted from the ZINC Zig
  inference engine.
- **ZML** — the tagged-tensor approach (axis tags carried in the type,
  operands aligned by name) was inspired by [ZML](https://github.com/zml/zml).
- **David Reid** (miniaudio), **Keller Jordan** (Muon), **Sebastiano Vigna**
  (splitmix64), **musl libc** and **ARM optimized-routines** (scalar/SIMD
  math lineage), **k2-fsa/Xiaomi** (OmniVoice), **NVIDIA NeMo** (Parakeet),
  **Boson AI** (Higgs Audio v2), **Meta** (HuBERT), **Descript** (DAC),
  **kitft** (natural-language autoencoders).

The complete inventory — what is vendored, what is ported, what is only a
parity reference, and under which license — is in `docs/THIRD-PARTY-NOTICES.md`.

## License

MIT — see `LICENSE`. One documented exception: `tools/gen_optim_goldens.py`
contains material derived from the APOLLO reference implementation and is
covered by its upstream CC-BY-NC-4.0 terms; see `docs/THIRD-PARTY-NOTICES.md`.

fucina

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README