nanoeuler

KO | EN

gitlite — search

#javascript #react #ai #llm #golang #deep-learning #nodejs #cli #kubernetes #vue #pytorch #openai #ios #mcp

(★ 111)

GPT-2-style LLM built from scratch in C/CUDA with hand-written backprop, BPE tokenizer, FlashAttention, pretraining, and SFT.

nanoeuler 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# nanoeuler

A GPT-2-class language model built **entirely from scratch in C/CUDA** — no PyTorch, no
autograd, no ML libraries. The forward **and** backward passes are written and verified by
hand, and the whole training pipeline lives in this repo: a hand-written **byte-level BPE
tokenizer**, **pretraining** on a books + web corpus, and **supervised fine-tuning** into a
chat model (RLHF/DPO planned). It runs on CPU (`libm` + OpenMP) for a small showcase model,
and a full from-scratch **CUDA engine** — cuBLAS matmuls, a hand-written **FlashAttention**,
validated against a CPU reference by a full-model gradient check — trains a **~116M-parameter**
model on a single RTX 4070.

> **Status & honesty.** This is a research/educational artifact, built in public. At ~116M
> parameters trained on a single consumer GPU, it is a *text generator in the spirit of
> GPT-2-small*: fluent-ish English, **no real world knowledge**. It is **not** a capable
> assistant — the chat model demonstrates that the pretrain→SFT pipeline works end to end, it
> is not a useful chatbot. The point of the project is the from-scratch engineering and the
> complete, understandable training pipeline.

```
make check              # verify the backward pass (gradient check, double precision)
make                    # build the training binary
./nanoeuler train       # train the small showcase model (~0.76M params)
./nanoeuler train big   # train the larger model (~10M params; meant for a GPU)
./nanoeuler chat        # REPL: type a prompt, the model continues it
```

## Why "Euler"?

A residual block computes

```
x = x + f(x)
```

Read it as a step of numerical integration. The **forward-Euler method** advances an
ordinary differential equation `dx/dt = f(x)` by

```
x(t+Δt) = x(t) + Δt · f(x(t))
```

With step size `Δt = 1` this is exactly the residual update. So a deep residual
network is a *discretized ODE*: **depth is integration time**, and each layer
integrates the hidden state forward by one Euler step. This is the view behind work
like Neural ODEs (a ResNet is the Euler discretization of a continuous flow). The
project is named after **Leonhard Euler**, who gave us that integration method.

## Example output

A sample from the ~116M model after a partial pretraining run on the books + web corpus
(prompt `Alessandro eat a`):

```
Alessandro eat a icing textile: the satisfied by the servants in order to keep your weight
[Using to a heated, collaborated young people that attend the metric process where the rank
is authorized and to contain the sedentary. Some state lawyers were able to insert ...
```

The content is not meaningful, but notice what it learned on its own: real grammar, long
clauses, and an encyclopedic register picked up from the web data. This is the expected
behaviour of a small model trained on a single GPU — fluent shape, shallow substance. More
training and (far) more data improve fluency; world knowledge needs scale this project does
not pretend to have.

## Architecture

Decoder-only transformer with the building blocks common to current models:

- **RMSNorm** (pre-norm, no bias)
- **Rotary position embeddings (RoPE)** applied to queries and keys
- **SwiGLU** feed-forward: `down(silu(gate(x)) * up(x))`
- **Grouped-query attention (GQA)**: query heads share a smaller set of key/value heads
- **Multi-token prediction (MTP)**: `K` output heads predict the next `K` tokens; the
  auxiliary heads improve the learned representation and enable speculative decoding.
  Generation uses head 0.
- **No biases** anywhere.
- **Byte-level BPE tokenizer**, hand-written, with GPT-2-style pretokenization (a single
  leading space attaches to the following word, so spaces are not wasted as standalone
  tokens). Merges are learned on a sample of the corpus; the GPU model uses a 4096-token
  vocabulary (~3.4 bytes/token on English).

Each block is `x = x + attn(rmsnorm(x))` followed by `x = x + swiglu(rmsnorm(x))`.
A residual connection `x = x + f(x)` is one step of the forward-Euler method for the
ODE `dx/dt = f(x)` — hence the name, and a nod to Leonhard Euler.

Configurations:

| where | dim | q/kv heads | layers | context | vocab | params |
|-------|-----|-----------|--------|---------|-------|--------|
| `small` (CPU, `nanoeuler.c`) | 128 | 4 / 2 | 4 | 128 | 512 | ~1.05M |
| **GPU pipeline** (`cuda/`, `run_train`) | 768 | 12 / 4 | 16 | 512 | 4096 | **~116M** |

The CPU `small` model trains in a few hours on 12 cores and is a self-contained showcase.
The ~116M GPU model is the real pipeline: it pretrains on the books + web mix and is then
fine-tuned into a chat model (see below). The head size is 64 (`768/12`), which fits the
FlashAttention kernel.

## Verified backward pass

Hand-written back-propagation is easy to get subtly wrong, so every analytic
gradient is compared against a central finite difference. The check runs in double
precision so floating-point cancellation does not hide correct gradients:

```
$ make check
  tok      : max rel err 1.02e-04
  qkvw     : max rel err 7.20e-07
  gatew    : max rel err 6.86e-08
  ...
max relative error: 1.02e-04
>>> backward OK (error < 1e-2)
```

Every parameter tensor is checked, including the less obvious backward passes of
RoPE, SwiGLU, GQA, and MTP.

## Build and performance

`make` builds with `-O3 -march=native -ffast-math -fopenmp`. Matrix multiplies and
attention are parallelized with OpenMP and vectorized; on a 12-core machine the
training loop uses all cores. `make check` builds a separate double-precision binary
used only for the gradient check.

No external dependencies. Tested with gcc 13 on Linux.

## Scope

This is a from-scratch text generator and a complete, understandable training pipeline —
not a product. A model of this size trained on one GPU produces fluent-looking English with
little real knowledge; the fine-tuned chat model answers in assistant *form* but its content
is shallow. A usable conversational model needs orders of magnitude more parameters, data and
compute (a ~135M model only becomes a basic assistant after ~600B training tokens; this repo
trains on a far smaller corpus on a single GPU). The goal is to own every piece — every
parameter, every gradient, the tokenizer, the kernels, the pretraining and the fine-tuning.

## GPU engine (CUDA)

`cuda/nanoeuler_cuda.cu` is a full from-scratch CUDA port — forward, backward, training
and inference on the GPU. Every kernel is validated on the device against a CPU reference,
and the whole model has a GPU gradient check (GPU grads vs CPU grads to ~1e-6).

Kernels: matmul (delegated to **cuBLAS** with TF32 tensor cores), RMSNorm, RoPE,
grouped-query attention with a hand-written **FlashAttention** (tiled, online softmax,
no T×T matrix in memory), SwiGLU, softmax/cross-entropy and AdamW. FlashAttention made
the training step about 3× faster.

Build (RTX 40-series = Ada = `sm_89`; the host-compiler flag avoids a gcc ICE on the large file):

```
cd cuda
nvcc -O3 -arch=sm_89 -Xcompiler -fno-tree-reassoc,-fno-tree-copy-prop nanoeuler_cuda.cu -o nanoeuler_cuda -lcublas
```

Modes:

```
./nanoeuler_cuda              # run all kernel self-tests (GPU vs CPU)
./nanoeuler_cuda g            # full-model gradient check (GPU grads vs CPU)
./nanoeuler_cuda t            # pretrain from scratch, checkpoint to ../nanoeuler.bin every 5k steps
./nanoeuler_cuda tr           # resume pretraining from the latest ../nanoeuler.bin checkpoint
./nanoeuler_cuda i "It was"   # autoregressive generation on GPU
./nanoeuler_cuda s            # supervised fine-tune on Alpaca, save ../nanoeuler_chat.bin
./nanoeuler_cuda c            # interactive chat with the fine-tuned model
```

Training checkpoints every 5000 steps, so a long run can be stopped (Ctrl-C) and resumed with
`tr`. A model trained on the GPU is saved in the CPU program's format, so `./nanoeuler chat`
can also load and run it.

## Chatbot: pretrain then fine-tune (SFT)

The chat pipeline is two stages. First **pretrain** the ~116M base on the books + web mix
(`./nanoeuler_cuda t`, resumable with `tr`). Then **supervised fine-tuning** turns it into an
assistant: `./nanoeuler_cuda s` loads the pretrained base, renders each
[Alpaca](https://github.com/tatsu-lab/stanford_alpaca) example with the standard instruction
template, and trains with the **loss masked to the response tokens only** (prompt and padding
positions get a target of `-1`, which the cross-entropy kernel turns into zero gradient). The
result is saved to `nanoeuler_chat.bin`; `./nanoeuler_cuda c` then wraps each line you type in
the same template and samples a reply, stopping at the `</s>` end marker.

After fine-tuning the model answers in the right *shape* — it follows the instruction→response
format, writes complete sentences and stops on its own. The *content*, though, is shallow and
often wrong: this is a small model trained on a single GPU, so it has little world knowledge to
express. SFT teaches the model *how* to respond, not *what* it knows — that comes from
pretraining and scale. This is a faithful, fully-from-scratch demonstration that the
pretrain→SFT pipeline works end to end, not a capable assistant.

## Data

Pretraining uses a real **books + web** mix:

- **Books** — `data/get_gutenberg.sh` downloads ~95 public-domain Project Gutenberg classics
  (Austen, Dickens, Dostoevsky, Tolstoy, Melville, the complete Shakespeare, ...). Each book's
  Project Gutenberg license header/footer is stripped (only the text between the
  `*** START ... ***` / `*** END ... ***` markers is kept) so the model trains on prose.
- **Web** — `data/get_web.sh` pulls a slice of [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
  (high-quality educational web text) straight from the Hugging Face parquet files using the
  **DuckDB** CLI (a single static binary — no Python, no libraries).

Then concatenate them into the pretraining corpus the trainer reads:

```
sh data/get_gutenberg.sh                       # books  -> data/gutenberg.txt
sh data/get_web.sh                             # web    -> data/web.txt (~1 GB by default)
cat data/gutenberg.txt data/web.txt > data/pretrain.txt
sh data/get_alpaca.sh                          # instruction data for SFT -> data/alpaca.json
```

Corpora and model checkpoints are git-ignored (regenerable).

## Roadmap

- ✅ Hand-written byte-level BPE with GPT-2-style pretokenization.
- ✅ From-scratch CUDA engine (cuBLAS + FlashAttention), validated by a full-model gradient check.
- ✅ Pretraining on a books + web mix, with checkpoint/resume.
- ✅ Supervised fine-tuning (Alpaca) with response-masked loss → a chat model.
- ⏳ **DPO** (preference optimization) — the alignment stage, next to build.
- ⏳ Scale the model and data (toward ~270M) and publish a trained checkpoint people can try.

## Files

```
nanoeuler.c             CPU model: forward, backward, training, sampling, chat REPL
cuda/nanoeuler_cuda.cu  GPU engine: BPE, kernels, FlashAttention, pretrain/SFT/infer/chat, gradient check
data/get_gutenberg.sh   downloads + cleans the Gutenberg books corpus
data/get_web.sh         downloads a FineWeb-Edu web slice via the DuckDB CLI (no Python)
data/get_alpaca.sh      downloads the Alpaca instruction data for fine-tuning
Makefile  LICENSE  shakespeare.txt  .gitignore
```

## License

MIT. See `LICENSE`.

nanoeuler

nanoeuler 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

nanoeuler

nanoeuler 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README