microgpt

KO | EN

gitlite — search

#python #hacktoberfest #go #golang #android #rust #linux #docker #nodejs #macos #windows #cli #pytorch #mcp

(★ 25)

A Common Lisp port of @karpathy microgpt.py — the most atomicimplementation of a GPT with a hand-written scalar autograd engine.

microgpt Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# microgpt.cl

A Common Lisp port of [@karpathy](https://github.com/karpathy)'s
[microgpt.py](https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95) — the most atomic
implementation of a GPT with a hand-written scalar autograd engine.

> "The most atomic way to train and run inference for a GPT in pure,
> dependency-free Python. This file is the complete algorithm.
> Everything else is just efficiency."

The port preserves the educational spirit of the original: no ML
libraries, no matrices, just plain lists of `Value` nodes wired
through a computation graph — now in idiomatic Common Lisp.

Warning, this port was completely done using LLM!

## What it does

1. Downloads a names dataset (English names from `karpathy/makemore`,
   or use input.txt file if it is already exists).
2. Trains a 1-layer GPT with scalar autograd and an Adam optimizer for
   N steps.
3. Generates new names by autoregressive sampling.

## Architecture

```
src/
  packages.lisp    — package definition
  autograd.lisp    — Value class + scalar autograd (backward pass)
  data.lisp        — dataset loading, shuffling, tokenizer
  model.lisp       — linear, softmax, rmsnorm, GPT forward pass
  train.lisp       — Adam optimizer training loop
  inference.lisp   — temperature-scaled autoregressive sampling
  main.lisp        — public run entry point
roswell/
  microgpt.ros     — Roswell CLI script
```

## Requirements

- [SBCL](https://www.sbcl.org/) (other implementations untested)
- [Quicklisp](https://www.quicklisp.org/)
- [dexador](https://github.com/fukamachi/dexador) (HTTP client, pulled
  via Quicklisp automatically)
- [Roswell](https://github.com/roswell/roswell) — optional, for CLI use

## Running from the REPL

```lisp
;; Load the system
(asdf:load-asd "/path/to/microgpt/microgpt.asd")
(ql:quickload "microgpt")

;; Run with defaults (1000 steps, 20 samples, temperature 0.5)
(microgpt:run)
```

`(microgpt:run)` will:
- Download `input.txt` if not present
- Print training loss each step
- Print 20 generated names after training

## Running from the CLI

Install Roswell script executable and run it:

```bash
ros install 40ants/microgpt

microgpt
```

Or if you cloned the repository, then do:

```
roswell/microgpt.ros
```

### CLI options

| Option | Default | Description |
|---|---|---|
| `--steps N` | `1000` | Number of training steps |
| `--samples N` | `20` | Number of names to generate |
| `--temperature T` | `0.5` | Sampling temperature (0.0–1.0, lower = sharper) |
| `--input PATH` | `input.txt` | Path to the names file |
| `--help` | | Print usage |

### Examples

```bash
# Quick run with 200 steps
./roswell/microgpt.ros --steps 200

# Use a custom names file with higher temperature
./roswell/microgpt.ros --input rus-names.txt --temperature 0.8 --samples 30

# Full training run
./roswell/microgpt.ros --steps 1000 --samples 20 --temperature 0.5
```

## Expected output

```
num docs: 32033
vocab size: 27
num params: 5152
step    1 / 1000 | loss 3.3241
step    2 / 1000 | loss 3.2847
...
step 1000 / 1000 | loss 2.1043

--- inference (new, hallucinated names) ---
sample  1: emra
sample  2: jorin
sample  3: alivia
...
```

Loss should start around 3.x and decrease over training. Results vary
per run unless you seed the random state explicitly.

## Seeding for reproducibility

```lisp
(setf *random-state* (sb-ext:seed-random-state 42))
(microgpt:run)
```

## Performance note

We conducted simple performance tests using the input.txt file stored in this repository, and the Python implementation was 9.5 times slower.

### Python version

```bash
[~/projects/lisp/microgpt]% time ./microgpt.py
time ./microgpt.py
num docs: 484
vocab size: 57
num params: 5152
step 1000 / 1000 | loss 2.3474
--- inference (new, hallucinated names) ---
sample  1: Небромир
sample  2: Миловета
sample  3: Милана
sample  4: Свеладр
sample  5: Милана
sample  6: Ратевоба
sample  7: Миловисла
sample  8: Крана
sample  9: Бородосл
sample 10: Промислав
sample 11: Воромир
sample 12: Додислав
sample 13: Малана
sample 14: Светомир
sample 15: Свемислава
sample 16: Вогона
sample 17: Мила
sample 18: Морислав
sample 19: Дробослава
sample 20: Милана
./microgpt.py  54.06s user 0.82s system 99% cpu 55.011 total
```

Common Lisp microgpt version:

```bash
[~/projects/lisp/microgpt]% time roswell/microgpt.ros
time roswell/microgpt.ros
num docs: 484
vocab size: 57
num params: 5152
step 1000 / 1000 | loss 1.1061

--- inference (new, hallucinated names) ---
sample  1: Додим
sample  2: Веромир
sample  3: Мирослав
sample  4: Добромир
sample  5: Робрислав
sample  6: Моломир
sample  7: Борослав
sample  8: Миромир
sample  9: Морислав
sample 10: Мирослава
sample 11: Воромир
sample 12: Завислав
sample 13: Яранислава
sample 14: Миломир
sample 15: Радислав
sample 16: Боромир
sample 17: Веромир
sample 18: Радислав
sample 19: Торослав
sample 20: Волимир
roswell/microgpt.ros  5.70s user 0.51s system 99% cpu 6.232 total
```

Although the originally ported code was only 6 times faster, replacing some lists with arrays, CLOS classes with data structures, and adding inline and ftype declarations made it almost 10 times faster. I believe there is still room for improvement, but I will leave this code as is to avoid overcomplicating it.

## License

Same spirit as the original: do whatever you want with it.

microgpt

microgpt Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

microgpt

microgpt Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README