llm-from-scratch

KO | EN

gitlite — search

#hacktoberfest #react #ai #go #android #docker #macos #java #nodejs #cli #vue #awesome-list #ios #open-source

llm-from-scratch

(★ 20)

Build a small GPT-style language model from scratch in seven guided, interactive missions.

llm-from-scratch Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LLM From Scratch Lab

### Seven missions. One empty notebook. A working GPT-style model.

This is not a repository you clone and forget. It is a guided lab where every
lesson fixes one visible limitation in the model you built in the lesson before it.

[**Start the 7-part course**](course/README.md) ·
[**Open the notebook**](notebooks/Build_Your_Own_LLM_From_Scratch.ipynb) ·
[**Try the trained model**](TRY_THE_MODEL.md) ·
[**Launch the mission map**](https://lerabyte.github.io/llm-from-scratch/)

</div>

> [!IMPORTANT]
> We are building a **small decoder-only Transformer from random weights**.
> It is not ChatGPT-sized and it is not an instruction-following assistant.
> The tokenizer, embeddings, causal attention, Transformer blocks, training loop,
> and generation process are all real.

## Your mission

Start with a text file. End with a model that can generate new text one token at
a time. Along the way, you will not merely run the code. You will see why each
component exists, what breaks without it, and how to prove that it works.

| Mission | The problem | What you build | Your proof |
|---|---|---|---|
| [01 · Find the signal](course/01-data/README.md) | The model has nothing to learn from | Dataset and train/validation split | Clean text with two separate splits |
| [02 · Invent an alphabet](course/02-tokenizer/README.md) | Neural networks cannot read text | Character tokenizer and batches | Text survives an encode/decode round trip |
| [03 · Make it speak badly](course/03-bigram/README.md) | We have data but no learner | Bigram language model and training loop | Loss falls and output stops looking fully random |
| [04 · Give it a memory](course/04-context/README.md) | The model sees only one token | Embeddings, positions, and causal context | Predictions use a sequence instead of one character |
| [05 · Teach it what matters](course/05-attention/README.md) | All previous tokens are treated equally | One causal self-attention head | A lower-triangular attention map |
| [06 · Assemble the Transformer](course/06-transformer/README.md) | One attention head is not a GPT | Multi-head attention and Transformer blocks | A complete forward pass and parameter count |
| [07 · Bring it to life](course/07-training/README.md) | The Transformer still has random weights | Full training, evaluation, and generation | Validation loss falls and the model generates text |

## Choose your route

New here? Use the two-minute [Start Here guide](START_HERE.md). If a term feels
unclear, open the [plain-language glossary](GLOSSARY.md). If code fails, go to the
[Bug Clinic](TROUBLESHOOTING.md) before changing several things at once.

Ready to put it online? Follow the [publishing checklist](PUBLISHING.md).

### I want to understand everything

Open the [numbered course folders](course/README.md), complete each checkpoint in order, and use
the notebook as your workspace. Every mission ends with a test and a small
experiment.

### I want to run the finished model first

Open the [complete notebook](notebooks/Build_Your_Own_LLM_From_Scratch.ipynb),
choose a GPU runtime in Google Colab, and select **Run all**. Then return to the
mission guides to unpack what happened.

### I want to train it on my own writing

Run the default dataset once. After the full pipeline works, upload one clean
UTF-8 text file, set `USE_CUSTOM_DATA = True`, and update `CUSTOM_DATA_PATH`.
Do not debug a new dataset and a new model at the same time.

## The complete model, in one view

```text
raw text
   ↓
character token IDs
   ↓
token embeddings + position embeddings
   ↓
causal multi-head self-attention
   ↓
feed-forward network + residual connections
   ↓
stacked Transformer blocks
   ↓
next-token logits → probabilities → sampled token
```

## All you need

- A computer
- A free Google Colab account
- Enough basic Python to follow variables, functions, and classes
- Curiosity about what the model is doing between the input and output

You do not need a local GPU or advanced calculus to complete the guided build.

## Quick start

### Google Colab

1. Open [`Build_Your_Own_LLM_From_Scratch.ipynb`](notebooks/Build_Your_Own_LLM_From_Scratch.ipynb).
2. Choose **Open in Colab** or upload the downloaded notebook to Colab.
3. Select **Runtime → Change runtime type → T4 GPU**.
4. Run the notebook from top to bottom.
5. Do not skip a failed cell. Later lessons depend on earlier variables.

### Local Python

```bash
git clone https://github.com/lerabyte/llm-from-scratch.git
cd llm-from-scratch
python -m venv .venv
```

Activate the environment, then install and run:

```bash
pip install -r requirements.txt
python src/lera_lm_course.py
```

Training without a supported GPU will be substantially slower, so Colab is the
recommended first experience.

## The learning loop

Every mission uses the same four-question check:

1. **What currently fails?** Identify the limitation before adding code.
2. **What is the smallest useful fix?** Build only the next required component.
3. **How can we prove it works?** Inspect shapes, loss, generated text, or weights.
4. **What is still missing?** Use the remaining failure to motivate the next mission.

That progression is the point of this repository. The final model matters, but
understanding why it needed every component matters more.

## Repo map

```text
llm-from-scratch/
├── README.md                   ← you are here
├── course/                     ← clearly numbered Part 01–07 folders
├── lessons/                    ← seven guided missions
├── notebooks/                 ← complete Colab-ready notebook
│   └── checkpoints/           ← one cumulative recovery notebook per mission
├── src/                        ← the same build as a Python script
├── docs/                       ← interactive GitHub Pages mission map
├── assets/                     ← repository visual identity
├── results/                    ← honest before/after graduation runs
├── TRY_THE_MODEL.md            ← prompt and temperature playground
├── START_HERE.md
├── GLOSSARY.md
├── TROUBLESHOOTING.md
├── requirements.txt
└── LICENSE
```

## Ground rules

- Run the baseline before changing it.
- Print tensor shapes whenever the data feels abstract.
- Change one major variable at a time.
- Keep training and validation loss separate.
- Never call an attention map “the model's thoughts.”
- Never call this small base model ChatGPT.
- Record the failure, the change, and the result of every experiment.

## What “finished” means

By the end, you should be able to explain:

- Why next-token prediction creates a language model
- How text becomes input and target tensors
- What the loss tells the optimizer
- Why token order needs positional information
- How causal attention blocks future answers
- What lives inside a Transformer block
- Why generation is repeated next-token sampling
- Why a base model is different from an assistant

If you can only run the final cell, you ran a notebook. If you can explain those
eight ideas, you built a language model.

## Acknowledgments

The educational progression is inspired by Andrej Karpathy's
[Neural Networks: Zero to Hero](https://github.com/karpathy/nn-zero-to-hero) and
[nanoGPT](https://github.com/karpathy/nanoGPT). The default corpus is the small
Tiny Shakespeare dataset commonly used for character-level language-model demos.

## License

MIT. Learn from it, remix it, and share what you discover.

llm-from-scratch

llm-from-scratch Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

llm-from-scratch

llm-from-scratch Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README