ptq_workshop

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 12 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Post Training Quantization (PTQ) Workshop

A minimalist Rust library proving the basic math of 8-bit quantization.
Built for a workshop — the entire core is ~50 lines of code with no external dependencies.

---

## The Big Idea

Large language models like GPT-3 have 175 billion parameters. At full 32-bit precision that is ~700 GB of memory — far beyond a single GPU. Quantization compresses those weights from 32-bit floats down to 8-bit integers, cutting memory by 4× with negligible accuracy loss.

This library gives a glimpse as to how that works, in plain Rust, with a dummy minimalist neural network stub you can run and inspect yourself.

---

## The Math

**Encode** (float → integer):

$$q = \text{round}\left(\frac{w}{\text{scale}}\right) \qquad \text{scale} = \frac{\max(|w|)}{127}$$

**Decode** (integer → float):

$$\tilde{w} = q \times \text{scale}$$

**Error** (what you lose):

$$\text{error} = w - \tilde{w} \leq \frac{\text{scale}}{2}$$

The error is bounded by half the scale — a rounding noise floor. One scale is shared across all weights in a block, so its memory cost is negligible.

---

## Project Structure

```
quant_demo/
├── src/
│   ├── lib.rs          # YOUR WORKSHOP EXERCISE — implement the todo!()s here
│   ├── lib_solution.rs # The library solution file - look here for inspiration if needed
│   └── main.rs         # Demo: 2-layer NN run in f32 vs Q8, printed side by side
├── tests/
│   └── nn_accuracy.rs  # 6 integration tests — all should pass when you are done
└── Cargo.toml
```

---

## What You Need to Implement

Open `src/lib.rs` and fill in the four `todo!()` stubs:

| Item | What it does |
|------|-------------|
| `Q8Block` | Struct holding one `f32` scale + a `Vec<i8>` of quantized weights |
| `quantize()` | Compresses `&[f32]` → `Q8Block` using the encode formula above |
| `dequantize()` | Restores `Q8Block` → `Vec<f32>` using the decode formula above |
| `rmse()` | Measures accuracy loss between original and restored values |

`linear_f32()` and `linear_q8()` are already implemented — they wire your functions into a simple neural network layer.

---

## Running It

```bash
# Check your work — run all 6 tests
cargo test

# Run the demo once all tests pass
cargo run
```

---

## The Tests

Run `cargo test` after each function to check your work. The tests tell you exactly what is expected:

| Test | What it proves |
|------|---------------|
| `quantize_then_dequantize_is_close` | Per-element error ≤ `scale / 2` (the theoretical bound) |
| `all_zeros_round_trip_perfectly` | RMSE < 1e-6 for all-zero input |
| `scale_equals_max_abs_over_127` | The scale formula is exactly correct |
| `integers_fit_in_i8_range` | All quantized values stay within [-127, 127] |
| `quantized_nn_output_close_to_f32` | Full 2-layer NN output RMSE < 1% |
| `quantized_hidden_layer_close_to_f32` | Hidden layer RMSE < 1% |

---

## Key Questions

Think about these as you work through the exercise:

- Why is one scale shared across all weights rather than one per weight?
- Why do we divide by 127 and not 128?
- Why does `.clamp()` exist — can you construct a case where it would actually fire?
- What is the maximum possible error per weight, expressed in terms of scale?
- Why does quantization work *better* on larger models?

---

## Further Reading

- [GPTQ paper](https://arxiv.org/abs/2210.17323) — 4-bit quantization of GPT-3 with negligible accuracy loss
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) — the original transformer paper
- [candle](https://github.com/huggingface/candle) — HuggingFace's Rust ML framework, where the production version of this quantization lives in `k_quants.rs`

ptq_workshop

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README