lora-speedrun

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 146 Open GitHub ↗

Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LoRA Speedrun 🏁

[![CI](https://github.com/Saivineeth147/lora-speedrun/actions/workflows/ci.yml/badge.svg)](https://github.com/Saivineeth147/lora-speedrun/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://raw.githubusercontent.com/Saivineeth147/lora-speedrun/main/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://raw.githubusercontent.com/Saivineeth147/lora-speedrun/main/CONTRIBUTING.md)

**How fast can you LoRA-fine-tune [Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) to ≥ 57% on GSM8K — on a single L40S?**

This is [modded-nanogpt](https://github.com/KellerJordan/modded-nanogpt) for fine-tuning: a
frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is
independently re-run 3× with fresh seeds on identical hardware before it counts.

**Attempting and verifying are free**: official timing runs on a Modal L40S sandbox, and
Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone
can re-verify any record with one command.

## Leaderboard

### Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S

**Current record: 0m 22s** by [@slippylolo](https://github.com/slippylolo) — Direct-from-safetensors Qwen load with a warmed page cache, then 12 optimizer updates: 8 full-network, then 4 through only the top decoder layer over a cached BF16 frozen-prefix boundary.

| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|------|--------|-----------|----------|---|-----------|
| 0 | 2026-07-18 | [@Saivineeth147](https://github.com/Saivineeth147) | 11m 57s | 59.4% | — | Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. ([report](records/verifications/000-baseline.md)) |
| 1 | 2026-07-18 | [@Saivineeth147](https://github.com/Saivineeth147) | 6m 05s | 61.1% | −49% | Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. ([report](records/verifications/001-saivineeth147.md)) |
| 2 | 2026-07-20 | [@stared](https://github.com/stared) | 1m 53s | 59.6% | −69% | One 4e-4 epoch over 3,000 examples with 2x optimizer updates (integrated-LR), 512-token packs. First outside record. ([report](records/verifications/002-pmigdal.md)) |
| 3 | 2026-07-20 | [@stared](https://github.com/stared) | 1m 44s | 60.3% | −8% | Shortest-4k data pruning, 1 aggressive-LR epoch, <<...>> annotations stripped, custom GPU-resident packed loop, chunked completion-only CE (no full logits). ([report](records/verifications/003-pmigdal.md)) |
| 4 | 2026-07-25 | [@slippylolo](https://github.com/slippylolo) | 0m 44s | 59.6% | −58% | 0.75-epoch six-target BF16 LoRA: shortest-4k pruning with down_proj dropped (13.76M params), deterministic best-fit packing, per-layer fused base+LoRA-A q/k/v & gate/up GEMMs, chunked completion-only CE. ([report](records/verifications/004-slippylolo.md)) |
| 5 | 2026-08-03 | [@slippylolo](https://github.com/slippylolo) | 0m 22s | 58.0% | −51% | Direct-from-safetensors Qwen load with a warmed page cache, then 12 optimizer updates: 8 full-network, then 4 through only the top decoder layer over a cached BF16 frozen-prefix boundary. ([report](records/verifications/005-slippylolo.md)) |

### Track 2 — SQuAD v1.1 · SmolLM2-1.7B · target ≥ 75.5% · 1× L40S

**Current record: 0m 50s** by [@abacon](https://github.com/abacon) — Track 2 port of the Track 1 stack: shortest-4k pruning, best-fit packing, fused base+LoRA-A GEMMs, chunked completion-only CE, 0.75 epoch on SmolLM2-1.7B.

| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|------|--------|-----------|----------|---|-----------|
| 0 | 2026-07-20 | [@Saivineeth147](https://github.com/Saivineeth147) | 8m 45s | 77.5% | — | Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks. ([report](records/verifications/t2-000-baseline.md)) |
| 1 | 2026-07-27 | [@abacon](https://github.com/abacon) | 0m 50s | 77.2% | −91% | Track 2 port of the Track 1 stack: shortest-4k pruning, best-fit packing, fused base+LoRA-A GEMMs, chunked completion-only CE, 0.75 epoch on SmolLM2-1.7B. ([report](records/verifications/t2-001-bacon.md)) |

Full history with verification reports: [records/RECORDS.md](./records/RECORDS.md) ·
Live leaderboard: [huggingface.co/spaces/vineeth98/lora-speedrun](https://huggingface.co/spaces/vineeth98/lora-speedrun)

## The tracks

Two frozen tracks, deliberately different model families and task types — so a technique
only proves general by winning on both. Same hardware, caps, and verification everywhere.

| | Track 1 | Track 2 |
|---|---|---|
| **Base model** | `Qwen/Qwen2.5-1.5B` | `HuggingFaceTB/SmolLM2-1.7B` |
| **Task** | GSM8K math → ≥ **57.0%** exact-match | SQuAD v1.1 QA → ≥ **75.5% EM** |
| **Training data** | GSM8K `train` split only | SQuAD `train` split only |
| **Metric** | **Training wall-clock. Lower wins.** | same |
| **Hardware** | 1× L40S (48 GB), Modal sandbox | same |
| **Constraint** | adapter-only, ≤ 30M trainable params | same |

Machine-readable specs: [`spec.yaml`](./spec.yaml) · [`spec-t2.yaml`](./spec-t2.yaml). Full rules: [TASK.md](./TASK.md).

You control everything else: LoRA rank and placement, quantization, learning-rate schedules,
sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on
1,000 well-chosen examples for 90 seconds if you can make it clear the bar.

## Why this exists

I fine-tune small models on a budget, and I couldn't tell which speedup claims were real.
Every technique — DoRA, rsLoRA, Unsloth, packing tricks — reports numbers on different
models, data, and hardware. In practice the claims are unfalsifiable.

The only format I've seen actually settle arguments like that is a frozen task with a
referee. That's what nanoGPT's speedrun did for pretraining optimizers (Muon came out of
it). This is the same arena for fine-tuning.

What it builds toward: a verified public record of which training tricks pay for
themselves in wall-clock and which don't survive replication. Every record has to explain
its mechanism in its write-up, so the leaderboard doubles as a lab notebook — the
rejected-variants sections are often as useful as the records. And because tricks can
overfit one setup, records live on tracks with different model families and task types:
a technique only proves general by transferring.

## Quickstart

**Official-hardware run (free).** Make a [Modal](https://modal.com) account, then:

```bash
git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup      # one-time browser auth

python harness/modal_verify.py --prefetch    # one-time: cache model + data in a volume

# one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py --submission submissions/000-baseline --runs 1

# full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py --submission submissions/000-baseline --runs 3
```

**Local iteration (optional).** Any 24 GB+ card runs the baseline for fast experimenting —
`bash scripts/setup_gpu.sh`, then `python harness/run_submission.py submissions/000-baseline --runs 1`.
Local times aren't official; the leaderboard clock is the Modal L40S.

Then copy `submissions/TEMPLATE/`, make it faster, and open a PR. See [CONTRIBUTING.md](./CONTRIBUTING.md).

## How records get verified

A real verification, replayed from the logs (seed 463953844, time-compressed): train → integrity + adapter audit → eval → 3-seed verdict.

1. You open a PR with your training script, config, notes, and self-reported numbers.
2. CI statically validates it, and an automated Claude security screen reviews the diff
   (exfiltration attempts, network use, harness tampering, test-set contact) and posts
   its findings publicly.
3. A maintainer reviews the code, then comments `/verify` — which re-runs your submission
   **3× with fresh seeds** in a network-blocked Modal sandbox on the spec L40S. All 3 runs
   must clear the target; official time is the mean.
4. The harness audits the adapter param count and re-verifies model/data content hashes
   (anti-tampering), and the verification report is posted on the PR and committed to
   [records/verifications/](./records/verifications/) with the accept/reject reasoning.

Full protocol, rubric, and threat model: [JUDGING.md](./JUDGING.md) · [SECURITY.md](./SECURITY.md).

## Ideas nobody has claimed yet

_Taken so far: sequence packing + completion-only masking (record #1)._

1-epoch aggressive-LR schedules · data pruning (train on the hardest 2k examples?) ·
block-diagonal/varlen packing attention · QLoRA NF4 vs bf16 tradeoff · rsLoRA / DoRA /
PiSSA init · LoRA+ (asymmetric LR for A/B) · NEFTune noise · curriculum ordering ·
rank/placement search (MLP-only vs attention-only) · `torch.compile` · Unsloth kernels ·
Liger kernels · fused cross-entropy · smarter warmup for short runs

Claim one, beat 0m 22s, get your name on the board.

## FAQ

**Is this LoRa the radio protocol?** No — LoRA ([Low-Rank Adaptation](https://en.wikipedia.org/wiki/LoRA_(machine_learning))),
the standard cheap way to fine-tune a language model: train a small adapter on top of a
frozen model. The competition: everyone fine-tunes the same model to the same score on
the same GPU, and the fastest verified training run holds the record.

**Won't techniques overfit to one model + one task?** That's exactly why there are two
tracks with different model families and task types — and more will follow the same
freeze-and-calibrate protocol. A trick that only wins on one track is a record, but the
techniques worth trusting are the ones that transfer. The track system makes that an
empirical question instead of an argument.

**Why Qwen2.5-1.5B? Isn't it pretrained on math?** Probably, like every modern base model.
It doesn't matter: the target is an *anchor*, not a claim about mathematical discovery. The
race is the interesting part — same reason nanoGPT speedrunning targets an arbitrary val loss.
(Track 2 uses a different family, SmolLM2, partly for this reason.)

**Why wall-clock instead of FLOPs or steps?** Because wall-clock is what you pay for, and it
forces kernels, data loading, and algorithms to compete in the same currency. Same rule as
modded-nanogpt.

**Why an L40S on Modal instead of a 4090 or H100?** Three reasons. It's one consistent
datacenter SKU, so times are actually comparable (rented consumer cards vary host-to-host).
It's free to use via Modal's monthly credits, so competing and re-verifying costs nothing.
And submissions are strangers' code — Modal sandboxes run them network-blocked and
secretless. (The L40S is the same AD102 silicon as the 4090, so consumer-GPU tricks transfer.)

**Can I train on other data / distill from a bigger model?** No. GSM8K train split only,
no teacher models, no synthetic data. See [TASK.md](./TASK.md) for the full banned list.

**Multiple GPUs?** No. One L40S. That's the point.

## License

MIT. Records, reports, and write-ups are as public as the code.

lora-speedrun

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README