lora-speedrun - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #golang #machine-learning #linux #docker #macos #java #awesome #cli #kubernetes #open-source

lora-speedrun

(★ 147)

Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.

파일 탐색기

최종 버전 다운로드 (.zip)

- - bug_report.md
  - dispute.md
  - rule-question.md
- - agent-review.yml
  - ci.yml
  - verify.yml
- PULL_REQUEST_TEMPLATE.md
- leaderboard.png
- verification-replay.gif
- evaluate_gsm8k.py
- evaluate_squad.py
- integrity_check.py
- modal_verify.py
- pins-t2.json
- pins.json
- prefetch.py
- run_submission.py
- validate_submission.py
- - 000-baseline.md
  - 001-saivineeth147.md
  - 002-pmigdal.md
  - 003-pmigdal.md
  - 004-slippylolo.md
  - 005-slippylolo.md
  - t2-000-baseline.md
  - t2-001-bacon.md
  - TEMPLATE.md
- records.json
- RECORDS.md
- generate_hero_card.py
- render_space.py
- setup_gpu.sh
- update_leaderboard.py
- index.html
- index.template.html
- README.md
- style.css
- - config.yaml
  - NOTES.md
  - train.py
- - config.yaml
  - NOTES.md
  - train.py
- - config.yaml
  - NOTES.md
  - train.py
- - config.yaml
  - NOTES.md
  - train.py
- - config.yaml
  - hot_loop_plan.py
  - NOTES.md
  - packing.py
  - train.py
- - adapter_writer.py
  - area_schedule.py
  - config.yaml
  - engineering_config.py
  - frozen_prefix.py
  - hot_loop_plan.py
  - NOTES.md
  - packing.py
  - production_config.py
  - progressive.py
  - projection_ablations.py
  - qwen_dispatch.py
  - startup_loader.py
  - static_lora.py
  - step_frontier.py
  - train.py
- - config.yaml
  - NOTES.md
  - train.py
- - config.yaml
  - hot_loop_plan.py
  - NOTES.md
  - packing.py
  - train.py
- - config.yaml
  - NOTES.md
  - train.py
.gitignore
CONTRIBUTING.md
env.lock
JUDGING.md
LICENSE
README.md
requirements.txt
SECURITY.md
spec-t2.yaml
spec.yaml
TASK.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LoRA Speedrun 🏁

[![CI](https://github.com/Saivineeth147/lora-speedrun/actions/workflows/ci.yml/badge.svg)](https://github.com/Saivineeth147/lora-speedrun/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://raw.githubusercontent.com/Saivineeth147/lora-speedrun/main/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://raw.githubusercontent.com/Saivineeth147/lora-speedrun/main/CONTRIBUTING.md)

**How fast can you LoRA-fine-tune [Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) to ≥ 57% on GSM8K — on a single L40S?**

This is [modded-nanogpt](https://github.com/KellerJordan/modded-nanogpt) for fine-tuning: a
frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is
independently re-run 3× with fresh seeds on identical hardware before it counts.

**Attempting and verifying are free**: official timing runs on a Modal L40S sandbox, and
Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone
can re-verify any record with one command.

<p align="center">
  <img src="docs/leaderboard.png" alt="LoRA Speedrun leaderboard — 11:57 baseline to 0:22 current record (−97%) across six verified records" width="760">
</p>

## Leaderboard

### Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S

**Current record: 0m 22s** by [@slippylolo](https://github.com/slippylolo) — Direct-from-safetensors Qwen load with a warmed page cache, then 12 optimizer updates: 8 full-network, then 4 through only the top decoder layer over a cached BF16 frozen-prefix boundary.

| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|------|--------|-----------|----------|---|-----------|
| 0 | 2026-07-18 | [@Saivineeth147](https://github.com/Saivineeth147) | 11m 57s | 59.4% | — | Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. ([report](records/verifications/000-baseline.md)) |
| 1 | 2026-07-18 | [@Saivineeth147](https://github.com/Saivineeth147) | 6m 05s | 61.1% | −49% | Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. ([report](records/verifications/001-saivineeth147.md)) |
| 2 | 2026-07-20 | [@stared](https://github.com/stared) | 1m 53s | 59.6% | −69% | One 4e-4 epoch over 3,000 examples with 2x optimizer updates (integrated-LR), 512-token packs. First outside record. ([report](records/verifications/002-pmigdal.md)) |
| 3 | 2026-07-20 | [@stared](https://github.com/stared) | 1m 44s | 60.3% | −8% | Shortest-4k data pruning, 1 aggressive-LR epoch, <<...>> annotations stripped, custom GPU-resident packed loop, chunked completion-only CE (no full logits). ([report](records/verifications/003-pmigdal.md)) |
| 4 | 2026-07-25 | [@slippylolo](https://github.com/slippylolo) | 0m 44s | 59.6% | −58% | 0.75-epoch six-target BF16 LoRA: shortest-4k pruning with down_proj dropped (13.76M params), deterministic best-fit packing, per-layer fused base+LoRA-A q/k/v & gate/up GEMMs, chunked completion-only CE. ([report](records/verifications/004-slippylolo.md)) |
| 5 | 2026-08-03 | [@slippylolo](https://github.com/slippylolo) | 0m 22s | 58.0% | −51% | Direct-from-safetensors Qwen load with a warmed page cache, then 12 optimizer updates: 8 full-network, then 4 through only the top decoder layer over a cached BF16 frozen-prefix boundary. ([report](records/verifications/005-slippylolo.md)) |

### Track 2 — SQuAD v1.1 · SmolLM2-1.7B · target ≥ 75.5% · 1× L40S

**Current record: 0m 50s** by [@abacon](https://github.com/abacon) — Track 2 port of the Track 1 stack: shortest-4k pruning, best-fit packing, fused base+LoRA-A GEMMs, chunked completion-only CE, 0.75 epoch on SmolLM2-1.7B.

| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|------|--------|-----------|----------|---|-----------|
| 0 | 2026-07-20 | [@Saivineeth147](https://github.com/Saivineeth147) | 8m 45s | 77.5% | — | Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks. ([report](records/verifications/t2-000-baseline.md)) |
| 1 | 2026-07-27 | [@abacon](https://github.com/abacon) | 0m 50s | 77.2% | −91% | Track 2 port of the Track 1 stack: shortest-4k pruning, best-fit packing, fused base+LoRA-A GEMMs, chunked completion-only CE, 0.75 epoch on SmolLM2-1.7B. ([report](records/verifications/t2-001-bacon.md)) |

Full history with verification reports: [records/RECORDS.md](./records/RECORDS.md) ·
Live leaderboard: [huggingface.co/spaces/vineeth98/lora-speedrun](https://huggingface.co/spaces/vineeth98/lora-speedrun)

## The tracks

Two frozen tracks, deliberately different model families and task types — so a technique
only proves general by winning on both. Same hardware, caps, and verification everywhere.

| | Track 1 | Track 2 |
|---|---|---|
| **Base model** | `Qwen/Qwen2.5-1.5B` | `HuggingFaceTB/SmolLM2-1.7B` |
| **Task** | GSM8K math → ≥ **57.0%** exact-match | SQuAD v1.1 QA → ≥ **75.5% EM** |
| **Training data** | GSM8K `train` split only | SQuAD `train` split only |
| **Metric** | **Training wall-clock. Lower wins.** | same |
| **Hardware** | 1× L40S (48 GB), Modal sandbox | same |
| **Constraint** | adapter-only, ≤ 30M trainable params | same |

Machine-readable specs: [`spec.yaml`](./spec.yaml) · [`spec-t2.yaml`](./spec-t2.yaml). Full rules: [TASK.md](./TASK.md).

You control everything else: LoRA rank and placement, quantization, learning-rate schedules,
sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on
1,000 well-chosen examples for 90 seconds if you can make it clear the bar.

## Why this exists

I fine-tune small models on a budget, and I couldn't tell which speedup claims were real.
Every technique — DoRA, rsLoRA, Unsloth, packing tricks — reports numbers on different
models, data, and hardware. In practice the claims are unfalsifiable.

The only format I've seen actually settle arguments like that is a frozen task with a
referee. That's what nanoGPT's speedrun did for pretraining optimizers (Muon came out of
it). This is the same arena for fine-tuning.

What it builds toward: a verified public record of which training tricks pay for
themselves in wall-clock and which don't survive replication. Every record has to explain
its mechanism in its write-up, so the leaderboard doubles as a lab notebook — the
rejected-variants sections are often as useful as the records. And because tricks can
overfit one setup, records live on tracks with different model families and task types:
a technique only proves general by transferring.

## Quickstart

**Official-hardware run (free).** Make a [Modal](https://modal.com) account, then:

```bash
git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup      # one-time browser auth

python harness/modal_verify.py --prefetch    # one-time: cache model + data in a volume

# one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py --submission submissions/000-baseline --runs 1

# full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py --submission submissions/000-baseline --runs 3
```

**Local iteration (optional).** Any 24 GB+ card runs the baseline for fast experimenting —
`bash scripts/setup_gpu.sh`, then `python harness/run_submission.py submissions/000-baseline --runs 1`.
Local times aren't official; the leaderboard clock is the Modal L40S.

Then copy `submissions/TEMPLATE/`, make it faster, and open a PR. See [CONTRIBUTING.md](./CONTRIBUTING.md).

## How records get verified

<p align="center">
  <img src="docs/verification-replay.gif" alt="Terminal replay of a real record verification: training, integrity + adapter audit, GSM8K eval, and the 3-seed verdict" width="760">
</p>

<p align="center"><em>A real verification, replayed from the logs (seed 463953844, time-compressed): train → integrity + adapter audit → eval → 3-seed verdict.</em></p>

1. You open a PR with your training script, config, notes, and self-reported numbers.
2. CI statically validates it, and an automated Claude security screen reviews the diff
   (exfiltration attempts, network use, harness tampering, test-set contact) and posts
   its findings publicly.
3. A maintainer reviews the code, then comments `/verify` — which re-runs your submission
   **3× with fresh seeds** in a network-blocked Modal sandbox on the spec L40S. All 3 runs
   must clear the target; official time is the mean.
4. The harness audits the adapter param count and re-verifies model/data content hashes
   (anti-tampering), and the verification report is posted on the PR and committed to
   [records/verifications/](./records/verifications/) with the accept/reject reasoning.

Full protocol, rubric, and threat model: [JUDGING.md](./JUDGING.md) · [SECURITY.md](./SECURITY.md).

## Ideas nobody has claimed yet

_Taken so far: sequence packing + completion-only masking (record #1)._

1-epoch aggressive-LR schedules · data pruning (train on the hardest 2k examples?) ·
block-diagonal/varlen packing attention · QLoRA NF4 vs bf16 tradeoff · rsLoRA / DoRA /
PiSSA init · LoRA+ (asymmetric LR for A/B) · NEFTune noise · curriculum ordering ·
rank/placement search (MLP-only vs attention-only) · `torch.compile` · Unsloth kernels ·
Liger kernels · fused cross-entropy · smarter warmup for short runs

Claim one, beat 0m 22s, get your name on the board.

## FAQ

**Is this LoRa the radio protocol?** No — LoRA ([Low-Rank Adaptation](https://en.wikipedia.org/wiki/LoRA_(machine_learning))),
the standard cheap way to fine-tune a language model: train a small adapter on top of a
frozen model. The competition: everyone fine-tunes the same model to the same score on
the same GPU, and the fastest verified training run holds the record.

**Won't techniques overfit to one model + one task?** That's exactly why there are two
tracks with different model families and task types — and more will follow the same
freeze-and-calibrate protocol. A trick that only wins on one track is a record, but the
techniques worth trusting are the ones that transfer. The track system makes that an
empirical question instead of an argument.

**Why Qwen2.5-1.5B? Isn't it pretrained on math?** Probably, like every modern base model.
It doesn't matter: the target is an *anchor*, not a claim about mathematical discovery. The
race is the interesting part — same reason nanoGPT speedrunning targets an arbitrary val loss.
(Track 2 uses a different family, SmolLM2, partly for this reason.)

**Why wall-clock instead of FLOPs or steps?** Because wall-clock is what you pay for, and it
forces kernels, data loading, and algorithms to compete in the same currency. Same rule as
modded-nanogpt.

**Why an L40S on Modal instead of a 4090 or H100?** Three reasons. It's one consistent
datacenter SKU, so times are actually comparable (rented consumer cards vary host-to-host).
It's free to use via Modal's monthly credits, so competing and re-verifying costs nothing.
And submissions are strangers' code — Modal sandboxes run them network-blocked and
secretless. (The L40S is the same AD102 silicon as the 4090, so consumer-GPU tricks transfer.)

**Can I train on other data / distill from a bigger model?** No. GSM8K train split only,
no teacher models, no synthetic data. See [TASK.md](./TASK.md) for the full banned list.

**Multiple GPUs?** No. One L40S. That's the point.

## License

MIT. Records, reports, and write-ups are as public as the code.