auto-sft

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 9)

Provide a data set and a base model and Auto-SFT finds the optimal settings for fine-tuning using unsloth as the underlying library. See inference test as you go and upload the best result to huggingface automatically. A straightforward 3-step process, inspired by Karpathy's Auto Research project.

auto-sft Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Auto-SFT

**Automated LoRA fine-tuning pipeline: hyperparameter search → best-config full fine-tune → GGUF export.** Inspired by Karpathy's autoresearch.

Point it at a base model and a dataset. It runs a fast, cheap hyperparameter search (short LoRA experiments, scored by an LLM judge or standard benchmarks), picks the winning config, then runs one full-length fine-tune with it — optionally exporting to GGUF and pushing to Hugging Face.

- 🔍 **Automated search** — random or hybrid explore→exploit sampling over LoRA rank/alpha/dropout/LR/batch size/etc.
- ⚡ **Unsloth or vanilla PEFT** — pick the training engine per run; the final fine-tune always matches whatever the search used.
- 🧠 **LLM-as-judge or lm-eval** — score with an adaptive rubric written for *your* dataset, or fall back to standardized benchmarks (ARC, HellaSwag, GSM8K, MMLU).
- 🖥️ **Web UI** — start/stop runs, watch live progress and loss curves, compare base vs. tuned samples, manage settings, kick off the final fine-tune — all from the browser.
- 📦 **GGUF export + optional Hugging Face upload** — merged model and quantized GGUFs, opt-in only.
- 🐳 **Docker-first**, with a native Python path too.

> **GPU required.** Full fine-tuning (`auto_sft.finetune`) requires CUDA and refuses to run on CPU — no silent fallback.

## Quick Start (Docker)

```bash
cp .env.example .env        # add your eval API key (OpenRouter/OpenAI/etc.)
docker compose up
```

Open **http://localhost:8765** and configure a run from the Settings page — or drive it entirely from the CLI (see below).

## Quick Start (CLI)

```bash
pip install -e ".[web]"
# For Unsloth-accelerated training (recommended):
pip install -e ".[unsloth]"

export AUTO_SFT_EVAL_API_KEY=your-key
export AUTO_SFT_EVAL_API_BASE=https://openrouter.ai/api/v1

# 1. Search for the best hyperparameters
python -m auto_sft.loop \
  --model Qwen/Qwen2.5-0.5B \
  --train-data data/train.jsonl \
  --eval-data data/eval.jsonl \
  --engine unsloth \
  --run-time 5 --max-runs 50 --max-hours 8

# 2. Fine-tune with the winning config (same --engine as the search!)
python -m auto_sft.finetune \
  --model Qwen/Qwen2.5-0.5B \
  --train-data data/train.jsonl \
  --best-config runs/best_config.json \
  --epochs 3 \
  --engine unsloth
```

The web UI (`python -m auto_sft.web`) starts automatically alongside `loop.py` unless `--no-web` is passed, and can also launch the final fine-tune for you (with `auto_finetune` in settings, or via the `/finetune` page).

## Installation

### Option 1: Docker (recommended for GPU servers)

Requires [Docker](https://docs.docker.com/get-docker/) and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).

```bash
cp .env.example .env   # fill in your eval API key
docker compose up
```

### Option 2: Native Python

```bash
pip install -e ".[web]"
pip install -e ".[unsloth]"    # recommended, faster training
```

**GPU wheel note (important on Windows/Linux):** PyPI only serves CPU-only `torch` wheels. This project's `pyproject.toml` pins `[tool.uv.sources]` to pull `torch` from PyTorch's CUDA index — if you're using `uv`, `uv sync` gets a CUDA build automatically. If you're using plain `pip`, install torch explicitly from the CUDA index first:

```bash
# RTX 50-series / Blackwell (sm_120) — needs CUDA 12.8+
pip install torch --index-url https://download.pytorch.org/whl/cu128
```

If `torch.cuda.is_available()` is ever `False` after install, that's almost always a CPU wheel — reinstall from the CUDA index above.

## Concept

**How it searches.** Runs start in a pure **exploration** phase — configs are sampled randomly and independently across the full search space (see [Parameter Search Space](#parameter-search-space)). Nothing is pruned up front, since a rank/LR/dropout combo that underperforms on one task can be optimal on another. After a configurable number of exploration runs (`--exploit-threshold`), `--search hybrid` switches to **exploitation**: new configs are perturbations around the best config found so far, narrowing the search into a promising region instead of continuing to sample blindly. Early stopping (`--early-stop-runs` / `--early-stop-hours`) cuts the session short once the best score stops improving.

**How it scores.** Two modes:
- **LLM-as-judge** (`--eval-mode lm-judge`, default) — a judge LLM scores sampled input/output pairs against a rubric. Auto-SFT first shows the judge real examples from *your* training data and asks it to write a rubric adapted to that dataset, so "helpful" means something different for a support bot than for a code assistant.
- **lm-eval benchmarks** (`--eval-mode lm-eval`) — standardized benchmarks (ARC, HellaSwag, GSM8K, MMLU, ...) via vLLM, for an objective signal instead of a subjective judge. **Experimental, and CLI-only** — the web UI always runs lm-judge; there's no `--eval-mode` field in Settings yet.

**How the data is split.** Before the first run, the dataset is split once for the
whole session: an evaluation **holdout** is carved out and *excluded from
training*, and every run then trains on identical rows and is scored on identical
held-out ones. Held-out rows are picked at an even stride across the dataset, not
off the tail, so a file ordered by topic or difficulty doesn't produce a holdout
representing only its last slice. Set the size with `--eval-split-percent` (or
*Evaluation Split (%)* in the UI); `0` holds out exactly `--eval-samples` rows,
the least training data that still keeps the two sets disjoint. The holdout never
exceeds half the dataset. Point `--eval-data` at a separate file or dataset and no
holdout is taken at all — that set is already disjoint.

**How much data a run sees.** A step budget and a dataset size are independent, so
a run can cover a fraction of the data or pass over it many times. Both are fine;
neither should be a surprise, so the coverage is reported as *"each entry will be
seen 0.64 times"* or *"entries are seen 2.7 times each"* — in the log for every
run, and live on the Configuration page as you change Max Steps. When a budget
won't complete one full pass, the UI says so. Partial passes are drawn at random
across the whole dataset (never the first N rows), and the data order is pinned to
the session seed so configs are compared on the same examples.

**The judge, and what happens when it misbehaves.** Credentials resolve in one
order everywhere — CLI flag, then `AUTO_SFT_EVAL_API_KEY`, then `OPENAI_API_KEY`
(and `AUTO_SFT_EVAL_API_BASE` → `OPENAI_BASE_URL` for the endpoint). Every session
logs which endpoint and which variable it used, with the key masked, and makes one
tiny preflight call before training so a wrong key fails in seconds instead of
after fifty runs of 0.0. Local judges that need no key pass the preflight.

Once running, judge calls retry through the failures worth retrying: rate limits,
dropped connections, timeouts and 5xx back off exponentially, and an *empty* reply
is retried too (usually a dropped completion, not a verdict). If a reply comes
back empty because the model hit its token limit mid-thought — common with
reasoning models — the retry gets double the budget rather than repeating the same
losing call. A sample that still can't be scored costs only that sample; the run
average ignores it. Authentication failures don't retry: they stop the search with
a clear message, because nothing in that session can be scored. There is no
fallback to a different provider.

**How the winner is picked.** The average judge score decides it. Equal averages are broken deterministically, in order: the tighter **sample spread** wins (an 8.0 earned on every sample beats an 8.0 propped up by one outlier), then the better worst-case sample, then the cheaper adapter (lower rank × fewer target modules). Failed judge calls score 0.0 and are excluded from both the average and the spread, so one API timeout can't decide a tie. A run that only ever equals the leader never takes the title unless it wins one of those tie-breaks.

**Why search and fine-tune are separate steps.** Search cheaply over many configs (short runs) to find what tends to work, then spend real compute on exactly one full-length run using the winner — rather than fine-tuning blind on a guess, or paying full training cost to compare configs.

## CLI Reference — Search (`auto_sft.loop`)

### Model & Data

| Flag | Default | Description |
|------|---------|-------------|
| `--model` | Qwen/Qwen2.5-0.5B | Base model (HuggingFace name) |
| `--train-data` | data/train.jsonl | Training data (path or HF dataset) |
| `--train-subset` | (blank) | HF subset/config for `--train-data` (e.g. `main` for `openai/gsm8k`) |
| `--eval-data` | (blank) | Eval data (path or HF dataset); blank = hold out part of the training data |
| `--eval-subset` | (blank) | HF subset/config for `--eval-data` |
| `--eval-split-percent` | 0 | Share of training data held out for scoring; 0 = exactly `--eval-samples` rows. Held-out rows are never trained on; capped at 50% |

### Training

| Flag | Default | Description |
|------|---------|-------------|
| `--run-time` | 5.0 | Wall-clock minutes per run (used when `--max-steps` is 0) |
| `--max-steps` | 0 | Fixed optimizer steps per run (0 = use `--run-time` instead) |
| `--max-runs` | 100 | Maximum number of runs |
| `--max-hours` | 8.0 | Maximum total session hours |
| `--seed` | 42 | Random seed |
| `--engine` | peft | Training engine: `peft` or `unsloth` |
| `--gradient-checkpointing` | off | Trade ~25% speed for VRAM headroom |
| `--force-local` | off | Skip VRAM-aware routing check, always run locally |

### Search Strategy

| Flag | Default | Description |
|------|---------|-------------|
| `--search` | random | `random` (pure exploration) or `hybrid` (explore → exploit) |
| `--exploit-threshold` | 3 | Runs before hybrid search starts exploiting the best config |
| `--early-stop-runs` | 12 | Stop if no improvement after N runs (0 = disabled) |
| `--early-stop-hours` | 1.0 | Stop if no improvement after N hours (0 = disabled) |

### Evaluation

| Flag | Default | Description |
|------|---------|-------------|
| `--eval-mode` | lm-judge | `lm-judge` (LLM scoring) or `lm-eval` (standardized benchmarks, experimental, **CLI-only** — not yet exposed in the web UI) |
| `--eval-criteria` | conversational | Criteria template name, or a custom prompt string |
| `--custom-criterion` | (none) | Path to a file with a custom scoring criterion (skips adaptive rubric generation) |
| `--eval-model` | gpt-5.5 | Model used for judge scoring. The web UI reads `eval_model` from `config/settings.json` (Settings page); there is no environment variable for the model |
| `--eval-api-base` | $AUTO_SFT_EVAL_API_BASE, then $OPENAI_BASE_URL | Judge API endpoint |
| `--eval-api-key` | $AUTO_SFT_EVAL_API_KEY, then $OPENAI_API_KEY | Judge API key |
| `--eval-samples` | 5 | Samples evaluated per run |
| `--lm-eval-tasks` | arc_easy,arc_challenge,hellaswag,gsm8k,mmlu | Comma-separated tasks (only with `--eval-mode lm-eval`) |

### Output & Web UI

| Flag | Default | Description |
|------|---------|-------------|
| `--output` | runs | Output directory |
| `--web-host` / `--web-port` | 0.0.0.0 / 8765 | Web UI bind address |
| `--no-web` | false | Disable the web UI |
| `--session-id` | (auto) | Session ID (auto-generated if not specified) |

## Full Fine-Tuning (`auto_sft.finetune`)

Runs one full-length training pass using a winning config from the search, then merges the LoRA adapter into a standalone model. **Requires a CUDA GPU — there is no CPU fallback**, and `--engine` must match whatever engine the search used (the web UI does this automatically).

**Budget: epochs or steps.** By default the run trains for a whole number of epochs. Pass `--max-steps N` (or pick *Max steps* under **Auto Fine-tune** in the web UI) to train for exactly N optimizer steps instead — the same trade-off the search offers, and the only way to hold run length constant when the dataset size changes.

| Flag | Default | Description |
|------|---------|-------------|
| `--model` | *required* | Base model |
| `--train-data` | *required* | Training data (path or HF dataset) |
| `--best-config` | *required* | Path to `best_config.json` from the search |
| `--epochs` | *required* | Number of epochs (ignored when `--max-steps` > 0) |
| `--max-steps` | 0 | Train for exactly N optimizer steps instead of whole epochs (0 = use `--epochs`) |
| `--engine` | peft | Must match the search's `--engine` |
| `--gradient-checkpointing` | off | Same VRAM/speed tradeoff as the search |
| `--output-dir` | finetuned | Output directory |
| `--eval-data` | (blank) | Eval samples for before/after comparison; blank = held out of the training data |
| `--eval-samples` | 5 | Number of before/after samples to generate |
| `--train-subset` / `--eval-subset` | (blank) | HF subset/config for each dataset |
| `--validation-rows` | 500 | Rows held out for validation loss (also where before/after samples are drawn from); ignored when `--eval-data` points at a separate file |
| `--no-early-stop` | off | Disable early stopping on validation loss (early stopping is on by default) |
| `--early-stop-patience` | 3 | Evals without validation-loss improvement before stopping early |
| `--gguf-export` | off | Export merged model to GGUF |
| `--gguf-quant-levels` | (none) | Comma-separated quant levels, e.g. `Q4_K_M,Q8_0` |
| `--hf-token` | $HF_TOKEN | Hugging Face token (upload is **opt-in**, off unless username + model name are set) |
| `--hf-username` / `--hf-model-name` | (none) | Target HF repo — required together to enable upload |
| `--no-hf-upload` | off | Force-disable HF upload even if credentials are set |

Output layout:

```
finetuned/<timestamp>/
├── state.json      # live progress (phase, loss, samples) — read by the web UI
├── adapter/         # saved LoRA adapter
├── merged/          # full merged model (safetensors)
└── gguf/            # quantized GGUF exports (if --gguf-export)
```

## Evaluation Criteria

Built-in criteria templates (`--eval-criteria`):

| Name | Description |
|------|-------------|
| `conversational` | Natural, flowing language |
| `helpful` | Direct, useful, not sycophantic |
| `anti_sycophantic` | Independent, honest responses |
| `persona` | Consistent voice/personality |
| `task_accuracy` | Follows instructions correctly |
| `coherence` | Logical, clear structure |

Or pass a custom prompt template as `--eval-criteria`, or a file path via `--custom-criterion`.

## Data Sources

**Local files:**
```bash
--train-data data/my_dataset.jsonl
```

| Extension | Notes |
|-----------|-------|
| `.jsonl` / `.ndjson` | One JSON object per line (recommended for large sets) |
| `.json` | An array of entries, a single entry, or a wrapper object (`{"data": [...]}`) |
| `.csv` / `.tsv` | Header row required; cells holding JSON (e.g. a `messages` column) are parsed |
| `.parquet` | The native format of most Hugging Face dataset downloads |
| `*.gz` | Any of the text formats above, gzipped (`train.jsonl.gz`) |

**Hugging Face datasets:**
```bash
--train-data tatsu-lab/alpaca
--train-data openai/gsm8k --train-subset main    # subset (a.k.a. config)
--train-data openai/gsm8k:main                   # same thing, shorthand
```

Some datasets are published as several **subsets** (what the `datasets` library
calls configs) and cannot be loaded without naming one. In the web UI the Subset
dropdown next to Training Data fills itself from the Hub: datasets with no
subsets leave it disabled, and a dataset that requires one says so. From the CLI
use `--train-subset` / `--eval-subset`, or the `repo:subset` shorthand. If a
dataset requires a subset and none is given, the first available one is used and
logged.

Auto-SFT always trains on the `train` split. Other splits a dataset ships
(`validation`, `test`) are listed in the UI for reference but not used.

**Supported entry formats** (auto-detected and normalized to `instruction` / `input` / `output`):

| Format | Fields |
|--------|--------|
| Alpaca | `instruction`, `input` (optional), `output` |
| ShareGPT | `conversations: [{from, value}]` |
| OpenAI Messages | `messages: [{role, content}]` |
| Simple | `prompt`, `response` |
| Q&A | `question`, `answer` |
| Completion | `prompt`, `completion` |
| Preference / DPO | `prompt`, `chosen`, `rejected` — trains on `chosen`, ignores `rejected` |
| FLAN / seq2seq | `inputs`, `targets` or `source`, `target` |
| Summarisation | `document` or `article`, plus `summary` or `highlights` |
| Input/output | `input`, `output` |

Field names are matched against a priority list, so near-miss variants
(`query`/`response`, `text`/`answer`, `instruction`/`response`, …) resolve too.
Extra context in `input`, `context`, or `passage` is appended to the instruction,
the way Alpaca's `input` field works.

**Chat transcripts.** Multi-turn `messages`/`conversations` entries are collapsed
into a single exchange: the last assistant turn becomes the target output, paired
with the user turn immediately before it. System turns are dropped — the internal
format has no system slot.

**Malformed entries** are skipped with a warning rather than aborting the load; if
more than 10% of a file is skipped, the log says so loudly (usually a sign the
format isn't what you think it is). A file where *nothing* is readable raises.

## Supported Model Architectures

There is no allowlist — models load through `AutoModelForCausalLM` (or Unsloth's
`FastLanguageModel`), so compatibility comes down to one thing: whether the
model's layers use the **Llama-style projection names** that the LoRA target
modules reference (see [Parameter Search Space](#parameter-search-space)):

```
q_proj, k_proj, v_proj, o_proj          # attention
gate_proj, up_proj, down_proj           # MLP
```

Any architecture using those names works. Architectures that fuse or rename them
do not.

### Fully supported

| Family | Notes |
|---|---|
| **Llama** (incl. Llama 3.x) | The reference naming; best-tested path |
| **Qwen2 / Qwen3** | Default base model is Qwen2.5 |
| **Mistral** | |
| **Gemma / Gemma 2 / Gemma 3** | |
| **SmolLM / SmolLM2** | `model_type: llama` — Llama under the hood |
| **SmolLM3** | Own architecture, standard naming |
| **OLMo / OLMo 2 / OLMo 3 / OLMo Hybrid** | Allen AI |
| **Tülu 3** | `model_type: llama` (post-trained Llama 3.1) |

Small models (SmolLM at 135M–1.7B, Qwen2.5-0.5B) suit the search especially
well — runs are cheap enough to afford a high `--max-runs` and actually explore
the space instead of sampling it thinly.

### Works, with caveats

| Family | Caveat |
|---|---|
| **OLMoE / Flex-OLMo** (MoE) | Standard names are present, but `gate_proj`/`up_proj`/`down_proj` live inside *every expert*. The all-MLP target preset therefore produces a far larger adapter than the same config on a dense model — watch VRAM headroom. The tie-break's adapter-cost heuristic (rank × module count) doesn't account for expert multiplication, so it under-estimates MoE cost. |
| **Phi** (phi-1 / phi-2) | Has `q/k/v_proj` but no `o_proj` or MLP projections. Presets still match partially, so runs succeed — but train a smaller adapter than the config implies (see below). |
| **Starcoder2** | Attention projections only; the all-MLP preset partially applies. |

### Not supported

| Family | Why |
|---|---|
| **Phi-3** | Fuses attention into `qkv_proj` and MLP into `gate_up_proj` — matches nothing, so **every run in the search fails** |
| **Falcon** | `query_key_value` |
| **GPT-NeoX** | `query_key_value` |
| **MPT** | `Wqkv` — two of the three target presets match nothing and crash, the third partially works, so a search fails intermittently |

### Two failure modes worth knowing

**A total mismatch is loud.** PEFT raises `ValueError: Target modules {...} not
found in the base model` when *none* of the targets match. On an unsupported
architecture every run fails this way, immediately.

**A partial mismatch is silent.** PEFT only errors when nothing matches, so an
architecture missing *some* of the names (Phi, Starcoder2) trains an adapter over
only the modules it found. The run succeeds and its score is real, but it isn't
testing the configuration the search believes it is — worth knowing before
comparing those results against a fully-supported model.

Supporting a fused-projection architecture like Phi-3 would mean making the
target modules architecture-aware rather than a fixed list; that's a feature, not
a configuration change.

## Web UI

```bash
python -m auto_sft.web --port 8765
```

Three pages:

- **Configuration** — model, data (browse/upload datasets, pick an HF subset), engine, eval, evaluation split, fine-tune budget, GGUF/HF-upload options. Shows the dataset's entry count, how it splits into train/holdout, and how much of it each run's budget will cover. **Start Search** launches the hyperparameter search, not the final fine-tune.
- **Parameter Search** — a per-run score chart with the min–max spread of each run's sample scores, the Best Run card, a collapsible All Runs list (collapse state is remembered), live progress, base vs. tuned samples, the active scoring criterion, the training log, and JSON/CSV export.
- **Training** — the full fine-tune: a stage banner at the top of the page naming the current stage ("Stage 3 of 8 · Training") with step/loss detail, the data split and coverage for the run, a phase stepper, live loss curve, before/after samples, saved-model paths, and storage cleanup.

Also here: start/stop controls for both search and fine-tune, and auto-launch of the full fine-tune when a search finishes above a score threshold (`auto_finetune` setting).

Copy `config/settings.example.json` to `config/settings.json` to seed your own defaults — the live `settings.json` is git-ignored since it holds your personal run configuration.

## Parameter Search Space

Kept intentionally broad (see [`params.py`](auto_sft/params.py)) — nothing is pruned based on results from any single model or dataset. During pure exploration, every value below is sampled independently and uniformly at random. The hybrid search's exploitation phase is the only place values get constrained, and even then only as a moving neighborhood around the current best (see [How it searches](#concept)).

**LoRA:**
- Rank: 4, 8, 16, 32, 64
- Alpha multiplier: 1x, 2x, 4x rank
- Dropout: 0.0–0.1
- Target modules: `[q_proj, v_proj]`, `[q_proj, v_proj, k_proj, o_proj]`, or all attention + MLP projections

**Training:**
- Learning rate: 1e-5 to 5e-4
- Batch size: 1, 2, 4
- Gradient accumulation steps: 1, 2, 4, 8
- Warmup ratio: 0.0, 0.03, 0.05, 0.1
- Max sequence length: 512, 1024, 2048

## Tips for Finding Good Parameters

Practical guidance for getting a useful search out of a limited compute/time budget:

- **Start with `--search random` and a generous `--max-runs`, then switch to `hybrid`
  once you've seen the score distribution.** Pure exploration tells you how much the
  score actually varies across the space before you commit to exploiting a region
  that might just be noise.
- **Keep runs short during search, and spend length on the final fine-tune instead.**
  `--run-time` (or `--max-steps`) controls the search; a 3–5 minute probe is usually
  enough to rank configs even though it undertrains every one of them. The point of
  the search is *relative* ranking, not an individually well-trained model.
- **Raise `--eval-samples` before you trust small score differences.** Five judge
  samples (the default) is enough to separate a good config from a bad one, but a
  0.2-point gap between two decent configs is likely noise at that sample size.
  Bump it to 10–15 if the leaderboard looks like a coin flip.
- **Watch "each entry will be seen N times" in the log.** If a run's budget doesn't
  even complete one pass over the training data, increasing rank or lowering the
  learning rate won't help it — it's data-starved, not undertuned. Either raise the
  run-time budget or shrink the dataset used for search.
- **Set `--exploit-threshold` relative to `--max-runs`, not to a fixed number.** A
  10-run session with the default threshold of 3 spends 70% of its budget
  exploiting one early guess; for short sessions, either raise the threshold or
  stay in `random` mode throughout.
- **Give early stopping room to work.** `--early-stop-runs` (default 12) and
  `--early-stop-hours` (default 1.0) both stop the session the first time
  improvement stalls. If your search space is wide or your run-time is short, low
  values can end the session before the exploitation phase has had a fair chance —
  raise one or both if runs are ending earlier than you expect.
- **Pick an `--eval-criteria` that matches what you actually want to reward.**
  `helpful` and `anti_sycophantic` push toward direct answers; `conversational` and
  `persona` reward tone and consistency; a search optimized for the wrong criterion
  will find a config that's confidently wrong for your use case. Write a
  `--custom-criterion` file if none of the built-ins fit.
- **Re-use the winning config's neighborhood, don't just take the top score
  verbatim.** `best_config.json` is one sample from the exploitation region; if two
  or three configs near the top score are close together on rank/LR, that
  neighborhood is more trustworthy than any single winner picked by a small
  judge sample.

## Architecture

```
auto_sft/
├── loop.py            # Hyperparameter search loop (LoRA experiments)
├── finetune.py         # Full fine-tune with the winning config → merge → GGUF/HF
├── params.py           # Search-space sampling (random + hybrid explore/exploit)
├── budget.py           # Step/epoch budget → how much of the data a run covers
├── eval.py             # LLM-as-judge evaluation, adaptive rubric generation
├── benchmark_eval.py   # lm-eval benchmark backend (experimental)
├── logger.py           # Run/session logging
├── routing.py          # VRAM-aware local/remote routing decisions
├── orchestrator.py     # High-level pipeline orchestration
├── dataset_builder.py  # HF dataset search, gap analysis, dataset assembly
├── data_gen.py         # Dynamic supplemental data generation between cycles
├── web.py              # FastAPI web server + REST API
├── data/
│   └── loader.py       # Data loading, normalization, subsets, train/eval split
└── templates/
    ├── dashboard.html   # Search monitoring UI
    ├── finetune.html    # Full fine-tune UI
    ├── settings.html    # Settings UI
    └── _coverage.html   # Shared coverage-estimate JS, included by settings/finetune
```

## Tests

```bash
python test_data_split.py     # train/eval split, subsets, budget coverage
python test_resilience.py     # judge credentials/retries, run logging, sessions
python test_criterion_guard.py # scoring-criterion mismatch guard
```

No GPU or network needed for those three — judge calls are faked. `test_pipeline.py`
is a separate end-to-end script that drives a live server and does need both.

## License

MIT

auto-sft

auto-sft Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

auto-sft

auto-sft Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README