mola

KO | EN

gitlite — search

#hacktoberfest #typescript #android #deep-learning #macos #java #nodejs #windows #awesome #kubernetes #vue #security #mcp #open-source

(★ 20)

Multi-LoRA inference server for Apple Silicon -- one base model, many adapters, zero reload

mola Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">MOLA</h1>
 Multi-adapter Orchestration LoRA Apple
 Multi-LoRA inference server for Apple Silicon: one base model, many adapters, no reload.

---

MOLA serves multiple LoRA adapters from one MLX base model on Apple Silicon. The base model stays resident in memory, adapters are selected per request, and same-adapter traffic is batched automatically.

> **Status:** Alpha. The published benchmark below uses `mlx-community/Qwen3.5-9B-MLX-4bit` with 8 resident adapters on `mlx-lm 0.31.1`.

| Approach | Runtime shape |
|---|---|
| Separate fine-tuned models | one full model per specialty, higher memory, reloads when switching |
| MOLA | one base model plus many LoRA adapters, lower memory, no model reloads |

This is the practical tradeoff MOLA is built for: keep one base model resident, switch adapters per request, and avoid reloading full fine-tuned checkpoints.

## Features

- OpenAI-compatible chat completions API
- Per-request adapter selection via the `model` field
- Multiple LoRA adapters loaded at once on one base model
- Same-adapter batching and stable mixed-adapter serving
- Hot-load and hot-unload adapters at runtime
- Streaming responses
- Runtime metrics via `/v1/engine/metrics`

## Quickstart

### Install

```bash
git clone https://github.com/bstnxbt/mola.git
cd mola
python3 -m venv .venv
./.venv/bin/python -m ensurepip --upgrade
./.venv/bin/python -m pip install -e ".[dev]"
./.venv/bin/python devtools/apply_mlx_lm_detached_batch_api.py
```

### Recommended Setup

Start the recommended runtime profile:

```bash
./.venv/bin/python devtools/run_mola_current_architecture.py start \
  --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --adapter rust ./path/to/rust-adapter \
  --adapter sql ./path/to/sql-adapter \
  --port 8000
```

Equivalent explicit command:

```bash
./.venv/bin/python -m mola.cli -v serve \
  --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --adapter rust ./path/to/rust-adapter \
  --adapter sql ./path/to/sql-adapter \
  --adapter support ./path/to/support-adapter \
  --max-inflight-tokens 131072 \
  --max-batch-size 128 \
  --prefill-batch-size 32 \
  --enable-routed-decode-reference \
  --strict-routed-decode-reference \
  --routed-decode-backend gather-mm \
  --enable-mixed-decode-migration \
  --prestep-mixed-decode-migration \
  --cache-routed-decode-sessions \
  --detached-shared-decode-owner \
  --port 8000
```

Stop or restart:

```bash
./.venv/bin/python devtools/run_mola_current_architecture.py stop --port 8000
./.venv/bin/python devtools/run_mola_current_architecture.py restart --port 8000
```

### Query

The `model` field is a strict selector. Unknown adapter names return `404`.

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rust",
    "messages": [{"role": "user", "content": "Implement a lock-free queue with crossbeam"}],
    "stream": true
  }'
```

### Manage Adapters At Runtime

```bash
curl http://localhost:8000/v1/adapters

curl -X POST http://localhost:8000/v1/adapters \
  -H "Content-Type: application/json" \
  -d '{"name": "medical", "path": "./path/to/medical-adapter"}'

curl -X DELETE http://localhost:8000/v1/adapters/medical
```

## API Summary

- `POST /v1/chat/completions`: OpenAI-compatible chat completions
- `GET /v1/models`: base model and loaded adapters
- `GET /v1/adapters`: loaded adapter metadata
- `POST /v1/adapters`: hot-load an adapter
- `DELETE /v1/adapters/{name}`: unload an adapter
- `GET /v1/engine/metrics`: runtime counters
- `GET /health`: health and model summary

## Published Benchmark

Reproduce the published benchmark with:

```bash
./.venv/bin/python scripts/bench_server.py \
  --routed-validation \
  --concurrency 1,16,64 \
  --repeats 3 \
  --json-out /tmp/mola-qwen35-9b-current-architecture-bench-1-16-64.json
```

Published profile:
- model: `mlx-community/Qwen3.5-9B-MLX-4bit`
- adapters: `rust`, `sql`, `medical`, `cyber`, `solidity`, `devops`, `math`, `legal`
- backend: `gather-mm`
- batch sizes: `128 / 32`
- machine: `Apple M5 Max 64GB`

This benchmark shows how much performance MOLA keeps when traffic moves from one adapter at a time to a real mixed multi-adapter workload.

| Concurrency | Same tok/s | Mixed tok/s | Multi-LoRA overhead | Mixed p95 |
|---|---:|---:|---:|---:|
| 1 | 76.4 | 76.4 | 0% | 843 ms |
| 16 | 308.8 | 241.4 | -22% | 4220 ms |
| 64 | 732.3 | 555.5 | -24% | 7372 ms |

At concurrency 1, same and mixed are effectively the same shape; the useful signal starts once requests overlap.

At moderate to high load, mixed multi-adapter traffic adds about 22-24% throughput overhead relative to same-adapter traffic.

From the same run:

- `long-decode-mixed` tok/s: `81.1` at `1`, `283.4` at `16`, `691.1` at `64`
- `hot/cold skew mix` tok/s: `77.4` at `1`, `241.3` at `16`, `559.1` at `64`

If you benchmark MOLA on another Apple Silicon machine or model, feel free to open an issue with your hardware, model, and results.

## Adapter Format

MOLA loads standard PEFT / mlx-lm adapter directories:

```text
my-adapter/
├── adapter_config.json
└── adapters.safetensors
```

Train adapters with [mlx-lm](https://github.com/ml-explore/mlx-lm), [mlx-tune](https://github.com/ARahim3/mlx-tune), or any tool that outputs PEFT-compatible safetensors.

## Requirements

- Apple Silicon Mac (M1 or later)
- Python 3.11+
- macOS 13+
- `mlx-lm 0.31.1` recommended

## Limitations

- Alpha release
- Apple Silicon only
- A local `mlx-lm` patch is still required for the recommended setup
- Switching adapters inside one conversation invalidates KV cache reuse
- Mixed prefill and deeper KV/adaptor residency management are still open problems

## License

Apache 2.0

mola

mola Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

mola

mola Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README