KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #claude-code #codex #cli #agent-skills #ai #dsh-plugin #mcp #windows #agent

ezlocalai

★ 94 Open GitHub ↗

ezlocalai is an easy to set up local artificial intelligence server with OpenAI Style Endpoints.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ezlocalai

[![GitHub](https://img.shields.io/badge/GitHub-ezLocalai-blue?logo=github&style=plastic)](https://github.com/DevXT-LLC/ezlocalai) [![Dockerhub](https://img.shields.io/badge/Docker-ezlocalai-blue?logo=docker&style=plastic)](https://hub.docker.com/r/joshxt/ezlocalai)

ezlocalai is an easy set up artificial intelligence server that allows you to easily run multimodal artificial intelligence from your computer. It is designed to be as easy as possible to get started with running local models. It automatically handles downloading the model of your choice and configuring the server based on your CPU, RAM, and GPU specifications. It also includes [OpenAI Style](https://pypi.org/project/openai/) endpoints for easy integration with other applications using ezlocalai as an OpenAI API proxy with any model. Additional functionality is built in for voice cloning text to speech and a voice to text for easy voice communication as well as image generation and video generation entirely offline after the initial setup.

## Prerequisites

- [Python 3.10+](https://www.python.org/downloads/)
- [Docker Desktop](https://docs.docker.com/docker-for-windows/install/) (Windows or Mac)
- [CUDA Toolkit (May Need 12.4)](https://developer.nvidia.com/cuda-12-4-0-download-archive) (NVIDIA GPU only)
- [ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/index.html) (AMD GPU only - Linux)

<details>
  <summary>Additional Linux Prerequisites</summary>

- [Docker](https://docs.docker.com/get-docker/)
- [Docker Compose](https://docs.docker.com/compose/install/)
- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) (NVIDIA GPU only)
- [ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/index.html) (AMD GPU only - Radeon RX 6000/7000/9000 series, Radeon PRO, and Ryzen APUs)

</details>

## Quick Start (Recommended)

Install the CLI and start ezlocalai with a single command:

```bash
pip install ezlocalai
ezlocalai start
```

It will take several minutes to download the models on the first run. Once running, access the API at <http://localhost:8091>.

### CLI Commands

```bash
# Start with defaults (auto-detects GPU, uses Qwen3-VL-4B)
ezlocalai start

# Start with a specific model
ezlocalai start --model unsloth/gemma-3-4b-it-GGUF

# Start with custom options
ezlocalai start --model unsloth/Qwen3-VL-4B-Instruct-GGUF \
                --uri http://localhost:8091 \
                --api-key my-secret-key \
                --ngrok <your-ngrok-token>

# Other commands
ezlocalai stop      # Stop the container
ezlocalai restart   # Restart the container
ezlocalai status    # Check if running and show configuration
ezlocalai logs      # Show container logs (use -f to follow)
ezlocalai update    # Pull/rebuild latest images

# Send prompts directly from the CLI
ezlocalai prompt "Hello, world!"
ezlocalai prompt "What's in this image?" -image ./photo.jpg
ezlocalai prompt "Explain quantum computing" -m unsloth/Qwen3-VL-4B-Instruct-GGUF -temp 0.7
```

### CLI Options

| Option | Default | Description |
|--------|---------|-------------|
| `--model`, `-m` | `unsloth/Qwen3-VL-4B-Instruct-GGUF` | HuggingFace GGUF model(s), comma-separated |
| `--uri` | `http://localhost:8091` | Server URL |
| `--api-key` | None | API key for authentication |
| `--ngrok` | None | ngrok token for public URL |

### Prompt Command Options

| Option | Default | Description |
|--------|---------|-------------|
| `-m`, `--model` | Auto-detected | Model to use for the prompt |
| `-temp`, `--temperature` | Model default | Temperature for response generation (0.0-2.0) |
| `-tp`, `--top-p` | Model default | Top-p (nucleus) sampling parameter (0.0-1.0) |
| `-image`, `--image` | None | Path to local image file or URL to include with prompt |
| `-stats`, `--stats` | Off | Show statistics (tokens, speed, timing) after response |

For additional options (Whisper, image model, etc.), edit `~/.ezlocalai/.env`:

### Data Persistence

All data is stored in `~/.ezlocalai/`:

| Directory | Contents |
|-----------|----------|
| `~/.ezlocalai/data/models/` | Downloaded GGUF model files |
| `~/.ezlocalai/data/hf/` | HuggingFace cache |
| `~/.ezlocalai/data/voices/` | Voice cloning samples |
| `~/.ezlocalai/data/outputs/` | Generated images/audio |
| `~/.ezlocalai/.env` | Your configuration |

Models persist across container updates - you won't re-download them when updating the CLI or rebuilding the CUDA image.

## Benchmarks

Performance tested on Intel i9-12900KS + RTX 4090 (24GB):

| Model | Size | Speed | Notes |
|-------|------|-------|-------|
| **Qwen3-VL-4B** | 4B | ~210 tok/s | Vision-capable, great for chat |
| **Qwen3-Coder-30B** | 30B (MoE) | ~65 tok/s | Coding model, hot-swappable |

Both models pre-calibrate at startup and hot-swap in ~1 second.

## Distributed Fallback / Multi-Machine Setup

ezlocalai supports a distributed fallback system where multiple instances can fall back to each other when local resources (VRAM/RAM) are exhausted, or fall back to any OpenAI-compatible API. This enables:

- **Load balancing**: When one machine is busy, requests automatically route to another
- **Redundancy**: If one server is overloaded, the fallback handles requests
- **Resource optimization**: Each machine handles what it can, forwarding the rest
- **Hybrid deployment**: Mix local ezlocalai instances with cloud APIs

### Configuration

Set these environment variables in your `.env` file or pass them to the container:

```bash
# Fallback server URL - can be another ezlocalai instance OR any OpenAI-compatible API
FALLBACK_SERVER=http://192.168.1.100:8091  # Another ezlocalai instance
# Or use a cloud provider:
# FALLBACK_SERVER=https://api.openai.com/v1

# Authentication for the fallback server
FALLBACK_API_KEY=your-api-key

# Optional: Override model for OpenAI-compatible fallback (pass-through by default)
# If not set, the originally requested model is passed through to the fallback server
# FALLBACK_MODEL=gpt-4o-mini

# Combined memory threshold (VRAM + RAM) in GB - fallback triggers when below this
# Models can offload to system RAM, so combined memory is more accurate than VRAM alone
FALLBACK_MEMORY_THRESHOLD=8.0
```

The system automatically detects whether `FALLBACK_SERVER` points to another ezlocalai instance or an OpenAI-compatible API by checking for the `/v1/resources` endpoint. If it's another ezlocalai server, full endpoint forwarding is used (preserving the original request). Otherwise, it falls back to standard OpenAI API calls, passing through the originally requested model (or using `FALLBACK_MODEL` if set as an override).

### Example: Two-Machine Setup

**Machine A** (Primary with RTX 4090):
```bash
EZLOCALAI_URL=http://0.0.0.0:8091
EZLOCALAI_API_KEY=shared-key
FALLBACK_SERVER=http://machine-b:8091
FALLBACK_API_KEY=shared-key
```

**Machine B** (Fallback with RTX 3080):
```bash
EZLOCALAI_URL=http://0.0.0.0:8091
EZLOCALAI_API_KEY=shared-key
FALLBACK_SERVER=http://machine-a:8091
FALLBACK_API_KEY=shared-key
```

Both machines fall back to each other - creating a resilient two-node cluster.

### Example: Local + Cloud Hybrid

Run a local ezlocalai with OpenAI as the fallback:

```bash
EZLOCALAI_URL=http://0.0.0.0:8091
FALLBACK_SERVER=https://api.openai.com/v1
FALLBACK_API_KEY=sk-your-openai-key
FALLBACK_MODEL=gpt-4o-mini
```

### Monitoring Fallback Status

Check the fallback status via API:
```bash
# Get resource status including fallback info
curl http://localhost:8091/v1/resources

# Check fallback availability and models
curl http://localhost:8091/v1/fallback/status
```

### What Gets Forwarded

When fallback is triggered to another ezlocalai instance, these endpoints are automatically forwarded:
- `/v1/chat/completions` - Chat completions (including streaming)
- `/v1/completions` - Text completions
- `/v1/embeddings` - Text embeddings
- `/v1/audio/transcriptions` - Speech-to-text
- `/v1/audio/speech` - Text-to-speech
- `/v1/audio/music` - Music generation
- `/v1/images/generations` - Image generation
- `/v1/images/edits` - Image editing (image + text to image)
- `/v1/videos/generations` - Video generation

For OpenAI-compatible APIs, only chat completions and embeddings are forwarded.

## Embeddings

ezlocalai serves `/v1/embeddings` with a dedicated GGUF embedding model, independent
of `TEXT_SERVER`. By default it uses Qwen3-Embedding-0.6B Q8_0 with a 32k context:

```bash
EMBEDDING_ENABLED=true
EMBEDDING_MODEL=Qwen/Qwen3-Embedding-0.6B-GGUF
EMBEDDING_MODEL_ALIAS=Qwen3-Embedding-0.6B
EMBEDDING_QUANT_TYPE=Q8_0
EMBEDDING_CONTEXT_LENGTH=32768
EMBEDDING_N_PARALLEL=1
EMBEDDING_GPU_LAYERS=auto
EMBEDDING_KV_CACHE_TYPE=f16
```

When using the router, workers advertise the `embedding` capability only when
`EMBEDDING_ENABLED=true`, so embedding requests route to workers that can serve
them and the dashboard shows the active embedding model. `EMBEDDING_N_PARALLEL`
controls how many full-context embedding model instances are loaded and reported
to the router; each instance keeps the full `EMBEDDING_CONTEXT_LENGTH` per request
instead of splitting the context across internal xllamacpp slots. With
`EMBEDDING_GPU_LAYERS=auto`, ezlocalai estimates the 32k embedding cache footprint
for each instance and partially offloads layers to CPU when VRAM is tight.

## Image And Video

Image and video workers are opt-in so setting `IMG_MODEL` or `VIDEO_MODEL` alone
does not warm-load or advertise those capabilities:

```bash
IMAGE_ENABLED=false
IMG_MODEL=
VIDEO_ENABLED=false
VIDEO_MODEL=unsloth/LTX-2.3-GGUF
```

Set `IMAGE_ENABLED=true` with `IMG_MODEL` to serve local image generation, or
`VIDEO_ENABLED=true` to serve local video generation. When `VIDEO_MODEL` is
omitted or blank, ezlocalai defaults to `unsloth/LTX-2.3-GGUF`. Enabled media
models report `image` or `video` capacity to the router. They warm-load and stay
resident when enough GPU headroom is available; on single-GPU LLM workers, video
can lazy-load after the LLM handoff so LTX initializes with freed VRAM. Workers
with an `IMAGE_SERVER` URL configured still delegate media requests instead of
loading local models.

On a single-GPU worker, leave `VIDEO_UNLOAD_LLM_DURING_GENERATION=auto`. When a
resident LLM is occupying the only GPU, ezlocalai marks the text/vision slots
temporarily unavailable, unloads the LLM plus idle aux GPU models, initializes
LTX-2.3 with the freed VRAM, runs generation, unloads video if needed, and
reloads persistent LLMs when `VIDEO_RELOAD_LLM_AFTER_GENERATION=true`.
`VIDEO_GPU_RESIDENCY=auto` chooses full GPU residency only on very large GPUs,
model CPU offload when enough VRAM was freed for the requested clip size, and
sequential CPU offload as the constrained fallback for long scenes. Short clips
can use model offload when at least `VIDEO_SHORT_MODEL_OFFLOAD_MIN_FREE_GB`
remains after LTX loads; longer scenes stay sequential unless
`VIDEO_MODEL_OFFLOAD_MIN_FREE_GB`/`VIDEO_FULL_GPU_MIN_FREE_GB` say the GPU has
room. If a more aggressive mode OOMs,
`VIDEO_RETRY_SEQUENTIAL_ON_OOM=true` reloads LTX with sequential offload and
retries once. Set `VIDEO_GPU_RESIDENCY=full` to force a full-GPU attempt.

LTX-2.3 requires dimensions divisible by 32 and frame counts in the `8n+1`
pattern. The music-video helper handles frame planning automatically.

## Music Generation

ezlocalai serves `/v1/audio/music` and `/v1/audio/music/generations` by
starting an internal ACE-Step 1.5 `acestep.cpp` server inside the same Docker
container. The GGUF model files are downloaded from
`Serveurperso/ACE-Step-1.5-GGUF` into `models/ace-step`; `acestep.cpp` expects
one LM GGUF, one Qwen3 embedding/text encoder GGUF, one DiT GGUF, and
`vae-BF16.gguf` in that directory.

For the normal container-local setup, enable music on a worker with:

```bash
MUSIC_ENABLED=true
```

With no other music environment variables set, ezlocalai downloads and serves
these ACE-Step 1.5 GGUF files:

```bash
MUSIC_MODEL=Serveurperso/ACE-Step-1.5-GGUF
ACE_STEP_SERVER_URL=
ACE_STEP_AUTO_START=true
ACE_STEP_MODELS_DIR=models/ace-step
ACE_STEP_LM_MODEL=acestep-5Hz-lm-4B-Q8_0.gguf
ACE_STEP_TEXT_ENCODER_MODEL=Qwen3-Embedding-0.6B-Q8_0.gguf
ACE_STEP_DIT_MODEL=acestep-v15-turbo-Q4_K_M.gguf
ACE_STEP_VAE_MODEL=vae-BF16.gguf
ACE_STEP_TIMEOUT=1800
```

The default LM is the 4B Q8 model for better music planning and lyric
structure. Smaller nodes can override `ACE_STEP_LM_MODEL` to
`acestep-5Hz-lm-0.6B-Q8_0.gguf` to save disk, RAM, and VRAM at the cost of
quality.

Set `ACE_STEP_SERVER_URL` only if you intentionally run an external
`acestep.cpp` process. When it is empty, ezlocalai starts
`/opt/acestep.cpp/build/ace-server` on `127.0.0.1:8085` during startup.

On smaller single-GPU nodes, leave
`MUSIC_UNLOAD_LLM_DURING_GENERATION=auto`. For container-local ACE-Step this
marks the worker's text/vision slots unavailable, unloads resident LLMs before
generation, runs the music job, then reloads persistent LLMs when
`MUSIC_RELOAD_LLM_AFTER_GENERATION=true`.

Example request:

```bash
curl http://localhost:8091/v1/audio/music \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Heavy metal anthem about the Pythagorean theorem, double-kick drums, distorted guitars, soaring vocals, and a triumphant chorus.",
    "lyrics": "[Verse]\nOn a right triangle battlefield\nTwo short sides raise their shields\n\n[Chorus]\nA squared plus B squared, lightning in the night\nEquals C squared, hypotenuse burning bright",
    "duration": 45,
    "keyscale": "E minor"
  }'
```

Music requests require `prompt`, `lyrics`, `duration`, and `keyscale`. The
OpenAI-style `model` field is optional; if it is omitted or names an alias this
worker does not advertise, ezlocalai uses the configured available music model
(`MUSIC_MODEL`, default `Serveurperso/ACE-Step-1.5-GGUF`) instead of rejecting
the request. Generation controls such as `seed`, `bpm`, `timesignature`,
`vocal_language`, `response_format`, `output_format`, `inference_steps`,
`guidance_scale`, `shift`, `solver`, `lm_model`, and `synth_model` are optional
per-request overrides. Defaults are `response_format=url`, `output_format=wav16`,
`bpm=128`, `timesignature=4/4`, `vocal_language=en`, `inference_steps=16`,
`guidance_scale=1.0`, `shift=3.0`, and `solver=euler`.

For a live proof test against the internal or external ACE-Step server:

```bash
ACE_STEP_LIVE_TEST=true python -m unittest test_music_generation.LiveAceStepMusicProofTest
```

## Music Video Generation

Workers that have both `MUSIC_ENABLED=true` and `VIDEO_ENABLED=true` advertise a
combined `music_video` capability to the router. The router dashboard lists the
combined music and video models next to that worker, and requests are routed to a
worker that can perform both parts locally.

```bash
MUSIC_ENABLED=true
VIDEO_ENABLED=true
```

The music-video endpoint first generates the song with ACE-Step, then generates
one or more LTX-2.3 video scenes in a single video session and muxes the
generated song audio into the final MP4:

```bash
curl http://localhost:8091/v1/videos/music \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Heavy metal song about the Pythagorean theorem",
    "lyrics": "[Verse]\nA squared roars, B squared screams\n[Chorus]\nC squared lights the hypotenuse",
    "duration": 30,
    "keyscale": "E minor"
  }'
```

The required fields are `prompt`, `lyrics`, `duration`, and `keyscale`.
Optional fields include `model`, `music_model`, `video_model`, `video_prompt`,
`scene_prompts`, `scene_duration`, `size`, `frame_rate`, `num_inference_steps`,
`guidance_scale`, `video_guidance_scale`, `response_format`, `seed`,
`music_seed`, `audio_url`, `include_scene_audio`, `storyboard`,
`scene_images`, `bpm`, `timesignature`, `vocal_language`, `inference_steps`,
`music_inference_steps`,
`music_guidance_scale`, `shift`, `solver`, `lm_model`, `synth_model`,
`audio_codes`, `image`, and `conditions`.
When `audio_url` points at an existing ezlocalai `outputs/` audio file, the
endpoint skips ACE-Step audio generation and only renders/muxes the video.
Intermediate scene clips are video-only by default; set
`include_scene_audio=true` only when debugging LTX's own generated audio.
The final mux trims or pads the audio stream to the requested `duration`.

Music-video scenes use `storyboard=true` by default. When no explicit image is
provided, ezlocalai draws deterministic first-frame storyboards from the prompt
and lyrics, then uses LTX image-to-video so each scene has visible lyric/thematic
anchors instead of drifting into generic concert footage. For maximum control,
pass `scene_images` as base64/data-URL/HTTP images, one per planned scene.
Pythagorean-theorem storyboards also burn in an exact `a squared + b squared =
c squared` equation overlay using superscript-style exponents, since generated
video models are unreliable at preserving formula text frame to frame.

LTX-2.3 audio-to-video is documented for roughly 20 seconds per request, with
longer videos created by chaining clips. ezlocalai therefore defaults
`MUSIC_VIDEO_SCENE_DURATION=5` and caps each planned scene with
`MUSIC_VIDEO_MAX_SCENE_DURATION=20`; the final response includes the generated
song URL, each scene URL, the scene plan, and the final MP4 URL.

## Router / Load Balancer Mode

For setups that outgrow point-to-point fallback (3+ machines, friends contributing GPUs, bittensor miners coming and going), ezlocalai can run as a dedicated **router**. The router itself loads no models — it accepts the normal OpenAI-compatible API and forwards each request to the best registered worker.

### How it works

```
┌────────────┐  OpenAI API   ┌────────────────────┐   proxied request   ┌─────────────────┐
│  Client    │ ────────────▶ │  ezlocalai-router  │ ──────────────────▶ │  Worker (5090)  │
└────────────┘               │   (no models)      │                     ├─────────────────┤
                             │                    │ ◀── heartbeat ───── │  Worker (4090)  │
                             │  selects worker    │ ◀── heartbeat ───── │  Worker (voice) │
                             │  based on free     │ ◀── heartbeat ───── │  Worker (image) │
                             │  VRAM + queue +    │ ◀── heartbeat ───── │  Friend's 3090  │
                             │  capability +      │                     └─────────────────┘
                             │  model availability│
                             └────────────────────┘
```

Each worker is a normal `ezlocalai` instance with `ROUTER_URL` set. On startup the worker registers itself, then sends heartbeats every `WORKER_HEARTBEAT_INTERVAL` seconds containing free VRAM, queue depth, loaded models, and advertised capabilities (`text`, `vision`, `tts`, `stt`, `embedding`, `image`, `video`, `music`, `music_video`). Workers that miss `ROUTER_WORKER_TTL` seconds of heartbeats are pruned. If a request arrives and no suitable worker is free, the router keeps it queued until a worker becomes available when `ROUTER_WAIT_TIMEOUT=0`, or waits up to the configured positive timeout before returning `503`.

### Run the router

```bash
docker compose -f docker-compose-router.yml up -d
```

The router listens on port `8092` by default and exposes the same OpenAI-compatible endpoints as a normal ezlocalai server, plus:

- `GET  /dashboard`         — live HTML dashboard (auto-refresh, no auth)
- `GET  /v1/router/dashboard` — same data as JSON (requires client key)
- `GET  /v1/router/workers` — list registered workers and their live state
- `GET  /v1/router/health`  — router health + live worker count
- `POST /v1/router/register` / `heartbeat` / `deregister` — worker protocol

### Point a worker at the router

For most setups, **`ROUTER_URL` is the only env var you need to add** to a worker. The worker self-reports its capabilities, GPUs (model names + VRAM), loaded models, and per-model context windows. The router uses the worker's existing `EZLOCALAI_URL` as the callback address, falling back to the connection source IP when that isn't reachable from the router.

Minimum config:

```bash
ROUTER_URL=http://router-host:8092
```

Optional overrides:

```bash
ROUTER_API_KEY=shared-key             # match the router's EZLOCALAI_API_KEY (or ROUTER_REGISTER_KEY)
WORKER_LABEL=main-5090                # friendly name (defaults to hostname)
WORKER_HEARTBEAT_INTERVAL=10          # seconds between heartbeats
```

> **Behind NAT or a tunnel?** Set `EZLOCALAI_URL` to the public callback address (this is the same env var the worker already uses to advertise itself). Otherwise the router uses the source IP it sees on the registration, which is what you want for LAN workers.

### Example: your current setup

Every worker just needs `ROUTER_URL` (and optionally `WORKER_LABEL`):

| Machine               | Address              | Role                          | Env additions                                                                |
|-----------------------|----------------------|-------------------------------|------------------------------------------------------------------------------|
| Main GPU (5090+3090Ti)| `192.168.1.135:8091` | text/vision (Qwen3.6-35B-A3B) | `ROUTER_URL=http://192.168.1.50:8092` `WORKER_LABEL=main-5090`               |
| Fallback GPU (4090)   | `192.168.1.243:8091` | text/vision (Qwen3.6-35B-A3B) | `ROUTER_URL=...` `WORKER_LABEL=fallback-4090`                                |
| Voice server          | `192.168.1.82:8091`  | TTS/STT/wake word             | `ROUTER_URL=...` `WORKER_LABEL=voice`                                        |
| Small + image         | `192.168.1.214:8091` | small text + image gen        | `ROUTER_URL=...` `WORKER_LABEL=img-small`                                    |
| Friend's 3090         | external             | text/vision                   | `ROUTER_URL=https://router.you.com:8092` (set `EZLOCALAI_URL=https://gpu.friend.com:8091` so the router can reach back) |
| Friend's 3090 (CGNAT) | no public IP         | text/vision                   | `ROUTER_URL=https://router.you.com:8092` `WORKER_TUNNEL=true` (worker dials out, no port forwarding needed) |

Clients then point to the router as if it were a single ezlocalai server:

```bash
curl https://router.you.com:8092/v1/chat/completions \
  -H "Authorization: Bearer shared-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"unsloth/Qwen3.6-35B-A3B-GGUF","messages":[{"role":"user","content":"Hi"}]}'
```

The router picks the highest-scoring idle compatible worker at request time. If the fastest worker is already handling a request, the router spills over immediately to the best idle worker that can serve the request. If no compatible text/vision worker is free, it queues the request until a worker becomes available when `ROUTER_WAIT_TIMEOUT=0`, or waits up to the configured positive timeout before returning `503`.

### Reverse tunnel (workers without a public IP)

Some workers (CGNAT, friend's home machines, anything you can't port-forward) have no inbound network path the router can dial. Set `WORKER_TUNNEL=true` on the worker and that's it — the worker dials *out* over WebSocket to `wss://<router>/v1/router/tunnel` and the router multiplexes inference requests back through the same connection. No public IP, no port forward, no extra container.

```bash
# On the worker (only env vars required):
ROUTER_URL=https://router.you.com:8092
WORKER_TUNNEL=true
WORKER_LABEL=friend-3090            # optional
ROUTER_API_KEY=shared-key           # optional, must match router's EZLOCALAI_API_KEY
```

The worker registers with a sentinel URL `tunnel://<worker_id>`; you'll see it in `/v1/router/workers` and the dashboard like any other worker. Reconnects with exponential backoff (2s → 60s) are automatic. Streaming endpoints (chat SSE, audio) flow through unchanged. The only requirement is the worker can make outbound HTTPS to the router — same direction as the existing heartbeat.

### Selection algorithm

Workers are scored each request as:

```
priority_tier = best_tier - 5 if tunneled else best_tier
score = priority_tier * 10  +  slots_left * 5  +  free_vram_gb  -  in_flight * 4
```

`best_tier` is derived from the worker's fastest GPU model (e.g. RTX 5090 ≈ 90, RTX 4090 = 80, RTX 3090 = 50, CPU = 2) and dominates the score, so an idle 5090 beats an idle 3090. Tunneled workers keep their reported `best_tier` but receive a 5-point priority-tier penalty so similarly capable direct workers are preferred. The load penalty (`in_flight * 4`) and idle-worker requirement keep burst traffic from stacking on a busy top-tier GPU while another compatible worker is free.

By default, text/vision routing requires an idle worker, so one long-running request on the 5090 sends the next compatible request to an idle 4090/3090 instead of stacking it onto the 5090. `ROUTER_CROSS_MODEL_GRACE=0` means the router does not wait for a busy same-model worker before using the best available compatible fallback. `ROUTER_IDLE_TIER_WINDOW=0` means the router does not hold back lower-tier idle workers while a higher-tier worker is busy; set a positive value to restrict idle spillover to workers within that many tier points of the fastest compatible tier. Set `ROUTER_BUSY_SLOT_FALLBACK=true` to restore slot-based routing when every compatible text/vision worker is already busy.

For capability-only voice routing, the router defaults `ROUTER_PREFER_DEDICATED_CAPABILITIES=stt`, so large STT transcription jobs prefer workers that are not also serving `text` or `vision`. TTS routes by the normal score/tier calculation by default so low-latency playback can use faster mixed-capability workers. Stale `ROUTER_PREFER_DEDICATED_CAPABILITIES=stt,tts` values are treated as STT-only for TTS unless `ROUTER_ALLOW_DEDICATED_TTS_PREFERENCE=true` is also set. Large transcription jobs also use `ROUTER_STT_TIMEOUT` (default `7200` seconds) instead of the generic `REQUEST_TIMEOUT`.

Workers missing the required capability (`text` / `vision` / `tts` / `stt` / `embedding` / `image` / `video` / `music`) or the requested model are filtered out before scoring. Stale workers (no heartbeat for `ROUTER_WORKER_TTL` seconds) are also excluded.

You can inspect the live registry, including each worker's reported GPUs, tier, free VRAM, queue depth, and per-model context windows:

```bash
curl https://router.you.com:8092/v1/router/workers \
  -H "Authorization: Bearer shared-key"
```

### Open pool vs. authenticated pool

Auth is intentionally simple: there is one shared secret for inference clients (`EZLOCALAI_API_KEY`) and one for workers (`ROUTER_REGISTER_KEY`, which falls back to `EZLOCALAI_API_KEY` if unset).

- **Both empty** → the router runs as an **open pool**: any client can submit requests, any worker can register. This is fine for a closed LAN; it is **not** safe to expose publicly. The router logs a `[Router] OPEN POOL: ...` warning at startup so you don't deploy this by accident.
- **`EZLOCALAI_API_KEY` set, `ROUTER_REGISTER_KEY` unset** → both clients and workers must present that one key.
- **Both set, distinct values** → clients use `EZLOCALAI_API_KEY`, workers use `ROUTER_REGISTER_KEY`. Use this when exposing the router publicly so you can rotate the worker secret independently of the client secret.

## Dedicated Voice Server

ezlocalai supports offloading TTS (text-to-speech) and STT (speech-to-text) processing to a dedicated voice server. This is useful when you want to:

- **Separate workloads**: Run voice models on a dedicated GPU while the main server handles LLMs
- **Optimize resources**: Keep voice models always loaded on a server with spare VRAM
- **Reduce latency**: Avoid lazy loading delays for voice requests

### Configuration

Set the `VOICE_SERVER` environment variable:

```bash
# Option 1: Point to another ezlocalai server for voice processing
VOICE_SERVER=http://192.168.1.100:8091
VOICE_SERVER_API_KEY=your-api-key  # Optional, uses EZLOCALAI_API_KEY if not set

# Option 2: Make THIS server the voice server (keeps TTS/STT loaded)
VOICE_SERVER=true

# Optional: load/report multiple local voice model instances
TTS_N_PARALLEL=1
STT_N_PARALLEL=1
```

### Voice Server Mode (`VOICE_SERVER=true`)

When set to `true`, this server becomes a dedicated voice server:
- TTS (Qwen-TTS) and STT (Whisper) models are **pre-loaded at startup** and stay resident
- Voice models are **never unloaded** after requests (no lazy load/unload cycle)
- LLM models are still lazy-loaded as needed
- Ideal for a secondary server with a dedicated GPU for voice processing

`TTS_N_PARALLEL` and `STT_N_PARALLEL` control how many separate local voice model
instances are available. In voice server mode, or when `LAZY_LOAD_VOICE=false`,
ezlocalai warm-loads that many resident instances and reports that capacity to
the router. With lazy voice loading, the same values cap how many transient local
TTS/STT instances may run at once.

Qwen-TTS voices live in the `voices/` directory as `.wav` reference samples. If a
same-name `.txt` transcript exists, ezlocalai uses it for transcript-conditioned
voice cloning; custom voices without a transcript automatically fall back to
speaker-embedding-only cloning.

Streaming Qwen-TTS sends one generated PCM block per text chunk. The default
chunking keeps the first chunk short for fast startup
(`QWEN_TTS_STREAM_FIRST_CHUNK_CHARS=120`) and uses longer follow-up chunks
(`QWEN_TTS_STREAM_CHUNK_CHARS=280`, capped by `QWEN_TTS_MAX_CHUNK_CHARS`) so
playback has more audio buffered while Qwen generates the next block. PCM frames
are flushed to clients in `QWEN_TTS_STREAM_WRITE_BYTES=16384` byte writes with a
short `QWEN_TTS_STREAM_FRAME_DRAIN_SECONDS=0.1` handoff and an
`QWEN_TTS_STREAM_FLUSH_SILENCE_MS=80` valid-silence frame so the next Qwen
generation does not hold the previous frame tail in server buffers.

### Voice Passthrough Mode (`VOICE_SERVER=<url>`)

When set to a URL, voice requests are forwarded to that server:
- TTS and STT requests first try the voice server
- If the voice server fails or is unavailable, falls back to local processing
- LLM models run locally as usual
- No voice models are loaded locally unless the voice server is unavailable

### Example: Two-Machine Voice Offload Setup

**Machine A** (Main LLM server with RTX 4090):
```bash
DEFAULT_MODEL=unsloth/Qwen3-Coder-30B-GGUF
VOICE_SERVER=http://machine-b:8091
VOICE_SERVER_API_KEY=shared-key
```

**Machine B** (Voice server with RTX 3090):
```bash
DEFAULT_MODEL=unsloth/Qwen3-4B-Instruct-GGUF  # Smaller LLM for basic tasks
VOICE_SERVER=true  # Keep voice models loaded
```

Machine A handles LLM inference while Machine B handles all voice processing with models always ready.

## Wake Word Detection and Training

ezlocalai includes a complete wake word training and inference system that enables custom voice activation for AI assistants. When `VOICE_SERVER=true`, the wake word endpoints are enabled.

### Custom Wake Word Architecture

ezlocalai trains wake word models using TTS-generated samples and exports them in multiple formats for cross-platform deployment:

```mermaid
graph LR
    A[User selects wake word] --> B[ezlocalai trains model]
    B --> C[Export: PyTorch .pt]
    B --> D[Export: ONNX .onnx]
    B --> E[Export: ESPDL .espdl]
    C --> F[Server inference]
    D --> G[Mobile App<br/>ONNX Runtime Mobile]
    E --> H[ESP32-S3<br/>ESP-DL Framework]
```

### Supported Export Formats

| Format | Extension | Size | Target Platform | Framework |
|--------|-----------|------|-----------------|-----------|
| **PyTorch** | `.pt` | ~1.7MB | Server | PyTorch |
| **ONNX** | `.onnx` | ~1.7MB | Mobile | ONNX Runtime Mobile |
| **ESPDL** | `.espdl` | ~460KB | ESP32-S3 | ESP-DL v2.0+ |

The ESPDL format is quantized to int8 for efficient inference on microcontrollers, reducing the model size by ~75% while maintaining accuracy.

### Model Architecture

- **Input**: 40 MFCC coefficients × 150 frames (1.5 seconds of audio at 16kHz)
- **Architecture**: Compact CNN with 3 conv layers + batch norm + max pooling
- **Output**: Single sigmoid probability [0.0, 1.0]
- **Typical Accuracy**: 95-98% validation accuracy

### API Endpoints

```bash
# Train a new wake word model
curl -X POST "http://localhost:8091/v1/wakeword/train" \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"word": "hey jarvis"}'

# Check training status
curl "http://localhost:8091/v1/wakeword/jobs/{job_id}" \
  -H "Authorization: Bearer your-api-key"

# List available models
curl "http://localhost:8091/v1/wakeword/models" \
  -H "Authorization: Bearer your-api-key"

# Download model for mobile (ONNX format)
curl "http://localhost:8091/v1/wakeword/models/hey%20jarvis?format=onnx" \
  -H "Authorization: Bearer your-api-key" \
  -o hey_jarvis_model.onnx

# Download model for ESP32-S3 (ESPDL format)
curl "http://localhost:8091/v1/wakeword/models/hey%20jarvis?format=espdl" \
  -H "Authorization: Bearer your-api-key" \
  -o hey_jarvis_model.espdl

# Download PyTorch model for server-side inference
curl "http://localhost:8091/v1/wakeword/models/hey%20jarvis?format=pytorch" \
  -H "Authorization: Bearer your-api-key" \
  -o hey_jarvis_model.pt
```

### Client Integration

**Mobile App (Flutter/Dart):**
```dart
// Download model from ezlocalai
final response = await http.get(
  Uri.parse('$serverUrl/v1/wakeword/models/$wakeWord?format=onnx'),
  headers: {'Authorization': 'Bearer $apiKey'},
);
// Save and load with ONNX Runtime Mobile
final session = await OrtSession.create(modelPath);
```

**ESP32-S3 (C with ESP-DL):**
```c
#include "custom_wakeword.h"

// Initialize the custom wake word system
custom_wakeword_init();

// Download model from server (stores in NVS)
custom_wakeword_download_model(
    "http://192.168.1.100:8091",  // ezlocalai server URL
    "your-api-key",               // API key (or NULL)
    "hey jarvis"                  // Wake word to download
);

// Start detection with callback
void on_wake_word(const char *word, float confidence, void *ctx) {
    printf("Wake word '%s' detected (%.2f confidence)\n", word, confidence);
}
custom_wakeword_start(on_wake_word, NULL);
```

### Training Process

The wake word trainer:
1. **Generates TTS samples** using gTTS and Edge-TTS with various voices
2. **Applies audio augmentation** (noise injection, speed/pitch variations, reverb)
3. **Trains a CNN model** with MFCC features (40 coefficients)
4. **Exports to all formats**:
   - PyTorch (.pt) - full precision for server
   - ONNX (.onnx) - for mobile deployment
   - ESPDL (.espdl) - int8 quantized for ESP32

Training typically takes 3-5 minutes depending on your hardware.

## OpenAI Style Endpoint Usage

OpenAI Style endpoints available at `http://<YOUR LOCAL IP ADDRESS>:8091/v1/` by default. Documentation can be accessed at that <http://localhost:8091> when the server is running.

```python
import requests

response = requests.post(
    "http://localhost:8091/v1/chat/completions",
    headers={"Authorization": "Bearer your-api-key"},  # Change this if you configured an API key
    json={
        "model": "unsloth/Qwen3-VL-4B-Instruct-GGUF",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe each stage of this image."},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://www.visualwatermark.com/images/add-text-to-photos/add-text-to-image-3.webp"
                        },
                    },
                ],
            },
        ],
        "max_tokens": 8192,
        "temperature": 0.7,
        "top_p": 0.8,
    },
)
print(response.json()["choices"][0]["message"]["content"])
```

For examples on how to use the server to communicate with the models, see the [Examples Jupyter Notebook](tests.ipynb) once the server is running. We also have an [example to use in Google Colab]([ezlocalai-colab.ipynb](https://colab.research.google.com/github/DevXT-LLC/ezlocalai/blob/main/ezlocalai-colab.ipynb)).

## Workflow

```mermaid
graph TD
   A[app.py] --> B[FASTAPI]
   B --> C[Pipes]
   C --> D[LLM]
   C --> E[STT]
   C --> F[CTTS]
   C --> G[IMG]
   D --> H[llama_cpp]
   D --> I[tiktoken]
   D --> J[torch]
   E --> K[faster_whisper]
   E --> L[pyaudio]
   E --> M[webrtcvad]
   E --> N[pydub]
   F --> O[TTS]
   F --> P[torchaudio]
   G --> Q[diffusers]
   Q --> J
   A --> R[Uvicorn]
   R --> S[ASGI Server]
   A --> T[API Endpoint: /v1/completions]
   T --> U[Pipes.get_response]
   U --> V{completion_type}
   V -->|completion| W[LLM.completion]
   V -->|chat| X[LLM.chat]
   X --> Y[LLM.generate]
   W --> Y
   Y --> Z[LLM.create_completion]
   Z --> AA[Return response]
   AA --> AB{stream}
   AB -->|True| AC[StreamingResponse]
   AB -->|False| AD[JSON response]
   U --> AE[Audio transcription]
   AE --> AF{audio_format}
   AF -->|Exists| AG[Transcribe audio]
   AG --> E
   AF -->|None| AH[Skip transcription]
   U --> AI[Audio generation]
   AI --> AJ{voice}
   AJ -->|Exists| AK[Generate audio]
   AK --> F
   AK --> AL{stream}
   AL -->|True| AM[StreamingResponse]
   AL -->|False| AN[JSON response with audio URL]
   AJ -->|None| AO[Skip audio generation]
   U --> AP[Image generation]
   AP --> AQ{IMG enabled}
   AQ -->|True| AR[Generate image]
   AR --> G
   AR --> AS[Append image URL to response]
   AQ -->|False| AT[Skip image generation]
   A --> AU[API Endpoint: /v1/chat/completions]
   AU --> U
   A --> AV[API Endpoint: /v1/embeddings]
   AV --> AW[LLM.embedding]
   AW --> AX[LLM.create_embedding]
   AX --> AY[Return embedding]
   A --> AZ[API Endpoint: /v1/audio/transcriptions]
   AZ --> BA[STT.transcribe_audio]
   BA --> BB[Return transcription]
   A --> BC[API Endpoint: /v1/audio/generation]
   BC --> BD[CTTS.generate]
   BD --> BE[Return audio URL or base64 audio]
   A --> BF[API Endpoint: /v1/models]
   BF --> BG[LLM.models]
   BG --> BH[Return available models]
   A --> BI[CORS Middleware]
   BJ[.env] --> BK[Environment Variables]
   BK --> A
   BL[setup.py] --> BM[ezlocalai package]
   BM --> BN[LLM]
   BM --> BO[STT]
   BM --> BP[CTTS]
   BM --> BQ[IMG]
   A --> BR[API Key Verification]
   BR --> BS[verify_api_key]
   A --> BT[Static Files]
   BT --> BU[API Endpoint: /outputs]
   A --> BV[Ngrok]
   BV --> BW[Public URL]
```

ezlocalai

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README