vllm-chatterbox-stream

KO | EN

gitlite — search

#javascript #php #java #dotfiles #python #golang #hacktoberfest #groovy #alfred-workflow #r #rails #html

vllm-chatterbox-stream

★ 56 Open GitHub ↗

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# vLLM Chatterbox Stream — OpenAI-compatible TTS with real streaming

Multilingual text-to-speech ([Chatterbox](https://github.com/resemble-ai/chatterbox))
served on **vLLM**, exposed through the **OpenAI `/v1/audio/speech` API** with **real
audio streaming** and low time-to-first-byte. 23 languages, voice cloning.

- **Two-stage TTS:** an autoregressive **T3** model (a Llama backbone) emits speech
  tokens — served by vLLM's `AsyncLLMEngine` with continuous batching — and **S3Gen**
  (CosyVoice2 flow-matching decoder + HiFi-GAN vocoder) turns them into a 24 kHz waveform.
- **Real streaming:** with `stream: true` the server emits raw PCM chunks as they are
  generated, so playback starts in ~1 second instead of waiting for the whole clip.

More detail: [`docs/serving.md`](docs/serving.md) (full API & tuning) ·
[`docs/maintainers-runbook.md`](docs/maintainers-runbook.md) (ops) ·
[`docs/technical-primer.md`](docs/technical-primer.md) (how it works).

---

## Install

Linux + NVIDIA GPU. Requires `git` and Python 3.10+ (and
[`uv`](https://pypi.org/project/uv/) for the fast path).

```bash
git clone https://github.com/wuxuedaifu/vllm-chatterbox-stream.git
cd vllm-chatterbox-stream
uv venv && source .venv/bin/activate
uv pip install -e .            # or: pip install -e .
```

Model weights download automatically from the Hugging Face Hub on first run.

## Start the server

```bash
chatterbox-serve --variant multilingual --host 0.0.0.0 --port 8000
```

Recommended low-latency flag (≈12% faster TTFB, identical audio):

```bash
CHATTERBOX_T3_CUDA_GRAPHS=1 chatterbox-serve --variant multilingual --port 8000
```

| Flag | Default | Meaning |
|---|---|---|
| `--variant` | `multilingual` | `multilingual` (23 langs) or `english` |
| `--host` / `--port` | `0.0.0.0` / `8000` | bind address |
| `--voices` | `voices/voices.json` | voice-preset file |
| `--max-batch-size` | `16` | vLLM `max_num_seqs` + scheduler concurrency budget |
| `--max-model-len` | `1000` | max speech tokens per request |

---

## Run with Docker

The image bundles the server and runs `chatterbox-serve` on port 8000. An NVIDIA GPU
and the NVIDIA Container Toolkit are required at runtime.

**Build:**

```bash
docker build -t vllm-chatterbox-stream .
```

**Run** (mount a Hugging Face cache so weights download only once):

```bash
docker run --gpus all -p 8000:8000 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  vllm-chatterbox-stream
```

Enable the low-latency CUDA-graphs path:

```bash
docker run --gpus all -p 8000:8000 \
  -e CHATTERBOX_T3_CUDA_GRAPHS=1 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  vllm-chatterbox-stream
```

The server is then reachable at `http://localhost:8000/v1/audio/speech`.

---

## Real streaming & TTFB

With `stream: true` the response is a chunked `audio/pcm` body: T3 speech tokens are
consumed as deltas and an `S3GenStreamer` re-decodes the growing prefix into ~1 s audio
chunks (HiFi-GAN `cache_source` + overlap-fade), so the first audio arrives long before
generation finishes.

**Measured on one A100 (vLLM 0.10.0):**

| Metric | Streaming | Non-streaming |
|---|---|---|
| Time-to-first-byte (Arabic, ~17 s clip) | **≈ 1.1 s** | ≈ 7.6 s (~7× slower) |
| Single-request TTFB p50 (short EN, CUDA graphs on) | **≈ 0.68 s** | — |
| Inter-chunk gap | ≈ 0.6 s | n/a |
| Real-time factor (RTF) | ≈ 0.5 | ≈ 0.5 |

### TTFB tuning (streaming)

| Setting | Default | Effect (measured) |
|---|---|---|
| Startup warmup | **on** | Throwaway generation at startup so the first real request skips cold-start (~1.5 s → ~0.81 s). Disable with `CHATTERBOX_SKIP_WARMUP=1`. |
| `CHATTERBOX_T3_CUDA_GRAPHS=1` | off | vLLM CUDA graphs for T3. **TTFB 809 → 714 ms (−12%)**, audio identical (corr 0.9999). **Recommended on.** |
| `CHATTERBOX_STREAM_FIRST_HOP` | 12 | Smaller first hop emits the first chunk sooner (tune to taste). |
| `CHATTERBOX_STREAM_NTIMESTEPS` | — | Fewer CFM decode steps roughly halve decode time (some quality cost). |

> Note: PCM is currently the only streaming format (`mp3`/`opus` containers are
> non-streaming for now). Streaming re-decodes the prefix each chunk (O(N²) total
> compute — fine for sentence-length inputs; incremental caching is a future optimization).

---

## Concurrency

`chatterbox-serve` runs on vLLM's `AsyncLLMEngine` with continuous batching. The
scheduler admits up to `--max-batch-size` (default 16) requests so they batch together;
a separate, smaller semaphore bounds concurrent S3Gen decodes
(`CHATTERBOX_S3GEN_CONC`, default 2) so they don't thrash VRAM.

**Closed-loop HTTP load test (A100, T3 CUDA graphs on, 79-char EN input)** — at each
concurrency level `N`, `N` virtual users each stream a request, await completion, and
repeat for a fixed window. Metrics over successful requests:

| Concurrency (N) | TTFB p50 | TTFB p95 | Throughput (req/s) | Errors |
|---|---|---|---|---|
| 1 | 0.68 s | 0.71 s | 0.45 | 0% |
| 2 | 1.07 s | 1.19 s | 0.33 | 0% |
| 4 | 1.98 s | — | 0.47 | 0% |
| 8 | 3.72 s | — | 0.43 | 0% |
| 16 | 4.47 s | — | 0.37 | 0% |
| 32 | 18.6 s | — | 0.45 | 0% |

TTFB stays roughly flat up to the batch capacity (16); beyond it, requests queue and
TTFB climbs — raise `--max-batch-size` (more VRAM) or add GPU replicas. **Throughput
holds at ~0.4 req/s regardless of N: that is the single-GPU compute ceiling** (≈2
concurrent requests already saturate the GPU), not a bug. Scale total throughput with
faster per-request compute (CUDA graphs, fewer CFM steps) or more replicas.

Reproduce with `test_scripts/concurrency_test.py` (closed-loop tester; writes a
CSV + Markdown table).

---

## API endpoints

| Method & path | Purpose |
|---|---|
| `POST /v1/audio/speech` | synthesize speech (non-streaming or streaming) |
| `GET /v1/models` | list the served model (`chatterbox`) |
| `GET /health` | readiness probe → `{"status":"ok"}` |

### Usage

**curl — non-streaming** (returns the whole clip):

```bash
curl -s -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox","input":"Hello from Chatterbox.","voice":"uae_man","language":"en","response_format":"mp3"}' \
  -o out.mp3
```

**curl — streaming** (low TTFB; raw PCM chunks piped straight to a player):

```bash
curl -N -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox","input":"مرحبا بالعالم","voice":"uae_man","language":"ar","stream":true,"response_format":"pcm"}' \
  --output - | aplay -r 24000 -f S16_LE -c 1
```

**OpenAI Python client** (Chatterbox extras go through `extra_body`):

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# Non-streaming
resp = client.audio.speech.create(
    model="chatterbox",
    voice="uae_man",
    input="Hello from Chatterbox.",
    response_format="mp3",
    extra_body={"language": "en", "exaggeration": 0.5},
)
resp.stream_to_file("out.mp3")

# Streaming: write PCM chunks as they arrive
with client.audio.speech.with_streaming_response.create(
    model="chatterbox",
    voice="uae_man",
    input="Hello from Chatterbox.",
    response_format="pcm",
    extra_body={"language": "en", "stream": True},
) as resp:
    with open("out.pcm", "wb") as f:
        for chunk in resp.iter_bytes():
            f.write(chunk)   # raw int16, 24 kHz, mono
```

### Request — `POST /v1/audio/speech` (JSON)

| Field | Type | Default | Meaning |
|---|---|---|---|
| `model` | string | `chatterbox` | model id |
| `input` | string | **required** | text to synthesize |
| `voice` | string | `default` | preset name from `voices/voices.json` |
| `reference_audio` | string | `null` | base64 or `http(s)` URL clip to clone (overrides `voice`) |
| `response_format` | string | `mp3` | `mp3` \| `wav` \| `opus` \| `flac` \| `pcm` \| `aac` |
| `stream` | bool | `false` | stream audio chunks as generated (use `pcm`) |
| `speed` | float | `1.0` | playback speed, `0 < speed ≤ 4` |
| `language` | string | `en` | one of 23 codes (see below) |
| `exaggeration` | float | `0.5` | emotion intensity, `0–1` |
| `vad` | bool | `true` | drop hallucinated non-speech (Silero VAD; non-streaming) |
| `temperature` | float | `0.8` | sampling |
| `top_p` / `min_p` / `repetition_penalty` | float | `0.8` / `0.1` / `2.0` | sampling |

### Response

- **Non-streaming:** HTTP body = the audio file, `Content-Type: audio/<response_format>`.
- **Streaming** (`stream: true`): chunked `audio/pcm` body emitted as generated — raw
  little-endian **int16, 24 kHz, mono**.
- **Errors:** JSON `{"error": "..."}` — `400` (bad voice/language), `503` (engine not
  ready), `500` (other).

**Voices:** edit `voices/voices.json` (`name → reference clip path`, or `null` for the
built-in default). Keep reference clips ~6–10 s of clean speech.

---

## Language samples

23 languages, all generated by this server. Codes:
`ar da de el en es fi fr he hi it ja ko ms nl no pl pt ru sv sw tr zh`.

| Language | Code | Sample | Language | Code | Sample |
|---|---|---|---|---|---|
| Arabic | `ar` | [▶](docs/samples/lang/lang_ar.mp3) | Japanese | `ja` | [▶](docs/samples/lang/lang_ja.mp3) |
| Danish | `da` | [▶](docs/samples/lang/lang_da.mp3) | Korean | `ko` | [▶](docs/samples/lang/lang_ko.mp3) |
| German | `de` | [▶](docs/samples/lang/lang_de.mp3) | Malay | `ms` | [▶](docs/samples/lang/lang_ms.mp3) |
| Greek | `el` | [▶](docs/samples/lang/lang_el.mp3) | Dutch | `nl` | [▶](docs/samples/lang/lang_nl.mp3) |
| English | `en` | [▶](docs/samples/lang/lang_en.mp3) | Norwegian | `no` | [▶](docs/samples/lang/lang_no.mp3) |
| Spanish | `es` | [▶](docs/samples/lang/lang_es.mp3) | Polish | `pl` | [▶](docs/samples/lang/lang_pl.mp3) |
| Finnish | `fi` | [▶](docs/samples/lang/lang_fi.mp3) | Portuguese | `pt` | [▶](docs/samples/lang/lang_pt.mp3) |
| French | `fr` | [▶](docs/samples/lang/lang_fr.mp3) | Russian | `ru` | [▶](docs/samples/lang/lang_ru.mp3) |
| Hebrew | `he` | [▶](docs/samples/lang/lang_he.mp3) | Swedish | `sv` | [▶](docs/samples/lang/lang_sv.mp3) |
| Hindi | `hi` | [▶](docs/samples/lang/lang_hi.mp3) | Swahili | `sw` | [▶](docs/samples/lang/lang_sw.mp3) |
| Italian | `it` | [▶](docs/samples/lang/lang_it.mp3) | Turkish | `tr` | [▶](docs/samples/lang/lang_tr.mp3) |
|  |  |  | Chinese | `zh` | [▶](docs/samples/lang/lang_zh.mp3) |

Regenerate the set with `test_scripts/multilang_test.py`.

---

## Known limitation — audio hallucinations

Chatterbox itself has a known defect: generated speech can occasionally contain
**hallucinated, blurry, or garbled non-speech artifacts** (buzzing, breaths, or
meaningful-sounding rambling past the end of the text). The non-streaming path mitigates
this with a **Silero VAD post-filter** (enabled by default; disable per request with
`"vad": false`) that keeps only speech regions, plus a text-proportional `max_tokens`
cap to bound rambling. These reduce the problem substantially but **do not eliminate it
entirely** — it is an upstream model limitation, not a server bug. VAD is not yet applied
to the streaming path (it needs a causal hangover buffer — planned).

---

## License

vllm-chatterbox-stream

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README