mstar

KO | EN

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #dsh #claude-code #codex #developer-tools #react

★ 68 Open GitHub ↗

A high-performance, universal serving framework for any-to-any models.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h3 align="center">A universal serving system for composite, any-to-any multimodal models</h3>

Models are dataflow graphs &nbsp;·&nbsp; requests are Walks &nbsp;·&nbsp; one runtime serves them all

<a href="#quickstart">Quickstart</a> &nbsp;·&nbsp;
 <a href="#supported-models">Models</a> &nbsp;·&nbsp;
 <a href="#how-it-works">How it works</a> &nbsp;·&nbsp;
 <a href="https://m-star.org/mstar/">Docs</a> &nbsp;·&nbsp;
 <a href="https://m-star.org/">Blog</a> &nbsp;·&nbsp;
 <a href="https://arxiv.org/abs/2606.12688">Paper</a>

---

One runtime that matches or beats state-of-the-art inference systems. Full methodology and current numbers in the <a href="https://m-star.org/">blog post</a> and <a href="https://arxiv.org/abs/2606.12688">paper</a>.

## What is M*?

**M\*** (pronounced *"M-star"*) is a serving system for the new generation of **composite multimodal models** — models built from structurally distinct components (vision encoders, transformer backbones, diffusion and flow heads, audio codecs, action generators, world-model predictors) whose execution path changes with the input and the task.

LLM serving stacks assume inference is a single autoregressive loop. Composite models broke that assumption. M\*'s core idea is the **Walk Graph**: a model is a dataflow graph of its components, and every request is a *Walk* over that graph. A single runtime serves unified multimodal models, omni models, speech LMs, vision-language-action policies, and world models — at or above the performance of engines specialized for each.

**Fast** — per-component fast paths, matched to each component's bottleneck:
- Paged attention (FlashInfer) and continuous batching for autoregressive backbones
- CUDA-graph capture for encoders and decode
- Classifier-free-guidance parallelism for diffusion / flow
- Tensor parallelism and Ulysses sequence parallelism, composable as a TP × SP mesh per component
- Sliding-window chunk streaming for audio codecs
- Component-level disaggregation with pluggable tensor transport (shared memory, TCP, RDMA)

**Flexible** — the abstraction mirrors the model:
- One small Python file per model declares its component graph and its Walks
- A YAML file maps components to GPUs at per-component, per-walk granularity — arbitrary disaggregation, no code changes
- Text, image, audio, video, and robot actions, in and out
- A **Python SDK**, an **OpenAI-compatible API**, and a native streaming endpoint

> **Roadmap.** M\* is evolving toward *many-model, agentic* multimodal serving — routing requests across many models and tools within one graph-scheduled runtime.

## Quickstart

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install --torch-backend=auto -e .[all]      # install M*
mstar serve bagel          # one command — launch a server (default: http://localhost:8000)
```

To enable flash-attn support (required for Qwen3-Omni, recommended for BAGEL),
```bash
# torch built for CUDA 12.x (cu12)
uv pip install \
  "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.9cxx11abiTRUE-cp312-cp312-linux_x86_64.whl"

# torch built for CUDA 13.x (cu13)
uv pip install \
  "https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu13torch2.9cxx11abiTRUE-cp312-cp312-linux_x86_64.whl"
```

Other models: `mstar serve cosmos3` · `mstar serve cosmos3_droid` · `mstar serve qwen3_omni` · `mstar serve orpheus` · `mstar serve pi05` · `mstar serve vjepa2`

**Python SDK** — works for every model (text, image, audio, video):

```python
from mstar import MStarClient
client = MStarClient("http://localhost:8000")

client.chat("What is the capital of France?").text          # text
client.generate_image("a cat in a hat")                     # → PNG bytes   (BAGEL, Cosmos3)
client.tts("Hello there", voice="tara").to_wav("out.wav")   # → speech      (Orpheus)

for event in client.chat("Tell me a story", stream=True):   # streaming
    print(getattr(event, "text", ""), end="", flush=True)
```

**OpenAI-compatible API** — drop-in for `bagel`, `cosmos3`, `qwen3_omni`, and `orpheus`:

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

client.chat.completions.create(model="bagel", messages=[{"role": "user", "content": "hi"}])
client.audio.speech.create(model="orpheus", input="hi", voice="tara")   # text-to-speech
client.images.generate(model="bagel", prompt="a cat")                   # image generation
```

Runnable scripts and `curl` examples live in [`examples/`](examples/). Power users can launch any
deployment with an explicit config: `mstar-serve --config configs/<model>.yaml`.

_Note_: The **first request(s) on a fresh environment can be slow** — often tens of seconds to a few minutes. mstar ``torch.compile``s the model on first use, and that compilation happens lazily on the first request that exercises each path.

## Supported models

| Model | Family | Input → Output | Endpoints |
|-------|--------|----------------|-----------|
| [BAGEL](https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT) | Unified multimodal | text, image → text, image | `/v1/chat/completions`, `/v1/images/generations` |
| [Qwen3-Omni](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) | Omni | text, image, audio, video → text, speech | `/v1/chat/completions` |
| [Orpheus](https://huggingface.co/canopylabs/orpheus-3b-0.1-ft) | Speech LM | text → speech | `/v1/audio/speech` |
| [Cosmos3 Nano / Super](https://huggingface.co/nvidia/Cosmos3-Nano) | World model | text, image, video → image, video (+ sound), robot actions | `/v1/images/generations`, `/v1/videos/generations` |
| [Cosmos3 Policy DROID](https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID) | Robot policy | text, image, video → robot actions, video | `/generate`, `/v1/images/generations`, `/v1/videos/generations` |
| [Pi0.5](https://huggingface.co/lerobot/pi05_base) | Vision-language-action | text, image, state → robot actions | `/generate` |
| [V-JEPA 2 / 2-AC](https://huggingface.co/facebook/vjepa2-vitl-fpc64-256) | World model | video (+ actions) → latents, rollouts | `/generate` |
| [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) | Video diffusion | text, image → video | `/v1/videos/generations`, `/generate` |

Every model is reachable through the SDK and the native `/generate` endpoint; the OpenAI-compatible
routes cover the chat, speech, image, and video models.

## How it works

```
HTTP / SDK  →  API Server  →  Conductor  →  Workers (one per GPU)  →  streaming results
                                  │              │
                          walks the graph,   own subgraphs; route tensors
                          schedules walks    directly to one another
```

A model declares a **computation graph** of components and a set of named **Walks** (e.g.
`prefill`, `decode`, `image_gen`). The **Conductor** turns each request into a walk over that graph
and schedules it; **Workers** each own a subgraph on their GPU and stream tensors directly to one
another. Logical graph structure is decoupled from physical placement, so the same model runs
single-GPU or fully disaggregated by changing only the YAML `node_groups`. Four composable
primitives — `Sequential`, `Parallel`, `Loop`, and a cross-partition
`StreamingGraphEdge` — express every model family above. See the [paper](https://arxiv.org/abs/2606.12688) for the full design.

### Optional: Rust ZMQ transport

The ZeroMQ control mesh can run over a Rust transport (vendored in [`rust/`](rust/)) instead of
pyzmq — wire-compatible, selectable per process with `MSTAR_RUST_ZMQ` (default `AUTO`).
Note what AUTO-by-default means: on any machine where the extension is built, the whole
mesh switches to the Rust transport with no configuration change — each process logs its
choice at startup (`control mesh transport: ...`), and `MSTAR_RUST_ZMQ=0` pins pyzmq. Build it with
[maturin](https://www.maturin.rs): `uv pip install maturin && maturin develop --release -m rust/Cargo.toml`.
See the [installation docs](https://m-star.org/mstar/installation.html) and
[environment variables](https://m-star.org/mstar/environment_variables.html).

## Performance

Across every model we benchmark, M\* matches or beats the system specialized for that family — unified
models (BAGEL), omni and speech models (Qwen3-Omni, Orpheus), and world models (Cosmos3, V-JEPA 2) — by executing
only the components each request needs and giving each its own fast path: paged attention and continuous
batching for autoregressive backbones, classifier-free-guidance parallelism for diffusion, chunk streaming
for audio codecs, and persistent-cache loops for world-model rollouts.

Benchmark numbers shift as systems evolve — ours and everyone else's — so rather than freeze figures here
that go stale, we keep the current results and full methodology in the
[blog post](https://m-star.org/) and the [paper](https://arxiv.org/abs/2606.12688).

## Contributing

Issues and pull requests are welcome. Found a bug, or want a model or feature supported?
**[Open an issue](https://github.com/mstar-project/mstar/issues).** To add a model yourself,
follow the [Adding a New Model](https://m-star.org/mstar/adding_models.html) guide.
PRs to `main` go through review and CI (`ruff`).

## Citation

If you use M\* in your research, please cite:

```bibtex
@article{mstar2026,
  title  = {M*: A Modular, Extensible, Serving System for Multimodal Models},
  author = {Jha, Atindra and Sagan, Naomi and Kamahori, Keisuke and Sivgin, Irmak and
            Sanda, Rohan and Gao, Steven and Horowitz, Mark and Zettlemoyer, Luke and
            Hsu, Olivia and Leskovec, Jure and Kasikci, Baris and Wang, Stephanie},
  year   = {2026},
  eprint = {2606.12688},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG}
}
```

From Stanford University & the University of Washington. Correspondence: `atindra@cs.stanford.edu`.

## Acknowledgments

M\* builds on ideas and proven primitives from the open-source community — paged attention and
continuous batching ([vLLM](https://github.com/vllm-project/vllm)),
[FlashInfer](https://github.com/flashinfer-ai/flashinfer) kernels, streaming speech serving
(VoxServe), and RDMA tensor transport ([Mooncake](https://github.com/kvcache-ai/Mooncake)).

## License

[Apache License 2.0](LICENSE).

mstar

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README