mlx-openai-server

KO | EN

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #claude-code #windows #codex #developer-tools #dsh

mlx-openai-server

★ 357 Open GitHub ↗

A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# mlx-openai-server

[![MIT License](https://img.shields.io/badge/license-MIT-green.svg)](https://raw.githubusercontent.com/cubist38/mlx-openai-server/main/LICENSE)
[![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/release/python-3110/)

OpenAI-compatible API server for local MLX models on Apple Silicon. It serves text, multimodal, image, embedding, and Whisper models through familiar OpenAI SDK endpoints.

> Requires macOS on Apple Silicon and Python 3.11+.

## Contents

- [Feature Launch](#feature-launch)
- [Install](#install)
- [Start a Server](#start-a-server)
- [API Usage](#api-usage)
- [Server Options](#server-options)
- [Multi-Model Config](#multi-model-config)
- [Long Context and Metal OOM](#long-context-and-metal-oom)
- [Advanced LM Options](#advanced-lm-options)
- [Troubleshooting](#troubleshooting)
- [Examples and Demos](#examples-and-demos)

## Feature Launch

Darwin 36B Opus is now available in MLX format for local text inference:

- Original model: [FINAL-Bench/Darwin-36B-Opus](https://huggingface.co/FINAL-Bench/Darwin-36B-Opus)
- MLX text-only 8-bit conversion: [GiaHuy/Darwin-36B-Opus-mlx-text-only-8bit](https://huggingface.co/GiaHuy/Darwin-36B-Opus-mlx-text-only-8bit)

Launch it with the required reasoning and tool-call parsers:

```bash
mlx-openai-server launch --model-path Darwin-36B-Opus-mlx-text-only-8bit --reasoning-parser qwen3_moe --tool-call-parser qwen3_coder --debug --served-model-name Darwin-36B-Opus
```

Darwin 36B Opus should be served with both `--reasoning-parser qwen3_moe` and `--tool-call-parser qwen3_coder`; without them, reasoning and tool-call output will not be parsed correctly.

## Install

```bash
python3.11 -m venv .venv
source .venv/bin/activate

uv pip install mlx-openai-server
```

Install from GitHub instead:

```bash
uv pip install git+https://github.com/cubist38/mlx-openai-server.git
```

Whisper transcription also needs ffmpeg:

```bash
brew install ffmpeg
```

## Start a Server

Text model:

```bash
mlx-openai-server launch \
  --model-type lm \
  --model-path mlx-community/Qwen3-Coder-Next-4bit \
  --reasoning-parser qwen3_moe \
  --tool-call-parser qwen3_coder
```

Point OpenAI-compatible clients to:

```text
http://localhost:8000/v1
```

Use any non-empty API key, for example `not-needed`.

Common launch modes:

```bash
# Multimodal text/image/audio
mlx-openai-server launch \
  --model-type multimodal \
  --model-path <mlx-vlm-model>

# Image generation
mlx-openai-server launch \
  --model-type image-generation \
  --model-path <flux-or-qwen-image-model> \
  --config-name flux-dev \
  --quantize 8

# Image editing
mlx-openai-server launch \
  --model-type image-edit \
  --model-path <flux-or-qwen-image-edit-model> \
  --config-name flux-kontext-dev \
  --quantize 8

# Embeddings
mlx-openai-server launch \
  --model-type embeddings \
  --model-path <embedding-model>

# Whisper transcription
mlx-openai-server launch \
  --model-type whisper \
  --model-path mlx-community/whisper-large-v3-mlx
```

Supported model types:

| Type | Backend | Endpoint family |
|------|---------|-----------------|
| `lm` | `mlx-lm` | chat, responses |
| `multimodal` | `mlx-vlm` | chat, responses |
| `image-generation` | `mflux` | image generation |
| `image-edit` | `mflux` | image editing |
| `embeddings` | `mlx-embeddings` | embeddings |
| `whisper` | `mlx-whisper` | audio transcription |

Image `--config-name` values:

- Generation: `flux-schnell`, `flux-dev`, `flux-krea-dev`, `flux2-klein-4b`, `flux2-klein-9b`, `qwen-image`, `z-image-turbo`, `fibo`
- Editing: `flux-kontext-dev`, `flux2-klein-edit-4b`, `flux2-klein-edit-9b`, `qwen-image-edit`

## API Usage

### Chat

```python
import openai