ChaosEngineAI

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 24 Open GitHub ↗

Local AI workstation — discover, run, chat, benchmark, and generate images from open-weight models. DFlash/DDTree speculative decoding, TurboQuant & TriAttention cache compression strategies, MLX + llama.cpp + vLLM + MTPLX backends.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">ChaosEngineAI</h1>

The local AI model runner for serious tinkerers. 
 Discover, convert, serve, chat with, benchmark, and generate images and video from open-weight models — all on your own machine.

> ⚠️ **Work in progress.** ChaosEngineAI is under active development. Expect rough edges, breaking changes between versions, and features that appear (and occasionally disappear) from one release to the next. Feedback and issue reports are very welcome.

---

## Why ChaosEngineAI

ChaosEngineAI is a desktop control plane for running large language models locally. It pairs a fast Tauri + React shell with a Python backend that drives `llama.cpp`, Apple MLX, and (optionally) vLLM, so you get a single window for everything from "I want to try this Hugging Face model" to "show me tokens-per-second across three quantizations on this exact prompt."

- **One app, the whole pipeline.** Discover models, download them, convert to MLX, load into a warm pool, serve over an OpenAI-compatible API, chat, benchmark, and generate images and video.
- **Real local performance.** First-class support for `llama.cpp` GGUF and Apple Silicon MLX for LLMs, plus local Stable Diffusion for image generation, plus diffusion DiT video models (Wan 2.1/2.2, LTX-Video 2.0/2.3, HunyuanVideo, CogVideoX, Mochi) via diffusers, mlx-video on Apple Silicon, and stable-diffusion.cpp scaffolding for cross-platform.
- **Pluggable cache compression.** Native f16 cache out of the box, with KV cache compression strategies — [TriAttention](https://github.com/WeianMao/triattention) and [TurboQuant](https://pypi.org/project/turboquant-mlx-full/). Install supported backends into the repo-local runtime, restart, and they appear in the UI.
- **Speculative decoding.** DFlash, DDTree, and MTPLX accelerate generation by 1.8-5x with zero quality loss. A small draft model proposes tokens; the target verifies them in one forward pass. DDTree extends this with tree-structured candidate exploration for even higher acceptance rates. MTPLX adds native Multi-Token Prediction for models that were trained with MTP heads (Qwen3.5/3.6, DeepSeek V3/R1, Qwen3-Coder-Next).
- **Hybrid local + remote workflows.** Scan multiple local model directories, convert Hugging Face checkpoints to MLX, or point the app at remote OpenAI-compatible providers when you want a cloud fallback.
- **Per-chat runtime profiles.** Each chat session remembers the exact model, cache strategy, quantization bits, context length, and speculative decoding settings used — switch between configurations without losing track.
- **Prompting + evaluation.** Built-in prompt templates, side-by-side compare mode, and benchmark modes for throughput, perplexity, and task accuracy keep experimentation in one place.
- **Agent tools.** Built-in web search, calculator, code executor, and file reader tools that models can call during conversations.
- **Built for power users.** Live runtime telemetry, structured logs, side-by-side benchmark history with model filtering and red/green delta comparisons, and granular launch preferences.
- **Polished, fast UI.** A focused dark workspace that gets out of the way and never blocks on the backend.
- **Self-updating.** Ships with a signed in-app updater — no more manual re-downloads when a new release drops.

---

## Feature Highlights

- 🧭 **Discover** open-weight models from curated catalogs and pull them with one click
- 💾 **My Models** library with format, size, context, and modified-date sorting
- 🔁 **Conversion** pipeline that turns Hugging Face checkpoints into MLX (Apple Silicon)
- 🚀 **Server** mode exposing an OpenAI-compatible REST API for your other tools
- 💬 **Chat** with the loaded model, including document attachments with inline citations, image inputs for vision-capable models, tool calls, collapsible reasoning traces, conversation branching, in-thread compare, mid-thread model swap, and retry on failed turns
- ⚖️ **Compare mode** to stream the same prompt through two models side-by-side with separate runtime settings and metrics
- 🧪 **HTML Challenge** — side-by-side HTML generation across 2-4 models with per-slot thinking mode, reasoning effort, and seed; sandboxed live previews, automated HTML validation, persistent history, retry + repair flows
- 🧰 **Sampler controls** in the composer — `top_p`, `top_k`, `min_p`, `repeat_penalty`, `seed`, `mirostat`, `reasoning_effort`, JSON-schema constrained output
- 🧷 **MCP client** — stdio JSON-RPC tool adapter so any local MCP server is callable from chat
- 📚 **Workspace knowledge stacks** — shared RAG corpus across sessions via `llama-embedding` + cosine retrieval
- 🗂️ **Prompt Library** with built-in starter templates plus searchable tags, categories, CRUD, fill-form variables, and one-click apply-to-chat
- 🎨 **Image Discover** curated catalog of Stable Diffusion models with one-click download from Hugging Face
- 🖼️ **Image Models** library showing installed image models ready for generation
- 🖌️ **Image Studio** for prompt-based image generation with aspect ratio, quality presets, and negative prompts
- 🏛️ **Image Gallery** to browse, filter, and reuse saved outputs — compare models and re-run with the same settings
- 🎬 **Video Studio** for prompt-based video generation across Wan 2.1/2.2, LTX-Video 2.0/2.3, HunyuanVideo, CogVideoX, and Mochi — with model picker, runtime status overlay, live progress, and cancellable generation
- 🍿 **Video catalog** curated set of diffusion DiT video models with one-click downloads and a separate Video Models library
- ⚡ **First Block Cache + TaylorSeer / MagCache / PyramidAttentionBroadcast / FasterCache** cross-platform diffusion cache hooks (≈1.8× FLUX.1-dev with imperceptible drift), plus TeaCache for FLUX / HunyuanVideo / LTX-Video / CogVideoX / Mochi
- 🖼️ **Live denoise thumbnails** in Image + Video Studio via TAESD/TAEHV preview VAE swap — see what's being generated in real time, not just a progress bar
- ✨ **MLX prompt enhancer** (Apple Silicon) — local Qwen2.5-0.5B rewrites your prompt into the active DiT's training distribution per family (FLUX / Wan / LTX / HunyuanVideo / SDXL / SD3)
- 🧬 **Distill LoRA + transformer support** — Hyper-SD-8step / Turbo-Alpha for FLUX.1-dev, CausVid for Wan 2.1, lightx2v 4-step distill for Wan 2.2 A14B I2V (bf16 + fp8_e4m3)
- 🎯 **Wan 2.1 / 2.2 mlx-video runtime** end-to-end on Apple Silicon — one-shot convert pipeline, GUI install panel, runtime routing for all `Wan-AI/Wan2.{1,2}-*` repos
- 🩺 **Diagnostics panel** in Settings — per-section error fallback, platform-aware repair actions, and a one-click reinstall for the Apple Silicon mlx-video runtime
- 📊 **Benchmarks** with throughput, perplexity, and task-accuracy modes plus a history view for A/B comparisons
- 📜 **Logs** streaming straight from the Python runtime
- ⚙️ **Settings** for data directories, Hugging Face tokens, remote providers, integrations, default launch preferences, and runtime tuning
- 🔍 **UI scale** — segmented 75% / 100% / 125% / 150% control in Settings → Display, applied app-wide via `useUiScale`
- ☁️ **Remote providers** for OpenAI-compatible APIs, with keys stored locally and masked in the UI
- 📁 **Model directories** for custom stores, Ollama, LM Studio, or shared model paths
- 🧠 **Warm pool** keeps recently-used models hot so subsequent loads are instant
- 🔄 **In-app updates** — signed, verified, cross-platform. Launch, prompt, relaunch.
- 🍎 **Notarized macOS builds** with a fully embedded, hardened-runtime Python runtime
- 🐧 **AppImage + .deb** for Linux, **NSIS installer** for Windows
- 🎚️ **Adaptive runtime controls** for balancing speed, memory footprint, and quality on local hardware
- 📈 **Live telemetry** on the dashboard: backend health, engine, loaded model, hardware, warm-pool state
- 🔮 **DFlash speculative decoding** for 3-5x faster generation with zero quality loss (Qwen3, Qwen3.5, LLaMA 3.1, gpt-oss, Kimi families)
- 🌳 **DDTree** tree-based speculative decoding — explores multiple draft paths in parallel for higher acceptance rates
- 🪄 **MTPLX** Multi-Token Prediction speculative decoding (Apple Silicon) — 1.8-2.2× speedup with zero quality loss for trained-with-MTP models (Qwen3.5/3.6, DeepSeek V3/R1, Qwen3-Coder-Next, Youssofal MTPLX-Optimized variants)
- 🔧 **Agent tools** — web search, calculator, code executor, and file reader for tool-augmented conversations
- 🎓 **Fine-tuning** with LoRA adapter support for MLX models
- 🧩 **Plugin system** with 5 extension types: cache strategies, inference engines, tools, model sources, and post-processors
- 🔌 **Integrations** with copy-paste connection snippets for Continue.dev, Goose, Cursor, and Claude Code via the local API

---

## Complete Feature Map

### Local runtimes & serving

- `llama.cpp` GGUF, Apple MLX, optional vLLM, plus remote OpenAI-compatible providers as fallbacks
- OpenAI-compatible local server with preferred port, optional LAN exposure, auto-start, live request counts, active connections, and copyable curl test commands
- Warm-pool activation so recently used runtime profiles can be re-activated instantly
- Signed in-app updates and bundled desktop runtimes for release builds

### Chat, prompting & agent workflows

- Multi-thread chat with pinned sessions, persistent history, inline thread renaming, and per-thread runtime memory
- Conversation branching — fork from any assistant message into a sibling thread
- In-thread compare — render sibling variants side-by-side under the assistant bubble
- Mid-thread model swap with one-turn override (try a different model for a single response, then revert)
- Retry failed assistant turns with one click, against the active runtime profile
- Document uploads with chunked retrieval, inline citations, and per-session document management
- Workspace knowledge stacks: shared RAG corpus across sessions via `llama-embedding` + cosine retrieval
- Image attachments for vision-capable models, with composer auto-gating from typed capability declarations
- Optional thinking mode with collapsible reasoning traces and cleanup of raw reasoning tokens
- Logprobs visualization (advanced-mode gated): per-message confidence summary, MLX logprobs streaming passthrough
- Substrate routing inspector + per-turn host strip surface engine, binary, CPU / GPU / RAM / temperature alongside tok/s and TTFT
- KV strategy chip in composer for per-turn cache override (native / turboquant / triattention) without touching launch settings
- Delve mode — critic-pass on assistant messages
- Tool-augmented conversations with `web_search`, `calculator`, `code_executor`, `file_reader`, plus stdio MCP client (JSON-RPC) so any local MCP server is callable from chat. Tool results render as table / code / markdown / image based on the returned shape.
- Side-by-side compare mode that streams the same prompt through two models with independent runtime settings and metrics
- HTML Challenge — 2-4 model slots, each with own thinking mode + reasoning effort + seed, all answering the same prompt with sandboxed live HTML previews, automated `valid` / `partial` / `script-error` / `blank-render` / `no-html` validation, persistent history, plus retry + repair flows
- OpenAI-compatible local server with full sampler-chain + embeddings parity (`top_p`, `top_k`, `min_p`, `repeat_penalty`, `seed`, `mirostat`, `reasoning_effort`, `json_schema`)
- Prompt Library with built-in starter templates, search, tagging, categories, CRUD, fill-form variables, and one-click apply-to-chat

### Models, discovery & media

- Curated text, image, and video catalogs with capability filters plus direct Hugging Face inspection
- Multiple local model directories, including custom paths, Ollama, LM Studio, or shared model stores
- Download management with progress, cancel/delete, and a metadata-rich local library
- Apple Silicon conversion from Hugging Face or local checkpoints to MLX with live progress
- Full local image workflow: Image Discover, Image Models, Image Studio, Image Gallery, preload/unload controls, seed reuse, and saved artifact metadata
- Full local video workflow: Video catalog, Video Models library, Video Studio with prompt enhancer, runtime probe/preload/unload, mlx-video LTX-2 subprocess engine on Apple Silicon, diffusers/MPS for Wan, and stable-diffusion.cpp scaffolding for cross-platform
- Hugging Face token management for gated model access

### Performance, evaluation & extensibility

- DFlash and DDTree speculative decoding with auto-resolved draft models and graceful fallback
- LLM cache strategies: native f16, TriAttention, TurboQuant
- Diffusion cache strategies — FBCache + TeaCache + TaylorSeer + MagCache + PyramidAttentionBroadcast (PAB) + FasterCache — applied to image + video DiTs (FLUX, SD3.5, Wan2.1/2.2, HunyuanVideo, LTX-Video, CogVideoX, Mochi)
- Runtime controls for cache bits, FP16 layers, fused attention, fit-in-memory behavior, context length, and speculative tree budget
- Benchmark modes for throughput, perplexity, and task accuracy (MMLU / HellaSwag), with persistent history, scatter plots, and diff tables
- LoRA adapter discovery plus fine-tuning hooks for local training workflows
- Plugin system spanning cache strategies, inference engines, tools, model sources, and post-processors
- Live logs, backend health, hardware telemetry, and orphaned-worker cleanup reporting

### Settings & integrations

- Data-directory relocation with migration/copy support
- Remote-provider configs stored locally with masked keys and HTTPS validation
- Integration snippets for Continue.dev, Goose, Cursor, and Claude Code through the local OpenAI-compatible API

---

## A Tour of the App

### Dashboard — *System overview*
![Dashboard](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/1.%20Dashboard_v2.png)

The launchpad. Surfaces backend health, the engine in use, the currently loaded model, hardware (platform, arch, memory), and quick stats from the warm pool. Big colored badges tell you instantly whether the runtime is online and what cache mode it will use on the next launch.

### Discover — *Browse and download AI models*
![Discover](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/2.%20Discover_v2.png)

Browse curated model families with capability tags (chat, coding, vision, reasoning, tools, multilingual, video, thinking). Expand a family to see every variant with its format, parameter count, and quant level, then queue downloads. Filter by format and capability across the whole catalog.

### Model Selection — *Configure and launch*
![Model Selection](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/4.%20Model_Selection_Modal_v2.png)

The launch modal. Pick a variant, set context length, choose the engine, and tune runtime strategy knobs in one place, pre-populated from your defaults.

### My Models — *Models on this machine*
![My Models](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/3.%20My_Models_v2.png)

Your local library. Sort by name, format, size, context length, or last-modified date. Each entry shows where it lives on disk, its quant scheme, and a one-click **Launch** that opens the launch modal with the right defaults pre-populated.

### Image Discover — *Browse and download image models*
![Image Discover](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/5.%20Image%20Discover.png)

Curated catalog of local image models. Discover Stable Diffusion models optimized for ChaosEngineAI, scout new releases from Hugging Face, and install them with one click. Filter by compatibility tags and see metadata at a glance.

### Image Models — *Installed image models*
![Image Models](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/6.%20Image%20Models.png)

Your installed image model library. See which Stable Diffusion models are ready for generation — each card shows size, diffusion pipeline, resolution, and a one-click **Generate** to jump straight into Image Studio.

### Image Studio — *Generate images from text*
![Image Studio](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/7.%20Image%20Studio.png)

Prompt-based image generation with full control. Choose a model, set aspect ratio and quality presets (square, portrait, landscape, wide), write positive and negative prompts, and generate. Recent outputs appear in the right panel with metadata and re-run options.

![Image Processing](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/8.%20Image%20Processing.png)

Live progress while the diffusion pipeline runs — step-by-step denoising visualization, elapsed time, and a running status log.

![Image Completion](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/9.%20Image%20Completion.png)

Completed image with full generation metadata — model, prompt, seed, steps, resolution, and timing. Open, reveal on disk, clone settings, or save directly from the completion modal.

### Image Gallery — *Browse and reuse generated images*
![Image Gallery](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/10.%20Image%20Gallery.png)

All your generated images in one place. Search by prompt, model, or runtime; filter by frame size and sort order. Each card shows the source model, generation settings, and quick actions to re-run with the same seed or open in Image Studio.

### Chat — *Local AI chat*
![Chat](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/12.%20Chat_v2.png)

A focused chat surface with multi-thread sessions in the left rail, pinned threads, document and image attachments, inline citations, tool-call cards, collapsible reasoning traces, and inline thread renaming. Threads persist across launches, remember the model/runtime profile that produced them, and can be reloaded directly from the thread toolbar.

### HTML Challenge — *Side-by-side HTML generation*

Pick 2 → 4 models, configure each slot's thinking mode (`off` / `auto`), reasoning effort (`low` / `medium` / `high`), and seed independently, then issue one prompt. Every slot streams its HTML response in parallel with a sandboxed live preview underneath the raw text — so you see what each model actually rendered, not just what it wrote.

Built-in HTML validation runs per slot (`valid` / `partial` / `script-error` / `blank-render` / `no-html`) and is fed back from the iframe sandbox so script crashes and blank renders surface immediately. Every run is saved to a persistent history with title + prompt + per-slot manifests; re-open earlier runs, delete, or open / reveal individual slot HTML files on disk. Failed slots support retry and a `continue` / `repair` flow that asks the same model to either resume from the partial response or fix its own broken HTML.

### Server — *OpenAI-compatible local API*
![Server](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/11.%20Server_v2.png)

Start, stop, and inspect a local OpenAI-compatible HTTP server backed by the loaded model. Shows the bind address, warm-pool entries, request count, active connections, LAN exposure, preferred port, auto-start controls, and a remote-test panel with copyable curl commands for `/health`, `/models`, and `/chat/completions`.

### Benchmarks — *Run a new benchmark*
![Benchmarks](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/13.%20Benchmarks_v2.png)

Configure a benchmark run: choose a model, throughput prompt preset or eval dataset, token budget, and decoding parameters, then watch live progress as the runner streams tokens-per-second, time-to-first-token, memory usage, or eval metrics.

![Benchmark running](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/14.%20Benchmark_Running_v2.png)

Live progress while a run is in flight — token/sec, TTFT, current prompt, and memory pressure all update in real time.

![Benchmark complete](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/15.%20Benchmark_Complete_v2.png)

Final report card: throughput, latency percentiles, generation samples, and the exact run parameters — saved automatically to history.

### History — *Compare saved runs*
![Benchmark History](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/16.%20Benchmark_History_v2.png)

Every benchmark you've ever run, side-by-side. Pick two runs and the page diffs them across throughput, latency, and quality metrics — perfect for proving that your new quant actually pays its keep.

### Conversion — *Convert models to MLX format* (macOS only)
![Conversion](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/17.%20Conversion_v2.png)

Apple Silicon only. Point at a Hugging Face checkpoint or local directory and convert it to MLX with optional runtime-specific compression settings. A conversion picker surfaces eligible source models.

![Conversion running](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/18.%20Conversion_Running_v2.png)

Layer-by-layer live progress while the conversion runs — bit budget per block, memory footprint, and a running log tail.

### Prompt Library — *Reusable system prompts*

Search, create, edit, tag, and delete reusable prompt templates, including five built-in starter personas. Any template can be applied to Chat with one click so you can switch between coding, summarization, translation, and other system-prompt presets without copy-paste.

### Plugins — *Extensions and plugin system*

Inspect built-in and external plugins across cache strategies, inference engines, tools, model sources, and post-processors. Plugins can be enabled or disabled from the UI, and external plugins are discovered from the app's plugin directory.

### Logs — *Runtime events*
![Logs](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/19.%20Logs_v2.png)

A live tail of the backend log stream — load events, server requests, errors, and runtime warnings — with level filtering. The first place to look when something feels off.

### Settings — *Directories and defaults*
![Settings](https://raw.githubusercontent.com/cryptopoly/ChaosEngineAI/staging/Screenshots/20.%20Settings_v2.png)

Configure model and cache directories, remote-provider fallbacks, Hugging Face tokens for gated models, data-directory migration, integration snippets for external tools, default launch preferences (cache strategy, FP16 layers, fused attention, context tokens, fit-in-memory toggle), and advanced runtime knobs. Every default in this panel is reused as the starting state for the launch modal.

---

## Download & Install

Head to the [Releases](https://github.com/cryptopoly/ChaosEngineAI/releases/latest) page for signed builds:

| Platform | File | Notes |
|---|---|---|
| **macOS** (Apple Silicon) | `ChaosEngineAI_*_aarch64.dmg` | Signed + notarized |
| **Linux** | `ChaosEngineAI_*_amd64.AppImage` | Portable, in-app updates supported |
| **Linux** (Debian/Ubuntu) | `ChaosEngineAI_*_amd64.deb` | Install via `dpkg`, update via apt |
| **Windows** | `ChaosEngineAI_*_x64-setup.exe` | Unsigned for now — SmartScreen will warn on first run |

From v0.4.21 onward, every install auto-updates from GitHub Releases on launch. Updates are cryptographically signed.

---

## Quick Start (from source)

Prereqs: Rust toolchain, Node 20+, Python 3.11+, and (on macOS) Xcode command-line tools.

```bash
# 1. Install JS dependencies

npm install

# 2. Stage the bundled Python runtime + llama.cpp binaries into src-tauri/
npm run stage:runtime

# 3. Launch the desktop app in dev mode (Tauri shell + Vite HMR)
npm run tauri:dev
```

That's it — the Python backend is spawned by the Tauri shell, the React UI hot-reloads, and you can start exploring.

---

## Architecture

ChaosEngineAI is three cooperating layers:

```
┌─────────────────────────────────────────────────────────┐
│  Tauri shell  (Rust)                                    │
│  ├─ React + TypeScript UI  (src/)                       │
│  ├─ In-app updater (signed releases from GitHub)        │
│  └─ Spawns and supervises the Python backend            │
└─────────────────────────────────────────────────────────┘
                          │  HTTP  /  IPC
                          ▼
┌─────────────────────────────────────────────────────────┐
│  Python backend  (backend_service/)                     │
│  ├─ Workspace + library + warm pool state               │
│  ├─ OpenAI-compatible HTTP server                       │
│  ├─ Engine adapters + plugin system                     │
│  └─ DFlash / DDTree speculative decoding                │
└─────────────────────────────────────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
  │  llama.cpp   │ │ MLX runtime  │ │   vLLM       │
  │  (GGUF)      │ │ (Apple Si)   │ │ (Linux/CUDA) │
  └──────────────┘ └──────────────┘ └──────────────┘
```

- **`src/`** — React 18 + TypeScript UI. Single-window workspace with a sidebar nav covering LLM and image generation screens.
- **`src-tauri/`** — Tauri 2 Rust shell + bundled runtime.
- **`backend_service/`** — Python service that owns model lifecycle, the warm pool, the OpenAI-compatible API, the benchmark runner, and speculative decoding (DFlash + DDTree).
- **`backend_service/routes/`** — FastAPI routes for chat, prompts, compare mode, benchmarks, plugins, images, server controls, and settings.
- **`cache_compression/`** — Pluggable cache/compression strategy system. Ships with native f16 and optional adapters for [TriAttention](https://github.com/WeianMao/triattention) and [TurboQuant](https://pypi.org/project/turboquant-mlx-full/).
- **`dflash/`** — DFlash speculative decoding integration: draft model registry, fuzzy matching for quantized variants, MLX and vLLM backend detection.

---

## Cache Compression Backends

ChaosEngineAI uses a pluggable cache strategy system. Out of the box, models run with the native f16 KV cache (no compression). Install any of the optional backends below and they automatically appear in the **Cache Strategy** selector in the launch modal — no configuration required.

| Backend | Install | Bits | Platforms | Description |
|---|---|---|---|---|
| **Native f16** | Built-in | — | All | Full-precision KV cache. Maximum quality, no compression. |
| **[TriAttention](https://github.com/WeianMao/triattention)** | `./.venv/bin/python3 -m pip install triattention vllm` | 1-4 | Linux + CUDA only (via vLLM) | Transparent KV cache compression integrated into vLLM's scheduler. Not supported on macOS. |
| **[TurboQuant](https://pypi.org/project/turboquant-mlx-full/)** | `./.venv/bin/python3 -m pip install turboquant-mlx-full` (Apple Silicon) or `scripts/build-llama-turbo.sh` (CUDA / Metal via llama.cpp fork) | 1-4 (MLX) / turbo2/3/4 (llama.cpp) | Apple Silicon (MLX), CUDA + Metal (llama.cpp fork) | Hadamard / Walsh-Hadamard rotation-based KV cache compression. The MLX path uses `turboquant-mlx-full` for native MLX caches; the llama.cpp path uses the forked `llama-server-turbo` binary built by `scripts/build-llama-turbo.sh`. |
| **[TeaCache](https://github.com/ali-vilab/TeaCache)** | Built-in (vendored `teacache_forward` patches under `cache_compression/_teacache_patches/`) | n/a (rel_l1_thresh) | Diffusion DiT (FLUX, HunyuanVideo, LTX-Video, CogVideoX, Mochi) | Diffusion-side cache that skips redundant forward passes between adjacent timesteps. Default `rel_l1_thresh=0.4`. |
| **[FBCache](https://github.com/huggingface/diffusers)** | Built-in (diffusers 0.36+ `apply_first_block_cache` hook) | n/a (threshold) | Diffusion DiT (FLUX, SD3.5, Wan, HunyuanVideo, LTX-Video, CogVideoX, Mochi) | Model-agnostic first-block cache for DiTs. Default threshold 0.12. |
| **TaylorSeer** | Built-in (diffusers 0.38+ `TaylorSeerCacheConfig`) | n/a | Diffusion DiT (FLUX, Wan, HunyuanVideo, LTX-Video, CogVideoX, Mochi) | Taylor-series predictor caches skipped forward passes. Surfaced in Image + Video Studio pickers. |
| **MagCache** | Built-in (diffusers 0.38+ `MagCacheConfig`) | n/a | Diffusion DiT (FLUX only without calibration) | Magnitude-aware caching using `FLUX_MAG_RATIOS`. CLI / API only until calibration UX lands. |
| **PyramidAttentionBroadcast (PAB)** | Built-in (diffusers 0.38+ `PyramidAttentionBroadcastConfig`) | n/a | Diffusion DiT | Broadcasts attention across timestep pyramids. Surfaced in Image + Video Studio pickers. |
| **FasterCache** | Built-in (diffusers 0.38+ `FasterCacheConfig`) | n/a | Diffusion DiT | Tight cache windowing for big DiTs. CLI / API only — not surfaced in the picker yet. |

Install optional backends into the backend runtime (`./.venv/bin/python3 -m pip install ...`), then restart ChaosEngineAI. TriAttention is Linux/CUDA only. The TurboQuant MLX path needs `turboquant-mlx-full` from PyPI; the TurboQuant llama.cpp path needs the forked `llama-server-turbo` binary built locally via `scripts/build-llama-turbo.sh`.

> **Removed in FU-030.** ChaosEngine (PCA-based; cryptopoly/ChaosEngine) and RotorQuant (IsoQuant / PlanarQuant rotation) were dropped from the strategy registry. ChaosEngine was eclipsed by KVTC at ICLR 2026 (same PCA approach but 8–32× compression vs 3.7×) and RotorQuant only ever aliased the TurboQuant llama.cpp fork. Persisted user configs that still reference `chaosengine` or `rotorquant` now coerce silently to `turboquant` via `CacheStrategyRegistry.resolve_legacy_id`.

LLM cache strategies are filtered out of the diffusion picker via the `appliesTo` domain field — TeaCache only appears on diffusion models, and the LLM-side strategies only appear in chat. The system is designed so new compression methods can be added as single-file adapters in `cache_compression/` without touching any other code.

---

## Speculative Decoding

ChaosEngineAI ships with three speculative decoding modes that accelerate generation by 1.8-5x with zero quality loss.

### DFlash

A small draft model proposes a block of tokens; the target model verifies them in a single forward pass. Accepted tokens are committed instantly; rejected tokens fall back to standard autoregressive generation.

**Supported model families:** Qwen3 (4B, 8B), Qwen3.5 (4B-35B / 122B-A10B), Qwen3-Coder (4B, 8B), LLaMA 3.1 8B, gpt-oss (20B, 120B), Kimi families (K2.5, K2.6), Gemma-4 (31B, 26B-A4B-it), MiniMax-M2.5 / M2.7.

**Requirements:** Apple Silicon + `dflash-mlx`, or Linux/CUDA + `dflash`. Native f16 cache (no compression). A compatible draft checkpoint is auto-resolved from the [z-lab](https://huggingface.co/z-lab) collection.

### DDTree (Diffusion Draft Tree)

Extends DFlash with tree-structured candidate exploration. Instead of verifying a single draft path, DDTree builds a tree of top-k candidates using a max-probability heap and verifies the entire tree in one forward pass with a tree-structured attention mask. The longest verified path is accepted, yielding higher acceptance rates than linear DFlash.

**Configuration:** Set the **tree budget** slider (0-64) in Runtime Controls when DFlash is enabled. Budget 0 = linear DFlash; higher budgets explore more branches at the cost of additional memory for the attention mask.

DDTree falls back to linear DFlash on failure, and DFlash falls back to standard generation.

### MTPLX (Multi-Token Prediction)

Native MTP speculative decoding for Apple Silicon, powered by the [`mtplx`](https://github.com/youssofal/mtplx) package (Apache 2.0). Instead of pairing a separate draft model with the target, MTPLX uses the target model's own trained MTP heads to propose multiple tokens per forward pass — yielding 1.8-2.2× speedup with zero quality loss on supported models.

**Supported model families:** Qwen3.5, Qwen3.6, DeepSeek V3 / R1, Qwen3-Coder-Next, plus the [Youssofal/MTPLX-Optimized](https://huggingface.co/Youssofal) variants. The full registry lives in `backend_service/inference/_mtp.py` (`MTP_MODEL_MAP` + `_MTP_ALIASES`).

**Install:** One-click "Install MTPLX" button in the Setup tab. Because `mtplx` ships its own forked `mlx` and can't share the main backend `.venv`, the installer provisions an isolated environment at `~/.chaosengine/mtplx-venv/`.

**Routing:** When speculative decoding is enabled and the loaded model has MTP heads, the backend's `_select_engine` auto-routes through MTPLX. If the MTPLX venv is missing or the model is not in the registry, the backend falls back to DFlash, then to standard MLX generation.

---

## Image & Video Generation

ChaosEngineAI includes a full local image and video generation pipeline powered by Hugging Face Diffusers, [mlx-video](https://github.com/Blaizzy/mlx-video) on Apple Silicon, and a [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) engine scaffold for cross-platform builds.

### Image models

| Model | Provider | Speed | Resolution |
|---|---|---|---|
| **FLUX.1 Schnell** | Black Forest Labs | ~4s | 1024x1024 |
| **FLUX.1 Dev** | Black Forest Labs | ~7s | 1024x1024 |
| **Stable Diffusion 3.5 Medium** | Stability AI | ~6s | 1024x1024 |
| **SD 3.5 Large Turbo** | Stability AI | ~2s | 1024x1024 |

### Video models

| Model | Provider | Engine | Notes |
|---|---|---|---|
| **Wan 2.1 T2V 1.3B / 14B** | Alibaba Wan-AI | diffusers (MPS / CUDA) | Text-to-video; 1.3B fits comfortably on a 64 GB Mac |
| **Wan 2.2** | Alibaba Wan-AI | diffusers | T2V successor with catalog metadata fixes |
| **Lightricks LTX-Video** | Lightricks | diffusers | Base LTX-Video; CUDA + MPS paths |
| **Lightricks LTX-2 / LTX-2.3** (distilled + dev) | Lightricks | mlx-video (Apple Silicon) | Subprocess engine via `prince-canuma/LTX-2*` repos |
| **HunyuanVideo** | Tencent | diffusers | TeaCache rescale coefficients vendored |
| **CogVideoX** | Zhipu | diffusers | TeaCache supported |
| **Mochi** | Genmo | diffusers | TeaCache supported |

Discover and download from the **Image Discover** and **Video** tabs, generate from prompts in **Image Studio** or **Video Studio** with aspect ratio, quality presets, negative prompts, prompt enhancer, and seed control, then browse and reuse outputs in **Image Gallery**.

---

## Building a Release

Releases are tag-driven. Push `vX.Y.Z` and the GitHub Actions release workflow builds signed bundles for macOS, Linux, and Windows in parallel, generates the `latest.json` updater manifest, and stages a draft release.

```bash
# locally, if you want to cut a build by hand:

npm run stage:runtime:release
npm run tauri:build
```

Release artifacts land in `src-tauri/target/release/bundle/`.

For an unsigned local macOS app + DMG without Apple signing/notarization or Tauri updater signing configured:

```bash

npm run release:macos -- --skip-sign --skip-notarize
```

That writes the local app + DMG to `releases/macos/`.

---

## Headless Automation — `chaosengine-cli`

Every feature in the desktop app is also reachable from the terminal via `scripts/chaosengine-cli` — a Python 3 wrapper (stdlib only, zero new dependencies) that talks to the same FastAPI backend the Tauri shell uses. It covers 100% of the 125 backend routes through a generic `call <METHOD> <PATH>` dispatcher plus 95 ergonomic typed shortcuts.

**Subcommand categories:** `serve`, `status`, `load`, `unload`, `prompt`, `bench`, `mtplx-install`, `mtplx-status`, `image-generate`, `video-generate`, `session-*`, `setup-*`, `diagnostics-*`, and more. JSON is written to stdout, errors to stderr — composable with `jq` or any other pipeline tool.

The backend on port 8876 must be running (either via the Tauri app or `./scripts/chaosengine-cli serve`).

```bash
# Start the backend (or just leave the Tauri app open)
./scripts/chaosengine-cli serve &

# Load a model with speculative decoding
./scripts/chaosengine-cli load Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed --spec

# Prompt with streaming + per-token metrics
./scripts/chaosengine-cli prompt "Write a Rust quicksort" --max-tokens 256 --stream --metrics

# Benchmark a model with 3 runs
./scripts/chaosengine-cli bench mlx-community/Qwen3.6-35B-A3B-4bit --spec --runs 3
```

Optional PATH symlink so you can call it from anywhere:

```bash
ln -s /Users/dan/ChaosEngineAI/scripts/chaosengine-cli ~/.local/bin/chaosengine-cli
```

**Headless install** (no GUI, no Tauri, no npm, no cargo):

```bash
git clone https://github.com/cryptopoly/ChaosEngineAI.git
cd ChaosEngineAI
python3 -m venv .venv
.venv/bin/pip install -e .
./scripts/chaosengine-cli serve
```

### E2E test suite

`scripts/e2e_test_suite.py` drives the CLI through 8 phases (env probe, chat, compare, HTML challenge, image studio, video studio, setup probes, diagnostics + cleanup) and writes JSON + Markdown reports to `~/.chaosengine/test-results/`. `--smoke` completes in ≤60s; the full sweep finishes in ≤5 min on a warm box. The suite is gated as phase 9 of 9 in `scripts/pre-build-check.sh`. See [`docs/E2E_TESTING.md`](./docs/E2E_TESTING.md) for the full phase-by-phase breakdown.

---

## Project Layout

```
ChaosEngineAI/
├── src/                  React UI (App.tsx is the workspace shell)
├── src-tauri/            Tauri Rust shell + bundled runtime
├── scripts/              Build, release + runtime staging scripts
├── backend_service/      Python backend (engine adapters + HTTP server)
├── cache_compression/    Pluggable cache/compression strategy adapters
├── dflash/               DFlash/DDTree speculative decoding integration
├── tests/                Backend integration tests
├── Screenshots/          UI screenshots used by this README
├── docs/                 Tour GIF + supporting docs
└── ChaosEngineAI_AppIcon.svg
```

---

## License & Credits

ChaosEngineAI is currently distributed under the Apache-2.0 license. See [`LICENSE`](./LICENSE).

See [`THIRD_PARTY_NOTICES.md`](./THIRD_PARTY_NOTICES.md) for optional third-party dependency notes.

Built on the shoulders of [`llama.cpp`](https://github.com/ggerganov/llama.cpp), [Apple MLX](https://github.com/ml-explore/mlx), [Tauri](https://tauri.app/), and the broader open-weights community.

ChaosEngineAI

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README