cascade

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 11 Open GitHub ↗

Extend LLM context windows beyond GPU memory limits with disk-backed KV cache.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

🇬🇧 <a href="README.md">English</a> | 🇨🇳 <a href="README.zh-CN.md">简体中文</a>

<h1 align="center">Cascade</h1>

Disk-backed KV cache for long-context LLM inference. 
 Extend vLLM beyond GPU memory with a Go metadata engine, POSIX/GDS storage backends, and a roadmap toward pooled NVMe clusters.

---

## Why Cascade

LLM serving is increasingly constrained by KV cache memory. GPU HBM is fast but expensive and finite; long prompts, agentic workloads, and repeated retrieval-heavy conversations quickly turn context length into an infrastructure problem.

**Cascade** treats local NVMe as a first-class KV cache tier. It keeps the inference engine API surface small, moves metadata and eviction into a compact Go service, and lets the Python vLLM connector save/load KV tensors through pluggable storage backends:

- **POSIX backend** for portable CPU-bounce-buffer storage.
- **GPUDirect Storage (GDS) backend** for GPU↔NVMe transfer on supported hosts.
- **Cluster roadmap** for RDMA-accessible pooled SSD and GPU-aware scheduling.

The goal is simple: **make long-context inference cheaper and more elastic without changing the model.**

### Why disk?

| | GPU HBM | Local NVMe | Remote NVMe (RDMA) |
|---|---:|---:|---:|
| Capacity | ~80 GB / GPU | 2–30 TB / node | Cluster-scale |
| Latency | ~1 µs | ~10–100 µs | ~100 µs+ |
| Bandwidth | ~2 TB/s | ~7 GB/s | 100–500 GB/s |
| Cost/GB | High | Low | Lower at scale |

NVMe is not a replacement for hot GPU KV. It is a high-capacity tier for cold or reusable KV blocks, enabling operators to trade a small amount of storage latency for a large reduction in HBM pressure.

---

## What works today

| Area | Status | Notes |
|---|---|---|
| vLLM integration | ✅ Working | `KVConnectorBase_V1` connector for vLLM 0.21-style V1 execution. |
| DiskCache engine | ✅ Working | Go service with HTTP API, Pebble metadata, LRU eviction, and persistent block/chunk indexes. |
| Storage backends | ✅ Working | POSIX backend plus GDS/NvFile backend with automatic fallback. |
| Cache-hit validation | ✅ Working | Isolated script starts disk-cache + vLLM and checks retrieved block counters and cached token stats. |
| A100 validation | ✅ Working | Real A100 run with Qwen2.5-7B-Instruct and vLLM 0.21.0. |
| Cluster gateway | 🧪 MVP | OpenAI-compatible routing across healthy vLLM replicas; HA, cache-aware matching, and distributed GC remain planned. |

---

## Validation snapshot

Observed on the project A100 validation host. These numbers are environment samples, not universal performance guarantees.

### Real vLLM + disk-cache smoke

| Item | Value |
|---|---|
| GPU | NVIDIA A100-PCIE-40GB |
| vLLM | 0.21.0 |
| Model | Qwen2.5-7B-Instruct |
| Prompt size | 6,629 prompt tokens (`REQUEST_REPETITIONS=200`, `max_model_len=8192`) |
| First request | `1.704s`, retrieved blocks `0` |
| Second request | `0.199s`, retrieved blocks `28`, cached tokens `6624` |

> Historical note: later inspection showed this T4 host mounted `/mnt/nvfile` on `/dev/sda3` XFS, with cuFile `use_compat_mode=true` and no loaded `nvidia_fs`. The following T4 “GDS” rows prove the NvFile/cuFile code path, not direct GDS DMA; see [benchmark-report.md](./benchmark-report.md) for the corrected 2026-07-23 results.

### T4 + nvfile POSIX benchmark (2026-07-22)

| Item | Value |
|---|---|
| GPU | NVIDIA Tesla T4 (16GB) |
| vLLM | 0.25.1 |
| Model | Qwen2.5-7B-Instruct-AWQ |
| Storage | nvfile (自研高性能存储集群), POSIX backend |
| Warmup TTFT | `8.105s` |
| Query TTFT | `0.390s` |
| Speedup | **20.8x** |
| Cache hit rate | 96% |
### T4 + nvfile GDS benchmark (2026-07-22)

| Item | Value |
|---|---|
| GPU | NVIDIA Tesla T4 (16GB) |
| vLLM | 0.25.1 |
| Model | Qwen2.5-7B-Instruct-AWQ |
| Storage | nvfile (自研高性能存储集群), GDS backend |
| Warmup TTFT | `0.381s` |
| Query TTFT | `0.389s` |
| Speedup (warmup) | **21x vs POSIX** |
| Cache hit rate | 98% |
### LMCache + GDS benchmark (2026-07-22)

| Item | Value |
|---|---|
| GPU | NVIDIA Tesla T4 (16GB) |
| vLLM | 0.25.1 |
| Model | Qwen2.5-7B-Instruct-AWQ |
| Storage | nvfile (自研高性能存储集群), GDS backend |
| Warmup TTFT | `8.242s` |
| Query TTFT | `0.092s` |
| Speedup (query) | **4.2x vs Cascade GDS** |

See [release notes](./docs/release-notes.md) for the full validation command and environment notes.

### POSIX vs GDS storage backend benchmark

| Backend | Selected implementation | Save median | Load median |
|---|---|---:|---:|
| POSIX | `PosixBackend` | `0.066639s` / `480.20 MiB/s` | `0.009055s` / `3534.14 MiB/s` |
| GDS | `NvFileBackend` | `0.020430s` / `1566.36 MiB/s` | `0.005160s` / `6202.14 MiB/s` |

See [A100 storage benchmark](./docs/storage-benchmark-a100.md) for the reproducible marker-aware report.

---

> **⚠️ Important: Disable vLLM's built-in prefix caching when using Cascade.**
>
> You **must** start vLLM with `--no-enable-prefix-caching`. If vLLM's built-in prefix caching is enabled (the default), it will cache KV blocks in GPU memory and **completely bypass Cascade's match interface**, resulting in `External prefix cache hit rate: 0.0%`. The official validation script (`scripts/validate_vllm_disk_cache.sh`) sets `DISABLE_PREFIX_CACHING=1` by default for this reason.

## Architecture

```text
┌──────────────────────────────────────────────────────────────────────┐
│                            vLLM / LLM Engine                         │
│                                                                      │
│   ┌──────────────────────────────────────────────────────────────┐   │
│   │ DiskCache KVConnector (Python)                               │   │
│   │ - scheduler-side prefix/cache-hit lookup                     │   │
│   │ - worker-side KV tensor save/load                            │   │
│   │ - POSIX/GDS backend selection                                │   │
│   └───────────────────────────────┬──────────────────────────────┘   │
└───────────────────────────────────┼──────────────────────────────────┘
                                    │ HTTP metadata + local file I/O
                                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                         Go DiskCache Engine                           │
│                                                                      │
│   ┌────────────────────┐   ┌────────────────────┐   ┌────────────┐  │
│   │ Pebble metadata    │   │ LRU eviction        │   │ HTTP stats │  │
│   │ blocks/sentinels   │   │ capacity control    │   │ diagnostics│  │
│   └────────────────────┘   └────────────────────┘   └────────────┘  │
│                                                                      │
│   ┌──────────────────────────────────────────────────────────────┐   │
│   │ Storage layer                                                │   │
│   │ POSIX today | GDS today | io_uring/RDMA planned              │   │
│   └──────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                        Cluster layer (roadmap)                        │
│ GPU-aware request routing | pooled SSD | RDMA | node registry         │
└──────────────────────────────────────────────────────────────────────┘
```

### Project layout

```text
cascade/
├── adapter/           # Inference engine adapters
│   └── vllm/          # vLLM KVConnector implementation
├── engine/            # Go disk-cache engine
│   ├── cmd/           # disk-cache, c-agent, cluster-server entry points
│   └── pkg/           # cache, metadata, server, storage, cluster packages
├── docs/              # Design docs, benchmark plans, release notes
├── scripts/           # Validation and benchmark scripts
├── test/              # Integration and benchmark helpers
└── images/            # README/community assets
```

---

## Quick start

### 1. Build the engine

```bash
make build-engine
# Output: bin/disk-cache
```

### 2. Run the disk-cache service

```bash
./bin/disk-cache \
  -cache-path /mnt/nvme/kv-cache \
  -metadata-path /tmp/disk-cache-meta \
  -max-size 100GB \
  -listen :9100
```

### 3. Prefer the reproducible vLLM smoke

On a GPU host with vLLM and a local model path:

```bash
MODEL_PATH=/tmp/models/Qwen2.5-7B-Instruct \
VLLM_EXTRA_ARGS='--tensor-parallel-size 1 --max-model-len 8192' \
REQUEST_REPETITIONS=200 \
MAX_TOKENS=8 \
make test-vllm-cache
```

The script builds the engine, starts isolated disk-cache and vLLM services, sends two repeated prompts, and fails if the second request does not retrieve cached KV chunks.

### 4. Manual vLLM integration

```bash
PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" \
vllm serve /path/to/model \
  --no-enable-prefix-caching \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --kv-transfer-config '{
    "kv_connector": "DiskCacheConnector",
    "kv_role": "kv_both",
    "kv_connector_module_path": "adapter.vllm.connector_v21",
    "kv_connector_extra_config": {
      "disk_cache_path": "/mnt/nvme/kv-cache",
      "disk_cache_engine_addr": "http://localhost:9100",
      "target_device": "auto",
      "storage_backend": "auto",
      "disk_cache_chunk_size_tokens": 256
    }
  }'
```

Use `target_device=auto` for multi-GPU safety: the connector saves/loads to the actual KV tensor device unless an explicit device override is required.

---

## Validation and CI targets

```bash
# Go packages and engine tests
make test-go

# Python adapter/helper tests
make test-adapter

# CI-friendly bundle: Go + adapter tests + POSIX storage smoke
make ci

# GPU host validation: CI bundle + POSIX/GDS benchmark
make ci-gpu

# Real vLLM + disk-cache smoke
make test-vllm-cache
```

Useful overrides:

```bash
# Storage smoke on a specific backend/device
STORAGE_BACKEND=gds STORAGE_DEVICE=cuda:0 make test-storage

# Storage backend benchmark
STORAGE_BENCH_BACKENDS=posix,gds \
STORAGE_BENCH_DEVICE=cuda:0 \
STORAGE_BENCH_MARKDOWN=docs/storage-benchmark-a100.md \
make bench-storage
```

---

## Roadmap

### Done

- [x] Go DiskCache engine: Pebble metadata, LRU eviction, HTTP API, stats.
- [x] vLLM `KVConnectorBase_V1` integration for save/load of KV tensors.
- [x] Persistent prefix/chunk metadata with sentinel entries and hit diagnostics.
- [x] POSIX storage backend using safetensors-compatible CPU bounce path.
- [x] GDS/NvFile backend with automatic POSIX fallback.
- [x] A100 validation workflow and marker-aware storage benchmark report.

### Next

- [ ] Better operator-facing docs and deployment examples.
- [ ] GPU-aware scheduling: VRAM/task/load monitoring and smarter dispatch.
- [ ] Pooled SSD design: RDMA-accessible shared NVMe across nodes.
- [ ] Async disk I/O experiments (`io_uring`) and larger benchmark matrix.
- [ ] SGLang adapter and additional inference-engine integration points.

### Later

- [ ] Cluster manager with node registry/discovery.
- [ ] Multi-GPU gang scheduling for tensor-parallel models.
- [ ] Fault tolerance, data migration, and production observability.
- [ ] Helm chart and managed deployment patterns.

---

## Benchmarking

```bash
# 1. Profile a local NVMe drive
python3 scripts/disk-bench.py /mnt/nvme

# 2. Exercise the cache engine over HTTP
python3 scripts/disk-bench-cache.py http://localhost:9100

# 3. Compare POSIX/GDS storage backends on a GPU host
make bench-storage

# 4. Refresh the checked-in A100 benchmark report
STORAGE_BENCH_MARKDOWN=docs/storage-benchmark-a100.md make bench-storage
```

See [Benchmark Plan](./docs/benchmark-plan.md) for inference benchmarking methodology and [A100 Storage Backend Benchmark](./docs/storage-benchmark-a100.md) for the latest checked-in storage report.

---

## Comparison

| Dimension | LMCache | Mooncake | **Cascade** |
|---|---|---|---|
| Primary role | Tiered KV cache | Distributed KV transfer | **Disk-backed KV cache + future pooled SSD** |
| Disk role | Warm tier | Overflow/offload target | **Primary high-capacity KV tier** |
| Metadata | Python/in-memory paths | Master/etcd service | **Pebble-backed Go engine** |
| vLLM integration | Deep integration | Integration plugin | **KVConnectorBase_V1** |
| GPU↔NVMe path | GDS backend exists | Not the main design center | **POSIX + GDS backend abstraction** |
| Cluster direction | Cache tiering | RDMA transfer | **GPU-aware scheduling + pooled NVMe roadmap** |

Cascade is intentionally small and infrastructure-oriented: keep the Python connector thin, keep metadata durable, make storage backends explicit, and evolve from one-node NVMe to pooled SSD clusters.

---

## Documentation

- [Design Document](./DESIGN.md) — architecture and rationale.
- [Benchmark Plan](./docs/benchmark-plan.md) — benchmarking methodology.
- [A100 Storage Backend Benchmark](./docs/storage-benchmark-a100.md) — reproducible POSIX/GDS sample report.
- [Release Notes](./docs/release-notes.md) — latest validation notes.
- [vLLM Baseline Setup](./docs/baseline-vllm-deepseek-v4.md) — reference deployment notes.

---

## Contributing

Cascade is early and practical. Good contributions include:

- Running the validation scripts on different GPUs, SSDs, filesystems, and vLLM versions.
- Improving storage backends, cache-hit diagnostics, or benchmark coverage.
- Hardening deployment, observability, and operational docs.
- Prototyping cluster scheduling, RDMA transfer, or SGLang integration.

Please open an issue or pull request. For major design changes, start with a short proposal.

---

## License

Cascade is **dual-licensed**:

- **Apache 2.0** — free for open-source projects, individual developers, and non-commercial use.
- **Commercial License** — required for embedding in hardware appliances, proprietary products, or commercial solutions.

See [COMMERCIAL_LICENSE.md](./COMMERCIAL_LICENSE.md) for details.

[Apache License 2.0](./LICENSE)

---

LLM inference should not be memory-bound. 
 Cascade — stretch context with NVMe, one cache block at a time.

---

## Community

<img src="./images/qr-community.jpg" width="280" alt="WeChat Cascade community QR code" /> 
 Scan to join the Cascade community and discuss KV cache, long-context inference, and LLM systems engineering.

cascade

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README