deepseek-v4-flash-2x-spark-1m

KO | EN

gitlite — search

#python #javascript #machine-learning #react #html #deep-learning #pytorch #chatgpt #openai #minecraft #css #reactjs

deepseek-v4-flash-2x-spark-1m

★ 36 Open GitHub ↗

DeepSeek V4 Flash @ 1M token context on 2x NVIDIA DGX Spark — production-tested recipe (45 tok/s decode, real 800K prompts served)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DeepSeek V4 Flash on 2x DGX Spark — 45.5 tok/s @ 1,000,000-token context

> Serving DeepSeek-V4-Flash (FP8, MLA + sparse indexer) with a true 1M-token context window on two DGX Sparks (GB10) wired back-to-back over 200GbE RDMA — 45.5 tok/s single-stream decode, no switch, no rack, no cloud.

## TL;DR

- **What you get:** a complete, production-tested recipe for DeepSeek-V4-Flash at a true 1,000,000-token context on two DGX Sparks — two boxes and a cable on a desk.
- **The numbers:** 45.5 tok/s single-stream decode (MTP n=2), a 2,842,610-token KV pool (2.84x concurrency at 1M ctx), and 924 / 786 tok/s prefill on real 400K / 800K prompts.
- **Going 500K → 1M costs zero decode** (45.3 → 45.5 tok/s), and tool calling is clean.
- **Who it's for:** this is not a benchmark stunt — it runs a 24/7 multi-agent fleet right now. It's a single-user/small-fleet box pair, not a throughput server; `--max-num-seqs 6` is honest sizing.

## Hardware

- 2x NVIDIA DGX Spark (GB10, 128GB unified each = 256GB total)
- 1x QSFP cable, direct ConnectX-7 ↔ ConnectX-7 (200GbE, RoCE/RDMA) — **no switch**
- That's it. Two boxes and a cable on a desk.

## Quick start

Prereqs: the two Sparks are wired and RDMA is up, the model is pre-pulled on both nodes, and you've edited `MASTER_ADDR` / `IFNAME` / `IB_HCA` at the top of the launcher (see [Setup](#setup-detailed)).

**Launch worker first, then head:**

```bash
# On the worker node:
./ds4-1m-launch.sh 1
# Then on the head node:
./ds4-1m-launch.sh 0
```

Wait for the model load + flashinfer autotune (first boot is the slow one), then smoke-test:

```bash
curl http://<head-node>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hello"}]}'
```

## Setup (detailed)

Full launcher: [`launch/ds4-1m-launch.sh`](launch/ds4-1m-launch.sh).

### Weights

- Model: `deepseek-ai/DeepSeek-V4-Flash` (FP8, MLA + sparse indexer).
- Pre-pull it into `~/.cache/huggingface` on **both** nodes — the launcher runs `HF_HUB_OFFLINE=1`.

### Image

- [`aidendle94/sparkrun-vllm-ds4-gb10:production-ready`](https://hub.docker.com/r/aidendle94/sparkrun-vllm-ds4-gb10) — a vLLM fork with b12x MoE kernels for GB10/SM121.

### Network

1. Wire the two Sparks ConnectX-7 to ConnectX-7 and set up RDMA/RoCE (see NVIDIA's [two-Spark playbook](https://build.nvidia.com/spark/connect-two-sparks)). Confirm `ibv_devices` shows your HCA on both nodes.
2. Edit the top of [`launch/ds4-1m-launch.sh`](launch/ds4-1m-launch.sh): set `MASTER_ADDR` to the head node's direct-link IP and the `IFNAME` / `IB_HCA` values to your interface and RDMA device names (`ip addr` / `ibv_devices`).

### Launch

Launch **worker first, then head** (`./ds4-1m-launch.sh 1`, then `./ds4-1m-launch.sh 0`). The flags that matter:

- `--tensor-parallel-size 2 --nnodes 2 --distributed-executor-backend mp` — TP=2 across the two boxes, torch-distributed multiprocessing, **no Ray**.
- `--kv-cache-dtype fp8` — half the KV footprint, no observed quality cost.
- `--speculative-config '{"method":"mtp","num_speculative_tokens":2}'` — DeepSeek MTP, the biggest single decode win.
- `--enable-prefix-caching` — free TTFT wins for agent workloads (see [Troubleshooting](#troubleshooting)).
- `--tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 --enable-auto-tool-choice`.
- `--enable-flashinfer-autotune` + `TORCH_CUDA_ARCH_LIST=12.1a FLASHINFER_CUDA_ARCH_LIST=12.1a`.
- `VLLM_USE_B12X_MOE=1` — the b12x MoE kernels (the reason this image beats stock builds on prefill ~2x).
- NCCL pinned to the RDMA link: `NCCL_IB_HCA`, `NCCL_IB_GID_INDEX=3`, `NCCL_SOCKET_IFNAME` (see [Troubleshooting](#troubleshooting) for the GID gotcha).

### Verify

Model load + flashinfer autotune + first-token warmup is minutes, not seconds. Send a throwaway warmup request before pointing real traffic (or agents) at it, then hit the endpoint:

```bash
curl http://<head-node>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hello"}]}'
```

## Benchmarks

Measured June 9, 2026 on production hardware:

| Metric | Result |
|---|---|
| Decode speed | **45.5 tok/s** single-stream (MTP n=2) |
| Real 400,022-token prompt | served in 433s (**924 tok/s prefill**) |
| Real 800,024-token prompt | served in 1,018s (**786 tok/s prefill**) |
| 33K-token prefill | 31.9s |
| KV cache pool | **2,842,610 tokens** (2.84x concurrency at 1M ctx) |
| Tool calling | clean (deepseek_v4 parser + auto tool choice) |
| Decode cost of going 500K → 1M | **zero** (45.3 → 45.5 tok/s) |

Community-reported numbers on the same recipe (0rand, NVIDIA forum): 30-45 tok/s @ 1M, 89/100 tool-calling eval, "zero fabric delays."

> **Benchmark with YOUR workload, not narrative text.** MTP acceptance (and therefore tok/s) is content-dependent — code and structured text accept far better than prose. Check `num_requests_running` to make sure you're measuring single-stream when comparing to these numbers.

## Configuration

### The whole 1M unlock is 3 flags

Two earlier attempts at 1M on this exact hardware died with `NVRM: NV_ERR_NO_MEMORY` during startup. The knobs that make 1M boot and survive:

| Flag / env | Value | Why |
|---|---|---|
| `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` | `256` | **The OOM killer.** Caps the DeepSeek sparse-indexer logits buffer, which otherwise blows up at long context. |
| `--block-size` | `256` | Fewer, bigger KV blocks — less paging overhead at 1M. |
| `--max-model-len` | `1000000` | The point. (`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` required.) |
| `--max-num-seqs` | `6` | Down from 8 — trades a little batch headroom for KV room. |
| `--gpu-memory-utilization` | `0.82` | The sweet spot. Higher OOMs at this context, lower wastes KV pool. |

If you're coming from a working 500K config with the indexer cap already set, the entire delta is: `max-model-len 500000→1000000`, `max-num-seqs 8→6`, `gpu-memory-utilization 0.8→0.82`.

Other decode knobs: KV cache is `fp8`; speculative decoding is MTP n=2 (n=3 tested, not better).

### 500K baseline / rollback

[`launch/ds4-500k-launch.sh`](launch/ds4-500k-launch.sh) is the identical recipe at 500K ctx / seqs 8 / util 0.80 — our previous stable prod. Same container name, so it stops/replaces in place:

```bash
# Both nodes, worker first:
./ds4-500k-launch.sh 1   # worker
./ds4-500k-launch.sh 0   # head
```

## Troubleshooting

Full bring-up bug list + fixes: [`TROUBLESHOOTING.md`](TROUBLESHOOTING.md). The honest notes:

- **Cold first request** after boot can hit client timeouts (autotune + warmup) — send a throwaway warmup prompt before pointing traffic at it.
- **Prefix-cache decay under multi-hour sustained load** has been reported upstream at larger scale (BlockPool alloc silently disabling cache hits). Watch your hit rate on long-running deployments; restart clears it.
- **Don't use `--rm`** in the docker run — add `--restart=unless-stopped` if you want crash auto-recovery (we do in fleet duty).
- The JSON in `--speculative-config` must stay single-quoted exactly as written or your shell will eat it.
- Node firmware/driver/OS **must match on both Sparks** — version skew tanks prefill before it breaks anything visibly.
- This is a single-user/small-fleet box pair, not a throughput server. `--max-num-seqs 6` is honest sizing.

## Credits & links

This recipe stands on a chain of community work:

- **[@aidendle94](https://hub.docker.com/u/aidendle94)** — the base image + b12x kernel build ("Aiden recipe", Reddit).
- **0rand** (NVIDIA developer forums, [thread 372268](https://forums.developer.nvidia.com/t/372268)) — first proven 1M-token operational config, including the sparse-indexer cap.
- **@11_p** — long-context debugging intel on the big DGX Spark thread ([370309](https://forums.developer.nvidia.com/t/370309)).
- **jasl's vLLM fork lineage** + vLLM PR43447 — the SM121/GB10 enablement this all builds on.
- **@tonyd615 / TonyD2Wild** — production validation: 24/7 multi-agent fleet, the 400K/800K real-prompt runs, this repo.

License: [MIT](LICENSE).

*Maintained by Tony (TonyD2Wild / Tech2Wild) — tested on real hardware, running in production. PRs with measured numbers welcome.*

deepseek-v4-flash-2x-spark-1m

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README