spark-doctor

KO | EN

gitlite — search

#docker #javascript #android #php #hacktoberfest #java #html #python #nodejs #game #scala #client

★ 101 Open GitHub ↗

Local diagnostic CLI for NVIDIA DGX Spark (GB10). Detects power caps, UMA pressure, thermal risk, CUDA 13/SM_121 wheel mismatches, Docker issues, and validates vLLM/Ollama/llama.cpp/SGLang recipes.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Spark Doctor

Local diagnostic CLI for NVIDIA DGX Spark. Collects system, GPU, memory, Docker, runtime, network, and recipe data, applies DGX Spark-specific rules, and prints a short answer: **what is wrong, why, and what to try next.**

Read-only. No dashboard. No auto-fixes. No telemetry.

## Why

DGX Spark is a new platform. When something goes wrong, the signal is scattered across `nvidia-smi`, `/proc/pressure/*`, `dmesg`, `docker info`, backend logs, forum threads, and Field Diagnostics. Owners hit the same issues repeatedly — GPU stuck in a 14 W low-power state, unified memory pressure stalling inference, thermal shutdowns, Docker runtime not registered, recipes with `tensor_parallel_size` set for a multi-GPU box.

Spark Doctor collects those signals in one command, applies DGX Spark-specific rules, and tells you the likely cause and the next step — in plain English, with the evidence attached.

## Install

```bash
git clone https://github.com/joeynyc/spark-doctor.git && cd spark-doctor
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
```

Requires Python 3.11+.

## Commands

```bash
spark-doctor scan                              # full scan + diagnosis
spark-doctor scan --json scan.json --markdown report.md
spark-doctor doctor --from scan.json           # re-run rules on saved scan
spark-doctor report --from scan.json --format {markdown,forum,github}
spark-doctor recipe check recipe.yaml
spark-doctor anonymize scan.json --out redacted.json
spark-doctor self-test
spark-doctor version
```

Exit codes: `0` clean · `1` warning · `2` critical · `3` collector failure.

## What it detects

| ID | Detects |
|---|---|
| `power.low_draw_under_load` | High GPU utilization with suspiciously low power draw (e.g. 14 W cap). |
| `thermal.shutdown_risk` | GPU temp ≥ 85/90 C or thermal events in logs. |
| `memory.uma_pressure` | Low `MemAvailable`, high memory PSI, or heavy swap use. |
| `runtime.docker_unhealthy` | Docker/NVIDIA container runtime missing or misconfigured. |
| `backend.multiple_heavy_models` | Two or more heavy model backends running concurrently. |
| `cuda.torch_cu12_wheel` | PyTorch built for CUDA 12 on a CUDA 13 / GB10 system. |
| `cuda.libcudart_missing` | Package linked against a CUDA runtime (`libcudart.so.N`) that is not installed. |
| `cuda.sm121_not_in_arch_list` | PyTorch build ships no SM_121 kernels for GB10. |
| `cuda.nvcc_toolkit_mismatch` | `nvcc` on PATH is older than the driver's CUDA version. |
| `backend.kv_cache_oom` | vLLM "No available memory for the cache blocks" — CUDA-graph memory squeezed out the KV cache (fix: `--enforce-eager`), a distinct failure from host `memory.uma_pressure`. |

Recipe validator checks tensor-parallel vs GPU count, container image registry, arm64 compatibility, memory budget, and aggressive `gpu_memory_utilization` / context lengths.

## Privacy

Reports are anonymized by default:

- Hostname, username, and home paths replaced.
- Private IPv4 and MAC addresses redacted unless `--include-network-identifiers`.
- HF, NGC, OpenAI, bearer, JWT, and SSH-key patterns redacted.
- Logs (`dmesg`, `journalctl`) only included with `--include-logs`.

## Safety

No package installs, driver updates, process kills, reboots, clock locking, or power changes. All fixes are instructions.

## Development

```bash
pip install -e '.[dev]'
pytest
```

New rules go in `src/spark_doctor/rules/`, register in `rules/engine.py`, add a fixture in `tests/fixtures/`, add a test.

## License

MIT.

spark-doctor

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README