spark-doctor - GitLite

KO | EN

gitlite — search

#javascript #typescript #android #machine-learning #rust #docker #nodejs #cli #kubernetes #awesome-list #vue #security #pytorch #ios

spark-doctor

(★ 101)

Local diagnostic CLI for NVIDIA DGX Spark (GB10). Detects power caps, UMA pressure, thermal risk, CUDA 13/SM_121 wheel mismatches, Docker issues, and validates vLLM/Ollama/llama.cpp/SGLang recipes.

File Explorer

Download Latest Version (.zip)

- spark-doctor-hero.png
- - __init__.py
  - cli.py
  - models.py
  - shell.py
- - cuda_env_healthy.json
  - cuda_libcudart_failure.json
  - cuda_wheel_mismatch.json
  - docker_cdi_no_named_runtime.json
  - docker_hook_no_named_runtime.json
  - docker_no_runtime_no_hook_no_cdi.json
  - healthy_minimal.json
  - kv_cache_oom.json
  - memory_pressure.json
  - power_limited_14w.json
  - recipe_ok.yaml
  - recipe_tp_too_high.yaml
  - thermal_shutdown_like.json
- __init__.py
- test_cli_smoke.py
- test_gpu_sampler.py
- test_recipe_validator.py
- test_redaction.py
- test_reports.py
- test_rules_cuda.py
- test_rules_kv_cache.py
- test_rules_memory.py
- test_rules_power.py
- test_rules_runtime.py
.gitignore
CLAUDE.md
LICENSE
pyproject.toml
README.md

# Installation Guide

1. Get the code

git clone https://github.com/joeynyc/spark-doctor

Downloads the entire project code from GitHub to your computer.

cd spark-doctor

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

python3 -m venv .venv && source .venv/bin/activate

Runs the Python script (or module).

pip install -e .

Installs the Python libraries listed in requirements.txt (or similar).

pip install -e '.[dev]'

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Spark Doctor

Local diagnostic CLI for NVIDIA DGX Spark. Collects system, GPU, memory, Docker, runtime, network, and recipe data, applies DGX Spark-specific rules, and prints a short answer: **what is wrong, why, and what to try next.**

Read-only. No dashboard. No auto-fixes. No telemetry.

<p align="center">
  <img src="assets/spark-doctor-hero.png" alt="Spark Doctor open-source diagnostic CLI for NVIDIA DGX Spark" width="100%">
</p>

## Why

DGX Spark is a new platform. When something goes wrong, the signal is scattered across `nvidia-smi`, `/proc/pressure/*`, `dmesg`, `docker info`, backend logs, forum threads, and Field Diagnostics. Owners hit the same issues repeatedly — GPU stuck in a 14 W low-power state, unified memory pressure stalling inference, thermal shutdowns, Docker runtime not registered, recipes with `tensor_parallel_size` set for a multi-GPU box.

Spark Doctor collects those signals in one command, applies DGX Spark-specific rules, and tells you the likely cause and the next step — in plain English, with the evidence attached.

## Install

```bash
git clone https://github.com/joeynyc/spark-doctor.git && cd spark-doctor
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
```

Requires Python 3.11+.

## Commands

```bash
spark-doctor scan                              # full scan + diagnosis
spark-doctor scan --json scan.json --markdown report.md
spark-doctor doctor --from scan.json           # re-run rules on saved scan
spark-doctor report --from scan.json --format {markdown,forum,github}
spark-doctor recipe check recipe.yaml
spark-doctor anonymize scan.json --out redacted.json
spark-doctor self-test
spark-doctor version
```

Exit codes: `0` clean · `1` warning · `2` critical · `3` collector failure.

## What it detects

| ID | Detects |
|---|---|
| `power.low_draw_under_load` | High GPU utilization with suspiciously low power draw (e.g. 14 W cap). |
| `thermal.shutdown_risk` | GPU temp ≥ 85/90 C or thermal events in logs. |
| `memory.uma_pressure` | Low `MemAvailable`, high memory PSI, or heavy swap use. |
| `runtime.docker_unhealthy` | Docker/NVIDIA container runtime missing or misconfigured. |
| `backend.multiple_heavy_models` | Two or more heavy model backends running concurrently. |
| `cuda.torch_cu12_wheel` | PyTorch built for CUDA 12 on a CUDA 13 / GB10 system. |
| `cuda.libcudart_missing` | Package linked against a CUDA runtime (`libcudart.so.N`) that is not installed. |
| `cuda.sm121_not_in_arch_list` | PyTorch build ships no SM_121 kernels for GB10. |
| `cuda.nvcc_toolkit_mismatch` | `nvcc` on PATH is older than the driver's CUDA version. |
| `backend.kv_cache_oom` | vLLM "No available memory for the cache blocks" — CUDA-graph memory squeezed out the KV cache (fix: `--enforce-eager`), a distinct failure from host `memory.uma_pressure`. |

Recipe validator checks tensor-parallel vs GPU count, container image registry, arm64 compatibility, memory budget, and aggressive `gpu_memory_utilization` / context lengths.

## Privacy

Reports are anonymized by default:

- Hostname, username, and home paths replaced.
- Private IPv4 and MAC addresses redacted unless `--include-network-identifiers`.
- HF, NGC, OpenAI, bearer, JWT, and SSH-key patterns redacted.
- Logs (`dmesg`, `journalctl`) only included with `--include-logs`.

## Safety

No package installs, driver updates, process kills, reboots, clock locking, or power changes. All fixes are instructions.

## Development

```bash
pip install -e '.[dev]'
pytest
```

New rules go in `src/spark_doctor/rules/`, register in `rules/engine.py`, add a fixture in `tests/fixtures/`, add a test.

## License

MIT.