gemma4-mtp-dgx-spark

KO | EN

gitlite — search

#react #ai #golang #machine-learning #deep-learning #linux #docker #macos #awesome #kubernetes #awesome-list #pytorch #mcp #openai

gemma4-mtp-dgx-spark

(★ 6)

No description available.

gemma4-mtp-dgx-spark 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Gemma 4 MTP on DGX Spark

Minimal Docker Compose template for running Gemma 4 26B and 31B with vLLM, including MTP/speculative decoding profiles.

## Models

| Profile | Target model | Assistant model | Default port |
| --- | --- | --- | --- |
| 26B | `google/gemma-4-26B-A4B-it` | - | `8100` |
| 26B MTP | `google/gemma-4-26B-A4B-it` | `google/gemma-4-26B-A4B-it-assistant` | `8101` |
| 31B | `google/gemma-4-31B-it` | - | `8102` |
| 31B MTP | `google/gemma-4-31B-it` | `google/gemma-4-31B-it-assistant` | `8103` |

The Docker image can fetch vLLM PR [`41745`](https://github.com/vllm-project/vllm/pull/41745) and overlay Gemma 4 model files into the installed vLLM package during build.

## Why the vLLM Patch Is Applied

Gemma 4 MTP uses assistant checkpoints for speculative decoding. vLLM PR [`41745`](https://github.com/vllm-project/vllm/pull/41745) adds the Gemma 4 MTP model path and proposer support needed to run those assistant models with vLLM.

This template targets DGX Spark / GB10 setups, where the pinned `vllm/vllm-openai:gemma4-0505-arm64-cu130` image may not include that merged support yet. Applying the patch during the Docker build makes the image usable for the 26B MTP and 31B MTP profiles without waiting for a newer published base image.

## Requirements

- Docker with Docker Compose v2
- NVIDIA GPU runtime available to Docker
- Access to the Gemma 4 model repositories on Hugging Face
- Hugging Face token with access to the required models

## Quick Start

Create a local environment file:

```bash
cp .env.example .env
```

Edit `.env` and set:

```env
HF_TOKEN=hf_your_token_here
```

Build the image:

```bash
docker compose build --pull --no-cache --progress=plain
```

Run one profile:

```bash
docker compose up --build gemma4-26b
docker compose up --build gemma4-26b-mtp
docker compose up --build gemma4-31b
docker compose up --build gemma4-31b-mtp
```

Stop all services:

```bash
docker compose down --remove-orphans
```

## Endpoints

Each service exposes an OpenAI-compatible vLLM API on host networking:

| Service | Base URL | Served model name |
| --- | --- | --- |
| `gemma4-26b` | `http://localhost:8100/v1` | `gemma4-26b-a4b` |
| `gemma4-26b-mtp` | `http://localhost:8101/v1` | `gemma4-26b-a4b-mtp` |
| `gemma4-31b` | `http://localhost:8102/v1` | `gemma4-31b` |
| `gemma4-31b-mtp` | `http://localhost:8103/v1` | `gemma4-31b-mtp` |

## Configuration

Copy `.env.example` to `.env` before running. The `.env` file is ignored by Git so local tokens and tuning values are not committed.

### Required

| Variable | Description |
| --- | --- |
| `HF_TOKEN` | Hugging Face token used by vLLM to download model weights. |

### Build and Patch Options

| Variable | Default | Description |
| --- | --- | --- |
| `VLLM_IMAGE` | `vllm/vllm-openai:gemma4-0505-arm64-cu130` | Base vLLM OpenAI-compatible image. |
| `APPLY_VLLM_PR_PATCH` | `1` | When `1`, fetches vLLM PR files and overlays them into the image. |
| `VLLM_PR_NUMBER` | `41745` | vLLM pull request number used for the overlay patch. |
| `VLLM_PATCH_REPO` | `https://github.com/vllm-project/vllm.git` | Git repository used to fetch the vLLM patch. |
| `PATCH_TRANSFORMERS_FOR_MTP` | `1` | When `1`, installs patched or newer Hugging Face dependencies for MTP support. |
| `HF_HUB_VERSION` | `1.13.0` | `huggingface-hub` version installed during the patch step. |
| `REGEX_VERSION` | `2025.10.23` | `regex` package version installed during the patch step. |
| `TOKENIZERS_VERSION` | `0.22.1` | `tokenizers` package version installed during the patch step. |
| `TRANSFORMERS_SOURCE` | `git` | Set to `git` to install Transformers from GitHub; any other value uses a prerelease `transformers>=5.7.0`. |

### vLLM Runtime Defaults

These values apply to every profile unless a service-specific override is set in `compose.yml`.

| Variable | Default | Description |
| --- | --- | --- |
| `HOST` | `0.0.0.0` | Bind address inside the container. |
| `MAX_MODEL_LEN` | `8192` | vLLM `--max-model-len`. |
| `MAX_NUM_BATCHED_TOKENS` | `8192` | vLLM `--max-num-batched-tokens`. |
| `MAX_NUM_SEQS` | `1` | vLLM `--max-num-seqs`. |
| `GPU_MEMORY_UTILIZATION` | `0.82` | vLLM `--gpu-memory-utilization`. |
| `KV_CACHE_DTYPE` | `fp8` | vLLM `--kv-cache-dtype`. |
| `ATTENTION_BACKEND` | `auto` | Optional vLLM attention backend. `auto` leaves the flag unset. |
| `LOAD_FORMAT` | `safetensors` | vLLM `--load-format`. |
| `LANGUAGE_MODEL_ONLY` | `1` | When `1`, passes `--language-model-only`. |
| `REASONING_PARSER` | `gemma4` | vLLM reasoning parser. |
| `ENABLE_AUTO_TOOL_CHOICE` | `1` | When `1`, enables vLLM automatic tool choice. |
| `TOOL_CALL_PARSER` | `gemma4` | Tool call parser used when automatic tool choice is enabled. |
| `PRESERVE_THINKING` | `0` | When `1`, passes chat template kwargs to preserve thinking content. |

### CPU and Compiler Tuning

| Variable | Default | Description |
| --- | --- | --- |
| `OMP_NUM_THREADS` | `16` | OpenMP thread count. |
| `VLLM_CPU_OMP_THREADS_BIND` | `1` | vLLM CPU OpenMP thread binding. |
| `TORCHINDUCTOR_MAX_AUTOTUNE` | `0` | TorchInductor autotune setting. |
| `TRITON_MAX_AUTOTUNE` | `0` | Triton autotune setting. |

### Model IDs

| Variable | Default | Description |
| --- | --- | --- |
| `GEMMA26_MODEL` | `google/gemma-4-26B-A4B-it` | Target model for 26B profiles. |
| `GEMMA31_MODEL` | `google/gemma-4-31B-it` | Target model for 31B profiles. |
| `GEMMA26_ASSISTANT_MODEL` | `google/gemma-4-26B-A4B-it-assistant` | Assistant model for 26B MTP. |
| `GEMMA31_ASSISTANT_MODEL` | `google/gemma-4-31B-it-assistant` | Assistant model for 31B MTP. |

### Ports and Served Names

| Variable | Default | Description |
| --- | --- | --- |
| `GEMMA26_PORT` | `8100` | Host port for `gemma4-26b`. |
| `GEMMA26_MODEL_NAME` | `gemma4-26b-a4b` | Served model name for `gemma4-26b`. |
| `GEMMA26_MTP_PORT` | `8101` | Host port for `gemma4-26b-mtp`. |
| `GEMMA26_MTP_MODEL_NAME` | `gemma4-26b-a4b-mtp` | Served model name for `gemma4-26b-mtp`. |
| `GEMMA31_PORT` | `8102` | Host port for `gemma4-31b`. |
| `GEMMA31_MODEL_NAME` | `gemma4-31b` | Served model name for `gemma4-31b`. |
| `GEMMA31_MTP_PORT` | `8103` | Host port for `gemma4-31b-mtp`. |
| `GEMMA31_MTP_MODEL_NAME` | `gemma4-31b-mtp` | Served model name for `gemma4-31b-mtp`. |

### MTP Settings

| Variable | Default | Description |
| --- | --- | --- |
| `GEMMA26_MTP_NUM_SPECULATIVE_TOKENS` | `4` | Speculative token count for 26B MTP. |
| `GEMMA31_MTP_NUM_SPECULATIVE_TOKENS` | `4` | Speculative token count for 31B MTP. |
| `GEMMA31_MTP_MAX_MODEL_LEN` | `8192` | 31B MTP-specific `--max-model-len`. |
| `GEMMA31_MTP_MAX_NUM_BATCHED_TOKENS` | `12288` | 31B MTP-specific `--max-num-batched-tokens`. |
| `GEMMA31_MTP_MAX_NUM_SEQS` | `1` | 31B MTP-specific `--max-num-seqs`. |
| `GEMMA31_MTP_GPU_MEMORY_UTILIZATION` | `0.72` | 31B MTP-specific GPU memory utilization. |
| `GEMMA31_MTP_LOAD_FORMAT` | `safetensors` | 31B MTP-specific load format. |
| `GEMMA31_MTP_ATTENTION_BACKEND` | `auto` | 31B MTP-specific attention backend. |
| `GEMMA31_MTP_MEM_LIMIT` | `96g` | Docker memory limit for `gemma4-31b-mtp`. |
| `GEMMA31_MTP_MEMSWAP_LIMIT` | `96g` | Docker memory plus swap limit for `gemma4-31b-mtp`. |
| `GEMMA31_MTP_RESTART` | `no` | Docker restart policy for `gemma4-31b-mtp`. |

### Extra vLLM Arguments

| Variable | Description |
| --- | --- |
| `VLLM_EXTRA_ARGS` | Extra arguments appended to every `vllm serve` command. |
| `GEMMA26_EXTRA_ARGS` | Extra arguments appended only to `gemma4-26b`. |
| `GEMMA26_MTP_EXTRA_ARGS` | Extra arguments appended only to `gemma4-26b-mtp`. |
| `GEMMA31_EXTRA_ARGS` | Extra arguments appended only to `gemma4-31b`. |
| `GEMMA31_MTP_EXTRA_ARGS` | Extra arguments appended only to `gemma4-31b-mtp`. |

## Storage

Docker Compose creates two local runtime directories:

| Path | Purpose |
| --- | --- |
| `models/hf` | Hugging Face model cache mounted at `/root/.cache/huggingface`. |
| `logs` | vLLM logs mounted at `/logs`. |

Both paths are ignored by Git.

## License

MIT. See [LICENSE](LICENSE).

gemma4-mtp-dgx-spark

gemma4-mtp-dgx-spark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

gemma4-mtp-dgx-spark

gemma4-mtp-dgx-spark 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README