diffusiongemma-26b-a4b-nvfp4-dgx-spark

KO | EN

gitlite — search

#react #typescript #go #android #rust #deep-learning #docker #java #windows #kubernetes #security #ios #mcp #openai

diffusiongemma-26b-a4b-nvfp4-dgx-spark

(★ 9)

No description available.

diffusiongemma-26b-a4b-nvfp4-dgx-spark Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DiffusionGemma 26B-A4B NVFP4 vLLM Template

Docker Compose template for serving `nvidia/diffusiongemma-26B-A4B-it-NVFP4` with vLLM behind an OpenAI-compatible API.

The template is designed for NVIDIA GPU hosts such as DGX Spark-class machines, while keeping the model, vLLM image, memory limits, port, and serving flags configurable through `.env`.

## Features

- vLLM OpenAI-compatible server
- Single Docker Compose workflow with host networking
- Hugging Face model cache mounted at `./models/hf`
- Runtime settings collected in `.env.example`

## Requirements

- Linux host with NVIDIA GPU support
- Docker Engine with Docker Compose v2
- NVIDIA Container Toolkit
- Hugging Face account and token with access to the model
- Sufficient GPU memory and system memory for the selected serving parameters

The model weights and model license are provided by NVIDIA on Hugging Face. This repository's code is MIT licensed, but model usage is governed by the upstream model terms:

https://huggingface.co/nvidia/diffusiongemma-26B-A4B-it-NVFP4

## Quick Start

Create a local environment file:

```bash
cp .env.example .env
```

Edit `.env` and set:

```bash
HF_TOKEN=hf_your_token_here
```

Start the server:

```bash
docker compose up -d
```

The first startup builds the local image and lets vLLM download the model into `./models/hf`.

Follow logs:

```bash
docker compose logs -f
```

Check the OpenAI-compatible model endpoint:

```bash
curl http://127.0.0.1:8000/v1/models
```

The default endpoint is:

```text
http://127.0.0.1:8000/v1
```

From another machine, replace `127.0.0.1` with the server host IP address.

## Configuration

Most settings live in `.env`.

| Variable | Default | Description |
| --- | --- | --- |
| `HF_TOKEN` | empty | Hugging Face token used to download and serve the model. |
| `VLLM_IMAGE` | `vllm/vllm-openai:gemma` | Base vLLM image used by the Dockerfile. |
| `IMAGE_NAME` | `diffusiongemma-26b-a4b-nvfp4-vllm:latest` | Locally built image name. |
| `CONTAINER_NAME` | `diffusiongemma-26b-a4b-nvfp4` | Docker container name. |
| `MODEL_ID` | `nvidia/diffusiongemma-26B-A4B-it-NVFP4` | Hugging Face model repository. |
| `SERVED_MODEL_NAME` | `diffusiongemma-26b-a4b-nvfp4` | Model name exposed through the OpenAI API. |
| `HOST` | `0.0.0.0` | Bind address inside the container. |
| `PORT` | `8000` | API port. |
| `MAX_MODEL_LEN` | `262144` | vLLM maximum context length. |
| `MAX_NUM_SEQS` | `4` | Maximum concurrent sequences. |
| `MAX_NUM_BATCHED_TOKENS` | `32768` | Maximum batched tokens. |
| `GPU_MEMORY_UTILIZATION` | `0.85` | vLLM GPU memory utilization target. |
| `ATTENTION_BACKEND` | `TRITON_ATTN` | vLLM attention backend. |
| `VLLM_EXTRA_ARGS` | empty | Additional arguments appended to `vllm serve`. |
| `MEM_LIMIT` | `112g` | Container memory limit. |
| `MEMSWAP_LIMIT` | `112g` | Container memory plus swap limit. |

Lower `MAX_MODEL_LEN`, `MAX_NUM_SEQS`, `MAX_NUM_BATCHED_TOKENS`, or `GPU_MEMORY_UTILIZATION` if the model does not fit on your hardware.

## Docker Commands

```bash
docker compose up -d          # Build if needed and start the server
docker compose logs -f        # Follow logs
docker compose ps             # Show container status
docker compose down           # Stop and remove the container
docker compose up -d --build  # Rebuild after changing Dockerfile or image args
```

## Repository Layout

## Notes

- `network_mode: host` is intended for Linux GPU hosts. On non-Linux systems, adjust `compose.yml` to use explicit port mappings.
- The vLLM image tag and runtime flags are based on the upstream model card recommendations for this checkpoint.
- Do not commit `.env`, downloaded model files, or logs.

## License

MIT License. See [LICENSE](LICENSE).

diffusiongemma-26b-a4b-nvfp4-dgx-spark

diffusiongemma-26b-a4b-nvfp4-dgx-spark Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

diffusiongemma-26b-a4b-nvfp4-dgx-spark

diffusiongemma-26b-a4b-nvfp4-dgx-spark Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README