llama-toolboxes-cockpit - GitLite

KO | EN

gitlite — search

#hacktoberfest #typescript #llm #go #machine-learning #nodejs #windows #kubernetes #awesome-list #vue #pytorch #ios #openai #open-source

llama-toolboxes-cockpit

(★ 100)

No description available.

파일 탐색기

최종 버전 다운로드 (.zip)

- - auto-version.yml
- .gitkeep
- - models.json
  - toolboxes.json
- __init__.py
- benchmark_runner.py
- config.py
- main.py
- model_manager.py
- server_runner.py
- toolbox_manager.py
- ubatch_calibration.py
- ubatch_profiles.py
- widgets.py
- test_shipped_defaults.py
- test_ubatch_calibration.py
- test_ubatch_commands.py
- test_ubatch_profiles.py
- test_vision_projectors.py
.gitignore
AGENTS.md
demo.gif
pyproject.toml
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/kyuz0/llama-toolboxes-cockpit

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd llama-toolboxes-cockpit

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. 공식 설치 스크립트

쉬움 추천

사전 준비물

pipx Python으로 만든 CLI 도구를 격리된 환경에 설치해주는 도구입니다.

pipx install git+https://github.com/kyuz0/llama-toolboxes-cockpit.git

pipx로 격리된 환경에 CLI 도구를 설치합니다.

✅ 설치 후 새 터미널을 열고, 프로그램의 버전 확인 명령(예: --version)으로 정상 설치됐는지 확인하세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Python

쉬움

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install .

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Llama Toolboxes Cockpit

A modern Terminal User Interface (TUI) for managing Llama.cpp toolbox containers and GGUF models across multiple AMD GPU platforms.

![Llama Cockpit Demo](https://raw.githubusercontent.com/kyuz0/llama-toolboxes-cockpit/main/demo.gif)

## Supported Platforms

| Platform | GPU Architecture | Resources |
| :--- | :--- | :--- |
| **AMD Strix Halo** | Ryzen AI Max (gfx1151) | [Website](https://strix-halo-toolboxes.com) · [GitHub](https://github.com/kyuz0/amd-strix-halo-toolboxes) · [Docker Hub](https://hub.docker.com/r/kyuz0/amd-strix-halo-toolboxes) |
| **AMD Radeon R9700** | Radeon AI PRO R9700 (gfx1201) | [GitHub](https://github.com/kyuz0/amd-r9700-ai-toolboxes) · [Docker Hub](https://hub.docker.com/r/kyuz0/amd-r9700-toolboxes) |
| **Intel Arc B70** | Intel Xe-HPG / Battlemage | [GitHub](https://github.com/kyuz0/intel-b70-ai-toolboxes) · [Docker Hub](https://hub.docker.com/r/kyuz0/intel-b70-ai-toolboxes) |

Each platform ships its own set of pre-built containers (ROCm + Vulkan backends). The cockpit lets you switch between platforms on the fly — the active choice is persisted across sessions.

## Features

- **Multi-Platform Support**: Switch between AMD hardware platforms from the banner. Each platform has its own registry, toolbox images, and backend configurations.
- **Interactive Toolboxes**: Create, enter, update, or batch-delete Llama.cpp CLI containers via Toolbx with Podman or Distrobox with Podman/Docker. The cockpit selects a compatible backend for the host.
- **Server Mode**: Launch a Llama.cpp OpenAI-compatible inference server directly from a container image — pick engine, image, model, context size, and extra args from the UI.
- **RDMA/RoCE**: Detect InfiniBand devices for Strix Halo interactive containers and native Podman/Docker server runs.
- **Benchmark Mode**: Measure fixed prefill and generation workloads at identical configurable starting KV depths, saving raw `llama-bench` JSONL and a combined CSV.
- **Model Manager**: Scan your local `~/models` directory for GGUF files, download curated models from Hugging Face, and manage sharded multi-file models.
- **Update Checker**: Check Docker Hub for newer image builds and batch-update toolboxes in one action.

## Installation

Install via `pipx` for an isolated environment:

```bash
# If you don't have pipx installed:
# Ubuntu/Debian: sudo apt install pipx
# Fedora: sudo dnf install pipx
# Arch: sudo pacman -S python-pipx

pipx install git+https://github.com/kyuz0/llama-toolboxes-cockpit.git
```

## Usage

```bash
llama-cockpit
```

Interactive containers use Toolbx with Podman when that combination is
available on Fedora-family hosts. Distrobox is preferred on Debian/Ubuntu and
supports either Podman or Docker. If both engines are installed and you want a
Docker-backed Distrobox, launch with:

```bash
DBX_CONTAINER_MANAGER=docker llama-cockpit
```

### Benchmark methodology

Benchmark mode produces a long-context throughput profile rather than a single
headline score. Here, **depth** means the number of tokens already present in
the KV cache, not model-layer depth.

For every selected model and toolbox, the cockpit runs two `llama-bench`
series over the same configurable starting KV depths:

| Series | Starting state | Timed workload | Default measured span |
| :--- | :--- | :--- | :--- |
| Prefill | `d` tokens already cached | Append a 2,048-token prompt chunk | `d` to `d + 2,048` |
| Generation | `d` tokens already cached | Decode 128 tokens sequentially | `d` to `d + 128` |

With the defaults, `d` is `0, 8192, 16384, ..., 65536`, and each point is
measured three times. `llama-bench` fills or restores the KV cache to `d`
before starting the timer, so that setup work is excluded. A prefill point at
32K therefore answers "how quickly can the next 2,048 prompt tokens be
processed after 32K tokens are already cached?" It does not measure the time
needed to ingest the first 32K tokens. Likewise, a generation point at 32K is
the average throughput while decoding the following 128 tokens.

Plot `starting_depth` on the x-axis and `avg_ts` (average tokens per second) on
the y-axis, with separate lines for `prefill` and `generation`. These curves
show how prompt ingestion and sequential decoding change as the KV cache grows.
The CSV also includes `stddev_ts` and the individual `samples_ts` values.

The benchmark uses synthetic token IDs and measures model execution only. It
does not include model loading, tokenization, sampling, server or network
overhead, streaming, concurrency, or output quality, so it should not be read
as end-to-end chat latency. Also note that the maximum setting is a **starting**
depth: with the defaults, the deepest prefill point ends at 67,584 tokens and
the deepest generation point ends at 65,664 tokens.

Raw results are saved as one JSONL file per model, toolbox, and series. The
cockpit combines them into `curve_summary.csv`; it does not render the graph
itself. Existing non-empty result files are skipped, so use a fresh results
directory when changing depths, workload sizes, repetitions, or extra
`llama-bench` arguments.

### Ubatch calibration

Calibrate one model/backend before comparing toolboxes:

```bash
llama-cockpit-calibrate-ubatch \
  --toolbox llama-rocm-7.14 \
  --model ~/models/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q8_K_XL.gguf
```

The command tests ubatches `256`, `512`, `1024`, and `2048` using the normal
nine-depth prefill curve and three repetitions. It selects the complete
candidate with the best depth-normalized throughput. Raw JSONL/log files remain
under the configured benchmark-results directory, and the measurements plus
selected value are recorded in
`~/.config/llama-cockpit/ubatch-calibrations.json`.

Benchmark and server modes use the recorded value automatically for the same
platform, backend family, and exact model filename. A benchmark ubatch override
or explicit server `-ub` argument still takes precedence. Image digests are not
part of the calibration key.

The cockpit ships these measured Strix Halo defaults, used when no local
calibration exists:

| Model catalog entry | ROCm | Vulkan |
| --- | ---: | ---: |
| DeepSeek V4 Flash 0731 | 2048 | 1024 |
| Qwen 3.6 27B MTP | 1024 | 256 |
| Qwen 3.6 35B A3B MTP | 2048 | 2048 |

Local calibration overrides these values. An explicit benchmark ubatch or
server `-ub` argument still overrides both.

### Configuration

User preferences are stored in `~/.llama-cockpit.conf`:

```json
{
    "active_platform": "strix-halo",
    "models_dir": "~/models",
    "benchmark_results_dir": "~/llamacpp_toolboxes_bench_results"
}
```

### Adding a New Platform

To add support for a new GPU platform, add a new entry to `src/assets/toolboxes.json` under the `platforms` array:

```json
{
  "id": "my-platform",
  "name": "My GPU Platform",
  "description": "Short description of the hardware",
  "registry": "docker.io/username/my-toolboxes",
  "groups": [
    {
      "name": "Official Toolboxes",
      "toolboxes": [
        {
          "name": "my-llama-rocm-7.2.2",
          "tag": "rocm-7.2.2",
          "description": "ROCm 7.2.2 backend",
          "supports_load_mode": true,
          "engine_args": ["--device", "/dev/dri", "..."]
        }
      ]
    }
  ]
}
```

Set `supports_load_mode` to `true` when the image's `llama-server` and
`llama-bench` support `--load-mode`; otherwise omit it to retain the legacy
`--no-mmap` / `-mmp` arguments. No code changes are required — the cockpit
picks up new platforms automatically.