memra

KO | EN

gitlite — search

#react #go #golang #terraform #python #hacktoberfest #javascript #windows #configuration #elixir #card #api

★ 310 Open GitHub ↗

Rust + CUDA inference engine for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. Serves safetensors and GGUF over an OpenAI-compatible API, with per-device tuned defaults and speculative decode gated byte-identical to plain decode. Hosted instance: inference.tiyuvta.ai

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# memra

[![ci](https://github.com/avifenesh/memra/actions/workflows/ci.yml/badge.svg)](https://github.com/avifenesh/memra/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/avifenesh/memra/main/LICENSE)
![Rust](https://img.shields.io/badge/rust-edition%202024-orange.svg)
![CUDA](https://img.shields.io/badge/CUDA-12.8%20%2F%2013.1-76B900.svg)
![arch](https://img.shields.io/badge/arch-sm__120a%20%2B%20sm__90a-black.svg)
![target](https://img.shields.io/badge/tuned%20for-RTX%20PRO%206000%20%2B%20RTX%205090-76B900.svg)

**Two card classes, tuned properly, instead of every card tuned adequately.** A mechanism that wins on
the RTX PRO 6000 and loses on the RTX 5090 becomes a per-device default keyed on the device — not
a compromise applied to both — so a naked command runs at that card's measured best. Speculative,
graphed and batched serving are each gated byte-identical to plain decode, per request, so the
speed is not paid for in silent output drift.

MIT. Clone it, point it at a supported checkpoint and a Blackwell card, and it serves — no
account, no key. No card? A hosted instance runs at
**[inference.tiyuvta.ai](https://inference.tiyuvta.ai)**; the lab is
**[tiyuvta.ai](https://tiyuvta.ai)**, same author, and its terms and prices are documented there
rather than here.

| | |
|---|---|
| **What** | Inference engine, Rust + CUDA, OpenAI-compatible serving |
| **Tuned for** | RTX PRO 6000 Blackwell (`sm_120a`) and RTX 5090, with a compile-gated H100 (`sm_90a`) lane |
| **Format** | **safetensors first** — the tuned path from here on. GGUF stays supported, and is still the path most models take |
| **Shape** | One model per GPU, replicas across cards behind an admission proxy, PP-2 when a model does not fit one card; placement validated through four GPUs. Tensor parallel, P2P and 3-stage PP are in progress |
| **Author** | [Avi Fenesh](https://github.com/avifenesh) · lab [tiyuvta.ai](https://tiyuvta.ai) · hosted instance [inference.tiyuvta.ai](https://inference.tiyuvta.ai) |
| **Licence** | MIT |

**Jump to** — [Quick start](#quick-start) · [Speed](#speed) · [Which models run](#which-models-run)
· [What the server does](#what-the-server-does) · [Docs](#docs)
· [Request a model](#request-a-model)

---

**Use it** if you serve on a PRO 6000 or a 50-series card and want a naked command to run at
that card's measured best.

**Look elsewhere** if you serve on datacenter parts — the `sm_90a` lane is compile-gated,
secondary, and sets no defaults — or if you want a Python library to import rather than a Rust
binary behind an HTTP surface. If the model you need is not in the table below,
[ask for it](#request-a-model): support is per (model, quantization, drafter), so a format alone
never carries a checkpoint here.

Current tag: [releases/latest](https://github.com/avifenesh/memra/releases/latest). `main` runs
ahead of it. A version number in prose is stale the day after it is written, so this file does
not repeat one.

## Install

The release installer is the shortest path. It selects the published `sm_120a`, `sm_90a`, or
`sm_89` prebuilt, verifies the release checksum, and installs `memra-server`, `run-gen`,
`run-spec`, and `kernel-check`.

```bash
curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
```

Prebuilt binaries require Linux x86_64, glibc 2.35 or newer, NVIDIA driver 580 or newer,
and the CUDA runtime libraries. They do not require `nvcc`. Set `MEMRA_INSTALL_DIR` to
override `~/.local/bin`.

To build all workspace binaries from source:

```bash
git clone https://github.com/avifenesh/memra.git
cd memra
cargo build --release
export PATH="$PWD/target/release:$PATH"
```

Source builds require Rust 1.85 or newer and the CUDA 13.1 toolkit. Architecture is detected
at build time; `MEMRA_CUDA_ARCH` is the documented override.

## Quick start

Point the command at a supported Hugging Face checkpoint directory or GGUF artifact to run one
chat-templated generation. New upstream model onboarding starts from the official checkpoint:

```bash
MEMRA_CHAT=1 run-gen /absolute/path/to/hf-checkpoint \
  --prompt "Explain KV caches in one sentence."
```

`run-gen` also accepts a supported GGUF or an `hf:owner/repo[:file]` spec. The `hf:` form
downloads and caches the selected artifact on first use.

Start the OpenAI-compatible server in one terminal:

```bash
MODEL=/absolute/path/to/hf-checkpoint
MEMRA_MODELS="qwen=$MODEL" memra-server
```

The default bind address is `127.0.0.1:8080`. From another terminal, stream a chat completion:

```bash
curl -sS -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Explain KV caches in one sentence."}],
    "max_tokens": 128,
    "stream": true
  }'
```

The `qwen` value is the alias from `MEMRA_MODELS`. Bearer authentication, multiple model
aliases, MTP drafter attachment, and deployment probes are covered in
[docs/SERVING.md](docs/SERVING.md).

## Speed

Flagship model, RTX PRO 6000 Blackwell, safetensors with the masked-ranks trim — the leading
path. Every figure carries its conditions in
[docs/PERFORMANCE.md](docs/PERFORMANCE.md), which is also where regression re-measures land.

| Metric | Measured |
|---|---|
| TTFT p50, cold | **0.156 s** (c=1) — ≤0.32 s through c=4 |
| TTFT, cached conversation turn | **0.130 s** on a 5.7k-token context (full prefix restore) |
| Decode p50, single stream | RTX PRO 6000: **140 tok/s** (rep medians 138–141) · RTX 5090 Laptop: 75 tok/s (range 71–80) — safetensors with the masked-ranks trim, the leading path; plain decode 75 and 44 |
| Sampled-config throughput | top-p/top-k/min-p requests sample on-device — sampled aggregate **equals greedy** (240–245 tok/s at c=16–32) |
| Aggregate completion | **238–245 tok/s** at c=16, flat to c=32, zero sheds across capacity mixes |
| Sustained soak | 576/576 requests, 0 errors, 0 sheds, −0.27% drift |
| Spec ON/OFF exactness | 8/8 byte-identical; verify gate: zero differing logits at T=1..4, K=1/3/8 |

**Image and video input** on the same endpoint (OpenAI `image_url` /
`video_url` content parts, base64 data URIs; videos as animated GIF, decoded
in-process) — the checkpoint's native ViT tower runs in-engine, gated by a per-token
cosine parity oracle against the HF reference before serving (images min-cos 0.9997,
video 0.99999); vision tokens bill as ordinary prompt tokens.

Tuning on Qwen3.8-27B is **finished on both paths, and safetensors is the leading one** — the
decode figures above are its numbers, measured with the masked-ranks trim. Step-3.7-Flash already
serves on the GGUF path and its current tuning is on **FP8**, not Q8; Gemma-4 31B is tuned on GGUF
with its NVFP4 safetensors path in tuning now. A default only ships for a card class it was
measured on.

Per-model detail lives in [docs/MODELS.md](docs/MODELS.md): the three published rank flavours and
the corpus behind each, why the trim cannot move output, what the prefix cache does and does not
cover on a GDN-hybrid, and two measured results that went against expectation — cache depth alone
swinging throughput 3.1x, and speculative decoding *losing* 4x on cache-carried shapes.

### Same-rig samples

The README carries only representative regression samples. The full boards, methodology,
thermal regime, N, and open cells live in [docs/PERFORMANCE.md](docs/PERFORMANCE.md).

| Model / scenario | Path | tok/s |
|---|---|---|
| Qwen3.5-9B — speculative decode, short / medium / long-agentic prompt classes | GGUF NVFP4, MTP + masked-ranks trim | **281.0 / 211.7 / 187.1** |
| Gemma-4 31B dense — speculative decode at 1.7k context (NVFP4 safetensors tuning in progress) | GGUF Q4_0, MTP K=6 + trim | **97.3** |
| Qwen3.6-35B-A3B MoE — plain decode, tg128 at 512-token context | GGUF IQ4_XS, plain decode | **187.0** |
| Qwen3.6-35B-A3B MoE — the same, at 6.3k-token context | GGUF IQ4_XS, plain decode | **177.1** |

*RTX 5090 Laptop, measured 2026-08-02 — medians of N=5 interleaved reps. The flagship's own numbers are in the table above; these are the rest of the roster on the second card. Conditions, thermal regime, open cells and the engine-to-engine pairings: [docs/PERFORMANCE.md](docs/PERFORMANCE.md); raw per-run logs: [research/tune-data/](research/tune-data/).*

## Which models run

Support is specific to a model, quantization and drafter — never to a format. A checkpoint
loading is not a model being supported: each family has its own tensor census, quantization
arithmetic and topology, and it counts as supported once it has passed its own gates.

**safetensors is the tuned path from here on.** GGUF stays supported and is still the path most
models take. There is no promise that a format keeps getting new work, in either direction — a
GGUF landing tomorrow may or may not be picked up, and a safetensors checkpoint can load
without being on the tuned path. Where a model is supported on both, both keep working.

| Family | Supported on | Tuning now |
|---|---|---|
| Qwen3.8-27B | **both**, both tuned — safetensors NVFP4 and NVFP4+Q5_K GGUF | done; safetensors leads |
| Gemma-4 31B | **GGUF** (QAT Q4_0), supported and tuned | NVFP4 safetensors |
| Step-3.7-Flash 196B-A11B | **GGUF** (IQ4_XS + Q8_0 MTP head, two-card PP-2) | FP8 |
| everything else | **GGUF** | — |

Full roster, per-card targets and the reasoning: **[docs/MODELS.md](docs/MODELS.md)**.

Tensor parallel, P2P and 3-stage pipeline parallel are being built now — named here as
unfinished rather than listed as features.

## What the server does

Table stakes, present and not worth a paragraph each: `/v1/chat/completions` and
`/v1/completions`, blocking or SSE; streaming `tool_calls`, separated reasoning output and
`reasoning_effort` mapped through each model's chat template; `response_format` `json_object`
and `json_schema` enforced during decode; prefix caching with same-window dedup and
session affinity; bearer auth, per-tenant cache boundaries and caps, health and readiness
probes, metrics, graceful drain; `/v1/models` reporting each loaded model's
`supported_parameters`; the model's full 262,144-token window.

Two things here are less common:

- **Vocab-masked draft heads.** The drafter proposes over 32,768 frequency-ranked tokens while
  verification runs the target's full 248,320-token vocabulary — so the mask moves the acceptance
  rate and cannot move output. On a safetensors trunk a published `.txt` of ranks drives the trim
  at load time (`MEMRA_FRSPEC_TRIM=<ranks.txt>`) and no separate draft file is needed; three
  flavours are published, one per workload
  ([docs/MODELS.md](docs/MODELS.md#qwen38-27b-in-detail)).
- **The decode mode is a per-request decision.** Speculation is not free: on shapes carried by
  the prefix cache it costs more than it returns, so the server drops to plain or batched decode
  where those measure faster, PP-2 included — and each mode is byte-identity gated against plain
  decode before it is allowed to serve.

Request fields, response shapes, capability gates, auth, cache semantics and admission
behaviour: [docs/SERVING.md](docs/SERVING.md).

## Docs

| Read this | For |
|---|---|
| [docs/MODELS.md](docs/MODELS.md) | Every supported model, which path it runs on, per-card targets |
| [docs/SERVING.md](docs/SERVING.md) | API contract, caching, auth, admission, PP-2, and operations |
| [docs/PERFORMANCE.md](docs/PERFORMANCE.md) | Full boards, methodology, rigs, gaps, and model qualification |
| [docs/FLAGS.md](docs/FLAGS.md) | Audited environment-variable catalog |
| [ARCHITECTURE.md](ARCHITECTURE.md) | Engine structure and the `sm_120a` implementation ledger |
| [ARCHITECTURE-H100.md](ARCHITECTURE-H100.md) | H100-specific mechanisms and evidence |
| [CONTRIBUTING.md](CONTRIBUTING.md) | Development workflow and GPU correctness gates |
| [docs/decisions/](docs/decisions/) | Why a default, format, target or arm was chosen — and what was rejected, with the measurement that settled it |
| [releases](https://github.com/avifenesh/memra/releases) | **The changelog.** Generated from conventional commits by `tools/changelog.sh` — what changed lives there, not in these docs |
| [docs/RELEASING.md](docs/RELEASING.md) | Versioning, target-rig battery, tags, and publication |

## Request a model

The support list is a series of decisions, not a plan — so the most useful thing you can send is
which model you want served and on what card. Open an issue or a discussion. A concrete
checkpoint with a reason carries more weight than a wishlist, and it gets read.

## Contributing

Issues and PRs are welcome. Start with [CONTRIBUTING.md](CONTRIBUTING.md); validation on a new
GPU should use the [hardware report template](.github/ISSUE_TEMPLATE/hardware-validation.md).

## License

MIT — see [LICENSE](LICENSE).

memra

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README