wayang.rs

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #llm #go #machine-learning #rust #linux #java #windows #kubernetes #vue #awesome-list #security

(★ 19)

Lightweight fast LLM.

wayang.rs Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# wayang.rs

CPU-focused **LLM inference** in Rust: load **GGUF** models natively, optional **paged KV cache**, **KV quantization** (contiguous mode), **layer streaming**, and an **OpenAI-compatible** HTTP server.

**[Documentation index](docs/README.md)** · **[API reference](docs/API.md)** · **[User guide](docs/user-guide.md)** · **[ROADMAP](ROADMAP.md)**

## Features

| Area | Details |
|------|---------|
| Models | GGUF; `general.architecture` routing (LLaMA/Mistral/Phi, Qwen2/3 subset); Mistral sliding window |
| Inference | `wayang-core`: GQA attention, RoPE, sampling, prefix-cache API (paged KV) |
| Memory | `eager` (all layers) or `stream` (one layer + prefetch) |
| Server | Axum: chat & text completions, SSE, `/metrics`, bounded inference queue |
| Hub | `hf:org/repo/path/file.gguf` + Hugging Face cache |

## Requirements

- **Rust** (stable, edition 2021)
- A compatible **GGUF** checkpoint (see [docs/models-and-hub.md](docs/models-and-hub.md))

## Quick start

```bash
# Build (recommended for inference)
make release
# Native CPU tuning: make release-native   (see Makefile)

# Run tests
make test

# One-shot chat
cargo run --release -p wayang-cli -- \
  --model /path/to/model.gguf \
  --prompt "Hello!"

# HTTP server (default port 11434)
cargo run --release -p wayang-server -- --model /path/to/model.gguf
```

## Workspace crates

| Crate | Role |
|-------|------|
| [wayang-core](crates/wayang-core) | Engine, GGUF, model forward, KV, sampling |
| [wayang-tokenizer](crates/wayang-tokenizer) | BPE tokenizer from GGUF metadata |
| [wayang-model-manager](crates/wayang-model-manager) | Local paths + Hugging Face downloads |
| [wayang-server](crates/wayang-server) | HTTP API library + binary |
| [wayang-cli](crates/wayang-cli) | CLI: chat, `serve`, `models` |
| [wayang-bench](crates/wayang-bench) | TTFT / tok/s benchmarks |

## Makefile targets

Run `make help` for a list. Common targets: `build`, `release`, `test`, `fmt`, `clippy`, `doc`, `clean`.

## API & docs

- **[docs/API.md](docs/API.md)** — public types and functions by crate (quick reference).
- **Rustdoc:** `make doc` then open `target/doc/wayang_core/index.html`.
- HTTP endpoints: [docs/server-and-api.md](docs/server-and-api.md).

## Contributing

See **[CONTRIBUTING.md](CONTRIBUTING.md)**.

## License

This project is licensed under the **MIT License** — see **[LICENSE](LICENSE)**.

wayang.rs

wayang.rs Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

wayang.rs

wayang.rs Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README