quant.cpp

(★ 399)

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

File Explorer

Download Latest Version (.zip)
  • .dockerignore
  • .gitignore
  • CHANGELOG.md
  • CLAUDE.md
  • CMakeLists.txt
  • CODE_OF_CONDUCT.md
  • CONTRIBUTING.md
  • docker-compose.yml
  • Dockerfile
  • LICENSE
  • Makefile
  • program.md
  • quant.h
  • README.ko.md
  • README.md
  • requirements.txt
  • ROADMAP.md
  • score.sh
  • SECURITY.md

# Installation Guide

1. Get the code
git clone https://github.com/quantumaikr/quant.cpp

Downloads the entire project code from GitHub to your computer.

cd quant.cpp

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended
Prerequisites
  • Python 3 Python is required to use pip.
pip install quantcpp

Installs the package published on PyPI directly — no need to clone the source.

pip install huggingface_hub

Installs the package published on PyPI directly — no need to clone the source.

✅ After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

3. Docker

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker build -t quant.cpp .

Builds a runnable image based on the Dockerfile.

docker run -v ./models:/models quant.cpp /models/SmolLM2-135M-Instruct-Q8_0.gguf -p "hello" -k uniform_4b -v q4

Runs the built image as an actual container.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

Pulled directly from this repo's README.

4. CMake

Medium
Prerequisites
  • Git Needed to download the project code from GitHub.
  • CMake The tool used to generate build configuration.
  • C/C++ 컴파일러 Windows needs Visual Studio (Community edition, free), macOS needs Xcode Command Line Tools, Linux needs the gcc/g++ package.
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j$(nproc)

Type this command into your terminal and run it.

cmake -B build && cmake --build build # build libllama

Type this command into your terminal and run it.

cc app.c -o app -lm -lpthread # that's it — no cmake, no framework

Type this command into your terminal and run it.

cmake -B build -DTQ_BUILD_SERVER=ON && cmake --build build

Type this command into your terminal and run it.

✅ Check that an executable was created inside the build folder, then run it directly (e.g. ./build/app_name).

Pulled directly from this repo's README.

5. Node.js

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Node.js Node.js must be installed to use npm. The LTS version is recommended.
cd wasm

This project's files live in a subfolder, so move into it first.

npm install

Downloads and installs the libraries listed in package.json.

npm start

Starts the development/run server.

✅ After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.

6. Python

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.
pip install quantcpp

Installs the package published on PyPI directly — no need to clone the source.

pip install huggingface_hub

Installs the package published on PyPI directly — no need to clone the source.

python3 -m http.server 8080 # Serve locally

Runs the Python script (or module).

cd bindings/python && pip install .

Moves into the project folder you just downloaded.

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

7. Make

Medium
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Make Usually pre-installed on Linux/macOS. On Windows, install separately (e.g. via MSYS2 or WSL).
make

Compiles the code based on the generated build configuration to produce an executable.

✅ If it finishes without errors, it worked. Try running the generated executable directly.

Today's picks

RuView ★ 90,357

π RuView turns commodity WiFi signals into real-time spatial intelligence, vital sign moni

Rust
JavaGuide ★ 157,804

Java 面试 & 后端通用面试指南，覆盖计算机基础、数据库、分布式、高并发、系统设计与 AI 应用开发

JavaScript
browser-use ★ 109,428

🌐 Make websites accessible for AI agents. Automate tasks online with ease.

Python
Awesome-Hacking ★ 118,448

A collection of various awesome lists for hackers, pentesters and security researchers
HelloGitHub ★ 171,210

:octocat: 分享 GitHub 上有趣、入门级的开源项目。Share interesting, entry-level open source projects on Gi

Python
create-react-app ★ 103,294

Set up a modern web app by running one command.

JavaScript
// repository documentation