Overload

KO | EN

gitlite — search

#ai #llm #go #machine-learning #rust #linux #docker #windows #awesome #kubernetes #vue #security #ios #mcp

(★ 17)

UNQUANTIZED Mistral 7b on ~2GB of RAM

Overload 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Overload

Run UNQUANTIZED Mistral 7b Instruct on about 2gb of RAM

This repository provides a minimal and memory-efficient way to run the Mistral-7B Instruct v0.3 model on low-resource devices (e.g. CPUs with limited RAM). It loads model weights layer-by-layer directly from disk, avoiding the need to hold the entire model in memory at once. The goal of this project is to allow anyone to run even 100B+ models on almost any modern-ish (Last 10 years or so) computer.

🔧 Features

- Layer-by-layer loading/inference from safetensors model shards

- Optional automatic preloading of layers into available RAM and/or GPU VRAM

- GPU acceleration supported (with user-selectable VRAM safety buffer + RAM safety buffer)

- Currently known to be compatible with Mistral-7B Instruct v0.3 from Hugging Face (More and larger models coming soon)

- CPU-only mode still supported, no GPU required

- Minimal dependencies and clean CLI interface

Explanation Videos:

https://youtu.be/53Do4whfrqE?si=2ODPnr8G8HFL9-Q6

https://youtu.be/yVNQHMtITfk?si=FpgJtQlO0lIDdBqa

📦 Requirements

Python 3.8+

4GB of RAM Minimum (More, or added gpu, directly decreases time per token)

Mistral 7B-Instruct v0.3 - https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

See requirements.txt for packages

🚀 Usage on PC

Clone this repo to your device/computer.

Download the Mistral-7B-Instruct-v0.3 files into the ./mistral_7b_instruct/ folder (The mistral_7b_instruct folder itself should be in whichever folder you have the Overload python script). You need the numbered .safetensors files (NOT the consolidated safetensors file!!!), plus all the other non-weight files

Install dependencies:

pip install -r requirements.txt

Run overload.py in a terminal window

🚀 Usage on Raspberry Pi 5

Clone this repo to your device/computer.

Download the Mistral-7B-Instruct-v0.3 files into the ./mistral_7b_instruct/ folder (The mistral_7b_instruct folder itself should be in whichever folder you have the Overload python script). You need the numbered .safetensors files (NOT the consolidated safetensors file!!!), plus all the other non-weight files

Then in a terminal, navigate to the folder where the Overload.py script exists, and run this command to install all dependencies for Pi5:

bash pi5_setup.sh

Follow the instructions from the install script to run Overload.

Overload

Overload 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Overload

Overload 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README