minigpt4.cpp - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #llm #golang #android #machine-learning #deep-learning #docker #java #pytorch #mcp #ios

minigpt4.cpp

(★ 573)

Port of MiniGPT4 in C++ (4bit, 5bit, 6bit, 8bit, 16bit CPU inference with GGML)

File Explorer

Download Latest Version (.zip)

- - run.yml
- minigpt4-demo1.gif
- webui_demo.png
- CMakeLists.txt
- main.cpp
- - llama.png
  - story.png
- colab_webui.ipynb
- convert.py
- minigpt4_library.py
- quantize.py
- quantize_all.py
- requirements.txt
- webui.py
.gitignore
CMakeLists.txt
LICENSE
minigpt4.cpp
minigpt4.h
README.md

# Installation Guide

1. Get the code

git clone https://github.com/Maknee/minigpt4.cpp

Downloads the entire project code from GitHub to your computer.

cd minigpt4.cpp

Moves into the project folder you just downloaded.

2. CMake

Medium Recommended

Prerequisites

Git Needed to download the project code from GitHub.
CMake The tool used to generate build configuration.
C/C++ 컴파일러 Windows needs Visual Studio (Community edition, free), macOS needs Xcode Command Line Tools, Linux needs the gcc/g++ package.

cmake .

Type this command into your terminal and run it.

cmake --build . --config Release

Type this command into your terminal and run it.

✅ Check that an executable was created inside the build folder, then run it directly (e.g. ./build/app_name).

Pulled directly from this repo's README.

3. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# minigpt4.cpp

<a href='https://huggingface.co/spaces/maknee/minigpt4.cpp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'>
[![Quickstart in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Maknee/minigpt4.cpp/blob/master/minigpt4/colab_webui.ipynb)

Inference of [MiniGPT4](https://github.com/Vision-CAIR/MiniGPT-4) in pure C/C++.

## Description

The main goal of `minigpt4.cpp` is to run minigpt4 using 4-bit quantization with using the [ggml](https://github.com/ggerganov/ggml) library.

## Demo

![minigpt1](https://raw.githubusercontent.com/Maknee/minigpt4.cpp/master/assets/webui_demo.png)

![minigpt1](https://raw.githubusercontent.com/Maknee/minigpt4.cpp/master/assets/minigpt4-demo1.gif)

## Usage

### 1. Clone repo

**Requirements**: [git](https://gitforwindows.org/)

```bash
git clone --recursive https://github.com/Maknee/minigpt4.cpp
cd minigpt4.cpp
```

### 2. Getting the library

#### Option 1: Download precompiled binary

##### Windows / Linux / MacOS

Go to [Releases](https://github.com/Maknee/minigpt4.cpp/releases) and extract `minigpt4` library file into the repository directory.

#### Option 2: Build library manually

##### Windows

**Requirements**: [CMake](https://cmake.org/download/), [Visual Studio](https://visualstudio.microsoft.com/) and [Git](https://gitforwindows.org/)

```commandline
cmake .
cmake --build . --config Release
```

`bin\Release\minigpt4.dll` should be generated

##### Linux

**Requirements**: CMake (Ubuntu: `sudo apt install cmake`)

```bash
cmake .
cmake --build . --config Release
```

`minigpt4.so` should be generated

##### MacOS

**Requirements**: CMake (MacOS: `brew install cmake`)

```sh
cmake .
cmake --build . --config Release
```

`minigpt4.dylib` should be generated

**Note:** If you build with opencv (allowing features such as loading and preprocessing image within the library itself), set `MINIGPT4_BUILD_WITH_OPENCV` to `ON` in `CMakeLists.txt` or build with `-DMINIGPT4_BUILD_WITH_OPENCV=ON` as a parameter to the cmake cli.

### 3. Obtaining the model

#### Option 1: Download pre-quantized MiniGPT4 model

Pre-quantized models are available on Hugging Face ~ [7B](https://huggingface.co/datasets/maknee/minigpt4-7b-ggml/tree/main) or [13B](https://huggingface.co/datasets/maknee/minigpt4-13b-ggml/tree/main).

Recommended for reliable results, but slow inference speed: [minigpt4-13B-f16.bin](https://huggingface.co/datasets/maknee/minigpt4-13b-ggml/blob/main/minigpt4-13B-f16.bin)

#### Option 2: Convert and quantize PyTorch model

**Requirements**: [Python 3.x](https://www.python.org/downloads/) and [PyTorch](https://pytorch.org/get-started/locally/).

Clone the [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4) repository and perform the setup

```sh
cd minigpt4
git clone https://github.com/Vision-CAIR/MiniGPT-4.git
cd MiniGPT-4
conda env create -f environment.yml
conda activate minigpt4
```

Download the pretrained checkpoint in the [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4) repository under `Checkpoint Aligned with Vicuna 7B` or `Checkpoint Aligned with Vicuna 13B` or download them from [Huggingface link for 7B](https://huggingface.co/datasets/maknee/minigpt4-7b-ggml/blob/main/pretrained_minigpt4_7b.pth) or [13B](https://huggingface.co/datasets/maknee/minigpt4-13b-ggml/blob/main/pretrained_minigpt4.pth)

Convert the model weights into ggml format

##### Windows

7B model
```commandline
cd minigpt4
python convert.py C:\pretrained_minigpt4_7b.pth --ftype=f16
```

13B model
```commandline
cd minigpt4
python convert.py C:\pretrained_minigpt4.pth --ftype=f16
```

##### Linux / MacOS

7B model
```sh
python convert.py ~/Downloads/pretrained_minigpt4_7b.pth --outtype f16
```

13B model
```sh
python convert.py ~/Downloads/pretrained_minigpt4.pth --outtype f16
```

`minigpt4-7B-f16.bin` or `minigpt4-13B-f16.bin` should be generated

#### 4. Obtaining the vicuna model

#### Option 1: Download pre-quantized vicuna-v0 model

Pre-quantized models are available on [Hugging Face](https://huggingface.co/datasets/maknee/ggml-vicuna-v0-quantized/tree/main)

Recommended for reliable results and decent inference speed: [ggml-vicuna-13B-v0-q5_k.bin](https://huggingface.co/datasets/maknee/ggml-vicuna-v0-quantized/blob/main/ggml-vicuna-13B-v0-q5_k.bin)

#### Option 2: Convert and quantize vicuna-v0 model

**Requirements**: [Python 3.x](https://www.python.org/downloads/) and [PyTorch](https://pytorch.org/get-started/locally/).

Follow the [guide from the MiniGPT4](https://github.com/Vision-CAIR/MiniGPT-4/blob/main/PrepareVicuna.md) to obtain the vicuna-v0 model.

Then, clone llama.cpp

```sh
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake .
cmake --build . --config Release
```

Convert the model to ggml

```sh
python convert.py <path-to-model>
```

Quantize the model

```sh
python quanitize <path-to-model> <output-model> Q4_1
```

#### 5. Running

Test if minigpt4 works by calling the following, replacing `minigpt4-13B-f16.bin` and `ggml-vicuna-13B-v0-q5_k.bin` with your respective models

```sh
cd minigpt4
python minigpt4_library.py minigpt4-13B-f16.bin ggml-vicuna-13B-v0-q5_k.bin
```

##### Webui

Install the requirements for the webui

```sh
pip install -r requirements.txt
```

Then, run the webui, replacing `minigpt4-13B-f16.bin` and `ggml-vicuna-13B-v0-q5_k.bin` with your respective models

```sh
python webui.py minigpt4-13B-f16.bin ggml-vicuna-13B-v0-q5_k.bin
```

The output should contain something like the following:

```sh
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
```

Go to `http://127.0.0.1:7860` in your browser and you should be able to interact with the webui.