onnx-llm

KO | EN

gitlite — search

#python #react #golang #go #machine-learning #rust #nodejs #macos #windows #cli #kubernetes #ios #mcp #open-source

(★ 49)

llm deploy project based onnx.

onnx-llm 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![onnx-llm](https://raw.githubusercontent.com/wangzhaode/onnx-llm/main/resource/logo.png)

# onnx-llm
[![License](https://img.shields.io/github/license/wangzhaode/onnx-llm)](https://raw.githubusercontent.com/wangzhaode/onnx-llm/main/LICENSE.txt)

## Build

### Steps

1. Download `onnxruntime` release package from [here](https://github.com/microsoft/onnxruntime/releases).
2. Extract the package to `onnx-llm/3rd_party/onnxruntime`.
3. Compile the project.

### Example
```base
wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.2/onnxruntime-osx-arm64-1.19.2.tgz
tar -xvf onnxruntime-osx-arm64-1.19.2.tgz
mv onnxruntime-osx-arm64-1.19.2 3rd_party/onnxruntime -T
mkdir build && cd build
cmake ..
make -j
```

## Usage

1. Model export using [llm-export](https://github.com/wangzhaode/llm-export)
2. Usage of onnx-llm same as [mnn-llm](https://github.com/wangzhaode/mnn-llm)

```base
(base) ➜  build git:(main) ✗ ./cli_demo qwen2-0.5b-instruct/config.json ../resource/prompt.txt
model path is ../../llm-export/model/config.json
load tokenizer
tokenizer_type = 3
load tokenizer Done
load ../../llm-export/model/llm.onnx ... Load Module Done!
prompt file is ../resource/prompt.txt
Hello! How can I assist you today?
我是来自阿里云的超大规模语言模型，我叫通义千问。
很抱歉，作为AI助手，我无法实时获取和显示当前的天气信息。建议您查看当地的气象预报或应用中的天气查询功能来获取准确的信息。

#################################
prompt tokens num  = 36
decode tokens num  = 64
prefill time = 0.32 s
 decode time = 2.00 s
prefill speed = 112.66 tok/s
 decode speed = 32.07 tok/s
##################################
```

## Reference
- [mnn-llm](https://github.com/wangzhaode/mnn-llm)
- [onnxruntime](https://github.com/microsoft/onnxruntime)

onnx-llm

onnx-llm 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

onnx-llm

onnx-llm 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README