chatbot-api

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #typescript #llm #docker #macos #java #awesome #cli #pytorch #mcp #ios

(★ 9)

A simple backend implemented with fastapi for deploying large language model chatbot locally

chatbot-api Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# chatbot-api

SEE [examples.ipynb](examples.ipynb) for request examples.

Now support:

* [llama](https://huggingface.co/decapoda-research/llama-7b-hf). `cfgs/llama-7b.json`
* [llama](https://huggingface.co/decapoda-research/llama-7b-hf) with [lora](https://huggingface.co/tloen/alpaca-lora-7b). `cfgs/llama-7b-lora.json`
* [chatglm](https://huggingface.co/THUDM/chatglm-6b). `cfgs/chatglm-6b.json`
* [InstructGLM](https://github.com/yanqiangmiffy/InstructGLM). `cfgs/chatglm-6b-alpaca-lora.json`
* [blip2chatglm](https://huggingface.co/Xipotzzz/blip2zh-chatglm-6b). `cfgs/blip2zh-chatglm-6b.json`.

# Setup

```
conda create -n llmapi python=3.8
conda activate llmapi
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt
```

# Run

```
uvicorn src:app --reload
```

chatbot-api supports model scheduling:

1. idle model instances will be closed
2. new model instances will be created if too many concurrent requests

You can modify [sched_config.json](sched_config.json) to change the scheduling strategy and model instances.

A typical config is:

```json
{
    "idle_check_period": 120,               // check idle models and close them every 120 seconds
    "models": {
        "blip2zh-chatglm-6b": {             // modelname should be the same as the config filename under cfgs/
            "max_instances": 1,             // at most 1 instance will be created
            "idle_time": 3600,              // if no request for 1 hours, the instance will be closed
            "create_threshold": {           // if 5 requests request blip2zh-chatglm-6b in 5 seconds,
                "n_requests": 5,            //    1 more instance will be created (not exceeding max_instances)
                "delay": 5
            }
        }
    }
}
```

# Format

## Request format

```json
{
  "model": "chatglm-6b",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": true,
  "max_tokens": 1024,
}
```

## Response format

A typical response:

```json
{
    "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I help you today?"}}]
}
```

You may refer to [examples.ipynb](examples.ipynb) for more examples.

chatbot-api

chatbot-api Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

chatbot-api

chatbot-api Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README