rust-dl-webserver

KO | EN

gitlite — search

#python #llm #go #rust #linux #nodejs #cli #awesome-list #vue #security #mcp #openai #ios #open-source

rust-dl-webserver

(★ 34)

🦀 Example of serving deep learning models in Rust with batched prediction

rust-dl-webserver 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# rust-dl-webserver

This project provides an example of serving a deep learning model with batched prediction using Rust.
In particular it runs a GPT2 model from [rust-bert](https://github.com/guillaume-be/rust-bert) to generate text based on input context.

## Features

- Batched prediction using [`batched-fn`](https://github.com/epwalsh/batched-fn) when GPU is detected.
- Back pressure mechanism that will return a 503 status code if the server gets back-logged with too many requests.

## Running the server

In order for the server to make use of your GPU (if you have one available) you'll need to compile it against the right
version of the LibTorch C++ library, which you can download from
[https://pytorch.org/get-started/locally/](https://pytorch.org/get-started/locally/).

After downloading, unzip the file and then run the server with

```bash
make run LIBTORCH=/path/to/libtorch
```

If you don't have a GPU available, just run

```bash
make run
```

Now in a separate terminal you can send several requests in to the server at once:

```bash
curl -d '{"text":"Hello, World!"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
curl -d '{"text":"Stay at home"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
curl -d '{"text":"Wash your hands"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
curl -d '{"text":"Do not touch your face"}' \
    -H "Content-Type: application/json" \
    http://localhost:3030/generate &
```

The logs from the server should look something like this:

![server output](https://raw.githubusercontent.com/epwalsh/rust-dl-webserver/master/img/server_output.png)

rust-dl-webserver

rust-dl-webserver 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

rust-dl-webserver

rust-dl-webserver 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README