KO
|
EN
gitlite — search
Search
#javascript
#python
#hacktoberfest
#react
#ai
#typescript
#llm
#go
#golang
#android
#machine-learning
#rust
#deep-learning
#linux
batching
★ 9
Open GitHub ↗
Dynamic Batching for Deep Learning Serving
Download README (.md)
Explore Similar Repositories
jacktris
:
a tetris-like block game implemented in Jack language
PlatformerTutorial
:
A simple platformer made in Unity 3D for demonstrating game feel and tuning
syllabi
:
Computer Science Syllabi Directory of Hunter College
dscamp_public
:
Materials for Data Science camp
symfony-upload-multiple
:
Live Coding : Upload d'images multiples avec Symfony 4 et 5
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
batching
?
Download (.md)
# Dynamic Batching for Deep Learning Serving [](https://golang.org/) [](https://goreportcard.com/report/github.com/kemingy/batching) [](https://godoc.org/github.com/kemingy/batching)  [](https://github.com/kemingy/batching/blob/master/LICENSE) [Ventu](https://github.com/kemingy/ventu) already implement this protocol, so it can be used as the worker for deep learning inference. ## Attention **This project is just a proof of concept. Check the [MOSEC](https://github.com/mosecorg/mosec) for production usage.** ## Features * dynamic batching with batch size and latency * invalid request won't affects others in the same batch * communicate with workers through Unix domain socket or TCP * load balancing If you are interested in the design, check my blog [Deep Learning Serving Framework](https://kemingy.github.io/blogs/deep-learning-serving/). ## Configs ```shell script go run service/app.go --help ``` ``` Usage app: -address string socket file or host:port (default "batch.socket") -batch int max batch size (default 32) -capacity int max jobs in the queue (default 1024) -host string host address (default "0.0.0.0") -latency int max latency (millisecond) (default 10) -port int service port (default 8080) -protocol string unix or tcp (default "unix") -timeout int timeout for a job (millisecond) (default 5000) ``` ## Demo ```shell script go run service/app.go python examples/app.py python examples/client.py ```
