mlapp

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #llm #android #rust #macos #java #windows #awesome #awesome-list #mcp #ios

(★ 72)

ML model serving app based on APIs

mlapp Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MLapp

A web application to serve model based on Fastai v1 library. Application is built on [Molten Framework](https://moltenframework.com/) due to it's inbuilt support for modern tooling: [Swagger](https://swagger.io/), [OpenAPI](https://www.openapis.org/), [Prometheus](https://prometheus.io/) etc.

## Motivation

I wanted to have an application which allow me to serve ML models in real production settings. Application should provide:

* Versioned APIs
* Metrics (prometheus)
* Ablility to run different version of models without unnecassary code changes
* User friendly API documentations
* Input data validation
* Authentication
* Inference device flexibility (CPU or GPU)
* Scale up or down instances based on incoming traffic (Kubernetes)

So that I can focus on my ML model improvment and don't need to bother about how to serve the model in produciton.

## Testing

To test the current model, just run the following commands

### Virtualenv

Run the following commands to set things up. You will need to have python 3.6 or above and `git-lfs` package installed.

```bash
git clone --depth 1 https://github.com/gurvindersingh/mlapp.git
cd mlapp
virtualenv .ve
source .ve/bin/activate
pip install -r requirements.txt
pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
gunicorn app:app
```

Now you can visit [localhost:8000](http://localhost:8000/_docs) to see Swagger UI and test APIs out. Or you can run `curl` as

```bash
curl -X POST "http://localhost:8000/v1/predict" -F "file=@loadtest/keeshond.jpg"
```

You can change the `config.json` based on your own model, classes and architecture to see predictions from your own model. Make sure to have your `.pth` located with correct name under `models` directory.

### Docker

```bash
CPU: docker run -ti --rm -p 8000:8000 gurvin/mlapp:v0.1.0-cpu
GPU & CPU: docker run -ti --rm -p 8000:8000 gurvin/mlapp:v0.1.0
```

## Configuration

Application uses `config.json` to provide various options.

```raw
{
    "model_name": "v1",     -> Specify the model from multiple models in your config as which one you want to run with this instance
    "token": "",            -> To enable authenticaiton to you APIs
    "v1": {                 -> Model name and its corresponding config. These can be different based on your model
        "device": "cpu",    -> Device on which to run inference (cpu, cuda)
        "url": "",          -> URL to fetch the weights file .pth from. If empty app assumes local file under models dir
        "arch": "resnet34", -> Model Architecutre
        "size": 224,        -> Image size
        "classes": []       -> Classes used during training from which prediciton will happen
    }
}
```

## Design

Application design is simple. `app.py` set up the routes and required methods to get data in and out. Following routes are exposed by application.

```
/               -> Returns health information and serve as liveness check for application
/_docs          -> Serves Swagger UI to provide API docs and user friendly API testing
/_schema        -> json formatted API spec in OpenAPI standard
/metrics        -> Endpoint to scrape metrics by Prometheus
/v1/predict     -> Receives end user data to make prediction on
/v1/feedback    -> User can provide feedback on prediction if there was any errors
```

API is **versioned** which makes evolution of our application possible without introducing any breaking changes to downstream applications. `app.py` **validates data** according to schema descibed in `model.py` file using Molten framework. The current application provides a simple CNN model which was described in Lesson 1 of [Fast v1](https://github.com/fastai/fastai/) course using schema shown below for `predict` endpoint. For more details see `model.py` or [Molten docs](http://moltenframework.com/)

```python
@schema
class ModelData():
    file: UploadedFile
```

Idea is that based on a given model, you can update the schema in this class and let Molten take care of validation for you and provide a user friendly docs and test interface as shown.

![alt text](https://raw.githubusercontent.com/gurvindersingh/mlapp/master/figures/swagger-docs.png "Swagger Docs UI")

To test API either using `curl` or swagger UI can be used.

![alt text](https://raw.githubusercontent.com/gurvindersingh/mlapp/master/figures/predict-ui.png "Predict Test")

## Metrics

Application uses Molten support for [Prometheus](https://prometheus.io/) to export request count, latency, GC Time and few other metrics.

![alt text](https://raw.githubusercontent.com/gurvindersingh/mlapp/master/figures/grafana.png "Prometheus Metrics Dashboard")

## Deployment

[Kubernetes](https://kubernetes.io/) deployment file is provided in `deployment` folder which also have support for [HorizontalPodAutoscalar](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) to scale the application based on CPU usage and later scale down when not needed.

## Load testing

## TODO

* Add support for ONNX

mlapp

mlapp Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

mlapp

mlapp Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README