lda-rest

KO | EN

gitlite — search

#hacktoberfest #ai #golang #android #machine-learning #rust #deep-learning #linux #docker #nodejs #windows #cli #awesome-list #pytorch

(★ 10)

REST web service to compute and query Latent Dirichlet Allocation models

lda-rest Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Latent Dirichlet Allocation REST Web Service

This library provides a Python REST Web Service to access a simple pipeline to create and query LDA Models. Created models information are stored in a Mongo database and models file are stored on a shared folder in the host filesystem.

## Project architecture

The system is composed by two Docker containers:

* Python container, contains the API library code. This container is mapped on `localhost` on port `5000`.
* Mongodb container, contains the database server. The container is available with the `db` hostname on port `27017`.

### Run the containers and the application

Download the code from the repository, then **edit the docker-compose.yml file** to update all the mount source directories that will be shared between the containers and the host.

Then, run:  
	
	docker-compose build
	docker-compose up
	docker-compose exec web python db/load_fake_data.py

### Python container composition

The main container is composed by

* a Flask application that exposes some REST routes (see below)
* a library with algorithms to compute and query the LDA model 
* some support tools to load initial data into the associated database
 
### Available APIs

Following a table describing the available APIs:

| Endpoint | Http request | Description | Parameters | 
| --- | --- | --- | --- |
| models/ | GET | Lists all models | - |
| models/ | PUT | Creates a new model w.r.t. the provided parameters | * `model_id`: str, the id of the model to be created; * `number_of_topics`: int, the number of topics to extract; * `language`: 'en', the language of the documents; * `use_lemmer`: bool, true to perform lemmatisation, false to perform stemming; * `min_df`: int, the minimum number of documents that should contain a term to consider it; * `max_df`: float, the maximum percentage of documents that should contain a term to consider it as valid; * `chunksize`: int, the size of a chunk in LDA; * `num_passes`: int, the minimum number of passes through the dataset during learning with LDA;  * `waiting_seconds`: int, the number of seconds to wait before starting the learning; * `data_filename`: str, the filename in the 'data' folder that contains data, the file should contain a json dump of documents each one with `doc_id` and `doc_content` keys; * `data`: json dictionary, a dictionary of documents, containing document_id as key and document_content as value; * `assign_topics`: bool, true to assign topics to the newly created model and to save on db, false to ignore assignments for the learning documents; |  
| models/`<model-id>` | GET | Shows detailed information about model with id `<model-id>` | - | 
| models/`<model-id>`/documents/ | GET | Lists all documents assigned to the model with id `<model-id>` | - |
| models/`<model-id>`/ | DELETE | Delete the model with the specified id, stops the computation if scheduled or performed | - |
| models/`<model-id>`/documents/`<doc-id>` | GET | Shows detailed information about document with id `<doc-id>` in model `<model-id>`| * `threshold`: float, the minimum probability that a topic should have to be returned as associated to the document.| 
| models/`<model-id>`/neighbors/ | GET | Computes and shows documents similar to the specified text.| * `text`: str, the text to categorize; * `limit`: int, the maximum number of similar documents to extract. |
| models/`<model-id>`/documents/`<doc-id>`/neighbors/ | GET | Computes and shows documents similar to the document identified with `<doc-id>`.| * `limit`: int, the maximum number of similar documents to extract. |
| models/`<model-id>`/topics/ | GET | Lists all topics related to the model with id `<model-id>` or extracts topics from a text if `text` is specified. | Only for extract topics from a text: * `text`, str, the text to compute topics for; * `threshold`, float, the min weight of a topic to be retrieved. |
| models/`<model-id>`/topics/ | SEARCH | Computes and returns all topics assigned to the text. | * `text`, str, the text to compute topics for; * `threshold`, float, the min weight of a topic to be retrieved. |
| models/`<model-id>`/topics/`<topic-id>` | GET | Shows detailed information about topic with id `<topic-id>` in model `<model-id>`| * `threshold`: float, the minimum probability that a topic should have to be returned as associated to the document.| 
| models/`<model-id>`/topics/`<topic-id>`/documents | GET | Shows all documents associated to the topic with id `<topic-id>` in model `<model-id>`| * `threshold`: float, the minimum probability of the topic that the document should have to be returned as associated to the topic.| 
| models/`<model-id>`/topics/`<topic-id>`/documents | PUT | Compute topics associated to the provided document (single if `doc_id` and `doc_content` are set, multiple if `documents` is set) in model `<model-id>`| * `documents`: json dictionary, optional, keys are document ids and values are document contents; * `doc_id`, string, optional, the document id (in single case); * `doc_content`, string, optional, the document content; * `save_on_db`, bool, default True, true to save documents and topic assignments on db, False to return and forget.| 
| models/`<model-id>`/topics/`<topic-id>` | PATCH | Update optional information of the topic with id `<topic-id>` in model `<model-id>`| * `label`: str, optional, the topic label. * `description`: str, optional, the optional topic description. |

### Load sample data

To invoke the modules that loads fake data into the database run, from the machine that is running Docker, the following command:

docker-compose exec web python db/load_fake_data.py

### Database

To connect directly to the mongodb instance:

* connect to the machine that hosts docker with ssh (optional: only if it is not the current machine)
* the db is available on host 'db' port 27017

#### Models

When asking for model's detailed information, the required model can be in one of the following statuses:

* `scheduled`, the model computation will start after the specified waiting period
* `computing`, the model computation has been started and is currently running
* `completed`, the model computation is finished and the model is stable
* `killed`, the model computation has been interrupted by an error

#### Languages
The language can be specified during in model creation message. Each model can handle only one language, chosen from:

* `en` for english documents
* `it` for italian documents, stopwords are available in `/app/resources` folder and lemmatisation is performed with `MorphIt`

#### Documents

During model computation it is possible to load documents in two ways:

* load from file: provide the `data_filename` field in the request. The file should be a json file and should be contained in the data folder. The json should be a list of dictionaries, each dictionary represent a document and contains the keys `doc_id` and `doc_content`. For example:

[
			{'doc_id': 'doc_1', 'doc_content': 'doc content 1'}, 
			{'doc_id': 'doc_2', 'doc_content': 'doc content 2'}, 
			{'doc_id': 'doc_3', 'doc_content': 'doc content 3'}
		]

* load directly: provide the documents in the `data` field. This field should contain a dictionary of key:values where keys are document ids and values are document contents.

{
			'doc_1': 'doc content 1', 
			'doc_2': 'doc content 2', 
			'doc_3': 'doc content 3'
		}

#### Useful commands

To build all containers

docker-compose build

To run all containers

docker-compose up

To exec a command within a running container, e.g. load fake data into the mongo database

docker-compose exec web COMANDO ARGS

lda-rest

lda-rest Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

lda-rest

lda-rest Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README