kafka-ml
Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)
File Explorer
Download Latest Version (.zip)- backend.yml
- build.yml
- federated_backend.yml
- federated_data_control_logger.yml
- federated_model_control_logger.yml
- federated_tensorflow_model_training.yml
- frontend.yml
- kafka_control_logger.yml
- pthexecutor.yml
- pytorch_model_inference.yml
- pytorch_model_training.yml
- tensorflow_model_inference.yml
- tensorflow_model_training.yml
- tfexecutor.yml
- job_manifest_generator.py
- configurations.py
- datasources.py
- deployments.py
- inferences.py
- iot_devices.py
- models.py
- training_results.py
- __init__.py
- admin.py
- apps.py
- models.py
- routing.py
- serializers.py
- test_views.py
- tests.py
- urls.py
- utils.py
- websockets.py
- Context.sol
- ERC20.sol
- IERC20.sol
- IERC20Metadata.sol
- __init__.py
- asgi.py
- create_blockchain_token.py
- settings.py
- urls.py
- wsgi.py
- __init__.py
- __init__.py
- __init__.py
- __init__.py
- Dockerfile
- nginx.conf
- Dockerfile
- manage.py
- README.md
- requirements.txt
- start.sh
- __init__.py
- avro_inference.py
- avro_sink.py
- federated_online_raw_sink.py
- federated_raw_sink.py
- online_raw_sink.py
- raw_sink.py
- README.md
- requirements.txt
- sink.py
- eurosat_dataset_inference_example.py
- eurosat_dataset_training_example.py
- README.md
- requirements.txt
- mnist_dataset_federated_incremental_training_example.py
- mnist_dataset_federated_training_example.py
- mnist_dataset_unsupervised_federated_incremental_training_example.py
- mnist_dataset_unsupervised_federated_training_example.py
- mnist_sample_input_format.json
- requirements.txt
- __init__.py
- data_scheme.avsc
- HCOPD_data_stream_producer.py
- HCOPD_Dataset.csv
- HCOPD_inference.py
- label_scheme.avsc
- README.md
- requirements.txt
- MLGPARK_Dataset.csv
- MLGPARK_dataset_inference_example.py
- MLGPARK_dataset_training_example.py
- README.md
- requirements.txt
- mnist_dataset_inference_example.py
- mnist_dataset_inference_example_v2.py
- mnist_dataset_online_training_example.py
- mnist_dataset_training_example.py
- mnist_dataset_unsupervised_online_training_example.py
- mnist_dataset_unsupervised_training_example.py
- requirements.txt
- visualization.json
- autoexec.be
- DatasetFiltered.csv
- no2_dataset_training_example.py
- README.md
- requirements.txt
- README.md
- requirements.txt
- so2sat_dataset_inference_example.py
- so2sat_dataset_training_example.py
- ampClamp_autoexec.be
- ampClamp_sendMqtt.be
- nilm_dataset_training_example.py
- README.md
- requirements.txt
- README.md
- requirements.txt
- vgg16_inference_example.py
- vgg16_training_example.py
- __init__.py
- __init__.py
- admin.py
- apps.py
- models.py
- serializers.py
- urls.py
- views.py
- __init__.py
- asgi.py
- settings.py
- urls.py
- wsgi.py
- __init__.py
- __init__.py
- __init__.py
- Dockerfile
- manage.py
- README.md
- requirements.txt
- start.sh
- Dockerfile
- federated_data_control_logger.py
- federated_data_control_logger.sh
- README.md
- requirements.txt
- Dockerfile
- federated_model_control_logger.py
- federated_model_control_logger.sh
- README.md
- requirements.txt
- .gitkeep
- blockchain_federated_training.py
- classic_federated_training.py
- config.py
- decoders.py
- Dockerfile
- federated_blockchainSingleClassicTraining.py
- federated_distributedClassicTraining.py
- federated_distributedIncrementalTraining.py
- federated_mainTraining.py
- federated_singleClassicTraining.py
- federated_singleIncrementalTraining.py
- federated_training.py
- FederatedKafkaMLAggregationSink.py
- KafkaModelEngine.py
- README.md
- requirements.txt
- start.sh
- utils.py
- federated-backend-deployment.yaml
- federated-backend-service.yaml
- federated-data-control-logger.yaml
- federated-kafkaml-configmap.yaml
- federated-model-control-logger.yaml
- role.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- README.md
- README.md
- app.e2e-spec.ts
- app.po.ts
- protractor.conf.js
- tsconfig.json
- configuration-list.component.css
- configuration-list.component.html
- configuration-list.component.spec.ts
- configuration-list.component.ts
- configuration-view.component.css
- configuration-view.component.html
- configuration-view.component.spec.ts
- configuration-view.component.ts
- confirm-dialog.component.css
- confirm-dialog.component.html
- confirm-dialog.component.spec.ts
- confirm-dialog.component.ts
- datasource-dialog.component.css
- datasource-dialog.component.html
- datasource-dialog.component.spec.ts
- datasource-dialog.component.ts
- datasource-list.component.css
- datasource-list.component.html
- datasource-list.component.spec.ts
- datasource-list.component.ts
- deployment-list.component.css
- deployment-list.component.html
- deployment-list.component.spec.ts
- deployment-list.component.ts
- deployment-view.component.css
- deployment-view.component.html
- deployment-view.component.spec.ts
- deployment-view.component.ts
- inference-iot-view.component.css
- inference-iot-view.component.html
- inference-iot-view.component.spec.ts
- inference-iot-view.component.ts
- inference-list.component.css
- inference-list.component.html
- inference-list.component.spec.ts
- inference-list.component.ts
- inference-view.component.css
- inference-view.component.html
- inference-view.component.spec.ts
- inference-view.component.ts
- iot-devices-list.component.css
- iot-devices-list.component.html
- iot-devices-list.component.spec.ts
- iot-devices-list.component.ts
- iot-devices-view.component.css
- iot-devices-view.component.html
- iot-devices-view.component.spec.ts
- iot-devices-view.component.ts
- header.component.css
- header.component.html
- header.component.spec.ts
- header.component.ts
- sidenav-list.component.css
- sidenav-list.component.html
- sidenav-list.component.spec.ts
- sidenav-list.component.ts
- model-list.component.css
- model-list.component.html
- model-list.component.spec.ts
- model-list.component.ts
- model-view.component.css
- model-view.component.html
- model-view.component.spec.ts
- model-view.component.ts
- plot-view.component.css
- plot-view.component.html
- plot-view.component.spec.ts
- plot-view.component.ts
- result-list.component.css
- result-list.component.html
- result-list.component.spec.ts
- result-list.component.ts
- configuration.service.spec.ts
- configuration.service.ts
- datasource.service.ts
- deployment.service.spec.ts
- deployment.service.ts
- inference.service.ts
- iot-devices.service.ts
- model.service.spec.ts
- model.service.ts
- result.service.ts
- visualization-ws.service.spec.ts
- visualization-ws.service.ts
- configuration.model.ts
- deployment.model.ts
- filter.ts
- inference.model.ts
- iot-inference.model.ts
- iotdevice.model.ts
- max-char.pipe.spec.ts
- max-char.pipe.ts
- ml.model.ts
- visualization.component.css
- visualization.component.html
- visualization.component.spec.ts
- visualization.component.ts
- app-routing.module.ts
- app.component.css
- app.component.html
- app.component.spec.ts
- app.component.ts
- app.module.ts
- .gitkeep
- env.js
- env.template.js
- environment.prod.ts
- environment.ts
- favicon.ico
- index.html
- main.ts
- polyfills.ts
- styles.css
- test.ts
- .dockerignore
- .editorconfig
- angular.json
- browserslist
- Dockerfile
- karma.conf.js
- nginx-custom.conf
- package-lock.json
- package.json
- README.md
- start.sh
- tsconfig.app.json
- tsconfig.json
- tsconfig.spec.json
- tslint.json
- classification.png
- configure-deployment.svg
- configure-distributed-deployment.png
- create-configuration.svg
- create-distributed-configuration.png
- create-distributed-model.png
- create-model-pytorch.svg
- create-model-tensorflow.svg
- create-model.png
- deploy-configuration.png
- deploy-distributed-configuration.png
- deploy-federated-configuration.png
- deploy-incremental-configuration-1.png
- deploy-incremental-configuration-2.png
- deploy-inference.svg
- deploy-unsupervised-configuration.png
- distributed-deploy-inference.png
- distributed-training-metrics.png
- distributed-training-results.png
- pipeline_.png
- plot-view.svg
- regression.png
- training-metrics.png
- training-metrics.svg
- training-results.png
- view_results.svg
- Dockerfile
- logger.py
- README.md
- requirements.txt
- start.sh
- backend-deployment.yaml
- backend-service.yaml
- frontend-deployment.yaml
- frontend-service.yaml
- kafka-control-logger-deployment.yaml
- kafkaml-configmap.yaml
- pth-executor-deployment.yaml
- pth-executor-service.yaml
- role.yaml
- tf-executor-deployment.yaml
- tf-executor-service.yaml
- kustomization.yaml
- kafka-deployment.yaml
- kafka-service.yaml
- namespace.yaml
- .gitignore
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- kustomization.yaml
- README.md
- app.py
- Dockerfile
- README.md
- requirements.txt
- start.sh
- .gitkeep
- app.py
- decoders.py
- Dockerfile
- README.md
- requirements.txt
- start.sh
- utils.py
- config.py
- decoders.py
- Dockerfile
- inference.py
- README.md
- requirements.txt
- start.sh
- utils.py
- config.py
- decoders.py
- Dockerfile
- inference.py
- README.md
- requirements.txt
- start.sh
- utils.py
- config.py
- Dockerfile
- README.md
- requirements.txt
- start.sh
- training.py
- TrainingKafkaDataset.py
- utils.py
- FederatedLearning.sol
- blockchain_utils.py
- blockchainSingleFederatedTraining.py
- callbacks.py
- cloudBasedTraining.py
- config.py
- decoders.py
- distributedClassicTraining.py
- distributedFederatedIncrementalTraining.py
- distributedFederatedTraining.py
- distributedIncrementalTraining.py
- Dockerfile
- edgeBasedTraining.py
- edgeBlockchainBasedTraining.py
- FederatedKafkaMLModelSink.py
- KafkaModelEngine.py
- mainTraining.py
- README.md
- requirements.txt
- singleClassicTraining.py
- singleFederatedIncrementalTraining.py
- singleFederatedTraining.py
- singleIncrementalTraining.py
- start.sh
- training.py
- utils.py
- .gitignore
- LICENSE
- README.md
# Installation Guide
1. Get the code
git clone https://github.com/ertis-research/kafka-ml
Downloads the entire project code from GitHub to your computer.
cd kafka-ml
Moves into the project folder you just downloaded.
2. Official Install Script
Easy RecommendedPrerequisites
- Node.js Node.js must be installed to use npm.
npm i -g @angular/cli@9.1.15
Installs the package published on the npm registry globally β no need to clone the source.
After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.
Pulled directly from this repo's README.
3. Docker
EasyPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Runs the built image as an actual container.
docker build --tag localhost:5000/backend .
Builds a runnable image based on the Dockerfile.
docker push localhost:5000/backend
Type this command into your terminal and run it.
docker build --tag localhost:5000/tfexecutor .
Builds a runnable image based on the Dockerfile.
docker push localhost:5000/tfexecutor
Type this command into your terminal and run it.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
Pulled directly from this repo's README.
4. Node.js
EasyPrerequisites
npm install # nvm install 10 & nvm use 10.24.1
Downloads and installs the libraries listed in package.json.
npm i -g @angular/cli@9.1.15
Installs the package published on the npm registry globally β no need to clone the source.
After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.
Pulled directly from this repo's README.
5. Python
EasyPrerequisites
pip install -r backend/requirements.txt
Installs the Python libraries listed in requirements.txt (or similar).
python <μ€νν νμΌλͺ
>.py # READMEμμ μ νν μ€ν νμΌλͺ
μ νμΈνμΈμ
Runs the Python script (or module).
If it runs without errors and prints output in the terminal, it worked.
// repository documentation
Was this content helpful?
(0 ratings)