incubator-liminal
Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.
File Explorer
Download Latest Version (.zip)- pre_commits.yml
- unittests.yml
- versioning.yml
- PULL_REQUEST_TEMPLATE.md
- LICENSE.txt
- RELEASE.md
- sign.sh
- hello_world.md
- iris_classification.md
- spark_app_demo.md
- emr.md
- index.rst
- kubernetes.md
- README.md
- index.rst
- python.md
- python_server.md
- README.md
- secret_util.md
- aws_cloudwatch.md
- index.rst
- README.md
- create_cloudformation_stack.md
- delete_cloudformation_stack.md
- index.rst
- python.md
- README.md
- spark.md
- advanced.liminal.yml.md
- extensibility.md
- index.rst
- liminal.yml.md
- monitoring.md
- pipelines.md
- README.md
- services.md
- airflow_main.png
- airflow_task_log.png
- airflow_view_dag.png
- airflow_view_log.png
- airflow_main.png
- airflow_task_log.png
- airflow_view_dag.png
- airflow_view_log.png
- k8s_dag.png
- liminal_arch_001.png
- liminal_arch_002.png
- liminal_logo.png
- liminal_aws_deploy.gif
- liminal_deployment_diagram.png
- How_to_install_liminal_in_airflow_on_kubernetes.md
- liminal_aws_deploy.sh
- architecture.md
- conf.py
- index.rst
- make.bat
- Makefile
- README.md
- aws-ml-app-demo.yaml
- __init__.py
- liminal.yml
- model_store.py
- requirements.txt
- serving.py
- training.py
- __init__.py
- custom_executor.py
- __init__.py
- Dockerfile
- __init__.py
- custom_image.py
- __init__.py
- custom_task.py
- __init__.py
- liminal.yml
- __init__.py
- hello_world.py
- __init__.py
- my_server.py
- __init__.py
- hello_world.json
- liminal.yml
- pip.conf
- requirements.txt
- liminal.yaml
- __init__.py
- data_preparation.py
- data_uploader.py
- __init__.py
- train.py
- inference.py
- liminal.yaml
- liminal_sm.py
- README.md
- requirements.txt
- sm_ops.py
- liminal.yml
- iris.csv
- spark-app-demo.yaml
- __init__.py
- data_cleanup.py
- liminal.yml
- model_store.py
- requirements.txt
- serving.py
- training.py
- airflow.png
- airflow_trigger.png
- k8s_running.png
- liminal_001.png
- liminal_002.png
- __init__.py
- Dockerfile
- python.py
- __init__.py
- Dockerfile
- liminal_python_server.py
- python_server.py
- python_server_requirements.txt
- __init__.py
- Dockerfile
- spark.py
- __init__.py
- __init__.py
- image_builder.py
- liminal_apps_builder.py
- python.py
- __init__.py
- liminal.yml
- __init__.py
- default_configs.py
- __init__.py
- config.py
- __init__.py
- class_util.py
- dict_util.py
- env_util.py
- extensible.py
- files_util.py
- __init__.py
- environment.py
- __init__.py
- __init__.py
- secret_util.py
- volume_util.py
- __init__.py
- logging_setup.py
- __init__.py
- __init__.py
- standalone_variable_backend.py
- __init__.py
- liminal_dags.py
- liminal_register_dags.py
- __init__.py
- airflow.py
- emr.py
- kubernetes.py
- __init__.py
- executor.py
- task.py
- __init__.py
- cloudformation.py
- job_status_operator.py
- operator_with_variable_resolving.py
- __init__.py
- airflow.py
- containerable.py
- create_cloudformation_stack.py
- delete_cloudformation_stack.py
- hadoop.py
- job_end.py
- job_start.py
- python.py
- spark.py
- sql.py
- __init__.py
- __init__.py
- __init__.py
- __init__.py
- settings.py
- __init__.py
- docker-compose.yml
- Dockerfile-airflow
- liminal
- requirements-airflow.txt
- webserver_config.py
- __init__.py
- test_apply_variables_substitution.py
- test_defaults_pipeline_config.py
- test_defaults_service_config.py
- test_defaults_tasks_config.py
- __init__.py
- test_config.py
- __init__.py
- test_dict_utils.py
- __init__.py
- __init__.py
- test_volume_util.py
- __init__.py
- __init__.py
- test_python_server_image_builder.py
- __init__.py
- __init__.py
- test_python_image_builder.py
- __init__.py
- test_spark_image_builder.py
- __init__.py
- test_liminal_apps_builder.py
- __init__.py
- __init__.py
- test_liminal_dags.py
- __init__.py
- test_airflow_executor.py
- test_emr.py
- __init__.py
- my_server.py
- __init__.py
- write_inputs.py
- __init__.py
- write_outputs.py
- __init__.py
- liminal.yml
- pip.conf
- requirements.txt
- __init__.py
- test_operator_with_variable_resolving.py
- __init__.py
- test_create_cloudformation_stack.py
- test_delete_cloudformation_stack.py
- test_job_end.py
- test_job_start.py
- test_python.py
- test_spark_task.py
- __init__.py
- liminal.yml
- __init__.py
- liminal.yml
- __init__.py
- liminal.yml
- __init__.py
- hellp_world.py
- __init__.py
- my_server.py
- __init__.py
- liminal.yml
- __init__.py
- requirements.txt
- wordcount.py
- words.txt
- __init__.py
- liminal.yml
- __init__.py
- __init__.py
- __init__.py
- test_clazz_leaf_1.py
- __init__.py
- test_clazz_leaf_2.py
- __init__.py
- test_clazz_child_1.py
- test_clazz_child_2.py
- __init__.py
- test_clazz_base.py
- __init__.py
- dag_test_utils.py
- test_class_utils.py
- __init__.py
- test_licenses.py
- .asf.yaml
- .bumpversion.cfg
- .gitignore
- .pre-commit-config.yaml
- CONTRIBUTING.md
- DISCLAIMER-WIP
- install.sh
- LICENSE
- liminal-arch.md
- MANIFEST.in
- NOTICE
- pyproject.toml
- README.md
- requirements.txt
- RETIRED.txt
- run_tests.sh
- setup.py
- yamllint-config.yml
# Installation Guide
1. Get the code
git clone https://github.com/apache/incubator-liminal
Downloads the entire project code from GitHub to your computer.
cd incubator-liminal
Moves into the project folder you just downloaded.
2. Official Install Script
Easy RecommendedPrerequisites
- Python 3 Python is required to use pip.
pip install git+https://github.com/apache/incubator-liminal.git
Installs the package published on PyPI directly β no need to clone the source.
After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.
Pulled directly from this repo's README.
3. Docker
EasyPrerequisites
- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker compose -f scripts/docker-compose.yml up -d --build
Runs the command against the services defined in the compose file.
Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.
4. Python
EasyPrerequisites
pip install git+https://github.com/apache/incubator-liminal.git
Installs the package published on PyPI directly β no need to clone the source.
If it runs without errors and prints output in the terminal, it worked.
Pulled directly from this repo's README.
// repository documentation
Was this content helpful?
(0 ratings)