airflow_project

KO | EN

gitlite — search

#hacktoberfest #react #typescript #llm #go #machine-learning #linux #docker #nodejs #macos #awesome #kubernetes #ios #open-source

airflow_project

(★ 85)

scaffold of Apache Airflow executing Docker containers

파일 탐색기

최종 버전 다운로드 (.zip)

- - pipeline.py
- - merge_xcom_dicts.py
- docker-compose-linux.yml
- docker-compose.yml
- Dockerfile
- requirements.txt
- - code.ipynb
  - Dockerfile
  - params.yaml
  - requirements.txt
- - code.ipynb
  - Dockerfile
  - params.yaml
  - requirements.txt
- - code.ipynb
  - Dockerfile
  - params.yaml
  - requirements.txt
- README.md
.gitignore
airflowlogo.png
build_images.py
LICENSE
on.png
README.md
result.png
trigger.png

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![airflow](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/airflowlogo.png)

# Yet Another Scalable Apache Airflow With Docker Example Setup
This repository contains the code for article https://medium.com/@tomaszdudek/yet-another-scalable-apache-airflow-with-docker-example-setup-84775af5c451. It features a scaffold of Apache Airflow, whose workers instead of executing the job themselves, run Docker containers and wait until they complete.

# Requirements
* Linux or MacOS
* Docker and docker-compose
* Python 3.6 (with `docker-py` installed)

# How to run example
* clone the project
* go to `./airflow/` and run `docker build . -t puckel-airflow-with-docker-inside:latest`
* make sure you have [docker-py](https://github.com/docker/docker-py) (tested with version `docker==3.7.0` - **careful because using `docker-py==1.10.x` won't work**) and run `python build_images.py`
* go to `./airflow` and run `docker-compose up -f docker-compose-linux.yml` (or just `up` when using OSX)
* launch http://localhost:8080/admin/, ensure that DAG is ON:

![on](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/on.png)

* trigger the DAG:

![trigger](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/trigger.png)

* watch the pipeline as it launches Docker containers:

![pip](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/result.png)

# Modify
* feel free to make PR's

airflow_project

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

airflow_project

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README