airflow_project

KO | EN

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #claude-code #cli #windows #codex #developer-tools #dsh

airflow_project

★ 85 Open GitHub ↗

scaffold of Apache Airflow executing Docker containers

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![airflow](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/airflowlogo.png)

# Yet Another Scalable Apache Airflow With Docker Example Setup
This repository contains the code for article https://medium.com/@tomaszdudek/yet-another-scalable-apache-airflow-with-docker-example-setup-84775af5c451. It features a scaffold of Apache Airflow, whose workers instead of executing the job themselves, run Docker containers and wait until they complete.

# Requirements
* Linux or MacOS
* Docker and docker-compose
* Python 3.6 (with `docker-py` installed)

# How to run example
* clone the project
* go to `./airflow/` and run `docker build . -t puckel-airflow-with-docker-inside:latest`
* make sure you have [docker-py](https://github.com/docker/docker-py) (tested with version `docker==3.7.0` - **careful because using `docker-py==1.10.x` won't work**) and run `python build_images.py`
* go to `./airflow` and run `docker-compose up -f docker-compose-linux.yml` (or just `up` when using OSX)
* launch http://localhost:8080/admin/, ensure that DAG is ON:

![on](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/on.png)

* trigger the DAG:

![trigger](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/trigger.png)

* watch the pipeline as it launches Docker containers:

![pip](https://raw.githubusercontent.com/tomaszdudek7/airflow_project/master/result.png)

# Modify
* feel free to make PR's

airflow_project

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README