airflow-operator - GitLite

KO | EN

gitlite — search

#ai #llm #golang #machine-learning #rust #deep-learning #nodejs #java #awesome #kubernetes #vue #awesome-list #pytorch #mcp

airflow-operator

(★ 297)

Kubernetes custom controller and CRDs to managing Airflow

File Explorer

Download Latest Version (.zip)

- - main.go
- - airflow_v1alpha1_airflowbase.yaml
  - airflow_v1alpha1_airflowcluster.yaml
- - kustomization.yaml
  - manager_image_patch.yaml
- - rbac_role.yaml
  - rbac_role_binding.yaml
- - airflow_v1alpha1_airflowbase.yaml
  - airflow_v1alpha1_airflowcluster.yaml
- airflow-base.png
- airflow-cluster.png
- airflow-multi-node.png
- airflow-pod.png
- airflow-region-spread.png
- airflow-zone-spread.png
- api.md
- design.md
- development.md
- quickstart.md
- userguide.md
- appcrd.yaml
- boilerplate.go.txt
- - addtoscheme_airflow_v1alpha1.go
  - apis.go
- - add_airflowbase.go
  - add_airflowcluster.go
  - controller.go
- - webhook.go
- airflow-configmap.yaml
- base-application.yaml
- cluster-application.yaml
- flower-sts.yaml
- headlesssvc.yaml
- mysql-sts.yaml
- nfs-sts.yaml
- pdb.yaml
- postgres-sts.yaml
- redis-sts.yaml
- rolebinding.yaml
- scheduler-sts.yaml
- secret.yaml
- serviceaccount.yaml
- sqlproxy-sts.yaml
- storage.yaml
- svc.yaml
- ui-sts.yaml
- worker-sts.yaml
- - base_test.go
  - cluster_test.go
  - gcp_test.go
.gitignore
cloudbuild.yaml
Dockerfile
Gopkg.lock
Gopkg.toml
LICENSE
Makefile
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Go Report Card](https://goreportcard.com/badge/github.com/GoogleCloudPlatform/airflow-operator)](https://goreportcard.com/report/github.com/GoogleCloudPlatform/airflow-operator)

**This is not an officially supported Google product.**

## Community

* Join our [Slack channel](https://kubernetes.slack.com/messages/CC1UAMYSV).

## Project Status

*Alpha*

The Airflow Operator is still under active development and has not been extensively tested in production environment. Backward compatibility of the APIs is not guaranteed for alpha releases.

## Prerequisites
* Version >= 1.9 of Kubernetes.
* Uses 1.9 of Airflow (1.10.1+ for k8s executor)
* Uses 4.0.x of Redis (for celery operator)
* Uses 5.7 of MySQL

## Get Started

[One Click Deployment](https://console.cloud.google.com/marketplace/details/google/airflow-operator) from Google Cloud Marketplace to your [GKE cluster](https://cloud.google.com/kubernetes-engine/)

Get started quickly with the Airflow Operator using the [Quick Start Guide](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/quickstart.md)

For more information check the [Design](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/design.md) and detailed [User Guide](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/userguide.md)

## Airflow Operator Overview
Airflow Operator is a custom [Kubernetes operator](https://coreos.com/blog/introducing-operators.html) that makes it easy to deploy and manage [Apache Airflow](https://airflow.apache.org/) on Kubernetes. Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Using the Airflow Operator, an Airflow cluster is split into 2 parts represented by the `AirflowBase` and `AirflowCluster` custom resources.
The Airflow Operator performs these jobs:
* Creates and manages the necessary Kubernetes resources for an Airflow deployment.
* Updates the corresponding Kubernetes resources when the `AirflowBase` or `AirflowCluster` specification changes.
* Restores managed Kubernetes resources that are deleted.
* Supports creation of Airflow schedulers with different Executors
* Supports sharing of the `AirflowBase` across mulitple `AirflowClusters`

Checkout out the [Design](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/design.md)

![Airflow Cluster](https://raw.githubusercontent.com/GoogleCloudPlatform/airflow-operator/master/docs/airflow-cluster.png)

## Development

Refer to the [Design](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/design.md) and [Development Guide](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/development.md).

## Managed Airflow solution

[Google Cloud Composer](https://cloud.google.com/composer/) is a fully managed workflow orchestration service targeting customers that need a workflow manager in the cloud.