Principles

Scalable

Apache Airflow® has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow™ is ready to scale to infinity.

Dynamic

Apache Airflow® pipelines are defined in Python, allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.

Extensible

Easily define your own operators and extend libraries to fit the level of abstraction that suits your environment.

Elegant

Apache Airflow® pipelines are lean and explicit. Parametrization is built into its core using the powerful Jinja templating engine.

Features

Pure Python

No more command-line or XML black-magic! Use standard Python features to create your workflows, including date time formats for scheduling and loops to dynamically generate tasks. This allows you to maintain full flexibility when building your workflows.

Useful UI

Monitor, schedule and manage your workflows via a robust and modern web application. No need to learn old, cron-like interfaces. You always have full insight into the status and logs of completed and ongoing tasks.

Robust Integrations