Build your first DAG
Create a DAG that runs daily, has catchup disabled, and prints a message in one Python task.
Data / Knowledge library
Apache Airflow (airflow)
python-based workflow orchestrator for data pipelines
Why learn Apache Airflow (airflow)?
Airflow helps teams author, schedule, observe, and retry complex data workflows as code (DAGs) instead of fragile cron chains and manual runbooks.
Where it came from
2014: Started at Airbnb to manage growing ETL dependency complexity. 2015: Open-sourced by Airbnb. 2016: Entered the Apache Incubator. 2019: Graduated to Apache Top-Level Project. 2020: Airflow 2.0 delivered major scheduler/UI improvements and TaskFlow API.
- 2014: Started at Airbnb to manage growing ETL dependency complexity.
- 2015: Open-sourced by Airbnb.
- 2016: Entered the Apache Incubator.
- 2019: Graduated to Apache Top-Level Project.
- 2020: Airflow 2.0 delivered major scheduler/UI improvements and TaskFlow API.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Create a DAG that runs daily, has catchup disabled, and prints a message in one Python task.
- Extend a 3-task pipeline (extract -> transform -> load). Add per-task retries and a 5-minute retry delay.
- Create two DAGs where DAG B triggers from a Dataset updated by DAG A. Validate behavior in the UI.
- Replace a long-polling sensor with a deferrable equivalent and explain worker-slot savings.
- Set up pools, SLA/alerts, and a backfill-safe strategy for late-arriving data.
More ways to practice
Add dependencies and retries
Extend a 3-task pipeline (extract -> transform -> load). Add per-task retries and a 5-minute retry delay.
Use data-aware scheduling
Create two DAGs where DAG B triggers from a Dataset updated by DAG A. Validate behavior in the UI.
Optimize idle wait time
Replace a long-polling sensor with a deferrable equivalent and explain worker-slot savings.
Production hardening
Set up pools, SLA/alerts, and a backfill-safe strategy for late-arriving data.
Documentation and sources
- Official docs (stable) (airflow.apache.org)
- Core concepts: DAGs (airflow.apache.org)
- Providers index (airflow.apache.org)
- Astronomer learning guides (www.astronomer.io)
Catalog research notes
- Recent 2.x releases expanded dynamic task mapping for scalable fan-out tasks.
- Deferrable operators + triggerer improved efficiency for long-wait tasks.
- Data-aware scheduling with Datasets enabled event-driven cross-DAG orchestration.
- REST API and UI coverage improved for deployment, monitoring, and automation workflows.
Connect the ideas
Explore next
Real World Example
Automated Retail Inventory Reconciliation
Retailers managing multi-channel inventory across physical stores and e-commerce platforms.
The business problem
Manual reconciliation processes lead to data silos, delayed stock updates, and frequent overselling of out-of-stock items due to inconsistent batch processing.
How this topic solves it
Apache Airflow orchestrates the end-to-end data pipeline by defining workflows as Python code, ensuring consistent execution of extraction, transformation, and load tasks across disparate systems.
- Define a DAG to encapsulate the reconciliation logic and schedule it for hourly execution.
- Implement extraction tasks using Python decorators to pull inventory logs from the e-commerce API.
- Utilize BashOperators to trigger local database cleanup scripts for physical store records.
- Set task dependencies using the bitshift operator to ensure data is cleaned before the final sync.
- Monitor the execution status and logs via the Airflow web UI to verify successful reconciliation.
The flow shows the sequence of tasks from data extraction to final synchronization.
Source of truth for online sales
Python task to fetch logs
BashOperator for DB maintenance
Final update to inventory system
- E-commerce API → data_fetch → Extract Task
- Extract Task → triggers_cleanup → Cleanup Task
- Cleanup Task → enables_sync → Sync Task
Implementation example
This code demonstrates defining a DAG with a Python task for extraction and a BashOperator for cleanup, linked by a dependency operator.
from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator
with DAG(dag_id='inventory_sync', schedule='@hourly'):
@task
def extract_online(): # Fetch e-commerce data
return 'data'
cleanup = BashOperator(task_id='db_cleanup', bash_command='rm -rf /tmp/old_logs')
extract_online() >> cleanup # Define dependencyExpected business outcome
Improved inventory accuracy and reduced manual intervention through reliable, automated, and observable pipeline execution.
Limitations and checks
Airflow is designed for batch-oriented workflows; it is not suitable for low-latency streaming requirements. Check this by verifying if the business requirement demands sub-second data propagation.