airflow-dag-patterns

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

When to Use This Skill

• Creating data pipeline orchestration with Airflow
• Designing DAG structures and dependencies
• Implementing custom operators and sensors
• Testing Airflow DAGs locally
• Setting up Airflow in production
• Debugging failed DAG runs

Core Concepts

1. DAG Design Principles

2. Task Dependencies

# Linear
task1 >> task2 >> task3
 
# Fan-out
task1 >> [task2, task3, task4]
 
# Fan-in
[task1, task2, task3] >> task4
 
# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4

Quick Start

# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator
 
default_args = {
    'owner'

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

• Use TaskFlow API - Cleaner code, automatic XCom
• Set timeouts - Prevent zombie tasks
• Use mode='reschedule' - For sensors, free up workers
• Test DAGs - Unit tests and integration tests
• Idempotent tasks - Safe to retry

Don'ts

• Don't use depends_on_past=True - Creates bottlenecks
• Don't hardcode dates - Use {{ ds }} macros
• Don't use global state - Tasks should be stateless
• Don't skip catchup blindly - Understand implications
• Don't put heavy logic in DAG file - Import from modules