Core Concepts

Here you can find detailed documentation about each one of Airflow’s core concepts and how to use them, as well as a high-level architectural overview.

Architecture

• Architecture Overview
  • Workloads
  • Control Flow
  • User interface

Workloads

• DAGs
  • Declaring a DAG
  • Loading DAGs
  • Running DAGs
  • DAG Assignment
  • Default Arguments
  • The DAG decorator
  • Control Flow
  • Dynamic DAGs
  • DAG Visualization
  • DAG & Task Documentation
  • SubDAGs
  • TaskGroups vs SubDAGs
  • Packaging DAGs
  • .airflowignore
  • DAG Dependencies
  • DAG pausing, deactivation and deletion
• DAG Runs
  • DAG Run Status
  • Cron Presets
  • Re-run DAG
  • External Triggers
  • To Keep in Mind
• Tasks
  • Relationships
  • Task Instances
  • Timeouts
  • SLAs
  • Special Exceptions
  • Zombie/Undead Tasks
  • Executor Configuration
• Operators
  • Jinja Templating
  • Reserved params keyword
• Sensors
• TaskFlow
  • Context
  • Logging
  • Passing Arbitrary Objects As Arguments
  • Sensors and the TaskFlow API
  • History
• Executor
  • Executor Types

Communication

• XComs
  • Custom XCom Backends
  • Working with Custom XCom Backends in Containers
  • Working with Custom Backends in K8s via Helm
• Variables
• Params
  • DAG-level Params
  • Task-level Params
  • Referencing Params in a Task
  • JSON Schema Validation
  • Use Params to Provide a Trigger UI Form
  • Disabling Runtime Param Modification