How-to Guides

Setting up the sandbox in the Quick Start section was easy; building a production-grade environment requires a bit more work!

These how-to guides will step you through common tasks in using and configuring an Airflow environment.

• Using the CLI
  • Set Up Bash/Zsh Completion
  • Creating a Connection
  • Exporting DAG structure as an image
  • Display DAGs structure
  • Formatting commands output
  • Purge history from metadata database
  • Export the purged records from the archive tables
  • Dropping the archived tables
  • Upgrading Airflow
  • Downgrading Airflow
  • Exporting Connections
• Add tags to DAGs and use it for filtering in the UI
• Add Owner Links to DAG
• Creating a notifier
  • Using a notifier
• Setting Configuration Options
• Configuring Flask Application for Airflow Webserver
• Set up a Database Backend
  • Choosing database backend
  • Database URI
  • Setting up a SQLite Database
  • Setting up a PostgreSQL Database
  • Setting up a MySQL Database
  • Setting up a MsSQL Database
  • Other configuration options
  • Initialize the database
  • What’s next?
• Using Operators
  • BashOperator
  • BashSensor
  • BranchDateTimeOperator
  • FileSensor
  • PythonOperator
  • PythonVirtualenvOperator
  • ExternalPythonOperator
  • ShortCircuitOperator
  • PythonSensor
  • TimeDeltaSensor
  • TimeDeltaSensorAsync
  • TimeSensor
  • TimeSensorAsync
  • BranchDayOfWeekOperator
  • DayOfWeekSensor
  • Cross-DAG Dependencies
• Customizing DAG Scheduling with Timetables
  • Timetable Registration
  • Define Scheduling Logic
  • Parameterized Timetables
  • Timetable Display in UI
  • Timetable Description Display in UI
  • Changing generated run_id
• Customize view of Apache from Airflow web UI
  • Custom view Registration
• Listener Plugin of Airflow
  • Listener Registration
• Customizing the UI
  • Customizing state colours
  • Customizing DAG UI Header and Airflow Page Titles
  • Add custom alert messages on the dashboard
• Creating a custom Operator
  • Hooks
  • User interface
  • Templating
  • Define an operator extra link
  • Sensors
• Creating Custom @task Decorators
• (Optional) Adding IDE auto-completion support
• Export dynamic environment variables available for operators to use
• Managing Connections
  • Storing connections in environment variables
  • Storing connections in a Secrets Backend
  • Storing connections in the database
  • Custom connection fields
  • URI format
• Managing Variables
  • Storing Variables in Environment Variables
  • Securing Variables
• Running Airflow behind a reverse proxy
• Running Airflow with systemd
• Using the Test Mode Configuration
• Define an operator extra link
  • Add or override Links to Existing Operators
• Email Configuration
  • Send email using SendGrid
  • Send email using AWS SES
• Dynamic DAG Generation
  • Dynamic DAGs with environment variables
  • Generating Python code with embedded meta-data
  • Dynamic DAGs with external configuration from a structured data file
  • Registering dynamic DAGs
• Running Airflow in Docker
  • Before you begin
  • Fetching docker-compose.yaml
  • Initializing Environment
  • Cleaning-up the environment
  • Running Airflow
  • Accessing the environment
  • Cleaning up
  • Using custom images
  • Special case - adding dependencies via requirements.txt file
  • Networking
  • FAQ: Frequently asked questions
  • What’s Next?
  • Environment variables supported by Docker Compose
• Upgrading from 1.10 to 2
  • Step 1: Switch to Python 3
  • Step 2: Upgrade to 1.10.15
  • Step 3: Run the Upgrade check scripts
  • Step 4: Switch to Backport Providers
  • Step 5: Upgrade Airflow DAGs
  • Step 6: Upgrade Configuration settings
  • Step 7: Upgrade to Airflow 2
  • Appendix