Executor

Executors are the mechanism by which task instances get run. They have a common API and are “pluggable”, meaning you can swap executors based on your installation needs.

Airflow can only have one executor configured at a time; this is set by the executor option in the [core] section of the configuration file.

Built-in executors are referred to by name, for example:

[core]
executor = KubernetesExecutor

You can also write your own custom executors, and refer to them by their full path:

[core]
executor = my_company.executors.MyCustomExecutor

Note

For more information on Airflow’s configuration, see Setting Configuration Options.

If you want to check which executor is currently set, you can use the airflow config get-value core executor command:

$ airflow config get-value core executor
SequentialExecutor

Executor Types

There are two types of executor - those that run tasks locally (inside the scheduler process), and those that run their tasks remotely (usually via a pool of workers). Airflow comes configured with the SequentialExecutor by default, which is a local executor, and the safest option for execution, but we strongly recommend you change this to LocalExecutor for small, single-machine installations, or one of the remote executors for a multi-machine/cloud installation.

Local Executors

• Testing DAGs with dag.test()
• Debugging Airflow DAGs on the command line
• Debug Executor (deprecated)
• Local Executor
• Sequential Executor

Remote Executors

• Celery Executor
• CeleryKubernetes Executor
• Dask Executor
• Kubernetes Executor
• LocalKubernetes Executor

Note

Something that often confuses new users of Airflow is that they don’t need to run a separate executor process. This is because the executor’s logic runs inside the scheduler process - if you’re running a scheduler, you’re running the executor.