BashOperator

Use the BashOperator to execute commands in a Bash shell.

airflow/example_dags/example_bash_operator.py[source]

run_this = BashOperator(
    task_id="run_after_loop",
    bash_command="echo 1",
)

Templating

You can use Jinja templates to parameterize the bash_command argument.

airflow/example_dags/example_bash_operator.py[source]

also_run_this = BashOperator(
    task_id="also_run_this",
    bash_command='echo "ti_key={{ task_instance_key_str }}"',
)

Warning

Care should be taken with “user” input or when using Jinja templates in the bash_command, as this bash operator does not perform any escaping or sanitization of the command.

This applies mostly to using “dag_run” conf, as that can be submitted via users in the Web UI. Most of the default template variables are not at risk.

For example, do not do this:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command='echo "Here is the message: \'{{ dag_run.conf["message"] if dag_run else "" }}\'"',
)

Instead, you should pass this via the env kwarg and use double-quotes inside the bash_command, as below:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command="echo \"here is the message: '$message'\"",
    env={"message": '{{ dag_run.conf["message"] if dag_run else "" }}'},
)

Skipping

In general a non-zero exit code produces an AirflowException and thus a task failure. In cases where it is desirable to instead have the task end in a skipped state, you can exit with code 99 (or with another exit code if you pass skip_exit_code).

airflow/example_dags/example_bash_operator.py[source]

this_will_skip = BashOperator(
    task_id="this_will_skip",
    bash_command='echo "hello world"; exit 99;',
    dag=dag,
)

Troubleshooting

Jinja template not found

Add a space after the script name when directly calling a Bash script with the bash_command argument. This is because Airflow tries to apply a Jinja template to it, which will fail.

t2 = BashOperator(
    task_id="bash_example",
    # This fails with 'Jinja template not found' error
    # bash_command="/home/batcher/test.sh",
    # This works (has a space after)
    bash_command="/home/batcher/test.sh ",
    dag=dag,
)

However, if you want to use templating in your bash script, do not add the space and instead put your bash script in a location relative to the directory containing the DAG file. So if your DAG file is in /usr/local/airflow/dags/test_dag.py, you can move your test.sh file to any location under /usr/local/airflow/dags/ (Example: /usr/local/airflow/dags/scripts/test.sh) and pass the relative path to bash_command as shown below:

t2 = BashOperator(
    task_id="bash_example",
    # "scripts" folder is under "/usr/local/airflow/dags"
    bash_command="scripts/test.sh",
    dag=dag,
)

Creating separate folder for bash scripts may be desirable for many reasons, like separating your script’s logic and pipeline code, allowing for proper code highlighting in files composed in different languages, and general flexibility in structuring pipelines.

It is also possible to define your template_searchpath as pointing to any folder locations in the DAG constructor call.

Example:

dag = DAG("example_bash_dag", template_searchpath="/opt/scripts")
t2 = BashOperator(
    task_id="bash_example",
    # "test.sh" is a file under "/opt/scripts"
    bash_command="test.sh ",
    dag=dag,
)

BashSensor

Use the BashSensor to use arbitrary command for sensing. The command should return 0 when it succeeds, any other value otherwise.

airflow/example_dags/example_sensors.py[source]

t3 = BashSensor(task_id="Sensor_succeeds", bash_command="exit 0")
t4 = BashSensor(task_id="Sensor_fails_after_3_seconds", timeout=3, soft_fail=True, bash_command="exit 1")