Audit Logs in Airflow

Overview

Audit logs are a critical component of any system that needs to maintain a high level of security and compliance. They provide a way to track user actions and system events, which can be used to troubleshoot issues, detect security breaches, and ensure regulatory compliance.

In Airflow, audit logs are used to track user actions and system events that occur during the execution of DAGs and tasks. They are stored in a database and can be accessed through the Airflow UI.

Level of Audit Logs

Audit logs exist at the task level and the user level.

• Task Level: At the task level, audit logs capture information related to the execution of a task, such as the start time, end time, and status of the task.

• User Level: At the user level, audit logs capture information related to user actions, such as creating, modifying, or deleting a DAG or task.

Location of Audit Logs

Audit logs can be accessed through the Airflow UI. They are located under the “Admin” tab, and can be viewed by selecting “Audit Logs” from the dropdown menu.

Types of Events

Airflow provides a set of predefined events that can be tracked in audit logs. These events include:

• action_trigger_dag: Triggering a DAG

• action_create: Creating a DAG or task

• action_edit: Modifying a DAG or task

• action_delete: Deleting a DAG or task

• action_failed: Setting a task as failed

• action_success: Setting a task as successful

• action_retry: Retrying a failed task

• action_clear: Clearing a task’s state

In addition to these predefined events, Airflow allows you to define custom events that can be tracked in audit logs. This can be done by calling the log method of the TaskInstance object.