Kerberos

Airflow has initial support for Kerberos. This means that Airflow can renew Kerberos tickets for itself and store it in the ticket cache. The hooks and DAGs can make use of ticket to authenticate against kerberized services.

Limitations

Please note that at this time, not all hooks have been adjusted to make use of this functionality. Also it does not integrate Kerberos into the web interface and you will have to rely on network level security for now to make sure your service remains secure.

Celery integration has not been tried and tested yet. However, if you generate a key tab for every host and launch a ticket renewer next to every worker it will most likely work.

Enabling Kerberos

Airflow

To enable Kerberos you will need to generate a (service) key tab.

  1. # in the kadmin.local or kadmin shell, create the airflow principal
  2. kadmin:  addprinc -randkey airflow/fully.qualified.domain.name@YOUR-REALM.COM
  4. # Create the airflow keytab file that will contain the airflow principal
  5. kadmin:  xst -norandkey -k airflow.keytab airflow/fully.qualified.domain.name

Now store this file in a location where the airflow user can read it (chmod 600). And then add the following to your airflow.cfg

  1. [core]
  2. security = kerberos
  4. [kerberos]
  5. keytab = /etc/airflow/airflow.keytab
  6. reinit_frequency = 3600
  7. principal = airflow

In case you are using Airflow in a docker container based environment, you can set the below environment variables in the Dockerfile instead of modifying airflow.cfg

  1. ENV AIRFLOW__CORE__SECURITY kerberos
  2. ENV AIRFLOW__KERBEROS__KEYTAB /etc/airflow/airflow.keytab
  3. ENV AIRFLOW__KERBEROS__INCLUDE_IP False

If you need more granular options for your Kerberos ticket the following options are available with the following default values:

  1. [kerberos]
  2. # Location of your ccache file once kinit has been performed
  3. ccache = /tmp/airflow_krb5_ccache
  4. # principal gets augmented with fqdn
  5. principal = airflow
  6. reinit_frequency = 3600
  7. kinit_path = kinit
  8. keytab = airflow.keytab
  10. # Allow kerberos token to be flag forwardable or not
  11. forwardable = True
  13. # Allow to include or remove local IP from kerberos token.
  14. # This is particularly useful if you use Airflow inside a VM NATted behind host system IP.
  15. include_ip = True

Keep in mind that Kerberos ticket are generated via kinit and will your use your local krb5.conf by default.

Launch the ticket renewer by

  1. # run ticket renewer
  2. airflow kerberos

Hadoop

If want to use impersonation this needs to be enabled in core-site.xml of your hadoop config.

  1. <property>
  2.   <name>hadoop.proxyuser.airflow.groups</name>
  3.   <value>*</value>
  4. </property>
  6. <property>
  7.   <name>hadoop.proxyuser.airflow.users</name>
  8.   <value>*</value>
  9. </property>
  11. <property>
  12.   <name>hadoop.proxyuser.airflow.hosts</name>
  13.   <value>*</value>
  14. </property>

Of course if you need to tighten your security replace the asterisk with something more appropriate.

Using Kerberos authentication

The Hive hook has been updated to take advantage of Kerberos authentication. To allow your DAGs to use it, simply update the connection details with, for example:

  1. { "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM"}

Adjust the principal to your settings. The _HOST part will be replaced by the fully qualified domain name of the server.

You can specify if you would like to use the DAG owner as the user for the connection or the user specified in the login section of the connection. For the login user, specify the following as extra:

  1. { "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "login"}

For the DAG owner use:

  1. { "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "owner"}

and in your DAG, when initializing the HiveOperator, specify:

  1. run_as_owner=True

To use kerberos authentication, you must install Airflow with the kerberos extras group:

  1. pip install 'apache-airflow[kerberos]'

You can read about some production aspects of Kerberos deployment at Kerberos-authenticated workers