Traefik & Kubernetes

The Kubernetes Ingress Controller, The Custom Resource Way.

Traefik used to support Kubernetes only through the Kubernetes Ingress provider, which is a Kubernetes Ingress controller in the strict sense of the term.

However, as the community expressed the need to benefit from Traefik features without resorting to (lots of) annotations, we ended up writing a Custom Resource Definition (alias CRD in the following) for an IngressRoute type, defined below, in order to provide a better way to configure access to a Kubernetes cluster.

Configuration Requirements

All Steps for a Successful Deployment

  • Add/update all the Traefik resources definitions
  • Add/update the RBAC for the Traefik custom resources
  • Use Helm Chart or use a custom Traefik Deployment
    • Enable the kubernetesCRD provider
    • Apply the needed kubernetesCRD provider configuration
  • Add all needed traefik custom resources

Initializing Resource Definition and RBAC

Traefik Resource Definition

  1. # All resources definition must be declared
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: ingressroutes.traefik.containo.us
  7. spec:
  8.   group: traefik.containo.us
  9.   version: v1alpha1
  10.   names:
  11.     kind: IngressRoute
  12.     plural: ingressroutes
  13.     singular: ingressroute
  14.   scope: Namespaced
  16. ---
  17. apiVersion: apiextensions.k8s.io/v1beta1
  18. kind: CustomResourceDefinition
  19. metadata:
  20.   name: middlewares.traefik.containo.us
  22. spec:
  23.   group: traefik.containo.us
  24.   version: v1alpha1
  25.   names:
  26.     kind: Middleware
  27.     plural: middlewares
  28.     singular: middleware
  29.   scope: Namespaced
  31. ---
  32. apiVersion: apiextensions.k8s.io/v1beta1
  33. kind: CustomResourceDefinition
  34. metadata:
  35.   name: ingressroutetcps.traefik.containo.us
  37. spec:
  38.   group: traefik.containo.us
  39.   version: v1alpha1
  40.   names:
  41.     kind: IngressRouteTCP
  42.     plural: ingressroutetcps
  43.     singular: ingressroutetcp
  44.   scope: Namespaced
  46. ---
  47. apiVersion: apiextensions.k8s.io/v1beta1
  48. kind: CustomResourceDefinition
  49. metadata:
  50.   name: ingressrouteudps.traefik.containo.us
  52. spec:
  53.   group: traefik.containo.us
  54.   version: v1alpha1
  55.   names:
  56.     kind: IngressRouteUDP
  57.     plural: ingressrouteudps
  58.     singular: ingressrouteudp
  59.   scope: Namespaced
  61. ---
  62. apiVersion: apiextensions.k8s.io/v1beta1
  63. kind: CustomResourceDefinition
  64. metadata:
  65.   name: tlsoptions.traefik.containo.us
  67. spec:
  68.   group: traefik.containo.us
  69.   version: v1alpha1
  70.   names:
  71.     kind: TLSOption
  72.     plural: tlsoptions
  73.     singular: tlsoption
  74.   scope: Namespaced
  76. ---
  77. apiVersion: apiextensions.k8s.io/v1beta1
  78. kind: CustomResourceDefinition
  79. metadata:
  80.   name: tlsstores.traefik.containo.us
  82. spec:
  83.   group: traefik.containo.us
  84.   version: v1alpha1
  85.   names:
  86.     kind: TLSStore
  87.     plural: tlsstores
  88.     singular: tlsstore
  89.   scope: Namespaced
  91. ---
  92. apiVersion: apiextensions.k8s.io/v1beta1
  93. kind: CustomResourceDefinition
  94. metadata:
  95.   name: traefikservices.traefik.containo.us
  97. spec:
  98.   group: traefik.containo.us
  99.   version: v1alpha1
  100.   names:
  101.     kind: TraefikService
  102.     plural: traefikservices
  103.     singular: traefikservice
  104.   scope: Namespaced
  106. ---
  107. apiVersion: apiextensions.k8s.io/v1beta1
  108. kind: CustomResourceDefinition
  109. metadata:
  110.   name: serverstransports.traefik.containo.us
  112. spec:
  113.   group: traefik.containo.us
  114.   version: v1alpha1
  115.   names:
  116.     kind: ServersTransport
  117.     plural: serverstransports
  118.     singular: serverstransport
  119.   scope: Namespaced

RBAC for Traefik CRD

  1. kind: ClusterRole
  2. apiVersion: rbac.authorization.k8s.io/v1beta1
  3. metadata:
  4.   name: traefik-ingress-controller
  6. rules:
  7.   - apiGroups:
  8.       - ""
  9.     resources:
  10.       - services
  11.       - endpoints
  12.       - secrets
  13.     verbs:
  14.       - get
  15.       - list
  16.       - watch
  17.   - apiGroups:
  18.       - extensions
  19.       - networking.k8s.io
  20.     resources:
  21.       - ingresses
  22.       - ingressclasses
  23.     verbs:
  24.       - get
  25.       - list
  26.       - watch
  27.   - apiGroups:
  28.       - extensions
  29.     resources:
  30.       - ingresses/status
  31.     verbs:
  32.       - update
  33.   - apiGroups:
  34.       - traefik.containo.us
  35.     resources:
  36.       - middlewares
  37.       - ingressroutes
  38.       - traefikservices
  39.       - ingressroutetcps
  40.       - ingressrouteudps
  41.       - tlsoptions
  42.       - tlsstores
  43.       - serverstransports
  44.     verbs:
  45.       - get
  46.       - list
  47.       - watch
  49. ---
  50. kind: ClusterRoleBinding
  51. apiVersion: rbac.authorization.k8s.io/v1beta1
  52. metadata:
  53.   name: traefik-ingress-controller
  55. roleRef:
  56.   apiGroup: rbac.authorization.k8s.io
  57.   kind: ClusterRole
  58.   name: traefik-ingress-controller
  59. subjects:
  60.   - kind: ServiceAccount
  61.     name: traefik-ingress-controller
  62.     namespace: default

Resource Configuration

When using KubernetesCRD as a provider, Traefik uses Custom Resource Definition to retrieve its routing configuration. Traefik Custom Resource Definitions are a Kubernetes implementation of the Traefik concepts. The main particularities are:

  • The usage of name and namespace to refer to another Kubernetes resource.
  • The usage of secret for sensible data like:
    • TLS certificate.
    • Authentication data.
  • The structure of the configuration.
  • The obligation to declare all the definitions.

The Traefik CRD are building blocks which you can assemble according to your needs. See the list of CRDs in the dedicated routing section.

LetsEncrypt Support with the Custom Resource Definition Provider

By design, Traefik is a stateless application, meaning that it only derives its configuration from the environment it runs in, without additional configuration. For this reason, users can run multiple instances of Traefik at the same time to achieve HA, as is a common pattern in the kubernetes ecosystem.

When using a single instance of Traefik with LetsEncrypt, no issues should be encountered, however this could be a single point of failure. Unfortunately, it is not possible to run multiple instances of Traefik 2.0 with LetsEncrypt enabled, because there is no way to ensure that the correct instance of Traefik will receive the challenge request, and subsequent responses. Previous versions of Traefik used a KV store to attempt to achieve this, but due to sub-optimal performance was dropped as a feature in 2.0.

If you require LetsEncrypt with HA in a kubernetes environment, we recommend using Traefik Enterprise where distributed LetsEncrypt is a supported feature.

If you want to continue to run Traefik Community Edition, LetsEncrypt HA can be achieved by using a Certificate Controller such as Cert-Manager. When using Cert-Manager to manage certificates, it will create secrets in your namespaces that can be referenced as TLS secrets in your ingress objects. When using the Traefik Kubernetes CRD Provider, unfortunately Cert-Manager cannot interface directly with the CRDs yet, but this is being worked on by our team. A workaround is to enable the Kubernetes Ingress provider to allow Cert-Manager to create ingress objects to complete the challenges. Please note that this still requires manual intervention to create the certificates through Cert-Manager, but once created, Cert-Manager will keep the certificate renewed.

Provider Configuration

endpoint

Optional, Default=empty

File (TOML)

  1. [providers.kubernetesCRD]
  2.   endpoint = "http://localhost:8080"
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     endpoint: "http://localhost:8080"
  4.     # ...

CLI

  1. --providers.kubernetescrd.endpoint=http://localhost:8080

The Kubernetes server endpoint as URL.

When deployed into Kubernetes, Traefik will read the environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT or KUBECONFIG to construct the endpoint.

The access token will be looked up in /var/run/secrets/kubernetes.io/serviceaccount/token and the SSL CA certificate in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. Both are provided mounted automatically when deployed inside Kubernetes.

The endpoint may be specified to override the environment variable values inside a cluster.

When the environment variables are not found, Traefik will try to connect to the Kubernetes API server with an external-cluster client. In this case, the endpoint is required. Specifically, it may be set to the URL used by kubectl proxy to connect to a Kubernetes cluster using the granted authentication and authorization of the associated kubeconfig.

token

Optional, Default=empty

File (TOML)

  1. [providers.kubernetesCRD]
  2.   token = "mytoken"
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     token: "mytoken"
  4.     # ...

CLI

  1. --providers.kubernetescrd.token=mytoken

Bearer token used for the Kubernetes client configuration.

certAuthFilePath

Optional, Default=empty

File (TOML)

  1. [providers.kubernetesCRD]
  2.   certAuthFilePath = "/my/ca.crt"
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     certAuthFilePath: "/my/ca.crt"
  4.     # ...

CLI

  1. --providers.kubernetescrd.certauthfilepath=/my/ca.crt

Path to the certificate authority file. Used for the Kubernetes client configuration.

namespaces

Optional, Default: all namespaces (empty array)

File (TOML)

  1. [providers.kubernetesCRD]
  2.   namespaces = ["default", "production"]
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     namespaces:
  4.     - "default"
  5.     - "production"
  6.     # ...

CLI

  1. --providers.kubernetescrd.namespaces=default,production

Array of namespaces to watch.

labelselector

Optional,Default: empty (process all resources)

File (TOML)

  1. [providers.kubernetesCRD]
  2.   labelselector = "app=traefik"
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     labelselector: "app=traefik"
  4.     # ...

CLI

  1. --providers.kubernetescrd.labelselector="app=traefik"

By default, Traefik processes all resource objects in the configured namespaces. A label selector can be defined to filter on specific resource objects only, this will apply only on Traefik Custom Resources and has no effect on Kubernetes Secrets, Endpoints and Services.

See label-selectors for details.

Warning

As the LabelSelector is applied to all Traefik Custom Resources, they all must match the filter.

ingressClass

Optional, Default: empty

File (TOML)

  1. [providers.kubernetesCRD]
  2.   ingressClass = "traefik-internal"
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     ingressClass: "traefik-internal"
  4.     # ...

CLI

  1. --providers.kubernetescrd.ingressclass=traefik-internal

Value of kubernetes.io/ingress.class annotation that identifies resource objects to be processed.

If the parameter is non-empty, only resources containing an annotation with the same value are processed. Otherwise, resources missing the annotation, having an empty value, or the value traefik are processed.

throttleDuration

Optional, Default: 0 (no throttling)

File (TOML)

  1. [providers.kubernetesCRD]
  2.   throttleDuration = "10s"
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     throttleDuration: "10s"
  4.     # ...

CLI

  1. --providers.kubernetescrd.throttleDuration=10s

allowCrossNamespace

Optional, Default: true

File (TOML)

  1. [providers.kubernetesCRD]
  2.   allowCrossNamespace = false
  3.   # ...

File (YAML)

  1. providers:
  2.   kubernetesCRD:
  3.     allowCrossNamespace: false
  4.     # ...

CLI

  1. --providers.kubernetescrd.allowCrossNamespace=false

If the parameter is set to false, an IngressRoute will not be able to reference any resources in another namespace than the IngressRoute namespace.

Deprecation

Please notice that the default value for this option will be set to false in a future version.

Further

Also see the full example with Let’s Encrypt.