Traefik & CRD & Let’s Encrypt

Traefik with an IngressRoute Custom Resource Definition for Kubernetes, and TLS Through Let’s Encrypt.

This document is intended to be a fully working example demonstrating how to set up Traefik in Kubernetes, with the dynamic configuration coming from the IngressRoute Custom Resource, and TLS setup with Let’s Encrypt. However, for the sake of simplicity, we’re using k3s docker image for the Kubernetes cluster setup.

Please note that for this setup, given that we’re going to use ACME’s TLS-ALPN-01 challenge, the host you’ll be running it on must be able to receive connections from the outside on port 443. And of course its internet facing IP address must match the domain name you intend to use.

In the following, the Kubernetes resources defined in YAML configuration files can be applied to the setup in two different ways:

  • the first, and usual way, is simply with the kubectl apply command.
  • the second, which can be used for this tutorial, is to directly place the files in the directory used by the k3s docker image for such inputs (/var/lib/rancher/k3s/server/manifests).

Kubectl Version

With the rancher/k3s version used in this guide (0.8.0), the kubectl version needs to be >= 1.11.

k3s Docker-compose Configuration

Our starting point is the docker-compose configuration file, to start the k3s cluster. You can start it with:

  1. docker-compose -f k3s.yml up
  1. server:
  2.   image: rancher/k3s:v1.17.2-k3s1
  3.   command: server --disable-agent --no-deploy traefik
  4.   environment:
  5.     - K3S_CLUSTER_SECRET=somethingtotallyrandom
  6.     - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
  7.     - K3S_KUBECONFIG_MODE=666
  8.   volumes:
  9.     # k3s will generate a kubeconfig.yaml in this directory. This volume is mounted
  10.     # on your host, so you can then 'export KUBECONFIG=/somewhere/on/your/host/out/kubeconfig.yaml',
  11.     # in order for your kubectl commands to work.
  12.     - /somewhere/on/your/host/out:/output
  13.     # This directory is where you put all the (yaml) configuration files of
  14.     # the Kubernetes resources.
  15.     - /somewhere/on/your/host/in:/var/lib/rancher/k3s/server/manifests
  16.   ports:
  17.     - 6443:6443
  19. node:
  20.   image: rancher/k3s:v1.17.2-k3s1
  21.   privileged: true
  22.   links:
  23.     - server
  24.   environment:
  25.     - K3S_URL=https://server:6443
  26.     - K3S_CLUSTER_SECRET=somethingtotallyrandom
  27.   volumes:
  28.     # this is where you would place a alternative traefik image (saved as a .tar file with
  29.     # 'docker save'), if you want to use it, instead of the traefik:v2.4 image.
  30.     - /sowewhere/on/your/host/custom-image:/var/lib/rancher/k3s/agent/images

Cluster Resources

Let’s now have a look (in the order they should be applied, if using kubectl apply) at all the required resources for the full setup.

IngressRoute Definition

First, the definition of the IngressRoute and the Middleware kinds. Also note the RBAC authorization resources; they’ll be referenced through the serviceAccountName of the deployment, later on.

  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4.   name: ingressroutes.traefik.containo.us
  6. spec:
  7.   group: traefik.containo.us
  8.   version: v1alpha1
  9.   names:
  10.     kind: IngressRoute
  11.     plural: ingressroutes
  12.     singular: ingressroute
  13.   scope: Namespaced
  15. ---
  16. apiVersion: apiextensions.k8s.io/v1beta1
  17. kind: CustomResourceDefinition
  18. metadata:
  19.   name: middlewares.traefik.containo.us
  21. spec:
  22.   group: traefik.containo.us
  23.   version: v1alpha1
  24.   names:
  25.     kind: Middleware
  26.     plural: middlewares
  27.     singular: middleware
  28.   scope: Namespaced
  30. ---
  31. apiVersion: apiextensions.k8s.io/v1beta1
  32. kind: CustomResourceDefinition
  33. metadata:
  34.   name: ingressroutetcps.traefik.containo.us
  36. spec:
  37.   group: traefik.containo.us
  38.   version: v1alpha1
  39.   names:
  40.     kind: IngressRouteTCP
  41.     plural: ingressroutetcps
  42.     singular: ingressroutetcp
  43.   scope: Namespaced
  45. ---
  46. apiVersion: apiextensions.k8s.io/v1beta1
  47. kind: CustomResourceDefinition
  48. metadata:
  49.   name: ingressrouteudps.traefik.containo.us
  51. spec:
  52.   group: traefik.containo.us
  53.   version: v1alpha1
  54.   names:
  55.     kind: IngressRouteUDP
  56.     plural: ingressrouteudps
  57.     singular: ingressrouteudp
  58.   scope: Namespaced
  60. ---
  61. apiVersion: apiextensions.k8s.io/v1beta1
  62. kind: CustomResourceDefinition
  63. metadata:
  64.   name: tlsoptions.traefik.containo.us
  66. spec:
  67.   group: traefik.containo.us
  68.   version: v1alpha1
  69.   names:
  70.     kind: TLSOption
  71.     plural: tlsoptions
  72.     singular: tlsoption
  73.   scope: Namespaced
  75. ---
  76. apiVersion: apiextensions.k8s.io/v1beta1
  77. kind: CustomResourceDefinition
  78. metadata:
  79.   name: tlsstores.traefik.containo.us
  81. spec:
  82.   group: traefik.containo.us
  83.   version: v1alpha1
  84.   names:
  85.     kind: TLSStore
  86.     plural: tlsstores
  87.     singular: tlsstore
  88.   scope: Namespaced
  90. ---
  91. apiVersion: apiextensions.k8s.io/v1beta1
  92. kind: CustomResourceDefinition
  93. metadata:
  94.   name: traefikservices.traefik.containo.us
  96. spec:
  97.   group: traefik.containo.us
  98.   version: v1alpha1
  99.   names:
  100.     kind: TraefikService
  101.     plural: traefikservices
  102.     singular: traefikservice
  103.   scope: Namespaced
  105. ---
  106. apiVersion: apiextensions.k8s.io/v1beta1
  107. kind: CustomResourceDefinition
  108. metadata:
  109.   name: serverstransports.traefik.containo.us
  111. spec:
  112.   group: traefik.containo.us
  113.   version: v1alpha1
  114.   names:
  115.     kind: ServersTransport
  116.     plural: serverstransports
  117.     singular: serverstransport
  118.   scope: Namespaced
  120. ---
  121. kind: ClusterRole
  122. apiVersion: rbac.authorization.k8s.io/v1beta1
  123. metadata:
  124.   name: traefik-ingress-controller
  126. rules:
  127.   - apiGroups:
  128.       - ""
  129.     resources:
  130.       - services
  131.       - endpoints
  132.       - secrets
  133.     verbs:
  134.       - get
  135.       - list
  136.       - watch
  137.   - apiGroups:
  138.       - extensions
  139.       - networking.k8s.io
  140.     resources:
  141.       - ingresses
  142.       - ingressclasses
  143.     verbs:
  144.       - get
  145.       - list
  146.       - watch
  147.   - apiGroups:
  148.       - extensions
  149.     resources:
  150.       - ingresses/status
  151.     verbs:
  152.       - update
  153.   - apiGroups:
  154.       - traefik.containo.us
  155.     resources:
  156.       - middlewares
  157.       - ingressroutes
  158.       - traefikservices
  159.       - ingressroutetcps
  160.       - ingressrouteudps
  161.       - tlsoptions
  162.       - tlsstores
  163.       - serverstransports
  164.     verbs:
  165.       - get
  166.       - list
  167.       - watch
  169. ---
  170. kind: ClusterRoleBinding
  171. apiVersion: rbac.authorization.k8s.io/v1beta1
  172. metadata:
  173.   name: traefik-ingress-controller
  175. roleRef:
  176.   apiGroup: rbac.authorization.k8s.io
  177.   kind: ClusterRole
  178.   name: traefik-ingress-controller
  179. subjects:
  180.   - kind: ServiceAccount
  181.     name: traefik-ingress-controller
  182.     namespace: default

Services

Then, the services. One for Traefik itself, and one for the app it routes for, i.e. in this case our demo HTTP server: whoami.

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: traefik
  6. spec:
  7.   ports:
  8.     - protocol: TCP
  9.       name: web
  10.       port: 8000
  11.     - protocol: TCP
  12.       name: admin
  13.       port: 8080
  14.     - protocol: TCP
  15.       name: websecure
  16.       port: 4443
  17.   selector:
  18.     app: traefik
  20. ---
  21. apiVersion: v1
  22. kind: Service
  23. metadata:
  24.   name: whoami
  26. spec:
  27.   ports:
  28.     - protocol: TCP
  29.       name: web
  30.       port: 80
  31.   selector:
  32.     app: whoami

Deployments

Next, the deployments, i.e. the actual pods behind the services. Again, one pod for Traefik, and one for the whoami app.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   namespace: default
  5.   name: traefik-ingress-controller
  7. ---
  8. kind: Deployment
  9. apiVersion: apps/v1
  10. metadata:
  11.   namespace: default
  12.   name: traefik
  13.   labels:
  14.     app: traefik
  16. spec:
  17.   replicas: 1
  18.   selector:
  19.     matchLabels:
  20.       app: traefik
  21.   template:
  22.     metadata:
  23.       labels:
  24.         app: traefik
  25.     spec:
  26.       serviceAccountName: traefik-ingress-controller
  27.       containers:
  28.         - name: traefik
  29.           image: traefik:v2.4
  30.           args:
  31.             - --api.insecure
  32.             - --accesslog
  33.             - --entrypoints.web.Address=:8000
  34.             - --entrypoints.websecure.Address=:4443
  35.             - --providers.kubernetescrd
  36.             - --certificatesresolvers.myresolver.acme.tlschallenge
  37.             - --certificatesresolvers.myresolver.acme.email=foo@you.com
  38.             - --certificatesresolvers.myresolver.acme.storage=acme.json
  39.             # Please note that this is the staging Let's Encrypt server.
  40.             # Once you get things working, you should remove that whole line altogether.
  41.             - --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
  42.           ports:
  43.             - name: web
  44.               containerPort: 8000
  45.             - name: websecure
  46.               containerPort: 4443
  47.             - name: admin
  48.               containerPort: 8080
  50. ---
  51. kind: Deployment
  52. apiVersion: apps/v1
  53. metadata:
  54.   namespace: default
  55.   name: whoami
  56.   labels:
  57.     app: whoami
  59. spec:
  60.   replicas: 2
  61.   selector:
  62.     matchLabels:
  63.       app: whoami
  64.   template:
  65.     metadata:
  66.       labels:
  67.         app: whoami
  68.     spec:
  69.       containers:
  70.         - name: whoami
  71.           image: traefik/whoami
  72.           ports:
  73.             - name: web
  74.               containerPort: 80

Port Forwarding

Now, as an exception to what we said above, please note that you should not let the ingressRoute resources below be applied automatically to your cluster. The reason is, as soon as the ACME provider of Traefik detects we have TLS routers, it will try to generate the certificates for the corresponding domains. And this will not work, because as it is, our Traefik pod is not reachable from the outside, which will make the ACME TLS challenge fail. Therefore, for the whole thing to work, we must delay applying the ingressRoute resources until we have port-forwarding set up properly, which is the next step.

  1. kubectl port-forward --address 0.0.0.0 service/traefik 8000:8000 8080:8080 443:4443 -n default

Also, and this is out of the scope if this guide, please note that because of the privileged ports limitation on Linux, the above command might fail to listen on port 443. In which case you can use tricks such as elevating caps of kubectl with setcaps, or using authbind, or setting up a NAT between your host and the WAN. Look it up.

Traefik Routers

We can now finally apply the actual ingressRoutes, with:

  1. kubectl apply -f 04-ingressroutes.yml
  1. apiVersion: traefik.containo.us/v1alpha1
  2. kind: IngressRoute
  3. metadata:
  4.   name: simpleingressroute
  5.   namespace: default
  6. spec:
  7.   entryPoints:
  8.     - web
  9.   routes:
  10.   - match: Host(`your.example.com`) && PathPrefix(`/notls`)
  11.     kind: Rule
  12.     services:
  13.     - name: whoami
  14.       port: 80
  16. ---
  17. apiVersion: traefik.containo.us/v1alpha1
  18. kind: IngressRoute
  19. metadata:
  20.   name: ingressroutetls
  21.   namespace: default
  22. spec:
  23.   entryPoints:
  24.     - websecure
  25.   routes:
  26.   - match: Host(`your.example.com`) && PathPrefix(`/tls`)
  27.     kind: Rule
  28.     services:
  29.     - name: whoami
  30.       port: 80
  31.   tls:
  32.     certResolver: myresolver

Give it a few seconds for the ACME TLS challenge to complete, and you should then be able to access your whoami pod (routed through Traefik), from the outside. Both with or (just for fun, do not do that in production) without TLS:

  1. curl [-k] https://your.example.com/tls
  1. curl http://your.example.com:8000/notls

Note that you’ll have to use -k as long as you’re using the staging server of Let’s Encrypt, since it is not an authorized certificate authority on systems where it hasn’t been manually added.