Traefik & CRD & Let’s Encrypt
Traefik with an IngressRoute Custom Resource Definition for Kubernetes, and TLS Through Let’s Encrypt.
This document is intended to be a fully working example demonstrating how to set up Traefik in Kubernetes, with the dynamic configuration coming from the IngressRoute Custom Resource, and TLS setup with Let’s Encrypt. However, for the sake of simplicity, we’re using k3s docker image for the Kubernetes cluster setup.
Please note that for this setup, given that we’re going to use ACME’s TLS-ALPN-01 challenge, the host you’ll be running it on must be able to receive connections from the outside on port 443. And of course its internet facing IP address must match the domain name you intend to use.
In the following, the Kubernetes resources defined in YAML configuration files can be applied to the setup in two different ways:
- the first, and usual way, is simply with the
kubectl apply
command. - the second, which can be used for this tutorial, is to directly place the files in the directory used by the k3s docker image for such inputs (
/var/lib/rancher/k3s/server/manifests
).
Kubectl Version
With the rancher/k3s
version used in this guide (0.8.0
), the kubectl version needs to be >= 1.11
.
k3s Docker-compose Configuration
Our starting point is the docker-compose configuration file, to start the k3s cluster. You can start it with:
docker-compose -f k3s.yml up
server:
image: rancher/k3s:v1.17.2-k3s1
command: server --disable-agent --no-deploy traefik
environment:
- K3S_CLUSTER_SECRET=somethingtotallyrandom
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- K3S_KUBECONFIG_MODE=666
volumes:
# k3s will generate a kubeconfig.yaml in this directory. This volume is mounted
# on your host, so you can then 'export KUBECONFIG=/somewhere/on/your/host/out/kubeconfig.yaml',
# in order for your kubectl commands to work.
- /somewhere/on/your/host/out:/output
# This directory is where you put all the (yaml) configuration files of
# the Kubernetes resources.
- /somewhere/on/your/host/in:/var/lib/rancher/k3s/server/manifests
ports:
- 6443:6443
node:
image: rancher/k3s:v1.17.2-k3s1
privileged: true
links:
- server
environment:
- K3S_URL=https://server:6443
- K3S_CLUSTER_SECRET=somethingtotallyrandom
volumes:
# this is where you would place a alternative traefik image (saved as a .tar file with
# 'docker save'), if you want to use it, instead of the traefik:v2.10 image.
- /somewhere/on/your/host/custom-image:/var/lib/rancher/k3s/agent/images
Cluster Resources
Let’s now have a look (in the order they should be applied, if using kubectl apply
) at all the required resources for the full setup.
IngressRoute Definition
First, you will need to install Traefik CRDs containing the definition of the IngressRoute
and the Middleware
kinds, and the RBAC authorization resources which will be referenced through the serviceAccountName
of the deployment.
# Install Traefik Resource Definitions:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
# Install RBAC for Traefik:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml
Services
Then, the services. One for Traefik itself, and one for the app it routes for, i.e. in this case our demo HTTP server: whoami.
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/user-guides/crd-acme/02-services.yml
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
ports:
- protocol: TCP
name: web
port: 8000
- protocol: TCP
name: admin
port: 8080
- protocol: TCP
name: websecure
port: 4443
selector:
app: traefik
---
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
ports:
- protocol: TCP
name: web
port: 80
selector:
app: whoami
Deployments
Next, the deployments, i.e. the actual pods behind the services. Again, one pod for Traefik, and one for the whoami app.
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/user-guides/crd-acme/03-deployments.yml
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: default
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.10
args:
- --api.insecure
- --accesslog
- --entrypoints.web.Address=:8000
- --entrypoints.websecure.Address=:4443
- --providers.kubernetescrd
- --certificatesresolvers.myresolver.acme.tlschallenge
- [email protected]
- --certificatesresolvers.myresolver.acme.storage=acme.json
# Please note that this is the staging Let's Encrypt server.
# Once you get things working, you should remove that whole line altogether.
- --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
ports:
- name: web
containerPort: 8000
- name: websecure
containerPort: 4443
- name: admin
containerPort: 8080
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: whoami
labels:
app: whoami
spec:
replicas: 2
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami
ports:
- name: web
containerPort: 80
Port Forwarding
Now, as an exception to what we said above, please note that you should not let the ingressRoute resources below be applied automatically to your cluster. The reason is, as soon as the ACME provider of Traefik detects we have TLS routers, it will try to generate the certificates for the corresponding domains. And this will not work, because as it is, our Traefik pod is not reachable from the outside, which will make the ACME TLS challenge fail. Therefore, for the whole thing to work, we must delay applying the ingressRoute resources until we have port-forwarding set up properly, which is the next step.
kubectl port-forward --address 0.0.0.0 service/traefik 8000:8000 8080:8080 443:4443 -n default
Also, and this is out of the scope if this guide, please note that because of the privileged ports limitation on Linux, the above command might fail to listen on port 443. In which case you can use tricks such as elevating caps of kubectl
with setcaps
, or using authbind
, or setting up a NAT between your host and the WAN. Look it up.
Traefik Routers
We can now finally apply the actual ingressRoutes, with:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/user-guides/crd-acme/04-ingressroutes.yml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: simpleingressroute
namespace: default
spec:
entryPoints:
- web
routes:
- match: Host(`your.example.com`) && PathPrefix(`/notls`)
kind: Rule
services:
- name: whoami
port: 80
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: ingressroutetls
namespace: default
spec:
entryPoints:
- websecure
routes:
- match: Host(`your.example.com`) && PathPrefix(`/tls`)
kind: Rule
services:
- name: whoami
port: 80
tls:
certResolver: myresolver
Give it a few seconds for the ACME TLS challenge to complete, and you should then be able to access your whoami pod (routed through Traefik), from the outside. Both with or (just for fun, do not do that in production) without TLS:
curl [-k] https://your.example.com/tls
curl http://your.example.com:8000/notls
Note that you’ll have to use -k
as long as you’re using the staging server of Let’s Encrypt, since it is not an authorized certificate authority on systems where it hasn’t been manually added.
Force TLS v1.2+
Nowadays, TLS v1.0 and v1.1 are deprecated. In order to force TLS v1.2 or later on all your IngressRoute, you can define the default
TLSOption:
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/user-guides/crd-acme/05-tlsoption.yml
---
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
name: default
namespace: default
spec:
minVersion: VersionTLS12
cipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 # TLS 1.2
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 # TLS 1.2
- TLS_AES_256_GCM_SHA384 # TLS 1.3
- TLS_CHACHA20_POLY1305_SHA256 # TLS 1.3
curvePreferences:
- CurveP521
- CurveP384
sniStrict: true