Automatically Rotating Control Plane TLS Credentials

Linkerd’s automatic mTLS feature uses a set of TLS credentials to generate TLS certificates for proxies: a trust anchor, and an issuer certificate and private key. While Linkerd automatically rotates the TLS certificates for data plane proxies every 24 hours, it does not rotate the TLS credentials used to issue these certificate. In this doc, we’ll describe how to automatically rotate the issuer certificate and private key, by using an external solution.

(Note that Linkerd’s trust anchor must still be manually rotated on long-lived clusters.)

Cert manager

Cert-manager is a popular project for making TLS credentials from external sources available to Kubernetes clusters.

As a first step, install cert-manager on your cluster.

Note

If you are installing cert-manager >= 1.0, you will need to have kubernetes >= 1.16. Legacy custom resource definitions in cert-manager for kubernetes <= 1.15 do not have a keyAlgorithm option, so the certificates will be generated using RSA and be incompatible with linkerd.

See v0.16 to v1.0 upgrade notes for more details on version requirements.

Cert manager as an on-cluster CA

In this case, rather than pulling credentials from an external source, we’ll configure it to act as an on-cluster CA and have it re-issue Linkerd’s issuer certificate and private key on a periodic basis.

First, create the namespace that cert-manager will use to store its Linkerd-related resources. For simplicity, we suggest the default Linkerd control plane namespace:

  1. kubectl create namespace linkerd

Save the signing key pair as a Secret

Next, using the step tool, create a signing key pair and store it in a Kubernetes Secret in the namespace created above:

  1. step certificate create root.linkerd.cluster.local ca.crt ca.key \
  2.   --profile root-ca --no-password --insecure &&
  3.   kubectl create secret tls \
  4.     linkerd-trust-anchor \
  5.     --cert=ca.crt \
  6.     --key=ca.key \
  7.     --namespace=linkerd

For a longer-lived trust anchor certificate, pass the --not-after argument to the step command with the desired value (e.g. --not-after=87600h).

Create an Issuer referencing the secret

With the Secret in place, we can create a cert-manager “Issuer” resource that references it:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: cert-manager.io/v1
  3. kind: Issuer
  4. metadata:
  5.   name: linkerd-trust-anchor
  6.   namespace: linkerd
  7. spec:
  8.   ca:
  9.     secretName: linkerd-trust-anchor
  10. EOF

Issuing certificates and writing them to a secret

Finally, we can create a cert-manager “Certificate” resource which uses this Issuer to generate the desired certificate:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: cert-manager.io/v1
  3. kind: Certificate
  4. metadata:
  5.   name: linkerd-identity-issuer
  6.   namespace: linkerd
  7. spec:
  8.   secretName: linkerd-identity-issuer
  9.   duration: 48h
  10.   renewBefore: 25h
  11.   issuerRef:
  12.     name: linkerd-trust-anchor
  13.     kind: Issuer
  14.   commonName: identity.linkerd.cluster.local
  15.   dnsNames:
  16.   - identity.linkerd.cluster.local
  17.   isCA: true
  18.   privateKey:
  19.     algorithm: ECDSA
  20.   usages:
  21.   - cert sign
  22.   - crl sign
  23.   - server auth
  24.   - client auth
  25. EOF

(In the YAML manifest above, the duration key instructs cert-manager to consider certificates as valid for 48 hours and the renewBefore key indicates that cert-manager will attempt to issue a new certificate 25 hours before expiration of the current one. These values can be customized to your liking.)

At this point, cert-manager can now use this Certificate resource to obtain TLS credentials, which will be stored in a secret named linkerd-identity-issuer. To validate your newly-issued certificate, you can run:

  1. kubectl get secret linkerd-identity-issuer -o yaml -n linkerd

Now we just need to inform Linkerd to consume these credentials.

Note

Due to a bug in cert-manager, if you are using cert-manager version 0.15 with experimental controllers, the certificate it issues are not compatible with with Linkerd versions <= stable-2.8.1.

Your linkerd-identity pods will likely crash with the following log output:

  1. "Failed to initialize identity service: failed to read CA from disk:
  2. unsupported block type: 'PRIVATE KEY'"

Some possible ways to resolve this issue are:

  • Upgrade Linkerd to the edge versions >= edge-20.6.4 which contains a fix.
  • Upgrade cert-manager to versions >= 0.16. (how to upgrade)
  • Turn off cert-manager experimental controllers. (docs)

Alternative CA providers

Instead of using Cert Manager as CA, you can configure it to rely on a number of other solutions such as Vault. More detail on how to setup the existing Cert Manager to use different type of issuers can be found here.

Third party cert management solutions

It is important to note that the mechanism that Linkerd provides is also usable outside of cert-manager. Linkerd will read the linkerd-identity-issuer Secret, and if it’s of type kubernetes.io/tls, will use the contents as its TLS credentials. This means that any solution that is able to rotate TLS certificates by writing them to this secret can be used to provide dynamic TLS certificate management.

You could generate that secret with a command such as:

  1. kubectl create secret tls linkerd-identity-issuer --cert=issuer.crt --key=issuer.key --namespace=linkerd

Where issuer.crt and issuer.key would be the cert and private key of an intermediary cert rooted at the trust root (ca.crt) referred above (check this guide to see how to generate them).

Note that the root cert (ca.crt) needs to be included in that Secret as well. You can just edit the generated Secret and include the ca.crt field with the contents of the file base64-encoded.

After setting up the linkerd-identity-issuer Secret, continue with the following instructions to install and configure Linkerd to use it.

Using these credentials with CLI installation

For CLI installation, the Linkerd control plane should be installed with the --identity-external-issuer flag, which instructs Linkerd to read certificates from the linkerd-identity-issuer secret. Whenever certificate and key stored in the secret are updated, the identity service will automatically detect this change and reload the new credentials.

Voila! We have set up automatic rotation of Linkerd’s control plane TLS credentials. And if you want to monitor the update process, you can check the IssuerUpdated events emitted by the service:

  1. kubectl get events --field-selector reason=IssuerUpdated -n linkerd

Installing with Helm

For Helm installation, rather than running linkerd install, set the identityTrustAnchorsPEM to the value of ca.crt in the linkerd-identity-issuer Secret:

  1. helm install linkerd2 \
  2.   --set-file identityTrustAnchorsPEM=ca.crt \
  3.   --set identity.issuer.scheme=kubernetes.io/tls \
  4.   --set installNamespace=false \
  5.   linkerd/linkerd2 \
  6.   -n linkerd

Note

For Helm versions < v3, --name flag has to specifically be passed. In Helm v3, It has been deprecated, and is the first argument as specified above.

See Automatically Rotating Webhook TLS Credentials for how to do something similar for webhook TLS credentials.