Manually Rotating Control Plane TLS Credentials

Linkerd’s automatic mTLS feature uses a set of TLS credentials to generate TLS certificates for proxies: a trust anchor, and an issuer certificate and private key. The trust anchor has a limited period of validity: 365 days if generated by linkerd install, or a customized value if generated manually.

Thus, for clusters that are expected to outlive this lifetime, you must manually rotate the trust anchor. In this document, we describe how to accomplish this without downtime.

Independent of the trust anchor, the issuer certificate and key pair can also expire (though it is possible to use cert-manager to set up automatic rotation. This document also covers how to rotate the issuer certificate and key pair without downtime.

Prerequisites

These instructions use the step and jq CLI tools.

Understanding the current state of your system

Begin by running:

  1. linkerd check --proxy

If your configuration is valid and your credentials are not expiring soon, you should see output similar to:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. √ trust roots are within their validity period
  6. √ trust roots are valid for at least 60 days
  7. √ issuer cert is using supported crypto algorithm
  8. √ issuer cert is within its validity period
  9. √ issuer cert is valid for at least 60 days
  10. √ issuer cert is issued by the trust root
  12. linkerd-identity-data-plane
  13. ---------------------------
  14. √ data plane proxies certificate match CA

However, if you see a message warning you that your trust anchor (“trust root”) or issuer certificates are expiring soon, then you must rotate them.

Note that this document only applies if the trust root and issuer certificate are currently valid. If your trust anchor or issuer certificate have expired, please follow the Replacing Expired Certificates Guide instead.

For example, if your issuer certificate has expired, you will see a message similar to:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. √ trust roots are within their validity period
  6. √ trust roots are valid for at least 60 days
  7. √ issuer cert is using supported crypto algorithm
  8. × issuer cert is within its validity period
  9. issuer certificate is not valid anymore. Expired on 2019-12-19T09:02:01Z
  10. see https://linkerd.io/checks/#l5d-identity-issuer-cert-is-time-valid for hints

If your trust anchor has expired, you will see a message similar to:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. × trust roots are within their validity period
  6. Invalid roots:
  7. * 79461543992952791393769540277800684467 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T09:11:30Z
  8. see https://linkerd.io/checks/#l5d-identity-roots-are-time-valid  for hints

Rotating the trust anchor

Rotating the trust anchor without downtime is a multi-step process: you must generate a new trust anchor, bundle it with the old one, rotate the issuer certificate and key pair, and finally remove the old trust anchor from the bundle. If you simply need to rotate the issuer certificate and key pair, you can skip directly to Rotating the identity issuer certificate and ignore the trust anchor rotation steps.

Generate a new trust anchor

First, generate a new trust anchor certificate and private key:

  1. step certificate create identity.linkerd.cluster.local ca-new.crt ca-new.key --profile root-ca --no-password --insecure --san identity.linkerd.cluster.local

Note that we use --no-password --insecure to avoid encrypting these files with a passphrase. Store the private key somewhere secure so that it can be used in the future to generate new issuer certificates.

Bundle your original trust anchor with the new one

Next, we need to bundle the trust anchor currently used by Linkerd together with the new anchor. The following command uses kubectl to fetch the Linkerd config, jq to extract the current trust anchor, and step to combine it with the newly generated trust anchor:

  1. kubectl -n linkerd get cm linkerd-config -o=jsonpath='{.data.global}' |  \
  2. jq -r .identityContext.trustAnchorsPem > original-trust.crt
  3. step certificate bundle ca-new.crt original-trust.crt bundle.crt
  4. rm original-trust.crt

Deploying the new bundle to Linkerd

At this point you can use the linkerd upgrade command to instruct Linkerd to work with the new trust bundle:

  1. linkerd upgrade --identity-trust-anchors-file=./bundle.crt | kubectl apply -f -

This will restart the proxies in the Linkerd control plane, and they will be reconfigured with the new trust anchor.

Finally, you must restart the proxy for all injected workloads in your cluster. For example, doing that for the emojivoto namespace would look like:

  1. kubectl -n emojivoto rollout restart deploy

Now you can run the check command to ensure that everything is ok:

  1. linkerd check --proxy

You might have to wait a few moments until all the pods have been restarted and are configured with the correct trust anchor. Meanwhile you might observe warnings:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. √ trust roots are within their validity period
  6. √ trust roots are valid for at least 60 days
  7. √ issuer cert is using supported crypto algorithm
  8. √ issuer cert is within its validity period
  9. ‼ issuer cert is valid for at least 60 days
  10.     issuer certificate will expire on 2019-12-19T09:51:19Z
  11.     see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
  12. √ issuer cert is issued by the trust root
  14. linkerd-identity-data-plane
  15. ---------------------------
  16. ‼ data plane proxies certificate match CA
  17.     Some pods do not have the current trust bundle and must be restarted:
  18.         * emojivoto/emoji-d8d7d9c6b-8qwfx
  19.         * emojivoto/vote-bot-588499c9f6-zpwz6
  20.         * emojivoto/voting-8599548fdc-6v64k
  21.         * emojivoto/web-67c7599f6d-xx98n
  22.         * linkerd/linkerd-sp-validator-75f9d96dc-rch4x
  23.         * linkerd/linkerd-tap-68d8bbf64-mpzgb
  24.         * linkerd/linkerd-web-849f74b7c6-qlhwc
  25.     see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

When the rollout completes, your check command should stop warning you that pods need to be restarted. It may still warn you, however, that your issuer certificate is about to expire soon:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. √ trust roots are within their validity period
  6. √ trust roots are valid for at least 60 days
  7. √ issuer cert is using supported crypto algorithm
  8. √ issuer cert is within its validity period
  9. ‼ issuer cert is valid for at least 60 days
  10.     issuer certificate will expire on 2019-12-19T09:51:19Z
  11.     see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
  12. √ issuer cert is issued by the trust root
  14. linkerd-identity-data-plane
  15. ---------------------------
  16. √ data plane proxies certificate match CA

Rotating the identity issuer certificate

To rotate the issuer certificate and key pair, first generate a new pair:

  1. step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key --ca ca-new.crt --ca-key ca-new.key --profile intermediate-ca --not-after 8760h --no-password --insecure --san identity.linkerd.cluster.local

Provided that the trust anchor has not expired and that, if recently rotated, all proxies have been updated to include a working trust anchor (as outlined in the previous section) it is now safe to rotate the identity issuer certificate by using the upgrade command again:

  1. linkerd upgrade  --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key | kubectl apply -f -

At this point Linkerd’s identity control plane service should detect the change of the secret and automatically update its issuer certificates.

To ensure this has happened, you can check for the specific Kubernetes event:

  1. kubectl get events --field-selector reason=IssuerUpdated -n linkerd
  3. LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
  4. 9s          Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity issuer

Restart the proxy for all injected workloads in your cluster to ensure that their proxies pick up certificates issued by the new issuer:

  1. kubectl -n emojivoto rollout restart deploy

Run the check command to make sure that everything is going as expected:

  1. linkerd check --proxy

You should see output without any certificate expiration warnings:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. √ trust roots are within their validity period
  6. √ trust roots are valid for at least 60 days
  7. √ issuer cert is using supported crypto algorithm
  8. √ issuer cert is within its validity period
  9. √ issuer cert is valid for at least 60 days
  10. √ issuer cert is issued by the trust root
  12. linkerd-identity-data-plane
  13. ---------------------------
  14. √ data plane proxies certificate match CA

Removing the old trust anchor

We can now remove the old trust anchor from the trust bundle we created earlier. The upgrade command can do that for the Linkerd components:

  1. linkerd upgrade  --identity-trust-anchors-file=./ca-new.crt  | kubectl apply -f -

Note that the ./ca-new.crt file is the same trust anchor you created at the start of this process. Additionally, you can use the rollout restart command to bring the configuration of your other injected resources up to date:

  1. kubectl -n emojivoto rollout restart deploy
  2. linkerd check --proxy

Finally the output of the check command should not produce any warnings or errors:

  1. linkerd-identity
  2. ----------------
  3. √ certificate config is valid
  4. √ trust roots are using supported crypto algorithm
  5. √ trust roots are within their validity period
  6. √ trust roots are valid for at least 60 days
  7. √ issuer cert is using supported crypto algorithm
  8. √ issuer cert is within its validity period
  9. √ issuer cert is valid for at least 60 days
  10. √ issuer cert is issued by the trust root
  12. linkerd-identity-data-plane
  13. ---------------------------
  14. √ data plane proxies certificate match CA

Congratulations, you have rotated your trust anchor! 🎉