Rotating webhooks certificates
Linkerd uses the Kubernetes admission webhooks and extension API server to implement some of its core features like automatic proxy injection and service profiles validation.
Also, the viz extension uses a webhook to make pods tappable, as does the jaeger extension to turn on tracing on pods.
To secure the connections between the Kubernetes API server and the webhooks, all the webhooks are TLS-enabled. The x509 certificates used by these webhooks are issued by the self-signed CA certificates embedded in the webhooks configuration.
By default, these certificates have a validity period of 365 days. They are stored in the following secrets:
- In the
linkerd
namespace:linkerd-proxy-injector-k8s-tls
andlinkerd-sp-validator-k8s-tls
- In the
linkerd-viz
namespace:tap-injector-k8s-tls
- In the
linkerd-jaeger
namespace:jaeger-injector-k8s-tls
The rest of this documentation provides instructions on how to renew these certificates.
Renewing the webhook certificates
To check the validity of all the TLS secrets (using step
):
# assuming you have viz and jaeger installed, otherwise trim down these arrays
# accordingly
SECRETS=("linkerd-proxy-injector-k8s-tls" "linkerd-sp-validator-k8s-tls" "tap-injector-k8s-tls" "jaeger-injector-k8s-tls")
NS=("linkerd" "linkerd" "linkerd-viz" "linkerd-jaeger")
for idx in "${!SECRETS[@]}"; do \
kubectl -n "${NS[$idx]}" get secret "${SECRETS[$idx]}" -ojsonpath='{.data.tls\.crt}' | \
base64 --decode - | \
step certificate inspect - | \
grep -iA2 validity; \
done
Manually delete these secrets and use upgrade
/install
to recreate them:
for idx in "${!SECRETS[@]}"; do \
kubectl -n "${NS[$idx]}" delete secret "${SECRETS[$idx]}"; \
done
linkerd upgrade | kubectl apply -f -
linkerd viz install | kubectl apply -f -
linkerd jaeger install | kubectl apply -f -
The above command will recreate the secrets without restarting Linkerd.
Note
For Helm users, use the helm upgrade
command to recreate the deleted secrets.
If you render the helm charts externally and apply them with kubectl apply
(e.g. in a CI/CD pipeline), you do not need to delete the secrets manually, as they wil be overwritten by a new cert and key generated by the helm chart.
Confirm that the secrets are recreated with new certificates:
for idx in "${!SECRETS[@]}"; do \
kubectl -n "${NS[$idx]}" get secret "${SECRETS[$idx]}" -ojsonpath='{.data.crt\.pem}' | \
base64 --decode - | \
step certificate inspect - | \
grep -iA2 validity; \
done
Ensure that Linkerd remains healthy:
linkerd check
Restarting the pods that implement the webhooks and API services is usually not necessary. But if the cluster is large, or has a high pod churn, it may be advisable to restart the pods manually, to avoid cascading failures.
If you observe certificate expiry errors or mismatched CA certs, restart their pods with:
kubectl -n linkerd rollout restart deploy \
linkerd-proxy-injector \
linkerd-sp-validator \
kubectl -n linkerd-viz rollout restart deploy tap tap-injector
kubectl -n linkerd-jaeger rollout restart deploy jaeger-injector