Automatically Rotating Webhook TLS Credentials

The Linkerd control plane contains several components, called webhooks, which are called directly by Kubernetes itself. The traffic from Kubernetes to the Linkerd webhooks is secured with TLS and therefore each of the webhooks requires a secret containing TLS credentials. These certificates are different from the ones that the Linkerd proxies use to secure pod-to-pod communication and use a completely separate trust chain. For more information on rotating the TLS credentials used by the Linkerd proxies, see Automatically Rotating Control Plane TLS Credentials.

By default, when Linkerd is installed with the Linkerd CLI or with the Linkerd Helm chart, TLS credentials are automatically generated for all of the webhooks. If these certificates expire or need to be regenerated for any reason, performing a Linkerd upgrade (using the Linkerd CLI or using Helm) will regenerate them.

This workflow is suitable for most users. However, if you need these webhook certificates to be rotated automatically on a regular basis, it is possible to use cert-manager to automatically manage them.

Automatically Rotating Webhook TLS Credentials - 图1

Linkerd Production Tip

This page contains best-effort instructions by the open source community. Production users with mission-critical applications should familiarize themselves with Linkerd production resources and/or connect with a commercial Linkerd provider.

Install Cert manager

As a first step, install cert-manager on your cluster and create the namespaces that cert-manager will use to store its webhook-related resources. For simplicity, we suggest using the default namespace linkerd uses:

  1. # control plane core
  2. kubectl create namespace linkerd
  3. kubectl label namespace linkerd \
  4. linkerd.io/is-control-plane=true \
  5. config.linkerd.io/admission-webhooks=disabled \
  6. linkerd.io/control-plane-ns=linkerd
  7. kubectl annotate namespace linkerd linkerd.io/inject=disabled
  8. # viz (ignore if not using the viz extension)
  9. kubectl create namespace linkerd-viz
  10. kubectl label namespace linkerd-viz linkerd.io/extension=viz
  11. # jaeger (ignore if not using the jaeger extension)
  12. kubectl create namespace linkerd-jaeger
  13. kubectl label namespace linkerd-jaeger linkerd.io/extension=jaeger

Save the signing key pair as a Secret

Next, we will use the step tool, to create a signing key pair which will be used to sign each of the webhook certificates:

  1. step certificate create webhook.linkerd.cluster.local ca.crt ca.key \
  2. --profile root-ca --no-password --insecure --san webhook.linkerd.cluster.local
  3. kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd
  4. # ignore if not using the viz extension
  5. kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd-viz
  6. # ignore if not using the jaeger extension
  7. kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd-jaeger

Create Issuers referencing the secrets

With the Secrets in place, we can create cert-manager “Issuer” resources that reference them:

  1. kubectl apply -f - <<EOF
  2. apiVersion: cert-manager.io/v1
  3. kind: Issuer
  4. metadata:
  5. name: webhook-issuer
  6. namespace: linkerd
  7. spec:
  8. ca:
  9. secretName: webhook-issuer-tls
  10. ---
  11. # ignore if not using the viz extension
  12. apiVersion: cert-manager.io/v1
  13. kind: Issuer
  14. metadata:
  15. name: webhook-issuer
  16. namespace: linkerd-viz
  17. spec:
  18. ca:
  19. secretName: webhook-issuer-tls
  20. ---
  21. # ignore if not using the jaeger extension
  22. apiVersion: cert-manager.io/v1
  23. kind: Issuer
  24. metadata:
  25. name: webhook-issuer
  26. namespace: linkerd-jaeger
  27. spec:
  28. ca:
  29. secretName: webhook-issuer-tls
  30. EOF

Issuing certificates and writing them to secrets

Finally, we can create cert-manager “Certificate” resources which use the Issuers to generate the desired certificates:

  1. kubectl apply -f - <<EOF
  2. apiVersion: cert-manager.io/v1
  3. kind: Certificate
  4. metadata:
  5. name: linkerd-policy-validator
  6. namespace: linkerd
  7. spec:
  8. secretName: linkerd-policy-validator-k8s-tls
  9. duration: 24h
  10. renewBefore: 1h
  11. issuerRef:
  12. name: webhook-issuer
  13. kind: Issuer
  14. commonName: linkerd-policy-validator.linkerd.svc
  15. dnsNames:
  16. - linkerd-policy-validator.linkerd.svc
  17. isCA: false
  18. privateKey:
  19. algorithm: ECDSA
  20. encoding: PKCS8
  21. usages:
  22. - server auth
  23. ---
  24. apiVersion: cert-manager.io/v1
  25. kind: Certificate
  26. metadata:
  27. name: linkerd-proxy-injector
  28. namespace: linkerd
  29. spec:
  30. secretName: linkerd-proxy-injector-k8s-tls
  31. duration: 24h
  32. renewBefore: 1h
  33. issuerRef:
  34. name: webhook-issuer
  35. kind: Issuer
  36. commonName: linkerd-proxy-injector.linkerd.svc
  37. dnsNames:
  38. - linkerd-proxy-injector.linkerd.svc
  39. isCA: false
  40. privateKey:
  41. algorithm: ECDSA
  42. usages:
  43. - server auth
  44. ---
  45. apiVersion: cert-manager.io/v1
  46. kind: Certificate
  47. metadata:
  48. name: linkerd-sp-validator
  49. namespace: linkerd
  50. spec:
  51. secretName: linkerd-sp-validator-k8s-tls
  52. duration: 24h
  53. renewBefore: 1h
  54. issuerRef:
  55. name: webhook-issuer
  56. kind: Issuer
  57. commonName: linkerd-sp-validator.linkerd.svc
  58. dnsNames:
  59. - linkerd-sp-validator.linkerd.svc
  60. isCA: false
  61. privateKey:
  62. algorithm: ECDSA
  63. usages:
  64. - server auth
  65. ---
  66. # ignore if not using the viz extension
  67. apiVersion: cert-manager.io/v1
  68. kind: Certificate
  69. metadata:
  70. name: tap
  71. namespace: linkerd-viz
  72. spec:
  73. secretName: tap-k8s-tls
  74. duration: 24h
  75. renewBefore: 1h
  76. issuerRef:
  77. name: webhook-issuer
  78. kind: Issuer
  79. commonName: tap.linkerd-viz.svc
  80. dnsNames:
  81. - tap.linkerd-viz.svc
  82. isCA: false
  83. privateKey:
  84. algorithm: ECDSA
  85. usages:
  86. - server auth
  87. ---
  88. # ignore if not using the viz extension
  89. apiVersion: cert-manager.io/v1
  90. kind: Certificate
  91. metadata:
  92. name: linkerd-tap-injector
  93. namespace: linkerd-viz
  94. spec:
  95. secretName: tap-injector-k8s-tls
  96. duration: 24h
  97. renewBefore: 1h
  98. issuerRef:
  99. name: webhook-issuer
  100. kind: Issuer
  101. commonName: tap-injector.linkerd-viz.svc
  102. dnsNames:
  103. - tap-injector.linkerd-viz.svc
  104. isCA: false
  105. privateKey:
  106. algorithm: ECDSA
  107. usages:
  108. - server auth
  109. ---
  110. # ignore if not using the jaeger extension
  111. apiVersion: cert-manager.io/v1
  112. kind: Certificate
  113. metadata:
  114. name: jaeger-injector
  115. namespace: linkerd-jaeger
  116. spec:
  117. secretName: jaeger-injector-k8s-tls
  118. duration: 24h
  119. renewBefore: 1h
  120. issuerRef:
  121. name: webhook-issuer
  122. kind: Issuer
  123. commonName: jaeger-injector.linkerd-jaeger.svc
  124. dnsNames:
  125. - jaeger-injector.linkerd-jaeger.svc
  126. isCA: false
  127. privateKey:
  128. algorithm: ECDSA
  129. usages:
  130. - server auth
  131. EOF

At this point, cert-manager can now use these Certificate resources to obtain TLS credentials, which are stored in the linkerd-proxy-injector-k8s-tls, linkerd-sp-validator-k8s-tls, tap-k8s-tls, tap-injector-k8s-tls and jaeger-injector-k8s-tls secrets respectively.

Now we just need to inform Linkerd to consume these credentials.

Using these credentials with CLI installation

To configure Linkerd to use the credentials from cert-manager rather than generating its own:

  1. # first, install the Linkerd CRDs
  2. linkerd install --crds | kubectl apply -f -
  3. # install the Linkerd control plane, using the credentials
  4. # from cert-manager
  5. linkerd install \
  6. --set policyValidator.externalSecret=true \
  7. --set-file policyValidator.caBundle=ca.crt \
  8. --set proxyInjector.externalSecret=true \
  9. --set-file proxyInjector.caBundle=ca.crt \
  10. --set profileValidator.externalSecret=true \
  11. --set-file profileValidator.caBundle=ca.crt \
  12. | kubectl apply -f -
  13. # ignore if not using the viz extension
  14. linkerd viz install \
  15. --set tap.externalSecret=true \
  16. --set-file tap.caBundle=ca.crt \
  17. --set tapInjector.externalSecret=true \
  18. --set-file tapInjector.caBundle=ca.crt \
  19. | kubectl apply -f -
  20. # ignore if not using the jaeger extension
  21. linkerd jaeger install
  22. --set webhook.externalSecret=true \
  23. --set-file webhook.caBundle=ca.crt \
  24. | kubectl apply -f -

Installing with Helm

A similar pattern can be used with Helm:

  1. # first install the linkerd-crds chart
  2. helm install linkerd-crds linkerd/linkerd-crds \
  3. -n linkerd --create-namespace
  4. # then install the linkerd-control-plane chart
  5. # (see note below)
  6. helm install linkerd-control-plane \
  7. --set-file identityTrustAnchorsPEM=... \
  8. --set-file identity.issuer.tls.crtPEM=... \
  9. --set-file identity.issuer.tls.keyPEM=... \
  10. --set policyValidator.externalSecret=true \
  11. --set-file policyValidator.caBundle=ca.crt \
  12. --set proxyInjector.externalSecret=true \
  13. --set-file proxyInjector.caBundle=ca.crt \
  14. --set profileValidator.externalSecret=true \
  15. --set-file profileValidator.caBundle=ca.crt \
  16. linkerd/linkerd-control-plane \
  17. -n linkerd
  18. # ignore if not using the viz extension
  19. helm install linkerd-viz \
  20. --set tap.externalSecret=true \
  21. --set-file tap.caBundle=ca.crt \
  22. --set tapInjector.externalSecret=true \
  23. --set-file tapInjector.caBundle=ca.crt \
  24. linkerd/linkerd-viz \
  25. -n linkerd-viz --create-namespace
  26. # ignore if not using the jaeger extension
  27. helm install linkerd-jaeger \
  28. --set webhook.externalSecret=true \
  29. --set-file webhook.caBundle=ca.crt \
  30. linkerd/linkerd-jaeger \
  31. -n linkerd-jaeger --create-namespace

Automatically Rotating Webhook TLS Credentials - 图2

Note

When installing the linkerd-control-plane chart, you must provide the issuer trust root and issuer credentials as described in Installing Linkerd with Helm.

See Automatically Rotating Control Plane TLS Credentials for details on how to do something similar for control plane credentials.