Vault as the Controller and Connect Inject Webhook Certificate Provider on Kubernetes

This topic describes how to configure the Consul Helm chart to use TLS certificates issued by Vault in the Consul controller and connect inject webhooks.

Overview

In a Consul Helm chart configuration that does not use Vault, webhook-cert-manager normally fulfills the role of ensuring that a valid certificate is updated to the mutatingwebhookconfiguration of either controller or connect inject to ensure that Kubernetes can communicate with each of these services.

When Vault is configured as the controller and connect inject Webhook Certificate Provider on Kubernetes:

  • webhook-cert-manager is no longer deployed to the cluster.
  • controller and connect inject each get their webhook certificates from its own Vault PKI mount via the injected Vault Agent.
  • controller and connect inject each need to be configured with its own Vault Role that has necessary permissions to receive certificates from its respective PKI mount.
  • controller and connect inject each locally update its own mutatingwebhookconfiguration so that Kubernetes can relay events.
  • Vault manages certificate rotation and rotates certificates to each webhook.

To use Vault as the controller and connect inject Webhook Certificate Provider, we will need to modify the steps outlined in the Data Integration section:

Setup per Consul datacenter

  1. Create a Vault policy that authorizes the desired level of access to the secret.
  2. (Added) Create Vault PKI roles for controller and connect inject each that establish the domains that each is allowed to issue certificates for.
  3. Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
  4. Configure the Vault Kubernetes auth roles in the Consul on Kubernetes helm chart.

Prerequisites

Complete the following prerequisites prior to implementing the integration described in this topic:

  1. Verify that you have completed the steps described in Systems Integration section of Vault as a Secrets Backend.
  2. You should be familiar with the Data Integration Overview section of Vault as a Secrets Backend.
  3. Configure Vault as the Server TLS Certificate Provider on Kubernetes
  4. Configure Vault as the Service Mesh Certificate Provider on Kubernetes
  5. Complete the Bootstrapping the PKI Engine for Controller and Connect Inject Webhooks section.

Bootstrapping the PKI Engine for Controller and Connect Inject Webhooks

The first step is to bootstrap the Vault cluster. Issue the following commands to enable and configure the PKI Secrets Engine to serve TLS certificates for the controller and connect inject webhooks:

  • Mount the PKI Secrets Engine for each:

    1. $ vault secrets enable -path=controller pki
    1. $ vault secrets enable -path=controller pki
    1. $ vault secrets enable -path=connect-inject pki
    1. $ vault secrets enable -path=connect-inject pki

  • Tune the engine mounts to enable longer TTL:

    1. $ vault secrets tune -max-lease-ttl=87600h controller
    1. $ vault secrets tune -max-lease-ttl=87600h controller
    1. $ vault secrets tune -max-lease-ttl=87600h connect-inject
    1. $ vault secrets tune -max-lease-ttl=87600h connect-inject

  • Generate the root CA for each:

    1. $ vault write -field=certificate controller/root/generate/internal \
    2.         common_name="<helm release name>-controller-webhook" \
    3.         ttl=87600h
    1. $ vault write -field=certificate controller/root/generate/internal \
    2.         common_name="<helm release name>-controller-webhook" \
    3.         ttl=87600h
    1. $ vault write -field=certificate connect-inject/root/generate/internal \
    2.         common_name="<helm release name>-connect-injector" \
    3.         ttl=87600h
    1. $ vault write -field=certificate connect-inject/root/generate/internal \
    2.         common_name="<helm release name>-connect-injector" \
    3.         ttl=87600h

Setup per Consul datacenter

You will need to preform the following steps for each datacenter that you would like to manage controller and connect inject webhook certificates in Vault. You will want to take care to create different names per datacenter for every pki mount, role, and policy.

Create a Vault policy that authorizes the desired level of access to the secret

To use Vault to issue controller or connect inject webhook certificates, you will need to create the Vault policies that will allow either controller or connect inject to access its respective certificate-issuing URL.

Create Vault Policies for the Controller and Connect Inject Webhook Certificates

Note: The PKI secret paths referenced by the Vault Policies below will be your global.secretsBackend.vault.controller.tlsCert.secretName and global.secretsBackend.vault.connectInject.tlsCert.secretName Helm values respectively.

The next step is to create a policy that allows ["create", "update"] access to the certificate issuing URL so Consul controller and connect inject can fetch a new certificate/key pair and provide it to the Kubernetes mutatingwebhookconfiguration. Issue the following commands to create the policy:

  1. $ vault policy write controller-tls-policy - <<EOF
  2. path controller/issue/controller-role {
  3.   capabilities = ["create", "update"]
  4. }
  5. EOF
  1. $ vault policy write controller-tls-policy - <<EOF
  2. path controller/issue/controller-role {
  3.   capabilities = ["create", "update"]
  4. }
  5. EOF
  1. $ vault policy write connect-inject-policy - <<EOF
  2. path connect-inject/issue/connect-inject-role {
  3.   capabilities = ["create", "update"]
  4. }
  5. EOF
  1. $ vault policy write connect-inject-policy - <<EOF
  2. path connect-inject/issue/connect-inject-role {
  3.   capabilities = ["create", "update"]
  4. }
  5. EOF

Create Vault Policies for the CA URL

Note: The PKI secret paths referenced by the Vault Policies below will be your global.secretsBackend.vault.controller.caCert.secretName and global.secretsBackend.vault.connectInject.caCert.secretName Helm values respectively.

Next, create a policy that allows ["read"] access to the CA URL. The policy is required so that Consul components can communicate with the Consul servers in order to fetch their auto-encryption certificates. Issue the following commands to create the policy:

  1. $ vault policy write controller-ca-policy - <<EOF
  2. path controller/cert/ca {
  3.   capabilities = ["read"]
  4. }
  5. EOF
  1. $ vault policy write controller-ca-policy - <<EOF
  2. path controller/cert/ca {
  3.   capabilities = ["read"]
  4. }
  5. EOF
  1. $ vault policy write connect-inject-ca-policy - <<EOF
  2. path connect-inject/cert/ca {
  3.   capabilities = ["read"]
  4. }
  5. EOF
  1. $ vault policy write connect-inject-ca-policy - <<EOF
  2. path connect-inject/cert/ca {
  3.   capabilities = ["read"]
  4. }
  5. EOF

Configure allowed domains for PKI certificates

Issue the following command to create a Vault role for the controller PKI engine and set the default parameters for issuing certificates:

  1. $ vault write controller/roles/controller-role \
  2.     allowed_domains="<Allowed-domains-string>" \
  3.     allow_subdomains=true \
  4.     allow_bare_domains=true \
  5.     allow_localhost=true \
  6.     generate_lease=true \
  7.     max_ttl="720h"
  1. $ vault write controller/roles/controller-role \
  2.     allowed_domains="<Allowed-domains-string>" \
  3.     allow_subdomains=true \
  4.     allow_bare_domains=true \
  5.     allow_localhost=true \
  6.     generate_lease=true \
  7.     max_ttl="720h"

Issue the following command to create a Vault role for the connect inject PKI engine and set the default parameters for issuing certificates:

  1. $ vault write connect-inject/roles/connect-inject-role \
  2.     allowed_domains="<Allowed-domains-string>" \
  3.     allow_subdomains=true \
  4.     allow_bare_domains=true \
  5.     allow_localhost=true \
  6.     generate_lease=true \
  7.     max_ttl="720h"
  1. $ vault write connect-inject/roles/connect-inject-role \
  2.     allowed_domains="<Allowed-domains-string>" \
  3.     allow_subdomains=true \
  4.     allow_bare_domains=true \
  5.     allow_localhost=true \
  6.     generate_lease=true \
  7.     max_ttl="720h"

To generate the <Allowed-domains-string> for each use the following script as a template:

  1. #!/bin/sh
  3. # NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
  4. export NAME=consulk8s
  5. # NAMESPACE is where the Consul on Kubernetes is installed
  6. export NAMESPACE=consul
  7. # DATACENTER is the value of `global.datacenter` from your Helm values config file
  8. export DATACENTER=dc1
  10. echo allowed_domains_controller=\"${NAME}-controller-webhook,${NAME}-controller-webhook.${NAMESPACE},${NAME}-controller-webhook.${NAMESPACE}.svc,${NAME}-controller-webhook.${NAMESPACE}.svc.cluster.local\""
  12. echo allowed_domains_connect_inject=\"${NAME}-connect-injector,${NAME}-connect-injector.${NAMESPACE},${NAME}-connect-injector.${NAMESPACE}.svc,${NAME}-connect-injector.${NAMESPACE}.svc.cluster.local\""
  1. #!/bin/sh
  3. # NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
  4. export NAME=consulk8s
  5. # NAMESPACE is where the Consul on Kubernetes is installed
  6. export NAMESPACE=consul
  7. # DATACENTER is the value of `global.datacenter` from your Helm values config file
  8. export DATACENTER=dc1
  10. echo allowed_domains_controller=\"${NAME}-controller-webhook,${NAME}-controller-webhook.${NAMESPACE},${NAME}-controller-webhook.${NAMESPACE}.svc,${NAME}-controller-webhook.${NAMESPACE}.svc.cluster.local\""
  12. echo allowed_domains_connect_inject=\"${NAME}-connect-injector,${NAME}-connect-injector.${NAMESPACE},${NAME}-connect-injector.${NAMESPACE}.svc,${NAME}-connect-injector.${NAMESPACE}.svc.cluster.local\""

Note: The Vault auth roles below will be your global.secretsBackend.vault.controllerRole and global.secretsBackend.vault.connectInjectRole Helm values respectively.

Finally, Kubernetes auth roles need to be created for controller and connect inject webhooks.

Role for Consul controller webhooks:

  1. $ vault write auth/kubernetes/role/controller-role \
  2.     bound_service_account_names=<Consul controller service account> \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=controller-ca-policy \
  5.     ttl=1h
  1. $ vault write auth/kubernetes/role/controller-role \
  2.     bound_service_account_names=<Consul controller service account> \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=controller-ca-policy \
  5.     ttl=1h

To find out the service account name of the Consul controller, you can run:

  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/controller-serviceaccount.yaml hashicorp/consul
  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/controller-serviceaccount.yaml hashicorp/consul

Role for Consul connect inject webhooks:

  1. $ vault write auth/kubernetes/role/connect-inject-role \
  2.     bound_service_account_names=<Consul connect inject service account> \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=connect-inject-ca-policy \
  5.     ttl=1h
  1. $ vault write auth/kubernetes/role/connect-inject-role \
  2.     bound_service_account_names=<Consul connect inject service account> \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=connect-inject-ca-policy \
  5.     ttl=1h

To find out the service account name of the Consul connect inject, use the command below.

  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/connect-inject-serviceaccount.yaml hashicorp/consul
  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/connect-inject-serviceaccount.yaml hashicorp/consul

Update the Consul on Kubernetes helm chart

Now that we’ve configured Vault, you can configure the Consul Helm chart to use the Server TLS certificates from Vault:

  1. global:
  2.   secretsBackend:
  3.     vault:
  4.       enabled: true
  5.       consulServerRole: "consul-server"
  6.       consulClientRole: "consul-client"
  7.       consulCARole: "consul-ca"
  8.       controllerRole: "controller-role"
  9.       connectInjectRole: "connect-inject-role"
  10.       controller:
  11.         caCert:
  12.           secretName: "controller/cert/ca"
  13.         tlsCert:
  14.           secretName: "controller/issue/controller-role"
  15.       connectInject:
  16.         caCert:
  17.           secretName: "connect-inject/cert/ca"
  18.         tlsCert:
  19.           secretName: "connect-inject/issue/connect-inject-role"
  20.   tls:
  21.     enabled: true
  22.     enableAutoEncrypt: true
  23.     caCert:
  24.       secretName: "pki/cert/ca"
  25. server:
  26.   serverCert:
  27.     secretName: "pki/issue/consul-server"
  28.   extraVolumes:
  29.     - type: "secret"
  30.       name: <vaultCASecret>
  31.       load: "false"
  32. connectInject:
  33.   enabled: true
  34. controller:
  35.   enabled: true

Webhook Certificates - 图1

values.yaml

  1. global:
  2.   secretsBackend:
  3.     vault:
  4.       enabled: true
  5.       consulServerRole: "consul-server"
  6.       consulClientRole: "consul-client"
  7.       consulCARole: "consul-ca"
  8.       controllerRole: "controller-role"
  9.       connectInjectRole: "connect-inject-role"
  10.       controller:
  11.         caCert:
  12.           secretName: "controller/cert/ca"
  13.         tlsCert:
  14.           secretName: "controller/issue/controller-role"
  15.       connectInject:
  16.         caCert:
  17.           secretName: "connect-inject/cert/ca"
  18.         tlsCert:
  19.           secretName: "connect-inject/issue/connect-inject-role"
  20.   tls:
  21.     enabled: true
  22.     enableAutoEncrypt: true
  23.     caCert:
  24.       secretName: "pki/cert/ca"
  25. server:
  26.   serverCert:
  27.     secretName: "pki/issue/consul-server"
  28.   extraVolumes:
  29.     - type: "secret"
  30.       name: <vaultCASecret>
  31.       load: "false"
  32. connectInject:
  33.   enabled: true
  34. controller:
  35.   enabled: true

The vaultCASecret is the Kubernetes secret that stores the CA Certificate that is used for Vault communication. To provide a CA, you first need to create a Kubernetes secret containing the CA. For example, you may create a secret with the Vault CA like so:

  1. $ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/
  1. $ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/