Vault as the Server TLS Certificate Provider on Kubernetes

Overview

To use an Vault as the Server TLS Certificate Provider on Kubernetes, we will need to modify the steps outlined in the Data Integration section:

One time setup in Vault

  1. Create a Vault policy that authorizes the desired level of access to the secret.

Setup per Consul datacenter

  1. (Added) Configure allowed domains for PKI certificates
  2. Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
  3. Update the Consul on Kubernetes helm chart.

Prerequisites

Prior to setting up the data integration between Vault and Consul on Kubernetes, you will need to have:

  1. Read and completed the steps in the Systems Integration section of Vault as a Secrets Backend.
  2. Read the Data Integration Overview section of Vault as a Secrets Backend.
  3. Complete the Bootstrapping the PKI Engine section.

Bootstrapping the PKI Engine

First, we need to bootstrap the Vault cluster by enabling and configuring the PKI Secrets Engine to be able to serve TLS certificates to Consul. The process can be as simple as the following, or more complicated such as in this example which also uses an intermediate signing authority.

  • Enable the PKI Secrets Engine:

    1. $ vault secrets enable pki
    1. $ vault secrets enable pki

  • Tune the engine to enable longer TTL:

    1. $ vault secrets tune -max-lease-ttl=87600h pki
    1. $ vault secrets tune -max-lease-ttl=87600h pki

  • Generate the root CA:

    Note: The common_name value is comprised of combining global.datacenter dot global.domain.

    1. $ vault write -field=certificate pki/root/generate/internal \
    2.         common_name="dc1.consul" \
    3.         ttl=87600h
    1. $ vault write -field=certificate pki/root/generate/internal \
    2.         common_name="dc1.consul" \
    3.         ttl=87600h

One time setup in Vault

Store the secret in Vault

This step is not valid to this use case because we are not storing a single secret. We are configuring Vault as a provider to mint certificates on an ongoing basis.

Create a Vault policy that authorizes the desired level of access to the secret

To use Vault to issue Server TLS certificates, you will need to create the following:

  1. Vault Policies that will allow the Consul server to access the certificate issuing url.
  2. Vault Policies that will allow the Consul components, e.g. ingress gateways, controller, to access the CA url.

Create Vault Policies for the Server TLS Certificates

Note: The PKI secret path referenced by the Vault Policy below will be your server.serverCert.secretName Helm value.

Next we will create a policy that allows ["create", "update"] access to the certificate issuing URL so the Consul servers can fetch a new certificate/key pair.

  1. path "pki/issue/consul-server" {
  2.   capabilities = ["create", "update"]
  3. }

Server TLS - 图1

consul-server-policy.hcl

  1. path "pki/issue/consul-server" {
  2.   capabilities = ["create", "update"]
  3. }

Apply the Vault policy by issuing the vault policy write CLI command:

  1. $ vault policy write consul-server consul-server-policy.hcl
  1. $ vault policy write consul-server consul-server-policy.hcl

Create Vault Policies for the CA URL

Next, we will create a policy that allows ["read"] access to the CA URL, this is required for the Consul components to communicate with the Consul servers in order to fetch their auto-encryption certificates.

  1. path "pki/cert/ca" {
  2.   capabilities = ["read"]
  3. }

Server TLS - 图2

ca-policy.hcl

  1. path "pki/cert/ca" {
  2.   capabilities = ["read"]
  3. }
  1. $ vault policy write ca-policy ca-policy.hcl
  1. $ vault policy write ca-policy ca-policy.hcl

Note: The PKI secret path referenced by the above Policy will be your global.tls.caCert.secretName Helm value.

Setup per Consul datacenter

Configure allowed domains for PKI certificates

Next, a Vault role for the PKI engine will set the default certificate issuance parameters:

  1. $ vault write pki/roles/consul-server \
  2.     allowed_domains="<Allowed-domains-string>" \
  3.     allow_subdomains=true \
  4.     allow_bare_domains=true \
  5.     allow_localhost=true \
  6.     generate_lease=true \
  7.     max_ttl="720h"
  1. $ vault write pki/roles/consul-server \
  2.     allowed_domains="<Allowed-domains-string>" \
  3.     allow_subdomains=true \
  4.     allow_bare_domains=true \
  5.     allow_localhost=true \
  6.     generate_lease=true \
  7.     max_ttl="720h"

To generate the <Allowed-domains-string> use the following script as a template:

  1. #!/bin/sh
  3. # NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
  4. export NAME=consulk8s
  5. # NAMESPACE is where the Consul on Kubernetes is installed
  6. export NAMESPACE=consul
  7. # DATACENTER is the value of `global.datacenter` from your Helm values config file
  8. export DATACENTER=dc1
  10. echo allowed_domains=\"$DATACENTER.consul, $NAME-server, $NAME-server.$NAMESPACE, $NAME-server.$NAMESPACE.svc\"
  1. #!/bin/sh
  3. # NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
  4. export NAME=consulk8s
  5. # NAMESPACE is where the Consul on Kubernetes is installed
  6. export NAMESPACE=consul
  7. # DATACENTER is the value of `global.datacenter` from your Helm values config file
  8. export DATACENTER=dc1
  10. echo allowed_domains=\"$DATACENTER.consul, $NAME-server, $NAME-server.$NAMESPACE, $NAME-server.$NAMESPACE.svc\"

Create three Vault auth roles, one for the Consul servers, one for the Consul clients, and one for Consul components, that link the policy to each Consul workload on Kubernetes service account that requires access.

Role for Consul servers:

  1. $ vault write auth/kubernetes/role/consul-server \
  2.     bound_service_account_names=<Consul server service account> \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=consul-server \
  5.     ttl=1h
  1. $ vault write auth/kubernetes/role/consul-server \
  2.     bound_service_account_names=<Consul server service account> \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=consul-server \
  5.     ttl=1h

To find out the service account name of the Consul server, you can run:

  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/server-serviceaccount.yaml hashicorp/consul
  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/server-serviceaccount.yaml hashicorp/consul

Role for Consul clients:

  1. $ vault write auth/kubernetes/role/consul-client \
  2.     bound_service_account_names=<Consul client service account>  \
  3.     bound_service_account_namespaces=default \
  4.     policies=ca-policy \
  5.     ttl=1h
  1. $ vault write auth/kubernetes/role/consul-client \
  2.     bound_service_account_names=<Consul client service account>  \
  3.     bound_service_account_namespaces=default \
  4.     policies=ca-policy \
  5.     ttl=1h

To find out the service account name of the Consul client, use the command below.

  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/client-serviceaccount.yaml hashicorp/consul
  1.  $ helm template --release-name ${RELEASE_NAME} --show-only templates/client-serviceaccount.yaml hashicorp/consul

Role for CA components:

  1. $ vault write auth/kubernetes/role/consul-ca \
  2.     bound_service_account_names="*" \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=ca-policy \
  5.     ttl=1h
  1. $ vault write auth/kubernetes/role/consul-ca \
  2.     bound_service_account_names="*" \
  3.     bound_service_account_namespaces=<Consul installation namespace> \
  4.     policies=ca-policy \
  5.     ttl=1h

The above Vault Roles will now be your Helm values for global.secretsBackend.vault.consulServerRole and global.secretsBackend.vault.consulCARole respectively.

Update the Consul on Kubernetes helm chart

Now that we’ve configured Vault, you can configure the Consul Helm chart to use the Server TLS certificates from Vault:

  1. global:
  2.   secretsBackend:
  3.     vault:
  4.       enabled: true
  5.       consulServerRole: consul-server
  6.       consulClientRole: consul-client
  7.       consulCARole: consul-ca
  8.   tls:
  9.     enableAutoEncrypt: true
  10.     enabled: true
  11.     caCert:
  12.       secretName: "pki/cert/ca"
  13. server:
  14.   serverCert:
  15.     secretName: "pki/issue/consul-server"
  16.   extraVolumes:
  17.     - type: "secret"
  18.       name: <vaultCASecret>
  19.       load: "false"

Server TLS - 图3

values.yaml

  1. global:
  2.   secretsBackend:
  3.     vault:
  4.       enabled: true
  5.       consulServerRole: consul-server
  6.       consulClientRole: consul-client
  7.       consulCARole: consul-ca
  8.   tls:
  9.     enableAutoEncrypt: true
  10.     enabled: true
  11.     caCert:
  12.       secretName: "pki/cert/ca"
  13. server:
  14.   serverCert:
  15.     secretName: "pki/issue/consul-server"
  16.   extraVolumes:
  17.     - type: "secret"
  18.       name: <vaultCASecret>
  19.       load: "false"

The vaultCASecret is the Kubernetes secret that stores the CA Certificate that is used for Vault communication. To provide a CA, you first need to create a Kubernetes secret containing the CA. For example, you may create a secret with the Vault CA like so:

  1. $ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/
  1. $ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/