Vault as the Service Mesh Certificate Provider on Kubernetes

This topic describes how to configure the Consul Helm chart to use TLS certificates issued by Vault for Consul service mesh communication.

Note: This feature requires Consul 1.11 or higher. As of v1.11, Consul allows using Kubernetes auth methods to configure Connect CA. This allows for automatic token rotation once the renewal is no longer possible.

Overview

To use an Vault as the Service Mesh Certificate Provider on Kubernetes, we will need to modify the steps outlined in the Data Integration section:

One time setup in Vault

1. Create a Vault policy that authorizes the desired level of access to the secret.

Setup per Consul datacenter

1. Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
2. Update the Consul on Kubernetes helm chart.

Prerequisites

Prior to setting up the data integration between Vault and Consul on Kubernetes, you will need to have:

1. Read and completed the steps in the Systems Integration section of Vault as a Secrets Backend.
2. Read the Data Integration Overview section of Vault as a Secrets Backend.

One time setup in Vault

Store the secret in Vault

This step is not valid to this use case as we are not storing any secrets for Service Mesh certificate, and we instead are Leveraging Vault CA as a provider to mint certificates on an ongoing basis.

Create a Vault policy that authorizes the desired level of access to the secret

To configure Vault as the provider for the Consul service mesh certificates, you will first need to decide on the type of policy that is suitable for you. To see the permissions that Consul would need in Vault, please see Vault ACL policies documentation.

Setup per Consul datacenter

Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access

Next, you will create Kubernetes auth roles for the Consul servers:

$ vault write auth/kubernetes/role/consul-server \
    bound_service_account_names=<Consul server service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=<Connect CA policy> \
    ttl=1h

$ vault write auth/kubernetes/role/consul-server \
    bound_service_account_names=<Consul server service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=<Connect CA policy> \
    ttl=1h

To find out the service account name of the Consul server, you can run:

$ helm template --release-name ${RELEASE_NAME} --show-only templates/server-serviceaccount.yaml hashicorp/consul

$ helm template --release-name ${RELEASE_NAME} --show-only templates/server-serviceaccount.yaml hashicorp/consul

Update the Consul on Kubernetes helm chart

Now you can configure the Consul Helm chart to use Vault as the Connect CA provider:

global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
      connectCA:
        address: <the address of the Vault server>
        rootPKIPath: <the path to root PKI>
        intermediatePKIPath: <the path to intermediate PKI>
      ca:
        secretName: <vaultCASecret>

values.yaml

global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
      connectCA:
        address: <the address of the Vault server>
        rootPKIPath: <the path to root PKI>
        intermediatePKIPath: <the path to intermediate PKI>
      ca:
        secretName: <vaultCASecret>

The address you provide to the connectCA configuration can be a Kubernetes DNS address if the Vault cluster is running the same Kubernetes cluster. The rootPKIPath and intermediatePKIPath should be the same as the ones defined in your Connect CA policy. Behind the scenes, Consul will authenticate to Vault using a Kubernetes service account using the Kubernetes auth method and will use the Vault token for any API calls to Vault. If the Vault token can not be renewed, Consul will re-authenticate to generate a new Vault token.

The vaultCASecret is the Kubernetes secret that stores the CA Certificate that is used for Vault communication. To provide a CA, you first need to create a Kubernetes secret containing the CA. For example, you may create a secret with the Vault CA like so:

$ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/ca

$ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/ca

Secondary Datacenters

To configure Vault as the Connect CA in secondary datacenters, you need to make sure that the Root CA path is the same, but the intermediate is different for each datacenter. In the connectCA Helm configuration for a secondary datacenter, you can specify a intermediatePKIPath that is, for example, prefixed with the datacenter for which this configuration is intended (e.g. dc2/connect-intermediate).