Federation Between Kubernetes Clusters with Vault as Secrets Backend

Note: This topic requires familiarity with Mesh Gateways, WAN Federation Via Mesh Gateways.

This page describes how you can federate multiple Kubernetes clusters using Vault as the secrets backend. See the Multi-Cluster Overview for more information on use cases and how it works.

Differences Between Using Kubernetes Secrets vs. Vault

The Federation Between Kubernetes Clusters page provides an overview of WAN Federation using Mesh Gateways with Kubernetes secrets as the secret backend. When using Vault as the secrets backend, there are different systems and data integration configuration that will be explained in the Usage section of this page. The other main difference is that when using Vault, there is no need for you to export and import a Federation Secret in each datacenter.

Usage

The expected use case is to create WAN Federation on Kubernetes clusters. The following procedure will result in a WAN Federation with Vault as the secrets backend between two clusters, dc1 and dc2. dc1 will act as the primary Consul cluster and will also contain the Vault server installation. dc2 will be the secondary Consul cluster.

Consul on Kubernetes with Vault as the Secrets Backend

The Vault Injectors in each cluster will ensure that every pod in cluster has a Vault agent inject into the pod.

Vault Injectors inject Vault agents into pods

The Vault Agents on each Consul pod will communicate directly with Vault on its externally accessible endpoint. Consul pods are also configured with Vault annotations that configure the secrets that the pod needs as well as the path that the Vault agent should locally store those secrets.

Vault agent and server communication

The two data centers will federated using mesh gateways. This communication topology is also described in the WAN Federation Via Mesh Gateways section of Multi-Cluster Federation Overview.

Mesh Federation via Mesh Gateways

Install Vault

In this setup, you will deploy Vault server in the primary datacenter (dc1) Kubernetes cluster, which is also the primary Consul datacenter. You will configure your Vault Helm installation in the secondary datacenter (dc2) Kubernetes cluster to use it as an external server. This way there will be a single vault server cluster that will be used by both Consul datacenters.

Note: For demonstration purposes, you will deploy a Vault server in dev mode. For production installations, this is not recommended. Please visit the Vault Deployment Guide for guidance on how to install Vault in a production setting.

  1. Change your current Kubernetes context to target the primary datacenter (dc1).

    1. $ kubectl config use-context <context for dc1>
    1. $ kubectl config use-context <context for dc1>

  2. Now, use the values files below for your Helm install.

    1. server:
    2.   dev:
    3.     enabled: true
    4.   service:
    5.     enabled: true
    6.     type: LoadBalancer
    7. ui:
    8.   enabled: true

    WAN Federation - 图5

    vault-dc1.yaml

    1. server:
    2.   dev:
    3.     enabled: true
    4.   service:
    5.     enabled: true
    6.     type: LoadBalancer
    7. ui:
    8.   enabled: true
    1. $ helm install vault-dc1 --values vault-dc1.yaml hashicorp/vault --wait
    1. $ helm install vault-dc1 --values vault-dc1.yaml hashicorp/vault --wait

Configuring your local environment

  1. Install Consul locally so that you can generate the gossip key. Please see the Precompiled Binaries section of the Install Consul page.

  2. Set the VAULT_TOKEN with a default value.

    1. $ export VAULT_ADDR=root
    1. $ export VAULT_ADDR=root

  3. Get the external IP or DNS name of the Vault server’s load balancer.

    WAN Federation - 图6

    WAN Federation - 图7

    On EKS, you can get the IP address of the Vault server’s load balancer with the following command:

    1. $ export VAULT_SERVER_HOST=$(kubectl get svc vault-dc1 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
    1. $ export VAULT_SERVER_HOST=$(kubectl get svc vault-dc1 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  4. Set the VAULT_ADDR environment variable.

    1. $ export VAULT_ADDR=http://${VAULT_SERVER_HOST}:8200
    1. $ export VAULT_ADDR=http://${VAULT_SERVER_HOST}:8200

Systems Integration

Overview

To use Vault as the Service Mesh Certificate Provider in Kubernetes, you must complete following systems integration actions:

  • One time setup in Vault
    1. Enabling Vault KV Secrets Engine - Version 2.
    2. Enabling Vault PKI Engine.
  • Setup per Consul datacenter
    1. Installing the Vault Injector within the Consul datacenter installation
    2. Configuring a Kubernetes Auth Method in Vault to authenticate and authorize operations from the Consul datacenter
    3. Enable Vault as the Secrets Backend in the Consul datacenter

One time setup on Vault

  1. Enable Vault KV secrets engine - Version 2 in order to store the Gossip Encryption Key and the ACL Replication token (global.acls.replicationToken).

    1. $ vault secrets enable -path=consul kv-v2
    1. $ vault secrets enable -path=consul kv-v2

  2. Enable Vault PKI Engine in order to leverage Vault for issuing Consul Server TLS certificates.

    1. $ vault secrets enable pki
    1. $ vault secrets enable pki
    1. $ vault secrets tune -max-lease-ttl=87600h pki
    1. $ vault secrets tune -max-lease-ttl=87600h pki

Setup per Consul datacenter

Primary Datacenter (dc1)

  1. Install the Vault Injector in your Consul Kubernetes cluster (dc1), which is used for accessing secrets from Vault.

    Note: In the primary datacenter (dc1), you will not have to configure injector.externalvaultaddr value because the Vault server is in the same primary datacenter (dc1) cluster.

    1. server:
    2.   dev:
    3.     enabled: true
    4.   service:
    5.     enabled: true
    6.     type: LoadBalancer
    7. injector:
    8.   enabled: true
    9.   authPath: auth/kubernetes-dc1
    10. ui:
    11.   enabled: true

    WAN Federation - 图8

    vault-dc1.yaml

    1. 1 2 3 4 5 6 7 8 9 1011server:
    2.   dev:
    3.     enabled: true
    4.   service:
    5.     enabled: true
    6.     type: LoadBalancer
    7. injector:
    8.   enabled: true
    9.   authPath: auth/kubernetes-dc1
    10. ui:
    11.   enabled: true

    Next, install Vault in the Kubernetes cluster.

    1. $ helm upgrade vault-dc1 --values vault-dc1.yaml hashicorp/vault --wait
    1. $ helm upgrade vault-dc1 --values vault-dc1.yaml hashicorp/vault --wait

  2. Configure the Kubernetes Auth Method in Vault for the primary datacenter (dc1).

    1. $ vault auth enable -path=kubernetes-dc1 kubernetes
    1. $ vault auth enable -path=kubernetes-dc1 kubernetes

    Because Consul is in the same datacenter cluster as Vault, the Vault Auth Method can use its own CA Cert and JWT to authenticate Consul dc1 service account requests. Therefore, you do not need to set token_reviewer and kubernetes_ca_cert on the dc1 Kubernetes Auth Method.

  3. Configure Auth Method with Kubernetes API host

    1. $ vault write auth/kubernetes-dc1/config kubernetes_host=https://kubernetes.default.svc
    1. $ vault write auth/kubernetes-dc1/config kubernetes_host=https://kubernetes.default.svc

  4. Enable Vault as the secrets backend in the primary datacenter (dc1). However, you will not yet apply the Helm install command. You will issue the Helm upgrade command after the Data Integration section.

    1. global:
    2.   secretsBackend:
    3.     vault:
    4.       enabled: true

    WAN Federation - 图9

    consul-dc1.yaml

    1. global:
    2.   secretsBackend:
    3.     vault:
    4.       enabled: true

Secondary Datacenter (dc2)

  1. Install the Vault Injector in the secondary datacenter (dc2).

    In the secondary datacenter (dc2), you will configure the externalvaultaddr value point to the external address of the Vault server in the primary datacenter (dc1).

    Change your Kubernetes context to target the secondary datacenter (dc2):

    1. $ kubectl config use-context <context for dc2>
    1. $ kubectl config use-context <context for dc2>
    1. server:
    2.   enabled: false
    3. injector:
    4.   enabled: true
    5.   externalVaultAddr: ${VAULT_ADDR}
    6.   authPath: auth/kubernetes-dc2

    WAN Federation - 图10

    vault-dc2.yaml

    1. server:
    2.   enabled: false
    3. injector:
    4.   enabled: true
    5.   externalVaultAddr: ${VAULT_ADDR}
    6.   authPath: auth/kubernetes-dc2

    Next, install Vault in the Kubernetes cluster.

    1. $ helm install vault-dc2 --values vault-dc2.yaml hashicorp/vault --wait
    1. $ helm install vault-dc2 --values vault-dc2.yaml hashicorp/vault --wait

  2. Configure the Kubernetes Auth Method in Vault for the datacenter

    1. $ vault auth enable -path=kubernetes-dc2 kubernetes
    1. $ vault auth enable -path=kubernetes-dc2 kubernetes

  3. Create a service account with access to the Kubernetes API in the secondary datacenter (dc2). For the secondary datacenter (dc2) auth method, you first need to create a service account that allows the Vault server in the primary datacenter (dc1) cluster to talk to the Kubernetes API in the secondary datacenter (dc2) cluster.

    1. $ cat <<EOF >> auth-method-serviceaccount.yaml
    2. # auth-method.yaml
    3. apiVersion: rbac.authorization.k8s.io/v1
    4. kind: ClusterRoleBinding
    5. metadata:
    6.   name: vault-dc2-auth-method
    7. roleRef:
    8.   apiGroup: rbac.authorization.k8s.io
    9.   kind: ClusterRole
    10.   name: system:auth-delegator
    11. subjects:
    12. - kind: ServiceAccount
    13.   name: vault-dc2-auth-method
    14.   namespace: default
    15. ---
    16. apiVersion: v1
    17. kind: ServiceAccount
    18. metadata:
    19.   name: vault-dc2-auth-method
    20.   namespace: default
    21. EOF
    1. $ cat <<EOF >> auth-method-serviceaccount.yaml
    2. # auth-method.yaml
    3. apiVersion: rbac.authorization.k8s.io/v1
    4. kind: ClusterRoleBinding
    5. metadata:
    6.   name: vault-dc2-auth-method
    7. roleRef:
    8.   apiGroup: rbac.authorization.k8s.io
    9.   kind: ClusterRole
    10.   name: system:auth-delegator
    11. subjects:
    12. - kind: ServiceAccount
    13.   name: vault-dc2-auth-method
    14.   namespace: default
    15. ---
    16. apiVersion: v1
    17. kind: ServiceAccount
    18. metadata:
    19.   name: vault-dc2-auth-method
    20.   namespace: default
    21. EOF
    1. $ kubectl apply --values auth-method-serviceaccount.yaml
    1. $ kubectl apply --values auth-method-serviceaccount.yaml

  4. Next, you will need to get the token and CA cert from that service account secret.

    1. $ export K8S_DC2_CA_CERT="$(kubectl get secret `kubectl get serviceaccounts vault-dc2-auth-method --output jsonpath='{.secrets[0].name}'` --output jsonpath='{.data.ca\.crt}' | base64 --decode)"
    1. $ export K8S_DC2_CA_CERT="$(kubectl get secret `kubectl get serviceaccounts vault-dc2-auth-method --output jsonpath='{.secrets[0].name}'` --output jsonpath='{.data.ca\.crt}' | base64 --decode)"
    1. $ export K8S_DC2_JWT_TOKEN="$(kubectl get secret `kubectl get serviceaccounts vault-dc2-auth-method --output jsonpath='{.secrets[0].name}'` --output jsonpath='{.data.token}' | base64 --decode)"
    1. $ export K8S_DC2_JWT_TOKEN="$(kubectl get secret `kubectl get serviceaccounts vault-dc2-auth-method --output jsonpath='{.secrets[0].name}'` --output jsonpath='{.data.token}' | base64 --decode)"

  5. Configure Auth Method with the JWT token of service account. You will have to get the externally reachable address of the secondary Consul datacenter (dc2) in the secondary Kubernetes cluster and set kubernetes_host within the Auth Method configuration.

    1. $ export KUBE_API_URL_DC2=$(kubectl config view --output jsonpath="{.clusters[?(@.name == \"$(kubectl config current-context)\")].cluster.server}")
    1. $ export KUBE_API_URL_DC2=$(kubectl config view --output jsonpath="{.clusters[?(@.name == \"$(kubectl config current-context)\")].cluster.server}")
    1. $ vault write auth/kubernetes-dc2/config \
    2.   kubernetes_host="${KUBE_API_URL_DC2}" \
    3.   token_reviewer_jwt="${K8S_DC2_JWT_TOKEN}" \
    4.   kubernetes_ca_cert="${K8S_DC2_CA_CERT}"
    1. $ vault write auth/kubernetes-dc2/config \
    2.   kubernetes_host="${KUBE_API_URL_DC2}" \
    3.   token_reviewer_jwt="${K8S_DC2_JWT_TOKEN}" \
    4.   kubernetes_ca_cert="${K8S_DC2_CA_CERT}"

  6. Enable Vault as the secrets backend in the secondary Consul datacenter (dc2). However, you will not yet apply the Helm install command. You will issue the Helm upgrade command after the Data Integration section.

    1. global:
    2.   secretsBackend:
    3.     vault:
    4.       enabled: true

    WAN Federation - 图11

    values-dc2.yaml

    1. global:
    2.   secretsBackend:
    3.     vault:
    4.       enabled: true

Data Integration

Overview

To use Vault as the Service Mesh Certificate Provider in Kubernetes, you must complete following data integration actions:

  • One time setup in Vault
    1. Store the secrets in Vault.
    2. Create a Vault policy that authorizes the desired level of access to the secrets.
  • Setup per Consul datacenter
    1. Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
    2. Update the Consul on Kubernetes helm chart.

One time setup in Vault

  1. Store the ACL Replication Token, Gossip Encryption Key, and Root CA certificate secrets in Vault.

    1. $ vault kv put consul/secret/gossip key="$(consul keygen)"
    1. $ vault kv put consul/secret/gossip key="$(consul keygen)"
    1. $ vault kv put consul/secret/replication token="$(uuidgen | tr '[:upper:]' '[:lower:]')"
    1. $ vault kv put consul/secret/replication token="$(uuidgen | tr '[:upper:]' '[:lower:]')"
    1. $ vault write pki/root/generate/internal common_name="Consul CA" ttl=87600h
    1. $ vault write pki/root/generate/internal common_name="Consul CA" ttl=87600h

  2. Create Vault policies that authorize the desired level of access to the secrets.

    1. $ vault policy write gossip - <<EOF
    2. path "consul/data/secret/gossip" {
    3.   capabilities = ["read"]
    4. }
    5. EOF
    1. $ vault policy write gossip - <<EOF
    2. path "consul/data/secret/gossip" {
    3.   capabilities = ["read"]
    4. }
    5. EOF
    1. $ vault policy write replication-token - <<EOF
    2. path "consul/data/secret/replication" {
    3.   capabilities = ["read"]
    4. }
    5. EOF
    1. $ vault policy write replication-token - <<EOF
    2. path "consul/data/secret/replication" {
    3.   capabilities = ["read"]
    4. }
    5. EOF
    1. $ vault policy write ca-policy - <<EOF
    2. path "pki/cert/ca" {
    3.   capabilities = ["read"]
    4. }
    5. EOF
    1. $ vault policy write ca-policy - <<EOF
    2. path "pki/cert/ca" {
    3.   capabilities = ["read"]
    4. }
    5. EOF

Setup per Consul datacenter

Pre-installation for Primary Datacenter (dc1)

  1. Change your Kubernetes context to target the primary datacenter (dc1):

    1. $ kubectl config use-context <context for dc1>
    1. $ kubectl config use-context <context for dc1>

Primary Datacenter (dc1)

  1. Create Server TLS and Service Mesh Cert Policies

    1. $ vault policy write consul-cert-dc1 - <<EOF
    2. path "pki/issue/consul-cert-dc1"
    3. {
    4.   capabilities = ["create","update"]
    5. }
    6. EOF
    1. $ vault policy write consul-cert-dc1 - <<EOF
    2. path "pki/issue/consul-cert-dc1"
    3. {
    4.   capabilities = ["create","update"]
    5. }
    6. EOF
    1. $ vault policy write connect-ca-dc1 - <<EOF
    2. path "/sys/mounts" {
    3.   capabilities = [ "read" ]
    4. }
    5. path "/sys/mounts/connect_root" {
    6.   capabilities = [ "create", "read", "update", "delete", "list" ]
    7. }
    8. path "/sys/mounts/dc1/connect_inter" {
    9.   capabilities = [ "create", "read", "update", "delete", "list" ]
    10. }
    11. path "/connect_root/*" {
    12.   capabilities = [ "create", "read", "update", "delete", "list" ]
    13. }
    14. path "/dc1/connect_inter/*" {
    15.   capabilities = [ "create", "read", "update", "delete", "list" ]
    16. }
    17. EOF
    1. $ vault policy write connect-ca-dc1 - <<EOF
    2. path "/sys/mounts" {
    3.   capabilities = [ "read" ]
    4. }
    5. path "/sys/mounts/connect_root" {
    6.   capabilities = [ "create", "read", "update", "delete", "list" ]
    7. }
    8. path "/sys/mounts/dc1/connect_inter" {
    9.   capabilities = [ "create", "read", "update", "delete", "list" ]
    10. }
    11. path "/connect_root/*" {
    12.   capabilities = [ "create", "read", "update", "delete", "list" ]
    13. }
    14. path "/dc1/connect_inter/*" {
    15.   capabilities = [ "create", "read", "update", "delete", "list" ]
    16. }
    17. EOF

  2. Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access. For each auth method in Vault, you will need roles for the Consul server, Consul client, the server-acl-init job, and Consul server CA.

    1. $ vault write auth/kubernetes-dc1/role/consul-server \
    2.         bound_service_account_names=consul-server \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip,connect-ca-dc1,consul-cert-dc1" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc1/role/consul-server \
    2.         bound_service_account_names=consul-server \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip,connect-ca-dc1,consul-cert-dc1" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc1/role/consul-client \
    2.         bound_service_account_names=consul-client \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc1/role/consul-client \
    2.         bound_service_account_names=consul-client \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc1/role/server-acl-init \
    2.         bound_service_account_names=consul-server-acl-init \
    3.         bound_service_account_namespaces="default" \
    4.         policies="replication-token" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc1/role/server-acl-init \
    2.         bound_service_account_names=consul-server-acl-init \
    3.         bound_service_account_namespaces="default" \
    4.         policies="replication-token" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc1/role/consul-ca \
    2.         bound_service_account_names="*" \
    3.         bound_service_account_namespaces="default" \
    4.         policies=ca-policy \
    5.         ttl=1h
    1. $ vault write auth/kubernetes-dc1/role/consul-ca \
    2.         bound_service_account_names="*" \
    3.         bound_service_account_namespaces="default" \
    4.         policies=ca-policy \
    5.         ttl=1h

  3. Create the server TLS Cert role.

    1. $ vault write pki/roles/consul-cert-dc1 \
    2.   allowed_domains="dc1.consul,consul-server,consul-server.default,consul-server.default.svc" \
    3.   allow_subdomains=true \
    4.   allow_bare_domains=true \
    5.   allow_localhost=true \
    6.   generate_lease=true \
    7.   max_ttl="720h"
    1. $ vault write pki/roles/consul-cert-dc1 \
    2.   allowed_domains="dc1.consul,consul-server,consul-server.default,consul-server.default.svc" \
    3.   allow_subdomains=true \
    4.   allow_bare_domains=true \
    5.   allow_localhost=true \
    6.   generate_lease=true \
    7.   max_ttl="720h"

  4. Configure and install Consul in the primary datacenter (dc1).

    1. global:
    2.   datacenter: "dc1"
    3.   name: consul
    4.   secretsBackend:
    5.     vault:
    6.       enabled: true
    7.       consulServerRole: consul-server
    8.       consulClientRole: consul-client
    9.       consulCARole: consul-ca
    10.       manageSystemACLsRole: server-acl-init
    11.       connectCA:
    12.         address: http://vault-dc1.default:8200
    13.         rootPKIPath: connect_root/
    14.         intermediatePKIPath: dc1/connect_inter/
    15.         authMethodPath: kubernetes-dc1
    16.   tls:
    17.     enabled: true
    18.     enableAutoEncrypt: true
    19.     caCert:
    20.       secretName: pki/cert/ca
    21.   federation:
    22.     enabled: true
    23.   acls:
    24.     manageSystemACLs: true
    25.     replicationToken:
    26.       secretName: consul/data/secret/replication
    27.       secretKey: token
    28.   gossipEncryption:
    29.     secretName: consul/data/secret/gossip
    30.     secretKey: key
    31. server:
    32.   replicas: 1
    33.   serverCert:
    34.     secretName: "pki/issue/consul-cert-dc1"
    35. connectInject:
    36.   replicas: 1
    37.   enabled: true
    38. controller:
    39.   enabled: true
    40. meshGateway:
    41.   enabled: true
    42.   replicas: 1

    WAN Federation - 图12

    consul-dc1.yaml

    1. global:
    2.   datacenter: "dc1"
    3.   name: consul
    4.   secretsBackend:
    5.     vault:
    6.       enabled: true
    7.       consulServerRole: consul-server
    8.       consulClientRole: consul-client
    9.       consulCARole: consul-ca
    10.       manageSystemACLsRole: server-acl-init
    11.       connectCA:
    12.         address: http://vault-dc1.default:8200
    13.         rootPKIPath: connect_root/
    14.         intermediatePKIPath: dc1/connect_inter/
    15.         authMethodPath: kubernetes-dc1
    16.   tls:
    17.     enabled: true
    18.     enableAutoEncrypt: true
    19.     caCert:
    20.       secretName: pki/cert/ca
    21.   federation:
    22.     enabled: true
    23.   acls:
    24.     manageSystemACLs: true
    25.     replicationToken:
    26.       secretName: consul/data/secret/replication
    27.       secretKey: token
    28.   gossipEncryption:
    29.     secretName: consul/data/secret/gossip
    30.     secretKey: key
    31. server:
    32.   replicas: 1
    33.   serverCert:
    34.     secretName: "pki/issue/consul-cert-dc1"
    35. connectInject:
    36.   replicas: 1
    37.   enabled: true
    38. controller:
    39.   enabled: true
    40. meshGateway:
    41.   enabled: true
    42.   replicas: 1

    Next, install Consul in the primary Kubernetes cluster (dc1).

    1. $ helm install consul-dc1 --values consul-dc1.yaml hashicorp/consul
    1. $ helm install consul-dc1 --values consul-dc1.yaml hashicorp/consul

Pre-installation for Secondary Datacenter (dc2)

  1. Update the Consul on Kubernetes helm chart. For secondary datacenter (dc2), you will need to get the address of the mesh gateway from the primary datacenter (dc1) cluster.

    Keep your Kubernetes context targeting dc1 and set the MESH_GW_HOST environment variable that you will use in the Consul Helm chart for secondary datacenter (dc2).

    1. $ kubectl config use-context <context for dc1>
    1. $ kubectl config use-context <context for dc1>

    Next, get mesh gateway address:

    WAN Federation - 图13

    WAN Federation - 图14

    1. $ export MESH_GW_HOST=$(kubectl get svc consul-mesh-gateway --output jsonpath='{.status.loadBalancer.ingress[0].hostname}')
    1. $ export MESH_GW_HOST=$(kubectl get svc consul-mesh-gateway --output jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  2. Change your Kubernetes context to target the primary datacenter (dc2):

    1. $ kubectl config use-context <context for dc2>
    1. $ kubectl config use-context <context for dc2>

Secondary Datacenter (dc2)

  1. Create Server TLS and Service Mesh Cert Policies

    1. $ vault policy write consul-cert-dc2 - <<EOF
    2. path "pki/issue/consul-cert-dc2"
    3. {
    4.   capabilities = ["create","update"]
    5. }
    6. EOF
    1. $ vault policy write consul-cert-dc2 - <<EOF
    2. path "pki/issue/consul-cert-dc2"
    3. {
    4.   capabilities = ["create","update"]
    5. }
    6. EOF
    1. $ vault policy write connect-ca-dc2 - <<EOF
    2. path "/sys/mounts" {
    3.   capabilities = [ "read" ]
    4. }
    5. path "/sys/mounts/connect_root" {
    6.   capabilities = [ "create", "read", "update", "delete", "list" ]
    7. }
    8. path "/sys/mounts/dc2/connect_inter" {
    9.   capabilities = [ "create", "read", "update", "delete", "list" ]
    10. }
    11. path "/connect_root/*" {
    12.   capabilities = [ "create", "read", "update", "delete", "list" ]
    13. }
    14. path "/dc2/connect_inter/*" {
    15.   capabilities = [ "create", "read", "update", "delete", "list" ]
    16. }
    17. EOF
    1. $ vault policy write connect-ca-dc2 - <<EOF
    2. path "/sys/mounts" {
    3.   capabilities = [ "read" ]
    4. }
    5. path "/sys/mounts/connect_root" {
    6.   capabilities = [ "create", "read", "update", "delete", "list" ]
    7. }
    8. path "/sys/mounts/dc2/connect_inter" {
    9.   capabilities = [ "create", "read", "update", "delete", "list" ]
    10. }
    11. path "/connect_root/*" {
    12.   capabilities = [ "create", "read", "update", "delete", "list" ]
    13. }
    14. path "/dc2/connect_inter/*" {
    15.   capabilities = [ "create", "read", "update", "delete", "list" ]
    16. }
    17. EOF

  2. Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access. For each auth method in Vault, you will need roles for the Consul server, Consul client, the server-acl-init job, and Consul server CA.

    1. $ vault write auth/kubernetes-dc2/role/consul-server \
    2.         bound_service_account_names=consul-server \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip,connect-ca-dc2,consul-cert-dc2,replication-token" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc2/role/consul-server \
    2.         bound_service_account_names=consul-server \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip,connect-ca-dc2,consul-cert-dc2,replication-token" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc2/role/consul-client \
    2.         bound_service_account_names=consul-client \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc2/role/consul-client \
    2.         bound_service_account_names=consul-client \
    3.         bound_service_account_namespaces="default" \
    4.         policies="gossip" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc2/role/server-acl-init \
    2.         bound_service_account_names=consul-server-acl-init \
    3.         bound_service_account_namespaces="default" \
    4.         policies="replication-token" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc2/role/server-acl-init \
    2.         bound_service_account_names=consul-server-acl-init \
    3.         bound_service_account_namespaces="default" \
    4.         policies="replication-token" \
    5.         ttl=24h
    1. $ vault write auth/kubernetes-dc2/role/consul-ca \
    2.         bound_service_account_names="*" \
    3.         bound_service_account_namespaces="default" \
    4.         policies=ca-policy \
    5.         ttl=1h
    1. $ vault write auth/kubernetes-dc2/role/consul-ca \
    2.         bound_service_account_names="*" \
    3.         bound_service_account_namespaces="default" \
    4.         policies=ca-policy \
    5.         ttl=1h

  3. Create the Server TLS Cert role.

    1. $ vault write pki/roles/consul-cert-dc2 \
    2.   allowed_domains="dc2.consul,consul-server,consul-server.default,consul-server.default.svc" \
    3.   allow_subdomains=true \
    4.   allow_bare_domains=true \
    5.   allow_localhost=true \
    6.   generate_lease=true \
    7.   max_ttl="720h"
    1. $ vault write pki/roles/consul-cert-dc2 \
    2.   allowed_domains="dc2.consul,consul-server,consul-server.default,consul-server.default.svc" \
    3.   allow_subdomains=true \
    4.   allow_bare_domains=true \
    5.   allow_localhost=true \
    6.   generate_lease=true \
    7.   max_ttl="720h"

  4. Configure and install Consul in the secondary datacenter (dc2).

    Note: To configure Vault as the Connect CA in secondary datacenters, you need to make sure that the Root CA path is the same. The intermediate path is different for each datacenter. In the connectCA Helm configuration for a secondary datacenter, you can specify a intermediatePKIPath that is, for example, prefixed with the datacenter for which this configuration is intended (e.g. dc2/connect-intermediate).

    1. global:
    2.   datacenter: "dc2"
    3.   name: consul
    4.   secretsBackend:
    5.     vault:
    6.       enabled: true
    7.       consulServerRole: consul-server
    8.       consulClientRole: consul-client
    9.       consulCARole: consul-ca
    10.       manageSystemACLsRole: server-acl-init
    11.       connectCA:
    12.         address: ${VAULT_ADDR}
    13.         rootPKIPath: connect_root/
    14.         intermediatePKIPath: dc2/connect_inter/
    15.         authMethodPath: kubernetes-dc2
    16.   tls:
    17.     enabled: true
    18.     enableAutoEncrypt: true
    19.     caCert:
    20.       secretName: "pki/cert/ca"
    21.   federation:
    22.     enabled: true
    23.     primaryDatacenter: dc1
    24.     k8sAuthMethodHost: ${KUBE_API_URL_DC2}
    25.     primaryGateways:
    26.     - ${MESH_GW_HOST}:443
    27.   acls:
    28.     manageSystemACLs: true
    29.     replicationToken:
    30.       secretName: consul/data/secret/replication
    31.       secretKey: token
    32.   gossipEncryption:
    33.     secretName: consul/data/secret/gossip
    34.     secretKey: key
    35. server:
    36.   replicas: 1
    37.   serverCert:
    38.     secretName: "pki/issue/consul-cert-dc2"
    39. connectInject:
    40.   replicas: 1
    41.   enabled: true
    42. controller:
    43.   enabled: true
    44. meshGateway:
    45.   enabled: true
    46.   replicas: 1

    WAN Federation - 图15

    consul-dc2.yaml

    1. global:
    2.   datacenter: "dc2"
    3.   name: consul
    4.   secretsBackend:
    5.     vault:
    6.       enabled: true
    7.       consulServerRole: consul-server
    8.       consulClientRole: consul-client
    9.       consulCARole: consul-ca
    10.       manageSystemACLsRole: server-acl-init
    11.       connectCA:
    12.         address: ${VAULT_ADDR}
    13.         rootPKIPath: connect_root/
    14.         intermediatePKIPath: dc2/connect_inter/
    15.         authMethodPath: kubernetes-dc2
    16.   tls:
    17.     enabled: true
    18.     enableAutoEncrypt: true
    19.     caCert:
    20.       secretName: "pki/cert/ca"
    21.   federation:
    22.     enabled: true
    23.     primaryDatacenter: dc1
    24.     k8sAuthMethodHost: ${KUBE_API_URL_DC2}
    25.     primaryGateways:
    26.     - ${MESH_GW_HOST}:443
    27.   acls:
    28.     manageSystemACLs: true
    29.     replicationToken:
    30.       secretName: consul/data/secret/replication
    31.       secretKey: token
    32.   gossipEncryption:
    33.     secretName: consul/data/secret/gossip
    34.     secretKey: key
    35. server:
    36.   replicas: 1
    37.   serverCert:
    38.     secretName: "pki/issue/consul-cert-dc2"
    39. connectInject:
    40.   replicas: 1
    41.   enabled: true
    42. controller:
    43.   enabled: true
    44. meshGateway:
    45.   enabled: true
    46.   replicas: 1

    Next, install Consul in the consul Kubernetes cluster (dc2).

    1. $ helm install consul-dc2 -f consul-dc2.yaml hashicorp/consul
    1. $ helm install consul-dc2 -f consul-dc2.yaml hashicorp/consul

Next steps

You have completed the process of federating the secondary datacenter (dc2) with the primary datacenter (dc1) using Vault as the Secrets backend. To validate that everything is configured properly, please confirm that all pods within both datacenters are in a running state.

For further detail on specific Consul secrets that are available to be stored in Vault, please checkout the detailed information in the Data Integration section of the Vault as a Secrets Backend area of the Consul on Kubernetes documentation.