Vault as a Service Mesh Certificate Authority

Vault as a Service Mesh Certificate Authority

You can configure Consul to use Vault as the certificate authority (CA) so that Vault can manage and sign certificates distributed to services in the mesh. The Vault CA provider uses the Vault PKI secrets engine to generate and sign certificates. This page describes how configure the Vault CA provider.

Tutorial: Complete the Vault as Consul Service Mesh Certification Authority tutorial for hands-on guidance on how to configure Vault as the Consul service mesh certification authority.

Requirements

Refer to Service Mesh Certificate Authority Overview for important background information about how Consul manages certificates with configurable CA providers.
Vault 0.10.3 to 1.10.x.

Compatibility note: If you use Vault 1.11.0+ as Consul’s service mesh CA, versions of Consul released before Dec 13, 2022 will develop an issue with Consul control plane or service mesh communication (GH-15525). Use or upgrade to a Consul version that includes the fix to avoid this problem.

Enable Vault as the CA

You can enable Vault as the CA by configuring Consul to use "vault" as the CA provider and including the required provider configuration options. You can provide the CA configuration in the server agents’ configuration file or in the body of a PUT request to the /connect/ca/configuration API endpoint. Refer to the Configuration Reference for details about configuration options and for example use cases.

The following example shows the required configurations for a default implementation:

Service mesh CA configuration

/etc/consul.d/config.hcl

# ...
connect {
  enabled = true
  ca_provider = "vault"
  ca_config {
    address = "http://localhost:8200"
    token = "<vault-token-with-necessary-policy>"
    root_pki_path = "connect-root"
    intermediate_pki_path = "connect-dc1-intermediate"
  }
}

{
  "Provider": "vault",
  "Config": {
    "Address": "http://localhost:8200",
    "Token": "<vault-token-with-necessary-policy>",
    "RootPKIPath": "connect-root",
    "IntermediatePKIPath": "connect-dc1-intermediate"
  }
}

Configuration Reference

You can specify the following configuration options. Note that a configuration option’s name may differ between API calls and the agent configuration file. The first key refers to the option name for use in API calls. The key after the slash refers to the corresponding option name in the agent configuration file.

Address / address (string: <required>) - The address of the Vault server.
Token / token (string: "") - A token for accessing Vault. This is write-only and will not be exposed when reading the CA configuration. This token must have proper privileges for the PKI paths configured. In Consul 1.8.5 and later, if the token has the renewable flag set, Consul will attempt to renew its lease periodically after half the duration has expired.

Warning: You must either provide a token or configure an auth method below.
AuthMethod / auth_method (map: nil) - Vault auth method to use for logging in to Vault. Please see Vault Auth Methods for more information on how to configure individual auth methods. If auth method is provided, Consul will obtain a new token from Vault when the token can no longer be renewed.
- Type/ type (string: "") - The type of Vault auth method. Valid options are “approle”, “aws”, “azure”, “gcp”, “jwt” and “kubernetes”.
- MountPath/ mount_path (string: <AuthMethod.Type>) - The mount path of the auth method. If not provided the auth method type will be used as the mount path.
- Params/params (map: nil) - The parameters to configure the auth method. The configuration parameters needed will depend on which auth type you are using. Please refer to the Vault Agent auto-auth method documentation for details on their configuration options: AppRole, AWS, Azure, GCP, JWT, Kubernetes.
  
  Only the authentication related fields (for example, JWT’s path and role) are supported. The optional management fields (for example: remove_jwt_after_reading) are not supported.
RootPKIPath / root_pki_path (string: <required>) - The path to a PKI secrets engine for the root certificate.

If the path does not exist, Consul will mount a new PKI secrets engine at the specified path with the RootCertTTL value as the root certificate’s TTL. If the RootCertTTL is not set, a max_lease_ttl of 87600 hours, or 10 years is applied by default as of Consul 1.11 and later. Prior to Consul 1.11, the root certificate TTL was set to 8760 hour, or 1 year, and was not configurable. The root certificate will expire at the end of the specified period.

When WAN Federation is enabled, each secondary datacenter must use the same Vault cluster and share the same root_pki_path with the primary datacenter.

To use an intermediate certificate as the primary CA in Consul, initialize the RootPKIPath in Vault with a PEM bundle. The first certificate in the bundle must be the intermediate certificate that Consul will use as the primary CA. The last certificate in the bundle must be a root certificate. The bundle must contain a valid chain, where each certificate is followed by the certificate that authorized it.
RootPKINamespace / root_pki_namespace (string: <optional>) - The absolute namespace that the RootPKIPath is in. Setting this parameter overrides the Namespace option for the RootPKIPath. Introduced in 1.12.3.
IntermediatePKIPath / intermediate_pki_path (string: <required>) - The path to a PKI secrets engine for the generated intermediate certificate. This certificate will be signed by the configured root PKI path. If this path does not exist, Consul will attempt to mount and configure this automatically.

When WAN Federation is enabled, every secondary datacenter must specify a unique intermediate_pki_path.
IntermediatePKINamespace / intermediate_pki_namespace (string: <optional>) - The absolute namespace that the IntermediatePKIPath is in. Setting this parameter overrides the Namespace option for the IntermediatePKIPath. Introduced in 1.12.3.
CAFile / ca_file (string: "") - Specifies an optional path to the CA certificate used for Vault communication. If unspecified, this will fallback to the default system CA bundle, which varies by OS and version.
CAPath / ca_path (string: "") - Specifies an optional path to a folder containing CA certificates to be used for Vault communication. If unspecified, this will fallback to the default system CA bundle, which varies by OS and version.
CertFile / cert_file (string: "") - Specifies the path to the certificate used for Vault communication. If this is set, then you also need to set key_file.
KeyFile / key_file (string: "") - Specifies the path to the private key used for Vault communication. If this is set, then you also need to set cert_file.
TLSServerName / tls_server_name (string: "") - Specifies an optional string used to set the SNI host when connecting to Vault via TLS.
TLSSkipVerify / tls_skip_verify (bool: false) - Specifies if SSL peer validation should be enforced.
Namespace / namespace (string: <optional>) - The Vault Namespace that the Token and PKI Certificates are a part of. Vault Namespaces are a Vault Enterprise feature. Added in Consul 1.11.0

Common CA Config Options

The following configuration options are supported by all CA providers:

CSRMaxConcurrent / csr_max_concurrent (int: 0) - Sets a limit on the number of Certificate Signing Requests that can be processed concurrently. Defaults to 0 (disabled). This is useful when you want to limit the number of CPU cores available to the server for certificate signing operations. For example, on an 8 core server, setting this to 1 will ensure that no more than one CPU core will be consumed when generating or rotating certificates. Setting this is recommended instead of csr_max_per_second when you want to limit the number of cores consumed since it is simpler to reason about limiting CSR resources this way without artificially slowing down rotations. Added in 1.4.1.
CSRMaxPerSecond / csr_max_per_second (float: 50) - Sets a rate limit on the maximum number of Certificate Signing Requests (CSRs) the servers will accept. This is used to prevent CA rotation from causing unbounded CPU usage on servers. It defaults to 50 which is conservative – a 2017 MacBook can process about 100 per second using only ~40% of one CPU core – but sufficient for deployments up to ~1500 service instances before the time it takes to rotate is impacted. For larger deployments we recommend increasing this based on the expected number of server instances and server resources, or use csr_max_concurrent instead if servers have more than one CPU core. Setting this to zero disables rate limiting. Added in 1.4.1.
LeafCertTTL / leaf_cert_ttl (duration: "72h") - The upper bound on the lease duration of a leaf certificate issued for a service. In most cases a new leaf certificate will be requested by a proxy before this limit is reached. This is also the effective limit on how long a server outage can last (with no leader) before network connections will start being rejected. Defaults to 72h. This value cannot be lower than 1 hour or higher than 1 year.

This value is also used when rotating out old root certificates from the cluster. When a root certificate has been inactive (rotated out) for more than twice the current leaf_cert_ttl, it will be removed from the trusted list.
RootCertTTL / root_cert_ttl (duration: "87600h") The time to live (TTL) for a root certificate. Defaults to 10 years as 87600h. This value, if provided, needs to be higher than the intermediate certificate TTL.

This setting applies to all Consul CA providers.

For the Vault provider, this value is only used if the backend is not initialized at first.
IntermediateCertTTL / intermediate_cert_ttl (duration: "8760h") The time to live (TTL) for any intermediate certificates signed by root certificate of the primary datacenter. This field is only valid in the primary datacenter. Defaults to 1 year as 8760h.

This setting applies to all Consul CA providers.

For the Vault provider, this value is only used if the backend is not initialized at first.
PrivateKeyType / private_key_type (string: "ec") - The type of key to generate for this CA. This is only used when the provider is generating a new key. If private_key is set for the Consul provider, or existing root or intermediate PKI paths given for Vault then this will be ignored. Currently supported options are ec or rsa. Default is ec.

It is required that all servers in a datacenter have the same config for the CA. It is recommended that servers in different datacenters use the same key type and size, although the built-in CA and Vault provider will both allow mixed CA key types.

Some CA providers (currently Vault) will not allow cross-signing a new CA certificate with a different key type. This means that if you migrate from an RSA-keyed Vault CA to an EC-keyed CA from any provider, you may have to proceed without cross-signing which risks temporary connection issues for workloads during the new certificate rollout. We highly recommend testing this outside of production to understand the impact, and suggest sticking to same key type where possible.

Note: This only affects CA keys generated by the provider. Leaf certificate keys are always EC 256 regardless of the CA configuration.
PrivateKeyBits / private_key_bits (string: "") - The length of key to generate for this CA. This is only used when the provider is generating a new key. If private_key is set for the Consul provider, or existing root or intermediate PKI paths given for Vault then this will be ignored.

Currently supported values are:
- private_key_type = ec (default): 224, 256, 384, 521 corresponding to the NIST P-* curves of the same name.
- private_key_type = rsa: 2048, 4096

Root and Intermediate PKI Paths

The Vault CA provider uses two separately configured PKI secrets engines for managing Consul service mesh certificates.

The RootPKIPath is the PKI engine for the root certificate. Consul uses this root certificate to sign the intermediate certificate. Consul does not attempt to write or modify any data within the root PKI path.

The IntermediatePKIPath is the PKI engine used for storing the intermediate signed with the root certificate. The intermediate signs all leaf certificates, and Consul may periodically generate new intermediates for automatic rotation. Therefore, Consul requires write access to this path.

For each path, if the path does not exist, Consul attempts to mount and initialize a PKI secrets engine at that path. For this operation to succeed, the provided Vault token must have the required Vault privileges. If the path does already exist, Consul uses the PKI secrets engine at that path as configured.

Configure Vault ACL policies

The Vault CA provider requires a Vault token with a specific set of Vault privileges depending on whether you would prefer to control mount configuration from Vault or to delegate that responsibility to Consul.

Use Vault-managed PKI paths to obtain the following benefits:

Enables use of a root CA external to Consul by instantiating the PKI secrets engine at RootPKIPath with an intermediate certificate
Retain full control over PKI mount creation, and over RootPKIPath mount configuration

Otherwise, use Consul-managed PKI paths to let Consul fully automate PKI management.

The following sections describe the Vault policy needed for both options. The policy snippets use placeholder values for RootPKIPath and IntermediatePKIPath. Replace them to match the path values in your CA provider configuration.

Define a policy for Vault-managed PKI paths

To use Vault-managed PKI paths, you must first instantiate and configure PKI secrets engines at RootPKIPath and IntermediatePKIPath.

Then, attach the following Vault ACL policy to the CA provider’s Vault token or auth method:

Allow Consul to read both PKI mounts and to manage the intermediate PKI mount configuration:

vault-managed-pki-policy.hcl

path "/sys/mounts/<root_pki_path>" {
  capabilities = [ "read" ]
}
path "/sys/mounts/<intermediate_pki_path>" {
  capabilities = [ "read" ]
}
path "/sys/mounts/<intermediate_pki_path>/tune" {
  capabilities = [ "update" ]
}

Allow Consul read-only access to the root PKI engine, to automatically rotate intermediate CAs as needed, and full use of the intermediate PKI engine:

vault-managed-pki-policy.hcl

path "/<root_pki_path>/" {
  capabilities = [ "read" ]
}
path "/<root_pki_path>/root/sign-intermediate" {
  capabilities = [ "update" ]
}
path "/<intermediate_pki_path>/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

Allow Consul to renew its Vault token if the token is renewable. The rule enables the token to be renewed whether it is provided directly in the CA provider configuration or presented in an auth method.

vault-managed-pki-policy.hcl
```
path "auth/token/renew-self" {
  capabilities = [ "update" ]
}
path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}
```

Define a policy for Consul-managed PKI paths

To use Consul-managed PKI paths, ensure no PKI secrets engines are mounted at RootPKIPath and IntermediatePKIPath.

Then, attach the following Vault ACL policy to the CA provider’s Vault token or auth method:

Allow Consul to create and manage both PKI engines:

consul-managed-pki-policy.hcl

path "/sys/mounts/<root_pki_path>" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "/sys/mounts/<intermediate_pki_path>" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "/sys/mounts/<intermediate_pki_path>/tune" {
  capabilities = [ "update" ]
}

Allow Consul full use of both PKI engines:

consul-managed-pki-policy.hcl

path "/<root_pki_path>/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "/<intermediate_pki_path>/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

Allow Consul to renew its Vault token if the token is renewable. The rule enables the token to be renewed whether it is provided directly in the CA provider configuration or presented in an auth method.

consul-managed-pki-policy.hcl
```
path "auth/token/renew-self" {
  capabilities = [ "update" ]
}
path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}
```

Additional Vault ACL policies for sensitive operations

Additional Vault permissions are required while you perform the following CA provider configuration changes:

Changing the provider from Vault to a different provider, such as Consul’s built-in provider
Changing the RootPKIPath

Those configuration modifications trigger a root CA change that requires an extremely privileged root cross-sign operation. For that operation to succeed, the CA provider’s Vault token or auth method must contain the following rule:

temporary-sensitive-operation-pki-policy.hcl

path "<root_pki_path>/root/sign-self-issued" {
  capabilities = [ "sudo", "update" ]
}

We recommend using the following process when performing such a CA provider configuration change to minimize the time that a privileged Vault token is in use:

Create a new Vault token that includes both the root/sign-self-issued permission and the standard permissions for the current Consul or Vault managed PKI paths.
Modify the CA provider configuration to use that new Vault token.
Perform the CA provider configuration change that triggers the extremely privileged root cross-sign operation. If the configuration change maintains Vault as the provider but modifies RootPKIPath, the configuration change must include a Vault token or auth method with standard permissions for the new Consul or Vault managed PKI paths.

Vault Integration