Onboard services while in transparent proxy mode

This topic describes how to run Consul in permissive mTLS mode so that you can safely onboard existing applications to Consul service mesh when transparent proxy mode is enabled.

Background

When transparent proxy mode is enabled, all service-to-service traffic is secured by mTLS. Until the services that you want to add to the network are fully onboarded, your network may have a mix of mTLS and non-mTLS traffic, which can result in broken service-to-service communication. This situation occurs because sidecar proxies for existing mesh services reject traffic from services that are not yet onboarded.

You can enable the permissive mTLS mode to ensure existing non-mTLS service-to-service traffic is allowed during the onboarding phase. The permissive mTLS mode enables sidecar proxies to accept both mTLS and non-mTLS traffic to an application. Using this mode enables you to onboard without downtime and without being required to reconfigure or redeploy your application.

We recommend enabling permissive mTLS as a temporary operating mode. After onboarding is complete, you should reconfigure all services to strict mTLS mode to ensure all service-to-service communication is automatically secured by Consul service mesh.

Security warning: We recommend that you disable permissive mTLS mode after onboarding services to prevent non-mTLS connections to the service. Intentions are not enforced and encryption is not enabled for non-mTLS connections.

Workflow

The workflow to configure mTLS settings depends on the applications you are onboarding and the order you intend to onboard them, but the following steps describe the general workflow:

1. Configure global settings: Configure the mesh to allow services to send non-mTLS messages to services outside the mesh. Additionally, configure the mesh to let services in the mesh use permissive mTLS mode.
2. Enable permissive mTLS mode: If you are onboarding an upstream service prior to its related downstream services, then enable permissive mTLS mode in the service defaults configuration entry. This allows the upstream service to send encrypted messages from the mesh when you register the service with Consul.
3. Configure intentions: Intentions are controls that authorize traffic between services in the mesh. Transparent proxy uses intentions to infer traffic routes between Envoy proxies. Consul does not enforce intentions for non-mTLS connections made while proxies are in permissive mTLS mode, but intentions are necessary for completing the onboarding process.
4. Register the service: Create the service definition and configure and deploy its sidecar proxy.
5. Re-secure the mesh: If you enabled permissive mTLS mode, switch back to strict mTLS mode and revert the global settings to disable non-mTLS traffic in the service mesh.

Requirements

Permissive mTLS is only supported for services running in transparent proxy mode. Transparent proxy mode is only available on Kubernetes deployments.

Configure global settings

Configure Consul to allow services that are already in the mesh to send non-mTLS messages to services outside the mesh. You can also Consul to allow services to run in permissive mTLS mode. Set both configurations in the mesh gateway configuration entry, which is the global configuration that defines service mesh proxy behavior.

Allow outgoing non-mTLS traffic

You can configure a global setting that allows services in the mesh to send non-mTLS messages to services outside the mesh.

Add the MeshDestinationsOnly property to the mesh configuration entry and set the property to false. If the services belong to multiple admin partitions, you must apply the setting in each partition:

Allow non-mTLS traffic

Allow non-mTLS traffic

Kind = "mesh"
TransparentProxy {
  MeshDestinationsOnly = false
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
  name: mesh
spec:
  allowEnablingPermissiveMutualTLS: true

{
    "Kind": "mesh",
    "TransparentProxy": [
        {
            "MeshDestinationsOnly": false
        }
    ]
}

Alternatively, you can selectively allow outgoing traffic on a per-service basis by configuring outbound port exclusions. This setting excludes outgoing traffic from traffic redirection imposed by transparent proxy. When changing this setting, you must redeploy your application.

Allow permissive mTLS modes for incoming traffic

Set the AllowEnablingPermissiveMutualTLS parameter in the mesh configuration entry to true so that services in the mesh are able to use permissive mTLS mode for incoming traffic. The parameter does not direct services to use permissive mTLS. It is a global parameter that allows services to run in permissive mTLS mode.

Allow permissive mTLS mode

Allow permissive mTLS mode

Kind = "mesh"
AllowEnablingPermissiveMutualTLS = true
TransparentProxy {
  MeshDestinationsOnly = false
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
  name: mesh
spec:
  allowEnablingPermissiveMutualTLS: true
  transparentProxy:
    meshDestinationsOnly: false

{
    "Kind": "mesh",
    "AllowEnablingPermissiveMutualTLS": true,
    "TransparentProxy": [
        {
            "MeshDestinationsOnly": false
        }
    ]
}

You can change this setting back to false at any time, even if there are services currently running in permissive mode. Doing so allows you to decide at which point during the onboarding process to stop allowing services to use permissive mTLS. When the MeshDestinationOnly is set to false, you must configure all new services added to the mesh with MutualTLSMode=strict for the Consul to securely route traffic throughout the mesh.

Enable permissive mTLS mode

Depending on the services you are onboarding, you may not need to enable permissive mTLS mode. If the service does not accept incoming traffic or accepts traffic from downstream services that are already part of the service mesh, then permissive mTLS mode is not required to continue.

To enable permissive mTLS mode for the service, set MutualTLSMode=permissive in the service defaults configuration entry for the service. The following example shows how to configure this setting for a service named example-service.

Enable permissive mTLS for applicable services

Enable permissive mTLS for applicable services

Kind      = "service-defaults"
Name      = "example-service"
MutualTLSMode = "permissive"

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: example-service
spec:
  mutualTLSMode: "permissive"

{
    "Kind": "service-defaults",
    "Name": "example-service",
    "MutualTLSMode": "permissive"
}

Refer to the service defaults configuration reference for information about all settings.

You can change this setting back to strict at any time to ensure mTLS is required for incoming traffic to this service.

Configure intentions

Service intentions are mechanisms in Consul that control traffic between services in the mesh.

We recommend creating intentions that restrict services to accepting only necessary traffic. You must identify the downstream services that send messages to the service you want to add to the mesh and then create an intention to allow traffic to the service from its downstreams.

When transparent proxy enabled and the MutualTLSMode parameter is set to permissive, incoming traffic from a downstream service to another upstream service is not secured by mTLS unless that upstream relationship is known to Consul. You must either define an intention so that Consul can infer the upstream relationship from the intention, or you must include an explicit upstream as part of the service definition for the downstream.

Refer to Service intentions for additional information about how intentions work and how to create them.

Add the service to the mesh

Register your service into the catalog and update your application to deploy a sidecar proxy. You should also monitor your service to verify its configuration. Refer to the Consul on Kubernetes service mesh overview for additional information.

Re-secure mesh traffic

If the newly added service was placed in permissive mTLS mode for onboarding, then you should switch to strict mode when it is safe to do so. You should also revert the global settings that allow services to send and receive non-mTLS traffic.

Disable permissive mTLS mode

Configure the service to operate in strict mTLS mode after the service is no longer receiving incoming non-mTLS traffic. After the downstream services that send messages to this service are all onboarded to the mesh, this service should no longer receive non-mTLS traffic.

Check the following Envoy listener statistics for the sidecar proxy to determine if the sidecar is receiving non-mTLS traffic:

• The tcp.permissive_public_listener.* statistics indicate non-mTLS traffic. If these metrics are static over a sufficient period of time, that indicates the sidecar is not receiving non-mTLS traffic.
• The tcp.public_listener.* statistics indicate mTLS traffic. If incoming traffic is expected to this service and these statistics are changing, then the sidecar is receiving mTLS traffic.

Refer to the service mesh observability overview and metrics configuration for Consul on Kubernetes documentation for additional information.

If your service is still receiving non-mTLS traffic, complete the following steps to determine the source of the non-mTLS traffic:

1. Verify the list of downstream services. Optionally, you can enable Envoy access logging to determine source IP addresses for incoming traffic, and cross-reference those IP addresses with services in your network.
2. Verify that each downstream is onboarded to the service mesh. If a downstream is not onboarded, consider onboarding it next.
3. Verify that each downstream has an intention that allows it to send traffic to the upstream service.

After you determine it is safe to move the service to strict mode, set MutualTLSMode=strict in the service defaults configuration entry.

Enable strict mTLS mode for applicable services

Enable strict mTLS mode for applicable services

Kind      = "service-defaults"
Name      = "example-service"
MutualTLSMode = "strict"

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: example-service
spec:
  mutualTLSMode: "strict"

{
    "Kind": "service-defaults",
    "MutualTLSMode": "strict",
    "Name": "example-service"
}

Disable non-mTLS traffic

After all services are onboarded, revert the global settings that allow non-mTLS traffic and verify that permissive mTLS mode is not being used in the mesh.

Set AllowEnablingPermissiveMutualTLS=false and MeshDestinationsOnly=true in the mesh config entry.

Disable non-mTLS traffic

Disable non-mTLS traffic

Kind = “mesh”
AllowEnablingPermissiveMutualTLS = false
TransparentProxy {
  MeshDestinationsOnly = true
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
  name: mesh
spec:
  allowEnablingPermissiveMutualTLS: false
  transparentProxy:
    meshDestinationsOnly: true

{
    "Kind": "mesh",
    "AllowEnablingPermissiveMutualTLS": false,
    "TransparentProxy": [
        {
            "MeshDestinationsOnly": true
        }
    ]
}

For each namespace, admin partition, and datacenter in your Consul deployment, run the consul config list and consul config read commands to verify that no services are using permissive mTLS mode.

The following command returns any service defaults configuration entries that contain 'MutualTLSMode = "permissive"':

$ consul config list -kind service-defaults -filter 'MutualTLSMode == "permissive"'

In each admin partition and datacenter, verify that MutualTLSMode = "permissive" is not set in the proxy defaults configuration entry . If MutualTLSMode is either empty or if the configuration entry is not found, then the mode is strict by default.

The following command fetches the proxy defaults configuration entry:

$ consul config read -kind proxy-defaults -name global
{
    "Kind": "proxy-defaults",
    "Name": "global",
    "Partition": "default",
    "Namespace": "default",
    "TransparentProxy": {},
    "MutualTLSMode": "",
    "MeshGateway": {},
    "Expose": {},
    "AccessLogs": {},
    "CreateIndex": 26,
    "ModifyIndex": 30
}