Envoy Proxy Configuration for Service Mesh

Consul service mesh has first class support for using Envoy as a proxy. Consul configures Envoy by optionally exposing a gRPC service on the local agent that serves Envoy’s xDS configuration API.

Consul can configure Envoy sidecars to proxy traffic over the following protocols:

On Consul 1.5.0 and older, Envoy proxies can only proxy TCP traffic at L4.

Some L7 features can be configured using configuration entries. You can add custom Envoy configurations to the proxy service definition to use Envoy features that are not currently exposed through configuration entries. Adding custom Envoy configurations to the service definition is an interim solution that enables you to use the more powerful features of Envoy.

Note: When using Envoy with Consul and not using the consul connect envoy command Envoy must be run with the --max-obj-name-len option set to 256 or greater for Envoy versions prior to 1.11.0.

Supported Versions

The following matrix describes Envoy compatibility for the currently supported n-2 major Consul releases. For previous Consul version compatibility please view the respective versioned docs for this page.

Envoy and Consul Client Agent

Consul supports four major Envoy releases at the beginning of each major Consul release. Consul maintains compatibility with Envoy patch releases for each major version so that users can benefit from bug and security fixes in Envoy. As a policy, Consul will add support for a new major versions of Envoy in a Consul major release. Support for newer versions of Envoy will not be added to existing releases.

Envoy and Consul Dataplane

The Consul dataplane component was introduced in Consul v1.14 as a way to manage Envoy proxies without the use of Consul clients. Each new minor version of Consul is released with a new minor version of Consul dataplane, which packages both Envoy and the consul-dataplane binary in a single container image. For backwards compatability reasons, each new minor version of Consul will also support the previous minor version of Consul dataplane to allow for seamless upgrades. In addition, each minor version of Consul will support the next minor version of Consul dataplane to allow for extended dataplane support via newer versions of Envoy.

Getting Started

To get started with Envoy and see a working example you can follow the Using Envoy with Consul service mesh tutorial.

Configuration

Envoy proxies require two types of configuration: an initial bootstrap configuration and a dynamic configuration that is discovered from a “management server”, in this case Consul.

The bootstrap configuration at a minimum needs to configure the proxy with an identity (node id) and the location of its local Consul agent from which it discovers all of its dynamic configuration. See Bootstrap Configuration for more details.

The dynamic configuration Consul service mesh provides to each Envoy instance includes:

• TLS certificates and keys to enable mutual authentication and keep certificates rotating.
• [Intentions] to enforce service-to-service authorization rules.
• Service-discovery results for upstreams to enable each sidecar proxy to load-balance outgoing connections.
• L7 configuration including timeouts and protocol-specific options.
• Configuration to expose specific HTTP paths.

For more information on the parts of the Envoy proxy runtime configuration that are currently controllable via Consul service mesh, refer to Dynamic Configuration.

We plan to enable more and more of Envoy’s features through Consul service mesh’s first-class configuration over time, however some advanced users will need additional control to configure Envoy in specific ways. To enable this, we provide several “escape hatch” options that allow users to provide low-level raw Envoy config syntax for some sub-components in each Envoy instance. This allows operators to have full control over and responsibility for correctly configuring Envoy and ensuring version support etc.

Intention Enforcement

Intentions are enforced using Envoy’s RBAC filters. Depending on the configured protocol of the proxied service, intentions are either enforced per-connection (L4) using a network filter, or per-request (L7) using an HTTP filter.

Note: Prior to Consul 1.9.0 intentions were exclusively enforced per-connection (L4) using an ext_authz network filter.

Fetching Certificates

Envoy will use the CONSUL_HTTP_TOKEN and CONSUL_HTTP_ADDR environment variables to contact Consul to fetch certificates if the following conditions are met:

• The CONSUL_HTTP_TOKEN environment variable contains a Consul ACL token.
• The Consul ACL token has the necessary permissions to read configuration for that service.

If TLS is enabled on Consul, you will also need to add the following environment variables prior to starting Envoy:

• CONSUL_CACERT
• CONSUL_CLIENT_CERT
• CONSUL_CLIENT_KEY
• CONSUL_HTTP_SSL

Bootstrap Configuration

Envoy requires an initial bootstrap configuration file. You can either create the file manually using the Consul command line or configure Consul Dataplane to generate the file.

Generate the bootstrap file on the Consul CLI

Connect to a local Consul client agent and run the consul connect envoy command to create the Envoy bootstrap configuration. The command either outputs the bootstrap configuration directly to stdout or generates the configuration and issues an exec command to the Envoy binary as a convenience wrapper. For more information about using exec to bootstrap Envoy, refer to Exec Security Details.

If you experience issues when bootstrapping Envoy proxies from the CLI, use the -enable-config-gen-logging flag to enable debug message logging. These logs can help you troubleshoot issues that occur during the bootstrapping process. For more information about available flags and parameters, refer to the consul connect envoy CLI reference.

Generate the bootstrap file from Consul Dataplane

Consul Dataplane automatically configures and manages an Envoy process. Consul Dataplane generates the Envoy bootstrap configuration file prior to starting Envoy. To configure how Consul Dataplane starts Envoy, refer to the Consul Dataplane CLI reference.

Control bootstrap configuration from proxy configuration

Consul service mesh can control some parts of the bootstrap configuration by specifying Envoy proxy configuration options.

Add the following configuration items to the global proxy-defaults configuration entry or override them directly in the proxy.config field of a proxy service definition. When connected to a Consul client agent, you can place the configuration in the proxy.config field of the sidecar_service block.

• envoy_statsd_url - A URL in the form udp://ip:port identifying a UDP StatsD listener that Envoy should deliver metrics to. For example, this may be udp://127.0.0.1:8125 if every host has a local StatsD listener. In this case users can configure this property once in the global proxy-defaults configuration entry for convenience. Currently, TCP is not supported.

  Note: currently the url must use an ip address not a dns name due to the way Envoy is setup for StatsD.

  Expansion of the environment variable HOST_IP is supported, e.g. udp://${HOST_IP}:8125.

  Users can also specify the whole parameter in the form $ENV_VAR_NAME, which will cause the consul connect envoy command to resolve the actual URL from the named environment variable when it runs. This, for example, allows each pod in a Kubernetes cluster to learn of a pod-specific IP address for StatsD when the Envoy instance is bootstrapped while still allowing global configuration of all proxies to use StatsD in the global proxy-defaults configuration entry. The env variable must contain a full valid URL value as specified above and nothing else.

• envoy_dogstatsd_url - The same as envoy_statsd_url with the following differences in behavior:

  • Envoy will use dogstatsd tags instead of statsd dot-separated metric names.
  • As well as udp://, a unix:// URL may be specified if your agent can listen on a unix socket (e.g. the dogstatsd agent).

• envoy_prometheus_bind_addr - Specifies that the proxy should expose a Prometheus metrics endpoint to the public network. It must be supplied in the form ip:port and port and the ip/port combination must be free within the network namespace the proxy runs. Typically the IP would be 0.0.0.0 to bind to all available interfaces or a pod IP address.

  Note: Envoy versions prior to 1.10 do not export timing histograms using the internal Prometheus endpoint.

• envoy_stats_bind_addr - Specifies that the proxy should expose the /stats prefix to the public network. It must be supplied in the form ip:port and the ip/port combination must be free within the network namespace the proxy runs. Typically the IP would be 0.0.0.0 to bind to all available interfaces or a pod IP address.

• envoy_stats_tags - Specifies one or more static tags that will be added to all metrics produced by the proxy.

• envoy_stats_flush_interval - Configures Envoy’s stats_flush_interval.

• envoy_telemetry_collector_bind_socket_dir - Specifies the directory where Envoy creates a Unix socket. Envoy sends metrics to the socket where a Consul telemetry collector can collect them. The socket is not configured by default.

The Advanced Configuration section describes additional configurations that allow incremental or complete control over the bootstrap configuration generated.

Bootstrap Envoy on Windows VMs

Complete the Connect Services on Windows Workloads to Consul Service Mesh tutorial to learn how to deploy Consul and use its service mesh on Windows VMs.

If you are running Consul on a Windows VM, attempting to bootstrap Envoy with the consul connect envoy command returns the following output:

Directly running Envoy is only supported on linux and macOS since envoy itself doesn't build on other plataforms currently.
Use the -bootstrap option to generate the JSON to use when running envoy on a supported OS or via a container or VM.

To bootstrap Envoy on Windows VMs, you must generate the bootstrap configuration as a .json file and then manually edit it to add both your ACL token and a valid access log path.

To generate the bootstrap configuration file, add the -bootstrap option to the command and then save the output to a file:

$ consul connect envoy -bootstrap > bootstrap.json

Then, open bootstrap.json and update the following sections with your ACL token and log path.

bootstrap.json

1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526  "admin": {
    "access_log_path": "/dev/null",
    "address": {
      "socket_address": {
        "address": "127.0.0.1",
        "port_value": 19000
      }
    }
  },
  ## ...
  "dynamic_resources": {
    ## ...
    "ads_config": {
      ## ...
      "grpc_services": {
        "initial_metadata": [
          {
            "key": "x-consul-token",
            "value": "<ACL-Token>"
          }
        ],
        ## ...
      }
    }
  }
}

To complete the bootstrap process, start Envoy and include the path to bootstrap.json:

$ envoy -c bootstrap.json

Security Note: The bootstrap JSON contains the ACL token and should be handled as a secret. Because this token authorizes the identity of any service it has service:write permissions for, it can be used to access upstream services.

Dynamic Configuration

Consul automatically generates Envoy’s dynamic configuration based on its knowledge of the cluster. Users may specify default configuration options for a service through the available fields in the service-defaults configuration entry. Consul will use this information to configure appropriate proxy settings for that service’s proxies and also for the upstream listeners used by the service.

One example is how users can define a service’s protocol in the Protocol field of service-defaults configuration entry. Agents with enable_central_service_config set to true will automatically discover the protocol when configuring a proxy for a service. The proxy will discover the main protocol of the service it represents and use this to configure its main public listener. It will also discover the protocols defined for any of its upstream services and automatically configure its upstream listeners appropriately too as below.

This automated discovery results in Consul auto-populating the proxy.config and proxy.upstreams[*].config fields of the proxy service definition that is actually registered.

To learn about other options that can be configured centrally see the Configuration Entries docs.

Proxy Config Options

These fields may also be overridden explicitly in proxy.config of the proxy service definition, or defined in the global proxy-defaults configuration entry to act as defaults that are inherited by all services.

• protocol - The protocol the service speaks. Consul service mesh’s Envoy integration currently supports the following protocol values:

  • tcp - Unless otherwise specified this is the default, which causes Envoy to proxy at L4. This provides all the security benefits of the service mesh’s mTLS and works for any TCP-based protocol. Load-balancing and metrics are available at the connection level.

  • http - This specifies that the service speaks HTTP/1.x. Envoy will setup an http_connection_manager and will be able to load-balance requests individually to available upstream services. Envoy will also emit L7 metrics such as request rates broken down by HTTP response code family (2xx, 4xx, 5xx, etc).

  • http2 - This specifies that the service speaks http2 (specifically h2c since Envoy will still only connect to the local service instance via plain TCP not TLS). This behaves much like http with L7 load-balancing and metrics but has additional settings that correctly enable end-to-end http2.

  • grpc - gRPC is a common RPC protocol based on http2. In addition to the http2 support above, Envoy listeners will be configured with a gRPC bridge filter that translates HTTP/1.1 calls into gRPC, and instruments metrics with gRPC-status trailer codes.

    Note: The protocol of a service should ideally be configured via the protocol field of a service-defaults config entry for the service. Configuring it in a proxy config will not fully enable some L7 features. It is supported here for backwards compatibility with Consul versions prior to 1.6.0.

• bind_address - Override the address Envoy’s public listener binds to. By default Envoy will bind to the service address or 0.0.0.0 if there is not explicit address on the service registration.

• bind_port - Override the port Envoy’s public listener binds to. By default Envoy will bind to the service port.

• local_connect_timeout_ms - The number of milliseconds allowed to make connections to the local application instance before timing out. Defaults to 5000 (5 seconds).

• local_request_timeout_ms - In milliseconds, the request timeout for HTTP requests to the local application instance. Applies to HTTP based protocols only. If not specified, inherits the Envoy default for route timeouts (15s). A value of 0 will disable request timeouts.

• local_idle_timeout_ms - In milliseconds, the idle timeout for HTTP requests to the local application instance. Applies to HTTP based protocols only. If not specified, inherits the Envoy default for route idle timeouts (15s). A value of 0 disables request timeouts.

• max_inbound_connections - The maximum number of concurrent inbound connections to the local application instance. If not specified, inherits the Envoy default (1024).

• balance_inbound_connections - The strategy used for balancing inbound connections across Envoy worker threads. Consul service mesh Envoy integration supports the following balance_inbound_connections values:

  • “” - Empty string (default). No connection balancing strategy is used. Consul does not balance inbound connections.
  • exact_balance - Inbound connections to the service use the Envoy Exact Balance Strategy.

Proxy Upstream Config Options

The following configuration items may be overridden directly in the proxy.upstreams[].config field of a proxy service definition or sidecar_service block.

• protocol - Same as above in main config but affects the listener setup for the upstream.

  Note: The protocol of a service should ideally be configured via the protocol field of a service-defaults config entry for the upstream destination service. Configuring it in a proxy upstream config will not fully enable some L7 features. It is supported here for backwards compatibility with Consul versions prior to 1.6.0.

• connect_timeout_ms - The number of milliseconds to allow when making upstream connections before timing out. Defaults to 5000 (5 seconds).

  Note: The connection timeout for a service should ideally be configured via the connect_timeout field of a service-resolver config entry for the upstream destination service. Configuring it in a proxy upstream config will override any values defined in config entries. It is supported here for backwards compatibility with Consul versions prior to 1.6.0.

• limits - A set of limits to apply when connecting to the upstream service. These limits are applied on a per-service-instance basis. The following limits are respected:

  • max_connections - The maximum number of connections a service instance will be allowed to establish against the given upstream. Use this to limit HTTP/1.1 traffic, since HTTP/1.1 has a request per connection.
  • max_pending_requests - The maximum number of requests that will be queued while waiting for a connection to be established. For this configuration to be respected, a L7 protocol must be defined in the protocol field.
  • max_concurrent_requests - The maximum number of concurrent requests that will be allowed at a single point in time. Use this to limit HTTP/2 traffic, since HTTP/2 has many requests per connection. For this configuration to be respected, a L7 protocol must be defined in the protocol field.

• passive_health_check - Passive health checks remove hosts from the upstream cluster that are unreachable or that return errors.

  • interval - The time in nanosecond between checks. Each check will cause hosts which have exceeded max_failures to be removed from the load balancer, and any hosts which have passed their ejection time to be returned to the load balancer. If not specified, it uses the default value. For example, 10s for Envoy proxy.
  • max_failures - The number of consecutive failures that cause a host to be removed from the upstream cluster. If not specified, Consul uses the proxy’s default value. For example, 5 for Envoy proxy.
  • enforcing_consecutive_5xx - A percentage representing the chance that a host will be actually ejected when the proxy detects an outlier status through consecutive errors in the 500 code range. If not specified, Consul uses the proxy’s default value. For example, 100 for Envoy proxy.
  • max_ejection_percent - The maximum percentage of hosts that can be ejected from a upstream cluster due to passive health check failures. If not specified, inherits Envoy’s default of 10% or at least one host.
  • base_ejection_time - The base time that a host is ejected for. The real time is equal to the base time multiplied by the number of times the host has been ejected and is capped by max_ejection_time (Default 300s). If not speficied, inherits Envoy’s default value of 30s.

• balance_outbound_connections - Specifies the strategy for balancing outbound connections across Envoy worker threads. Consul service mesh Envoy integration supports the following balance_outbound_connections values:

  • “” - Empty string (default). No connection balancing strategy is used. Consul does not balance outbound connections.
  • exact_balance - Outbound connections from the upstream use the Envoy Exact Balance Strategy.

Gateway Options

These fields may also be overridden explicitly in the proxy service definition, or defined in the global proxy-defaults configuration entry to act as defaults that are inherited by all services.

Prior to 1.8.0 these settings were specific to Mesh Gateways. The deprecated names such as envoy_mesh_gateway_bind_addresses and envoy_mesh_gateway_no_default_bind will continue to be supported.

• connect_timeout_ms - The number of milliseconds to allow when making upstream connections before timing out. Defaults to 5000 (5 seconds). If the upstream service has the configuration option connect_timeout_ms set for the service-resolver, that timeout value will take precedence over this gateway option.

• envoy_gateway_bind_tagged_addresses - Indicates that the gateway services tagged addresses should be bound to listeners in addition to the default listener address.

• envoy_gateway_bind_addresses - A map of additional addresses to be bound. This map’s keys are the name of the listeners to be created and the values are a map with two keys, address and port, that combined make the address to bind the listener to. These are bound in addition to the default address.

• envoy_gateway_no_default_bind - Prevents binding to the default address of the gateway service. This should be used with one of the other options to configure the gateway’s bind addresses.

• envoy_dns_discovery_type - Determines how Envoy will resolve hostnames. Defaults to LOGICAL_DNS. Must be one of STRICT_DNS or LOGICAL_DNS. Details for each type are available in the Envoy documentation. This option applies to terminating gateways that route to services addressed by a hostname, such as a managed database. It also applies to mesh gateways, such as when gateways in other Consul datacenters are behind a load balancer that is addressed by a hostname.

• envoy_gateway_remote_tcp_enable_keepalive - Enables TCP keepalive settings on remote upstream connections for mesh and terminating gateways. Defaults to false. Must be one of true or false. Details for this feature are available in the Envoy documentation.

• envoy_gateway_remote_tcp_keepalive_time - The number of seconds a connection needs to be idle before keep-alive probes start being sent. For more information, see the Envoy documentation. This option only applies to remote upstream connections for mesh and terminating gateways.

• envoy_gateway_remote_tcp_keepalive_interval - The number of seconds between keep-alive probes. For more information, see the Envoy documentation. This option only applies to remote upstream connections for mesh and terminating gateways.

• envoy_gateway_remote_tcp_keepalive_probes - Maximum number of keepalive probes to send without response before deciding the connection is dead. For more information, see the Envoy documentation. This option only applies to remote upstream connections for mesh and terminating gateways.

Advanced Configuration

To support more flexibility when configuring Envoy, several “lower-level” options exist that require knowledge of Envoy’s configuration format. Many options allow configuring a subsection of either the bootstrap or dynamic configuration using your own custom protobuf config.

We separate these into two sets, Advanced Bootstrap Options and Escape Hatch Overrides. Both require writing Envoy config in the protobuf JSON encoding. Advanced options cover smaller chunks that might commonly need to be set for tasks like configuring tracing. In contrast, escape hatches give almost complete control over the proxy setup, but require operators to manually code the entire configuration in protobuf JSON.

Advanced Topic! This section covers options that allow users to take almost complete control of Envoy’s configuration. We provide these options so users can experiment or take advantage of features not yet fully supported in Consul service mesh. We plan to retain this ability in the future, but it should still be considered experimental because it requires in-depth knowledge of Envoy’s configuration format. Users should consider Envoy version compatibility when using these features because they can configure Envoy in ways that are outside of Consul’s control. Incorrect configuration could prevent all proxies in your mesh from functioning correctly, or bypass the security guarantees Consul service mesh is designed to enforce.

Configuration Formatting

All configurations are specified as strings containing the serialized proto3 JSON encoding of the specified Envoy configuration type. They are full JSON types except where noted.

The JSON supplied may describe a protobuf types.Any message with an @type field set to the appropriate type (for example type.googleapis.com/envoy.config.listener.v3.Listener).

For example, given a tracing config:

Example envoy_tracing_json configuration

{
  "http": {
    "name": "envoy.tracers.zipkin",
    "typedConfig": {
      "@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
      "collector_cluster": "zipkin",
      "collector_endpoint_version": "HTTP_JSON",
      "collector_endpoint": "/api/v1/spans",
      "shared_span_context": false
    }
  }
}

JSON escape the value of tracing into a string, for example using https://codebeautify.org/json-escape-unescape, or using jq.

$ cat <<EOF | jq '. | @json'
{
  "http": {
    "name": "envoy.tracers.zipkin",
    "typedConfig": {
      "@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
      "collector_cluster": "zipkin",
      "collector_endpoint_version": "HTTP_JSON",
      "collector_endpoint": "/api/v1/spans",
      "shared_span_context": false
    }
  }
}
EOF
"{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"zipkin\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v1/spans\",\"shared_span_context\":false}}}"

Then use that as the value for envoy_tracing_json:

{
  "kind": "proxy-defaults",
  "name": "global",
  "config": {
    "envoy_tracing_json": "{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"zipkin\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v1/spans\",\"shared_span_context\":false}}}\n"
  }
}

If using HCL, this escaping is done automatically:

Kind = "proxy-defaults"
Name = "global"
Config {
  envoy_tracing_json = <<EOF
{
  "http": {
    "name": "envoy.tracers.zipkin",
    "typedConfig": {
      "@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
      "collector_cluster": "zipkin",
      "collector_endpoint_version": "HTTP_JSON",
      "collector_endpoint": "/api/v1/spans",
      "shared_span_context": false
    }
  }
}
EOF
}

Advanced Bootstrap Options

Users may add the following configuration items to the global proxy-defaults configuration entry or override them directly in the proxy.config field of a proxy service definition or sidecar_service block.

• envoy_extra_static_clusters_json - Specifies one or more Envoy clusters that will be appended to the array of static clusters in the bootstrap config. This enables you to add custom clusters for tracing sinks, for example. In order to configure a single cluster, specify a single JSON object with the cluster details. For multiple clusters, specify objects in a comma-separated list with no trailing comma. The cluster objects will be interpolated directly into a JSON array.

  Example envoy_extra_static_clusters_json

  {
    "name": "local-service-cluster",
    "load_assignment": {
      "cluster_name": "local-service-cluster",
      "endpoints": [
        {
          "lb_endpoints": [
            {
              "endpoint": {
                "address": {
                  "socket_address": {
                    "address": "127.0.0.1",
                    "port_value": 32769
                  }
                }
              }
            }
          ]
        }
      ]
    }
  }

• envoy_extra_static_listeners_json - Similar to envoy_extra_static_clusters_json but appends one or more Envoy listeners to the array of static listener definitions. Can be used to setup limited access that bypasses the service mesh’s mTLS or authorization for health checks or metrics.

  Example envoy_extra_static_listeners_json

  {
    "name": "test_envoy_mtls_bypass_listener",
    "address": {
      "socket_address": {
        "address": "0.0.0.0",
        "port_value": 20201
      }
    },
    "filter_chains": [
      {
        "filters": [
          {
            "name": "envoy.filters.network.http_connection_manager",
            "typedConfig": {
              "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
              "http_filters": [
                {
                  "name": "envoy.filters.http.router"
                }
              ],
              "route_config": {
                "name": "self_admin_route",
                "virtual_hosts": [
                  {
                    "name": "self_admin",
                    "domains": [
                      "*"
                    ],
                    "routes": [
                      {
                        "match": {
                          "path": "/"
                        },
                        "route": {
                          "cluster": "local-service-cluster"
                        }
                      }
                    ]
                  }
                ]
              },
              "stat_prefix": "envoy_mtls_bypass",
              "tracing": {
                "random_sampling": {}
              }
            }
          }
        ]
      }
    ]
  }

• envoy_extra_stats_sinks_json - Similar to envoy_extra_static_clusters_json but for stats sinks. These are appended to any sinks defined by use of the higher-level envoy_statsd_url or envoy_dogstatsd_url config options.

  Example envoy_extra_stats_sinks_json

  {
    "name": "envoy.stat_sinks.dog_statsd",
    "typed_config": {
      "@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
      "address": {
        "socket_address": {
          "protocol": "UDP",
          "port_value": 8125,
          "address": "172.31.20.6"
        }
      }
    }
  }

• envoy_stats_config_json - The entire stats config. If provided this will override the higher-level envoy_stats_tags. It allows full control over dynamic tag replacements etc.

  Example envoy_stats_config_json

  {
    "stats_matcher": {
      "reject_all": true
    },
    "stats_tags": [
      {
        "tag_name": "envoy.http_user_agent",
        "regex": "^http(?=\\.).*?\\.user_agent\\.((.+?)\\.)\\w+?$"
      }
    ],
    "use_all_default_tags": false
  }

• envoy_tracing_json - The entire tracing config. Most tracing providers will also require adding static clusters to define the endpoints to send tracing data to.

  Example envoy_tracing_json

  {
    "http": {
      "name": "envoy.tracers.zipkin",
      "typedConfig": {
        "@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
        "collector_cluster": "zipkin",
        "collector_endpoint_version": "HTTP_JSON",
        "collector_endpoint": "/api/v1/spans",
        "shared_span_context": false
      }
    }
  }

Escape-Hatch Overrides

Users may add the following configuration items to the global proxy-defaults configuration entry or override them directly in the proxy.config field of a proxy service definition or sidecar_service block.

• envoy_bootstrap_json_tpl - Specifies a template in Go template syntax that is used in place of the default template when generating bootstrap via consul connect envoy command. The variables that are available to be interpolated are documented here. This offers complete control of the proxy’s bootstrap although major deviations from the default template may break Consul’s ability to correctly manage the proxy or enforce its security model.

• envoy_public_listener_json - Specifies a complete Envoy listener to be delivered in place of the main public listener that the proxy used to accept inbound connections. This will be used verbatim with the following exceptions:

  • Every FilterChain added to the listener will have its TlsContext overridden by the Connect TLS certificates and validation context. This means there is no way to override the service mesh’s mutual TLS for the public listener.
  • Every FilterChain will have the envoy.filters.{network|http}.rbac filter prepended to the filters array to ensure that all inbound connections are authorized by the service mesh. Before Consul 1.9.0 envoy.ext_authz was inserted instead.

  Example envoy_public_listener_json

  Example envoy_public_listener_json

  {
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "public_listener",
    "address": {
      "socket_address": {
        "address": "127.0.0.1",
        "port_value": 21002
      }
    },
    "filter_chains": [
      {
        "filters": [
          {
            "name": "envoy.filters.network.http_connection_manager",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
              "stat_prefix": "ingress_http",
              "http_filters": [
                {
                  "name": "envoy.filters.http.router"
                }
              ],
              "route_config": {
                "name": "local_route",
                "virtual_hosts": [
                  {
                    "name": "local_service",
                    "domains": ["*"],
                    "routes": [
                      {
                        "match": {
                          "prefix": "/"
                        },
                        "route": {
                          "cluster": "local-service-cluster",
                        }
                      }
                    ]
                  }
                ]
              }
            }
          }
        ]
      }
    ],
   "traffic_direction": "INBOUND"
  }

  {
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "public_listener",
    "address": {
      "socket_address": {
        "address": "127.0.0.1",
        "port_value": 21002
      }
    },
    "filter_chains": [
      {
        "filters": [
          {
            "name": "envoy.filters.network.tcp_proxy",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
              "stat_prefix": "ingress_tcp",
              "cluster": "local-service-cluster"
            }
          }
        ]
      }
    ],
    "traffic_direction": "INBOUND"
  }

• envoy_listener_tracing_json - Specifies a tracing configuration to be inserted in the proxy’s public and upstreams listeners.

  Example envoy_listener_tracing_json

  {
    "@type" : "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.Tracing",
    "provider" : {
      "name" : "envoy.tracers.zipkin",
      "typed_config" : {
        "@type" : "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
        "collector_cluster" : "otelcolector",
        "collector_endpoint" : "/api/v2/spans",
        "collector_endpoint_version" : "HTTP_JSON",
        "shared_span_context" : false
      }
    },
    "custom_tags" : [
      {
        "tag" : "custom_header",
        "request_header" : {
          "name" : "x-custom-traceid",
          "default_value" : ""
        }
      },
      {
        "tag" : "alloc_id",
        "environment" : {
          "name" : "NOMAD_ALLOC_ID"
        }
      }
    ]
  }

• envoy_local_cluster_json - Specifies a complete Envoy cluster to be delivered in place of the local application cluster. This allows customization of timeouts, rate limits, load balancing strategy etc.

  Example envoy_local_cluster_json

  {
    "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
    "name": "local_app",
    "type": "STATIC",
    "connect_timeout": "5s",
    "circuit_breakers": {
      "thresholds": [
        {
          "priority": "DEFAULT",
          "max_connections": 2048
        }
      ]
    },
    "load_assignment": {
      "cluster_name": "local_app",
      "endpoints": [
        {
          "lb_endpoints": [
            {
              "endpoint": {
                "address": {
                  "socket_address": {
                    "address": "127.0.0.1",
                    "port_value": 8080
                  }
                }
              }
            }
          ]
        }
      ]
    }
  }

The following configuration items may be overridden directly in the proxy.upstreams[].config field of a proxy service definition or sidecar_service block.

Note: - When a service-router, service-splitter, or service-resolver config entry exists for a service the below escape hatches are ignored and will log a warning.

• envoy_listener_json - Specifies a complete Listener to be delivered in place of the upstream listener that the proxy exposes to the application for outbound connections. This will be used verbatim with the following exceptions:

  • Every FilterChain added to the listener will have its TlsContext overridden by the service mesh TLS certificates and validation context. This means there is no way to override the service mesh’s mutual TLS for the public listener.

  Example upstream envoy_listener_json

  Example upstream envoy_listener_json

  {
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "example-service",
    "address": {
      "socket_address": {
        "address": "0.0.0.0",
        "port_value": 14000
      }
    },
    "filter_chains": [
      {
        "filters": [
          {
            "name": "envoy.filters.network.http_connection_manager",
            "typedConfig": {
              "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
              "access_log": [
                {
                  "name": "envoy.access_loggers.file",
                  "typedConfig": {
                    "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
                    "path": "/var/log/envoy-access/example-service.log"
                  }
                }
              ],
              "http_filters": [
                {
                  "name": "envoy.filters.http.router"
                }
              ],
              "route_config": {
                "name": "example-service",
                "virtual_hosts": [
                  {
                    "name": "example-service",
                    "domains": [
                      "*"
                    ],
                    "routes": [
                      {
                        "match": {
                          "prefix": "/"
                        },
                        "route": {
                          "cluster": "example-service",
                          "timeout": "90s",
                          "retry_policy": {
                            "retry_on": "5xx,connect-failure",
                            "num_retries": 2,
                            "per_try_timeout": "60s"
                          }
                        }
                      }
                    ]
                  }
                ]
              },
              "stat_prefix": "example-service",
              "tracing": {
                "random_sampling": {}
              }
            }
          }
        ]
      }
    ],
    "traffic_direction": "OUTBOUND"
  }

• envoy_cluster_json - Specifies a complete Envoy cluster to be delivered in place of the discovered upstream cluster. This allows customization of timeouts, circuit breaking, rate limits, load balancing strategy etc.

  Example upstream envoy_cluster_json

  Example upstream envoy_cluster_json

  {
    "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
    "name": "example-service",
    "type": "EDS",
    "eds_cluster_config": {
      "eds_config": {
        "ads": {}
      }
    },
    "connect_timeout": "90s",
    "lb_policy": "ROUND_ROBIN",
    "circuit_breakers": {
      "thresholds": [
        {
          "priority": "DEFAULT",
          "max_connections": 1024,
          "max_pending_requests": 1024,
          "max_requests": 1024,
          "max_retries": 3
        }
      ]
    }
  }