Bootstrap

This documentation is for the Envoy v3 API.

As of Envoy v1.18 the v2 API has been removed and is no longer supported.

If you are upgrading from v2 API config you may wish to view the v2 API documentation:

config/bootstrap/v2/bootstrap.proto

This proto is supplied via the -c CLI flag and acts as the root of the Envoy v3 configuration. See the v3 configuration overview for more detail.

config.bootstrap.v3.Bootstrap

[config.bootstrap.v3.Bootstrap proto]

Bootstrap configuration overview.

  1. {
  2.   "node": "{...}",
  3.   "static_resources": "{...}",
  4.   "dynamic_resources": "{...}",
  5.   "cluster_manager": "{...}",
  6.   "hds_config": "{...}",
  7.   "flags_path": "...",
  8.   "stats_sinks": [],
  9.   "stats_config": "{...}",
  10.   "stats_flush_interval": "{...}",
  11.   "stats_flush_on_admin": "...",
  12.   "watchdog": "{...}",
  13.   "watchdogs": "{...}",
  14.   "tracing": "{...}",
  15.   "layered_runtime": "{...}",
  16.   "admin": "{...}",
  17.   "overload_manager": "{...}",
  18.   "enable_dispatcher_stats": "...",
  19.   "header_prefix": "...",
  20.   "stats_server_version_override": "{...}",
  21.   "use_tcp_for_dns_lookups": "...",
  22.   "dns_resolution_config": "{...}",
  23.   "typed_dns_resolver_config": "{...}",
  24.   "bootstrap_extensions": [],
  25.   "fatal_actions": [],
  26.   "default_socket_interface": "...",
  27.   "inline_headers": [],
  28.   "perf_tracing_file_path": "..."
  29. }

node

(config.core.v3.Node) Node identity to present to the management server and for instance identification purposes (e.g. in generated headers).

static_resources

(config.bootstrap.v3.Bootstrap.StaticResources) Statically specified resources.

dynamic_resources

(config.bootstrap.v3.Bootstrap.DynamicResources) xDS configuration sources.

cluster_manager

(config.bootstrap.v3.ClusterManager) Configuration for the cluster manager which owns all upstream clusters within the server.

hds_config

(config.core.v3.ApiConfigSource) Health discovery service config option. (core.ApiConfigSource)

flags_path

(string) Optional file system path to search for startup flag files.

stats_sinks

(repeated config.metrics.v3.StatsSink) Optional set of stats sinks.

stats_config

(config.metrics.v3.StatsConfig) Configuration for internal processing of stats.

stats_flush_interval

(Duration) Optional duration between flushes to configured stats sinks. For performance reasons Envoy latches counters and only flushes counters and gauges at a periodic interval. If not specified the default is 5000ms (5 seconds). Only one of stats_flush_interval or stats_flush_on_admin can be set. Duration must be at least 1ms and at most 5 min.

stats_flush_on_admin

(bool) Flush stats to sinks only when queried for on the admin interface. If set, a flush timer is not created. Only one of stats_flush_on_admin or stats_flush_interval can be set.

watchdog

(config.bootstrap.v3.Watchdog) Optional watchdog configuration. This is for a single watchdog configuration for the entire system. Deprecated in favor of watchdogs which has finer granularity.

watchdogs

(config.bootstrap.v3.Watchdogs) Optional watchdogs configuration. This is used for specifying different watchdogs for the different subsystems.

Tip

This extension category has the following known extensions:

tracing

(config.trace.v3.Tracing) Configuration for an external tracing provider.

Attention

This field has been deprecated in favor of HttpConnectionManager.Tracing.provider.

layered_runtime

(config.bootstrap.v3.LayeredRuntime) Configuration for the runtime configuration provider. If not specified, a “null” provider will be used which will result in all defaults being used.

admin

(config.bootstrap.v3.Admin) Configuration for the local administration HTTP server.

overload_manager

(config.overload.v3.OverloadManager) Optional overload manager configuration.

Attention

This field should be configured in the presence of untrusted downstreams.

This field should be configured in the presence of untrusted upstreams.

Example configuration for untrusted environments:

  1. overload_manager:
  2.   actions:
  3.   - name: envoy.overload_actions.shrink_heap
  4.     triggers:
  5.     - name: envoy.resource_monitors.fixed_heap
  6.       threshold:
  7.         value: 0.9
  8.   - name: envoy.overload_actions.stop_accepting_requests
  9.     triggers:
  10.     - name: envoy.resource_monitors.fixed_heap
  11.       threshold:
  12.         value: 0.95
  13.   refresh_interval: 0.25s
  14.   resource_monitors:
  15.   - name: envoy.resource_monitors.fixed_heap
  16.     typed_config:
  17.       '@type': type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
  18.       max_heap_size_bytes: 1073741824.0

enable_dispatcher_stats

(bool) Enable stats for event dispatcher, defaults to false. Note that this records a value for each iteration of the event loop on every thread. This should normally be minimal overhead, but when using statsd, it will send each observed value over the wire individually because the statsd protocol doesn’t have any way to represent a histogram summary. Be aware that this can be a very large volume of data.

header_prefix

(string) Optional string which will be used in lieu of x-envoy in prefixing headers.

For example, if this string is present and set to X-Foo, then x-envoy-retry-on will be transformed into x-foo-retry-on etc.

Note this applies to the headers Envoy will generate, the headers Envoy will sanitize, and the headers Envoy will trust for core code and core extensions only. Be VERY careful making changes to this string, especially in multi-layer Envoy deployments or deployments using extensions which are not upstream.

stats_server_version_override

(UInt64Value) Optional proxy version which will be used to set the value of server.version statistic if specified. Envoy will not process this value, it will be sent as is to stats sinks.

use_tcp_for_dns_lookups

(bool) Always use TCP queries instead of UDP queries for DNS lookups. This may be overridden on a per-cluster basis in cds_config, when dns_resolvers and use_tcp_for_dns_lookups are specified. This field is deprecated in favor of dns_resolution_config which aggregates all of the DNS resolver configuration in a single message.

dns_resolution_config

(config.core.v3.DnsResolutionConfig) DNS resolution configuration which includes the underlying dns resolver addresses and options. This may be overridden on a per-cluster basis in cds_config, when dns_resolution_config is specified. This field is deprecated in favor of typed_dns_resolver_config.

typed_dns_resolver_config

(config.core.v3.TypedExtensionConfig) DNS resolver type configuration extension. This extension can be used to configure c-ares, apple, or any other DNS resolver types and the related parameters. For example, an object of CaresDnsResolverConfig can be packed into this typed_dns_resolver_config. This configuration replaces the dns_resolution_config configuration. During the transition period when both dns_resolution_config and typed_dns_resolver_config exists, when typed_dns_resolver_config is in place, Envoy will use it and ignore dns_resolution_config. When typed_dns_resolver_config is missing, the default behavior is in place.

Tip

This extension category has the following known extensions:

bootstrap_extensions

(repeated config.core.v3.TypedExtensionConfig) Specifies optional bootstrap extensions to be instantiated at startup time. Each item contains extension specific configuration.

Tip

This extension category has the following known extensions:

The following extensions are available in contrib images only:

fatal_actions

(repeated config.bootstrap.v3.FatalAction) Specifies optional extensions instantiated at startup time and invoked during crash time on the request that caused the crash.

default_socket_interface

(string) Optional overriding of default socket interface. The value must be the name of one of the socket interface factories initialized through a bootstrap extension

inline_headers

(repeated config.bootstrap.v3.CustomInlineHeader) Specifies a set of headers that need to be registered as inline header. This configuration allows users to customize the inline headers on-demand at Envoy startup without modifying Envoy’s source code.

Note that the ‘set-cookie’ header cannot be registered as inline header.

perf_tracing_file_path

(string) Optional path to a file with performance tracing data created by “Perfetto” SDK in binary ProtoBuf format. The default value is “envoy.pftrace”.

config.bootstrap.v3.Bootstrap.StaticResources

[config.bootstrap.v3.Bootstrap.StaticResources proto]

  1. {
  2.   "listeners": [],
  3.   "clusters": [],
  4.   "secrets": []
  5. }

listeners

(repeated config.listener.v3.Listener) Static Listeners. These listeners are available regardless of LDS configuration.

clusters

(repeated config.cluster.v3.Cluster) If a network based configuration source is specified for cds_config, it’s necessary to have some initial cluster definitions available to allow Envoy to know how to speak to the management server. These cluster definitions may not use EDS (i.e. they should be static IP or DNS-based).

secrets

(repeated extensions.transport_sockets.tls.v3.Secret) These static secrets can be used by SdsSecretConfig

config.bootstrap.v3.Bootstrap.DynamicResources

[config.bootstrap.v3.Bootstrap.DynamicResources proto]

  1. {
  2.   "lds_config": "{...}",
  3.   "cds_config": "{...}",
  4.   "ads_config": "{...}"
  5. }

lds_config

(config.core.v3.ConfigSource) All Listeners are provided by a single LDS configuration source.

cds_config

(config.core.v3.ConfigSource) All post-bootstrap Cluster definitions are provided by a single CDS configuration source.

ads_config

(config.core.v3.ApiConfigSource) A single ADS source may be optionally specified. This must have api_type GRPC. Only ConfigSources that have the ads field set will be streamed on the ADS channel.

config.bootstrap.v3.Admin

[config.bootstrap.v3.Admin proto]

Administration interface operations documentation.

  1. {
  2.   "access_log": [],
  3.   "access_log_path": "...",
  4.   "profile_path": "...",
  5.   "address": "{...}",
  6.   "socket_options": [],
  7.   "ignore_global_conn_limit": "..."
  8. }

access_log

(repeated config.accesslog.v3.AccessLog) Configuration for access logs emitted by the administration server.

access_log_path

(string) The path to write the access log for the administration server. If no access log is desired specify ‘/dev/null’. This is only required if address is set. Deprecated in favor of access_log which offers more options.

profile_path

(string) The cpu profiler output path for the administration server. If no profile path is specified, the default is ‘/var/log/envoy/envoy.prof’.

address

(config.core.v3.Address) The TCP address that the administration server will listen on. If not specified, Envoy will not start an administration server.

socket_options

(repeated config.core.v3.SocketOption) Additional socket options that may not be present in Envoy source code or precompiled binaries.

ignore_global_conn_limit

(bool) Indicates whether global_downstream_max_connections should apply to the admin interface or not.

config.bootstrap.v3.ClusterManager

[config.bootstrap.v3.ClusterManager proto]

Cluster manager architecture overview.

  1. {
  2.   "local_cluster_name": "...",
  3.   "outlier_detection": "{...}",
  4.   "upstream_bind_config": "{...}",
  5.   "load_stats_config": "{...}"
  6. }

local_cluster_name

(string) Name of the local cluster (i.e., the cluster that owns the Envoy running this configuration). In order to enable zone aware routing this option must be set. If local_cluster_name is defined then clusters must be defined in the Bootstrap static cluster resources. This is unrelated to the --service-cluster option which does not affect zone aware routing.

outlier_detection

(config.bootstrap.v3.ClusterManager.OutlierDetection) Optional global configuration for outlier detection.

upstream_bind_config

(config.core.v3.BindConfig) Optional configuration used to bind newly established upstream connections. This may be overridden on a per-cluster basis by upstream_bind_config in the cds_config.

load_stats_config

(config.core.v3.ApiConfigSource) A management server endpoint to stream load stats to via StreamLoadStats. This must have api_type GRPC.

config.bootstrap.v3.ClusterManager.OutlierDetection

[config.bootstrap.v3.ClusterManager.OutlierDetection proto]

  1. {
  2.   "event_log_path": "..."
  3. }

event_log_path

(string) Specifies the path to the outlier event log.

config.bootstrap.v3.Watchdogs

[config.bootstrap.v3.Watchdogs proto]

Allows you to specify different watchdog configs for different subsystems. This allows finer tuned policies for the watchdog. If a subsystem is omitted the default values for that system will be used.

  1. {
  2.   "main_thread_watchdog": "{...}",
  3.   "worker_watchdog": "{...}"
  4. }

main_thread_watchdog

(config.bootstrap.v3.Watchdog) Watchdog for the main thread.

worker_watchdog

(config.bootstrap.v3.Watchdog) Watchdog for the worker threads.

config.bootstrap.v3.Watchdog

[config.bootstrap.v3.Watchdog proto]

Envoy process watchdog configuration. When configured, this monitors for nonresponsive threads and kills the process after the configured thresholds. See the watchdog documentation for more information.

  1. {
  2.   "actions": [],
  3.   "miss_timeout": "{...}",
  4.   "megamiss_timeout": "{...}",
  5.   "kill_timeout": "{...}",
  6.   "max_kill_timeout_jitter": "{...}",
  7.   "multikill_timeout": "{...}",
  8.   "multikill_threshold": "{...}"
  9. }

actions

(repeated config.bootstrap.v3.Watchdog.WatchdogAction) Register actions that will fire on given WatchDog events. See WatchDogAction for priority of events.

miss_timeout

(Duration) The duration after which Envoy counts a nonresponsive thread in the watchdog_miss statistic. If not specified the default is 200ms.

megamiss_timeout

(Duration) The duration after which Envoy counts a nonresponsive thread in the watchdog_mega_miss statistic. If not specified the default is 1000ms.

kill_timeout

(Duration) If a watched thread has been nonresponsive for this duration, assume a programming error and kill the entire Envoy process. Set to 0 to disable kill behavior. If not specified the default is 0 (disabled).

max_kill_timeout_jitter

(Duration) Defines the maximum jitter used to adjust the kill_timeout if kill_timeout is enabled. Enabling this feature would help to reduce risk of synchronized watchdog kill events across proxies due to external triggers. Set to 0 to disable. If not specified the default is 0 (disabled).

multikill_timeout

(Duration) If max(2, ceil(registered_threads * Fraction(multikill_threshold))) threads have been nonresponsive for at least this duration kill the entire Envoy process. Set to 0 to disable this behavior. If not specified the default is 0 (disabled).

multikill_threshold

(type.v3.Percent) Sets the threshold for multikill_timeout in terms of the percentage of nonresponsive threads required for the multikill_timeout. If not specified the default is 0.

config.bootstrap.v3.Watchdog.WatchdogAction

[config.bootstrap.v3.Watchdog.WatchdogAction proto]

  1. {
  2.   "config": "{...}",
  3.   "event": "..."
  4. }

config

(config.core.v3.TypedExtensionConfig) Extension specific configuration for the action.

event

(config.bootstrap.v3.Watchdog.WatchdogAction.WatchdogEvent)

Enum config.bootstrap.v3.Watchdog.WatchdogAction.WatchdogEvent

[config.bootstrap.v3.Watchdog.WatchdogAction.WatchdogEvent proto]

The events are fired in this order: KILL, MULTIKILL, MEGAMISS, MISS. Within an event type, actions execute in the order they are configured. For KILL/MULTIKILL there is a default PANIC that will run after the registered actions and kills the process if it wasn’t already killed. It might be useful to specify several debug actions, and possibly an alternate FATAL action.

UNKNOWN

(DEFAULT) ⁣

KILL

⁣

MULTIKILL

⁣

MEGAMISS

⁣

MISS

⁣

config.bootstrap.v3.FatalAction

[config.bootstrap.v3.FatalAction proto]

Fatal actions to run while crashing. Actions can be safe (meaning they are async-signal safe) or unsafe. We run all safe actions before we run unsafe actions. If using an unsafe action that could get stuck or deadlock, it important to have an out of band system to terminate the process.

The interface for the extension is Envoy::Server::Configuration::FatalAction. FatalAction extensions live in the envoy.extensions.fatal_actions API namespace.

  1. {
  2.   "config": "{...}"
  3. }

config

(config.core.v3.TypedExtensionConfig) Extension specific configuration for the action. It’s expected to conform to the Envoy::Server::Configuration::FatalAction interface.

config.bootstrap.v3.Runtime

[config.bootstrap.v3.Runtime proto]

Runtime configuration overview (deprecated).

  1. {
  2.   "symlink_root": "...",
  3.   "subdirectory": "...",
  4.   "override_subdirectory": "...",
  5.   "base": "{...}"
  6. }

symlink_root

(string) The implementation assumes that the file system tree is accessed via a symbolic link. An atomic link swap is used when a new tree should be switched to. This parameter specifies the path to the symbolic link. Envoy will watch the location for changes and reload the file system tree when they happen. If this parameter is not set, there will be no disk based runtime.

subdirectory

(string) Specifies the subdirectory to load within the root directory. This is useful if multiple systems share the same delivery mechanism. Envoy configuration elements can be contained in a dedicated subdirectory.

override_subdirectory

(string) Specifies an optional subdirectory to load within the root directory. If specified and the directory exists, configuration values within this directory will override those found in the primary subdirectory. This is useful when Envoy is deployed across many different types of servers. Sometimes it is useful to have a per service cluster directory for runtime configuration. See below for exactly how the override directory is used.

base

(Struct) Static base runtime. This will be overridden by other runtime layers, e.g. disk or admin. This follows the runtime protobuf JSON representation encoding.

config.bootstrap.v3.RuntimeLayer

[config.bootstrap.v3.RuntimeLayer proto]

  1. {
  2.   "name": "...",
  3.   "static_layer": "{...}",
  4.   "disk_layer": "{...}",
  5.   "admin_layer": "{...}",
  6.   "rtds_layer": "{...}"
  7. }

name

(string, REQUIRED) Descriptive name for the runtime layer. This is only used for the runtime GET /runtime output.

static_layer

(Struct) Static runtime layer. This follows the runtime protobuf JSON representation encoding. Unlike static xDS resources, this static layer is overridable by later layers in the runtime virtual filesystem.

Precisely one of static_layer, disk_layer, admin_layer, rtds_layer must be set.

disk_layer

(config.bootstrap.v3.RuntimeLayer.DiskLayer)

Precisely one of static_layer, disk_layer, admin_layer, rtds_layer must be set.

admin_layer

(config.bootstrap.v3.RuntimeLayer.AdminLayer)

Precisely one of static_layer, disk_layer, admin_layer, rtds_layer must be set.

rtds_layer

(config.bootstrap.v3.RuntimeLayer.RtdsLayer)

Precisely one of static_layer, disk_layer, admin_layer, rtds_layer must be set.

config.bootstrap.v3.RuntimeLayer.DiskLayer

[config.bootstrap.v3.RuntimeLayer.DiskLayer proto]

Disk runtime layer.

  1. {
  2.   "symlink_root": "...",
  3.   "subdirectory": "...",
  4.   "append_service_cluster": "..."
  5. }

symlink_root

(string) The implementation assumes that the file system tree is accessed via a symbolic link. An atomic link swap is used when a new tree should be switched to. This parameter specifies the path to the symbolic link. Envoy will watch the location for changes and reload the file system tree when they happen. See documentation on runtime atomicity for further details on how reloads are treated.

subdirectory

(string) Specifies the subdirectory to load within the root directory. This is useful if multiple systems share the same delivery mechanism. Envoy configuration elements can be contained in a dedicated subdirectory.

append_service_cluster

(bool) Append the service cluster to the path under symlink root.

config.bootstrap.v3.RuntimeLayer.AdminLayer

[config.bootstrap.v3.RuntimeLayer.AdminLayer proto]

Admin console runtime layer.

config.bootstrap.v3.RuntimeLayer.RtdsLayer

[config.bootstrap.v3.RuntimeLayer.RtdsLayer proto]

Runtime Discovery Service (RTDS) layer.

  1. {
  2.   "name": "...",
  3.   "rtds_config": "{...}"
  4. }

name

(string) Resource to subscribe to at rtds_config for the RTDS layer.

rtds_config

(config.core.v3.ConfigSource) RTDS configuration source.

config.bootstrap.v3.LayeredRuntime

[config.bootstrap.v3.LayeredRuntime proto]

Runtime configuration overview.

  1. {
  2.   "layers": []
  3. }

layers

(repeated config.bootstrap.v3.RuntimeLayer) The layers of the runtime. This is ordered such that later layers in the list overlay earlier entries.

config.bootstrap.v3.CustomInlineHeader

[config.bootstrap.v3.CustomInlineHeader proto]

Used to specify the header that needs to be registered as an inline header.

If request or response contain multiple headers with the same name and the header name is registered as an inline header. Then multiple headers will be folded into one, and multiple header values will be concatenated by a suitable delimiter. The delimiter is generally a comma.

For example, if ‘foo’ is registered as an inline header, and the headers contains the following two headers:

  1. foo: bar
  2. foo: eep

Then they will eventually be folded into:

  1. foo: bar, eep

Inline headers provide O(1) search performance, but each inline header imposes an additional memory overhead on all instances of the corresponding type of HeaderMap or TrailerMap.

  1. {
  2.   "inline_header_name": "...",
  3.   "inline_header_type": "..."
  4. }

inline_header_name

(string, REQUIRED) The name of the header that is expected to be set as the inline header.

inline_header_type

(config.bootstrap.v3.CustomInlineHeader.InlineHeaderType) The type of the header that is expected to be set as the inline header.

Enum config.bootstrap.v3.CustomInlineHeader.InlineHeaderType

[config.bootstrap.v3.CustomInlineHeader.InlineHeaderType proto]

REQUEST_HEADER

(DEFAULT) ⁣

REQUEST_TRAILER

⁣

RESPONSE_HEADER

⁣

RESPONSE_TRAILER

⁣