Configuring Envoy as an edge proxy

Envoy is a production-ready edge proxy, however, the default settings are tailored for the service mesh use case, and some values need to be adjusted when using Envoy as an edge proxy.

TCP proxies should configure:

HTTP proxies should additionally configure:

If Envoy is configured with RBAC filter or makes route selection based on URL path it is recommended to enable the following path normalization options to minimize probability of path confusion vulnerabilities. Path confusion vulnerabilities occur when parties participating in request use different path representations.

Additionally the path_with_escaped_slashes_action setting should be set according to following recommendations:

  • REJECT_REQUEST if dowstream clients are expected to use rfc3986 compliant normalized paths (i.e. gRPC clients).

  • UNESCAPE_AND_REDIRECT if downstream client supports HTTP redirect (i.e. a browser). This option minimizes possibility of path confusion by forcing request to be re-issued with the same path across all parties: downstream client, Envoy and upstream server. Note that gRPC requests will still be rejected with the INTERNAL (13) error code, as gRPC clients do not support redirect.

  • KEEP_UNCHANGED for servers that are not rfc3986 compliant and require encoded slashes.

  • UNESCAPE_AND_FORWARD for servers that are known to treat escaped and unescaped slashes equivalently. Choosing this option may increase probablity of path confusion vulnerabilities if intermediaries perform path based access control.

The following is a YAML example of the above recommendation (taken from the Google VRP edge server configuration):

  1. overload_manager:
  2.   refresh_interval: 0.25s
  3.   resource_monitors:
  4.   - name: "envoy.resource_monitors.fixed_heap"
  5.     typed_config:
  6.       "@type": type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
  7.       # TODO: Tune for your system.
  8.       max_heap_size_bytes: 2147483648  # 2 GiB
  9.   actions:
  10.   - name: "envoy.overload_actions.shrink_heap"
  11.     triggers:
  12.     - name: "envoy.resource_monitors.fixed_heap"
  13.       threshold:
  14.         value: 0.95
  15.   - name: "envoy.overload_actions.stop_accepting_requests"
  16.     triggers:
  17.     - name: "envoy.resource_monitors.fixed_heap"
  18.       threshold:
  19.         value: 0.98
  21. admin:
  22.   address:
  23.     socket_address:
  24.       address: 127.0.0.1
  25.       port_value: 9090
  27. static_resources:
  28.   listeners:
  29.   - address:
  30.       socket_address:
  31.         address: 0.0.0.0
  32.         port_value: 443
  33.     listener_filters:
  34.     - name: "envoy.filters.listener.tls_inspector"
  35.     # Uncomment if Envoy is behind a load balancer that exposes client IP address using the PROXY protocol.
  36.     # - name: envoy.filters.listener.proxy_protocol
  37.     #   typed_config:
  38.     #     "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
  39.     per_connection_buffer_limit_bytes: 32768  # 32 KiB
  40.     filter_chains:
  41.     - filter_chain_match:
  42.         server_names: ["example.com", "www.example.com"]
  43.       transport_socket:
  44.         name: envoy.transport_sockets.tls
  45.         typed_config:
  46.           "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
  47.           common_tls_context:
  48.             tls_certificates:
  49.             - certificate_chain: {filename: "certs/servercert.pem"}
  50.               private_key: {filename: "certs/serverkey.pem"}
  51.       filters:
  52.       - name: envoy.filters.network.http_connection_manager
  53.         typed_config:
  54.           "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
  55.           stat_prefix: ingress_http
  56.           use_remote_address: true
  57.           normalize_path: true
  58.           merge_slashes: true
  59.           path_with_escaped_slashes_action: UNESCAPE_AND_REDIRECT
  60.           common_http_protocol_options:
  61.             idle_timeout: 3600s  # 1 hour
  62.             headers_with_underscores_action: REJECT_REQUEST
  63.           http2_protocol_options:
  64.             max_concurrent_streams: 100
  65.             initial_stream_window_size: 65536  # 64 KiB
  66.             initial_connection_window_size: 1048576  # 1 MiB
  67.           stream_idle_timeout: 300s  # 5 mins, must be disabled for long-lived and streaming requests
  68.           request_timeout: 300s  # 5 mins, must be disabled for long-lived and streaming requests
  69.           http_filters:
  70.           - name: envoy.filters.http.router
  71.           route_config:
  72.             virtual_hosts:
  73.             - name: default
  74.               domains: ["*"]
  75.               routes:
  76.               - match: {prefix: "/"}
  77.                 route:
  78.                   cluster: service_foo
  79.                   idle_timeout: 15s  # must be disabled for long-lived and streaming requests
  80.   clusters:
  81.   - name: service_foo
  82.     per_connection_buffer_limit_bytes: 32768  # 32 KiB
  83.     load_assignment:
  84.       cluster_name: some_service
  85.       endpoints:
  86.       - lb_endpoints:
  87.         - endpoint:
  88.             address:
  89.               socket_address:
  90.                 address: 127.0.0.1
  91.                 port_value: 8080
  92.     typed_extension_protocol_options:
  93.       envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
  94.         "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
  95.         explicit_http_config:
  96.           http2_protocol_options:
  97.             initial_stream_window_size: 65536  # 64 KiB
  98.             initial_connection_window_size: 1048576  # 1 MiB
  100. layered_runtime:
  101.   layers:
  102.   - name: static_layer_0
  103.     static_layer:
  104.       envoy:
  105.         resource_limits:
  106.           listener:
  107.             example_listener_name:
  108.               connection_limit: 10000
  109.       overload:
  110.         global_downstream_max_connections: 50000