Envoy Statistics

The Envoy proxy keeps detailed statistics about network traffic.

Envoy’s statistics only cover the traffic for a particular Envoy instance. See Observability for persistent per-service Istio telemetry. The statistics the Envoy proxies record can provide more information about specific pod instances.

To see the statistics for a pod:

  1. $ kubectl exec "$POD" -c istio-proxy -- pilot-agent request GET stats

Envoy generates statistics about its behavior, scoping the statistics by proxy function. Examples include:

By default, Istio configures Envoy to record a minimal set of statistics to reduce the overall CPU and memory footprint of the installed proxies. The default collection keys are:

  • cluster_manager
  • listener_manager
  • server
  • cluster.xds-grpc

To see the Envoy settings for statistics data collection use istioctl proxy-config bootstrap and follow the deep dive into Envoy configuration. Envoy only collects statistical data on items matching the inclusion_list within the stats_matcher JSON element.

Note: The names of Envoy statistics can vary based on the composition of Envoy configuration. As a result, the exposed names of statistics for Envoys managed by Istio are subject to the configuration behavior of Istio. If you build or maintain dashboards or alerts based on Envoy statistics, it is strongly recommended that you examine the statistics in a canary environment before upgrading Istio.

To configure Istio proxy to record additional statistics, you can add ProxyConfig.ProxyStatsMatcher to your mesh config. For example, to enable stats for circuit breaker, retry, and upstream connections globally, you can specify stats matcher as follows:

Proxy needs to restart to pick up the stats matcher configuration.

  1. apiVersion: install.istio.io/v1alpha1
  2. kind: IstioOperator
  3. spec:
  4.   meshConfig:
  5.     defaultConfig:
  6.       proxyStatsMatcher:
  7.         inclusionRegexps:
  8.           - ".*circuit_breakers.*"
  9.         inclusionPrefixes:
  10.           - "upstream_rq_retry"
  11.           - "upstream_cx"

You can also override the global stats matching configuration per proxy by using the proxy.istio.io/config annotation. For example, to configure the same stats generation inclusion as above, you can add the annotation to a gateway proxy or a workload as follows:

  1. metadata:
  2.   annotations:
  3.     proxy.istio.io/config: |-
  4.       proxyStatsMatcher:
  5.         inclusionRegexps:
  6.         - ".*circuit_breakers.*"
  7.         inclusionPrefixes:
  8.         - "upstream_rq_retry"
  9.         - "upstream_cx"

Note: If you are using sidecar.istio.io/statsInclusionPrefixes, sidecar.istio.io/statsInclusionRegexps, and sidecar.istio.io/statsInclusionSuffixes, consider switching to the ProxyConfig-based configuration as it provides a global default and a uniform way to override at the gateway and sidecar proxy.