Administration interface

Envoy exposes a local administration interface that can be used to query and modify different aspects of the server:

Attention

The administration interface in its current form both allows destructive operations to be performed (e.g., shutting down the server) as well as potentially exposes private information (e.g., stats, cluster names, cert info, etc.). It is critical that access to the administration interface is only allowed via a secure network. It is also critical that hosts that access the administration interface are only attached to the secure network (i.e., to avoid CSRF attacks). This involves setting up an appropriate firewall or optimally only allowing access to the administration listener via localhost. This can be accomplished with a v2 configuration like the following:

  1. admin:
  2.   access_log_path: /tmp/admin_access.log
  3.   profile_path: /tmp/envoy.prof
  4.   address:
  5.     socket_address: { address: 127.0.0.1, port_value: 9901 }

In the future additional security options will be added to the administration interface. This work is tracked in this issue.

All mutations must be sent as HTTP POST operations. When a mutation is requested via GET, the request has no effect, and an HTTP 400 (Invalid Request) response is returned.

GET /

Render an HTML home page with a table of links to all available options.

GET /help

Print a textual table of all available options.

GET /certs

List out all loaded TLS certificates, including file name, serial number, subject alternate names and days until expiration in JSON format conforming to the certificate proto definition.

GET /clusters

List out all configured cluster manager clusters. This information includes all discovered upstream hosts in each cluster along with per host statistics. This is useful for debugging service discovery issues.

  • Cluster manager information

    • version_info string – the version info string of the last loaded CDS update. If Envoy does not have CDS setup, the output will read version_info::static.
  1. Cluster wide information
  3. -   [circuit breakers](https://www.envoyproxy.io/docs/envoy/v1.12.0/configuration/upstream/cluster_manager/cluster_circuit_breakers#config-cluster-manager-cluster-circuit-breakers) settings for all priority settings.
  5. -   Information about [outlier detection](https://www.envoyproxy.io/docs/envoy/v1.12.0/intro/arch_overview/upstream/outlier#arch-overview-outlier-detection) if a detector is installed. Currently [average success rate](https://www.envoyproxy.io/docs/envoy/v1.12.0/api-v2/data/cluster/v2alpha/outlier_detection_event.proto#envoy-api-field-data-cluster-v2alpha-outlierejectsuccessrate-cluster-average-success-rate), and [ejection threshold](https://www.envoyproxy.io/docs/envoy/v1.12.0/api-v2/data/cluster/v2alpha/outlier_detection_event.proto#envoy-api-field-data-cluster-v2alpha-outlierejectsuccessrate-cluster-success-rate-ejection-threshold) are presented. Both of these values could be `-1` if there was not enough data to calculate them in the last [interval](https://www.envoyproxy.io/docs/envoy/v1.12.0/api-v2/api/v2/cluster/outlier_detection.proto#envoy-api-field-cluster-outlierdetection-interval).
  7. -   `added_via_api` flag – `false` if the cluster was added via static configuration, `true` if it was added via the [CDS](https://www.envoyproxy.io/docs/envoy/v1.12.0/configuration/upstream/cluster_manager/cds#config-cluster-manager-cds) api.
  10. Per host statistics
  12. <table><colgroup><col> <col> <col> </colgroup><thead><tr><th><p>Name</p></th><th><p>Type</p></th><th><p>Description</p></th></tr></thead><tbody><tr><td><p>cx_total</p></td><td><p>Counter</p></td><td><p>Total connections</p></td></tr><tr><td><p>cx_active</p></td><td><p>Gauge</p></td><td><p>Total active connections</p></td></tr><tr><td><p>cx_connect_fail</p></td><td><p>Counter</p></td><td><p>Total connection failures</p></td></tr><tr><td><p>rq_total</p></td><td><p>Counter</p></td><td><p>Total requests</p></td></tr><tr><td><p>rq_timeout</p></td><td><p>Counter</p></td><td><p>Total timed out requests</p></td></tr><tr><td><p>rq_success</p></td><td><p>Counter</p></td><td><p>Total requests with non-5xx responses</p></td></tr><tr><td><p>rq_error</p></td><td><p>Counter</p></td><td><p>Total requests with 5xx responses</p></td></tr><tr><td><p>rq_active</p></td><td><p>Gauge</p></td><td><p>Total active requests</p></td></tr><tr><td><p>healthy</p></td><td><p>String</p></td><td><p>The health status of the host. See below</p></td></tr><tr><td><p>weight</p></td><td><p>Integer</p></td><td><p>Load balancing weight (1-100)</p></td></tr><tr><td><p>zone</p></td><td><p>String</p></td><td><p>Service zone</p></td></tr><tr><td><p>canary</p></td><td><p>Boolean</p></td><td><p>Whether the host is a canary</p></td></tr><tr><td><p>success_rate</p></td><td><p>Double</p></td><td><p>Request success rate (0-100). -1 if there was not enough <a class="reference internal" href="https://www.envoyproxy.io/docs/envoy/v1.12.0/api-v2/api/v2/cluster/outlier_detection.proto#envoy-api-field-cluster-outlierdetection-success-rate-request-volume">request volume</a> in the <a class="reference internal" href="https://www.envoyproxy.io/docs/envoy/v1.12.0/api-v2/api/v2/cluster/outlier_detection.proto#envoy-api-field-cluster-outlierdetection-interval">interval</a> to calculate it</p></td></tr></tbody></table>
  14. Host health status
  16. A host is either healthy or unhealthy because of one or more different failing health states. If the host is healthy the `healthy` output will be equal to *healthy*.
  18. If the host is not healthy, the `healthy` output will be composed of one or more of the following strings:
  20. */failed\_active\_hc*: The host has failed an [active health check](https://www.envoyproxy.io/docs/envoy/v1.12.0/configuration/upstream/cluster_manager/cluster_hc#config-cluster-manager-cluster-hc).
  22. */failed\_eds\_health*: The host was marked unhealthy by EDS.
  24. */failed\_outlier\_check*: The host has failed an outlier detection check.

GET /clusters?format=json

Dump the /clusters output in a JSON-serialized proto. See the definition for more information.

GET /config_dump

Dump currently loaded configuration from various Envoy components as JSON-serialized proto messages. See the response definition for more information.

Warning

The underlying proto is marked v2alpha and hence its contents, including the JSON representation, are not guaranteed to be stable.

GET /contention

Dump current Envoy mutex contention stats (MutexStats) in JSON format, if mutex tracing is enabled. See --enable-mutex-tracing.

POST /cpuprofiler

Enable or disable the CPU profiler. Requires compiling with gperftools. The output file can be configured by admin.profile_path.

POST /heapprofiler

Enable or disable the Heap profiler. Requires compiling with gperftools. The output file can be configured by admin.profile_path.

POST /healthcheck/fail

Fail inbound health checks. This requires the use of the HTTP health check filter. This is useful for draining a server prior to shutting it down or doing a full restart. Invoking this command will universally fail health check requests regardless of how the filter is configured (pass through, etc.).

POST /healthcheck/ok

Negate the effect of POST /healthcheck/fail. This requires the use of the HTTP health check filter.

GET /hot_restart_version

See --hot-restart-version.

GET /listeners

List out all configured listeners. This information includes the names of listeners as well as the addresses that they are listening on. If a listener is configured to listen on port 0, then the output will contain the actual port that was allocated by the OS.

GET /listeners?format=json

Dump the /listeners output in a JSON-serialized proto. See the definition for more information.

POST /logging

Enable/disable different logging levels on a particular logger or all loggers.

  • To change the logging level across all loggers, set the query parameter as level=.

  • To change a particular logger’s level, set the query parameter like so, =.

  • To list the loggers, send a POST request to the /logging endpoint without a query parameter.

Note

Generally only used during development.

POST /memory

Prints current memory allocation / heap usage, in bytes. Useful in lieu of printing all /stats and filtering to get the memory-related statistics.

POST /quitquitquit

Cleanly exit the server.

POST /reset_counters

Reset all counters to zero. This is useful along with GET /stats during debugging. Note that this does not drop any data sent to statsd. It just affects local output of the GET /stats command.

POST /drain_listeners

Drains all listeners.

  • POST /drain_listeners?inboundonly

Drains all inbound listeners. traffic_direction field in Listener is used to determine whether a listener is inbound or outbound.

Attention

This operation directly stops the matched listeners on workers. Once listeners in a given traffic direction are stopped, listener additions and modifications in that direction are not allowed.

GET /server_info

Outputs a JSON message containing information about the running server.

Sample output looks like:

  1. {
  2.   "version": "b050513e840aa939a01f89b07c162f00ab3150eb/1.9.0-dev/Modified/DEBUG",
  3.   "state": "LIVE",
  4.   "command_line_options": {
  5.     "base_id": "0",
  6.     "concurrency": 8,
  7.     "config_path": "config.yaml",
  8.     "config_yaml": "",
  9.     "allow_unknown_static_fields": false,
  10.     "admin_address_path": "",
  11.     "local_address_ip_version": "v4",
  12.     "log_level": "info",
  13.     "component_log_level": "",
  14.     "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v",
  15.     "log_path": "",
  16.     "hot_restart_version": false,
  17.     "service_cluster": "",
  18.     "service_node": "",
  19.     "service_zone": "",
  20.     "mode": "Serve",
  21.     "disable_hot_restart": false,
  22.     "enable_mutex_tracing": false,
  23.     "restart_epoch": 0,
  24.     "file_flush_interval": "10s",
  25.     "drain_time": "600s",
  26.     "parent_shutdown_time": "900s",
  27.     "cpuset_threads": false
  28.   },
  29.   "uptime_current_epoch": "6s",
  30.   "uptime_all_epochs": "6s"
  31. }

See the ServerInfo proto for an explanation of the output.

GET /ready

Outputs a string and error code reflecting the state of the server. 200 is returned for the LIVE state, and 503 otherwise. This can be used as a readiness check.

Example output:

  1. LIVE

See the state field of the ServerInfo proto for an explanation of the output.

GET /stats

Outputs all statistics on demand. This command is very useful for local debugging. Histograms will output the computed quantiles i.e P0,P25,P50,P75,P90,P99,P99.9 and P100. The output for each quantile will be in the form of (interval,cumulative) where interval value represents the summary since last flush interval and cumulative value represents the summary since the start of Envoy instance. “No recorded values” in the histogram output indicates that it has not been updated with a value. See here for more information.

  • GET /stats?usedonly

Outputs statistics that Envoy has updated (counters incremented at least once, gauges changed at least once, and histograms added to at least once).

  • GET /stats?filter=regex

Filters the returned stats to those with names matching the regular expression regex. Compatible with usedonly. Performs partial matching by default, so /stats?filter=server will return all stats containing the word server. Full-string matching can be specified with begin- and end-line anchors. (i.e. /stats?filter=^server.concurrency$)

GET /stats?format=json

Outputs /stats in JSON format. This can be used for programmatic access of stats. Counters and Gauges will be in the form of a set of (name,value) pairs. Histograms will be under the element “histograms”, that contains “supported_quantiles” which lists the quantiles supported and an array of computed_quantiles that has the computed quantile for each histogram.

If a histogram is not updated during an interval, the output will have null for all the quantiles.

Example histogram output:

  1. {
  2.   "histograms": {
  3.     "supported_quantiles": [
  4.       0, 25, 50, 75, 90, 95, 99, 99.9, 100
  5.     ],
  6.     "computed_quantiles": [
  7.       {
  8.         "name": "cluster.external_auth_cluster.upstream_cx_length_ms",
  9.         "values": [
  10.           {"interval": 0, "cumulative": 0},
  11.           {"interval": 0, "cumulative": 0},
  12.           {"interval": 1.0435787, "cumulative": 1.0435787},
  13.           {"interval": 1.0941565, "cumulative": 1.0941565},
  14.           {"interval": 2.0860023, "cumulative": 2.0860023},
  15.           {"interval": 3.0665233, "cumulative": 3.0665233},
  16.           {"interval": 6.046609, "cumulative": 6.046609},
  17.           {"interval": 229.57333,"cumulative": 229.57333},
  18.           {"interval": 260,"cumulative": 260}
  19.         ]
  20.       },
  21.       {
  22.         "name": "http.admin.downstream_rq_time",
  23.         "values": [
  24.           {"interval": null, "cumulative": 0},
  25.           {"interval": null, "cumulative": 0},
  26.           {"interval": null, "cumulative": 1.0435787},
  27.           {"interval": null, "cumulative": 1.0941565},
  28.           {"interval": null, "cumulative": 2.0860023},
  29.           {"interval": null, "cumulative": 3.0665233},
  30.           {"interval": null, "cumulative": 6.046609},
  31.           {"interval": null, "cumulative": 229.57333},
  32.           {"interval": null, "cumulative": 260}
  33.         ]
  34.       }
  35.     ]
  36.   }
  37. }
  • GET /stats?format=json&usedonly

Outputs statistics that Envoy has updated (counters incremented at least once, gauges changed at least once, and histograms added to at least once) in JSON format.

GET /stats?format=prometheus

or alternatively,

  • GET /stats/prometheus

Outputs /stats in Prometheus v0.0.4 format. This can be used to integrate with a Prometheus server.

You can optionally pass the usedonly URL query argument to only get statistics that Envoy has updated (counters incremented at least once, gauges changed at least once, and histograms added to at least once)

  • GET /stats/recentlookups

This endpoint helps Envoy developers debug potential contention issues in the stats system. Initially, only the count of StatName lookups is acumulated, not the specific names that are being looked up. In order to see specific recent requests, you must enable the feature by POSTing to /stats/recentlookups/enable. There may be approximately 40-100 nanoseconds of added overhead per lookup.

When enabled, this endpoint emits a table of stat names that were recently accessed as strings by Envoy. Ideally, strings should be converted into StatNames, counters, gauges, and histograms by Envoy code only during startup or when receiving a new configuration via xDS. This is because when stats are looked up as strings they must take a global symbol table lock. During startup this is acceptable, but in response to user requests on high core-count machines, this can cause performance issues due to mutex contention.

This admin endpoint requires Envoy to be started with option –use-fake-symbol-table 0.

See source/docs/stats.md for more details.

Note also that actual mutex contention can be tracked via GET /contention.

  • POST /stats/recentlookups/enable

Turns on collection of recent lookup of stat-names, thus enabling /stats/recentlookups.

See source/docs/stats.md for more details.

  • POST /stats/recentlookups/disable

Turns off collection of recent lookup of stat-names, thus disabling /stats/recentlookups. It also clears the list of lookups. However, the total count, visible as stat server.stats_recent_lookups, is not cleared, and continues to accumulate.

See source/docs/stats.md for more details.

  • POST /stats/recentlookups/clear

Clears all outstanding lookups and counts. This clears all recent lookups data as well as the count, but collection continues if it is enabled.

See source/docs/stats.md for more details.

GET /runtime

Outputs all runtime values on demand in JSON format. See here for more information on how these values are configured and utilized. The output include the list of the active runtime override layers and the stack of layer values for each key. Empty strings indicate no value, and the final active value from the stack also is included in a separate key. Example output:

  1. {
  2.   "layers": [
  3.     "disk",
  4.     "override",
  5.     "admin",
  6.   ],
  7.   "entries": {
  8.     "my_key": {
  9.       "layer_values": [
  10.         "my_disk_value",
  11.         "",
  12.         ""
  13.       ],
  14.       "final_value": "my_disk_value"
  15.     },
  16.     "my_second_key": {
  17.       "layer_values": [
  18.         "my_second_disk_value",
  19.         "my_disk_override_value",
  20.         "my_admin_override_value"
  21.       ],
  22.       "final_value": "my_admin_override_value"
  23.     }
  24.   }
  25. }

POST /runtime_modify?key1=value1&key2=value2&keyN=valueN

Adds or modifies runtime values as passed in query parameters. To delete a previously added key, use an empty string as the value. Note that deletion only applies to overrides added via this endpoint; values loaded from disk can be modified via override but not deleted.

Attention

Use the /runtime_modify endpoint with care. Changes are effectively immediately. It is critical that the admin interface is properly secured.

GET /hystrix_event_stream

This endpoint is intended to be used as the stream source for Hystrix dashboard. a GET to this endpoint will trigger a stream of statistics from Envoy in text/event-stream format, as expected by the Hystrix dashboard.

If invoked from a browser or a terminal, the response will be shown as a continuous stream, sent in intervals defined by the Bootstrap stats_flush_interval

This handler is enabled only when a Hystrix sink is enabled in the config file as documented here.

As Envoy’s and Hystrix resiliency mechanisms differ, some of the statistics shown in the dashboard had to be adapted:

  • Thread pool rejections - Generally similar to what’s called short circuited in Envoy, and counted by upstream_rq_pending_overflow, although the term thread pool is not accurate for Envoy. Both in Hystrix and Envoy, the result is rejected requests which are not passed upstream.

  • circuit breaker status (closed or open) - Since in Envoy, a circuit is opened based on the current number of connections/requests in queue, there is no sleeping window for circuit breaker, circuit open/closed is momentary. Hence, we set the circuit breaker status to “forced closed”.

  • Short-circuited (rejected) - The term exists in Envoy but refers to requests not sent because of passing a limit (queue or connections), while in Hystrix it refers to requests not sent because of high percentage of service unavailable responses during some time frame. In Envoy, service unavailable response will cause outlier detection - removing a node off the load balancer pool, but requests are not rejected as a result. Therefore, this counter is always set to ‘0’.

  • Latency information represents data since last flush. Mean latency is currently not available.

POST /tap

This endpoint is used for configuring an active tap session. It is only available if a valid tap extension has been configured, and that extension has been configured to accept admin configuration. See: