Proxy Metrics

The Linkerd proxy exposes metrics that describe the traffic flowing through theproxy. The following metrics are available at /metrics on the proxy's metricsport (default: :4191) in the Prometheus format.

Protocol-Level Metrics

  • request_total: A counter of the number of requests the proxy has received.This is incremented when the request stream begins.

  • response_total: A counter of the number of responses the proxy has received.This is incremented when the response stream ends.

  • response_latency_ms: A histogram of response latencies. This measurementreflects the time-to-first-byte (TTFB) by recording the elapsed timebetween the proxy processing a request's headers and the first data frame of theresponse. If a response does not include any data, the end-of-stream event isused. The TTFB measurement is used so that Linkerd accurately reflectsapplication behavior when a server provides response headers immediately but isslow to begin serving the response body.

  • route_request_total, route_response_latency_ms, and route_response_total:These metrics are analogous to request_total, response_latency_ms, andresponse_total except that they are collected at the route level. Thismeans that they do not have authority, tls, grpc_status_code or anyoutbound labels but instead they have:

    • dst: The authority of this request.
    • rt_route: The name of the route for this request.

  • control_request_total, control_response_latency_ms, and control_response_total:These metrics are analogous to request_total, response_latency_ms, andresponse_total but for requests that the proxy makes to the Linkerd controlplane. Instead of authority, direction, or any outbound labels, insteadthey have:

    • addr: The address used to connect to the control plane.

Note that latency measurements are not exported to Prometheus until the streamcompletes. This is necessary so that latencies can be labeled with the appropriateresponse classification.

Labels

Each of these metrics has the following labels:

  • authority: The value of the :authority (HTTP/2) or Host (HTTP/1.1)header of the request.
  • direction: inbound if the request originated from outside of the pod,outbound if the request originated from inside of the pod.
  • tls: true if the request's connection was secured with TLS.

Response Labels

The following labels are only applicable on response_* metrics.

  • classification: success if the response was successful, or failure ifa server error occurred. This classification is based onthe gRPC status code if one is present, and on the HTTPstatus code otherwise. Only applicable to response metrics.
  • grpc_status_code: The value of the grpc-status trailer. Only applicablefor gRPC responses.
  • status_code: The HTTP status code of the response.

Outbound labels

The following labels are only applicable if direction=outbound.

  • dst_deployment: The deployment to which this request is being sent.
  • dst_k8s_job: The job to which this request is being sent.
  • dst_replicaset: The replica set to which this request is being sent.
  • dst_daemonset: The daemon set to which this request is being sent.
  • dst_statefulset: The stateful set to which this request is being sent.
  • dst_replicationcontroller: The replication controller to which this requestis being sent.
  • dst_namespace: The namespace to which this request is being sent.
  • dst_service: The service to which this request is being sent.
  • dst_pod_template_hash: The pod-template-hash of the podto which this request is being sent. This labelselector roughly approximates a pod's ReplicaSet orReplicationController.

Prometheus Collector labels

The following labels are added by the Prometheus collector.

  • instance: ip:port of the pod.
  • job: The Prometheus job responsible for the collection, typicallylinkerd-proxy.
Kubernetes labels added at collection time

Kubernetes namespace, pod name, and all labels are mapped to correspondingPrometheus labels.

  • namespace: Kubernetes namespace that the pod belongs to.
  • pod: Kubernetes pod name.
  • pod_template_hash: Corresponds to the pod-template-hashKubernetes label. This value changes during redeploys androlling restarts. This label selector roughlyapproximates a pod's ReplicaSet orReplicationController.
Linkerd labels added at collection time

Kubernetes labels prefixed with linkerd.io/ are added to your application atlinkerd inject time. More specifically, Kubernetes labels prefixed withlinkerd.io/proxy-* will correspond to these Prometheus labels:

  • daemonset: The daemon set that the pod belongs to (if applicable).
  • deployment: The deployment that the pod belongs to (if applicable).
  • k8s_job: The job that the pod belongs to (if applicable).
  • replicaset: The replica set that the pod belongs to (if applicable).
  • replicationcontroller: The replication controller that the pod belongs to(if applicable).
  • statefulset: The stateful set that the pod belongs to (if applicable).

Example

Here's a concrete example, given the following pod snippet:

  1. name: vote-bot-5b7f5657f6-xbjjw
  2. namespace: emojivoto
  3. labels:
  4.   app: vote-bot
  5.   linkerd.io/control-plane-ns: linkerd
  6.   linkerd.io/proxy-deployment: vote-bot
  7.   pod-template-hash: "3957278789"
  8.   test: vote-bot-test

The resulting Prometheus labels will look like this:

  1. request_total{
  2.   pod="vote-bot-5b7f5657f6-xbjjw",
  3.   namespace="emojivoto",
  4.   app="vote-bot",
  5.   control_plane_ns="linkerd",
  6.   deployment="vote-bot",
  7.   pod_template_hash="3957278789",
  8.   test="vote-bot-test",
  9.   instance="10.1.3.93:4191",
  10.   job="linkerd-proxy"
  11. }

Transport-Level Metrics

The following metrics are collected at the level of the underlying transportlayer.

  • tcp_open_total: A counter of the total number of opened transportconnections.
  • tcp_close_total: A counter of the total number of transport connectionswhich have closed.
  • tcp_open_connections: A gauge of the number of transport connectionscurrently open.
  • tcp_write_bytes_total: A counter of the total number of sent bytes. This isupdated when the connection closes.
  • tcp_read_bytes_total: A counter of the total number of received bytes. Thisis updated when the connection closes.
  • tcp_connection_duration_ms: A histogram of the duration of the lifetime of aconnection, in milliseconds. This is updated when the connection closes.

Labels

Each of these metrics has the following labels:

  • direction: inbound if the connection was established either from outside thepod to the proxy, or from the proxy to the application,outbound if the connection was established either from theapplication to the proxy, or from the proxy to outside the pod.
  • peer: src if the connection was accepted by the proxy from the source,dst if the connection was opened by the proxy to the destination.

Note that the labels described above under the heading “Prometheus Collector labels”are also added to transport-level metrics, when applicable.

Connection Close Labels

The following labels are added only to metrics which are updated when aconnection closes (tcp_close_total and tcp_connection_duration_ms):

  • classification: success if the connection terminated cleanly, failure ifthe connection closed due to a connection failure.