Customizing Istio Metrics

This task shows you how to customize the metrics that Istio generates.

Istio generates telemetry that various dashboards consume to help you visualize your mesh. For example, dashboards that support Istio include:

• Grafana
• Kiali
• Prometheus

By default, Istio defines and generates a set of standard metrics (e.g. requests_total), but you can also customize them and create new metrics using the Telemetry API.

Before you begin

Install Istio in your cluster and deploy an application. Alternatively, you can set up custom statistics as part of the Istio installation.

The Bookinfo sample application is used as the example application throughout this task. For installation instructions, see deploying the Bookinfo application.

Enable custom metrics

To customize telemetry metrics, for example, to add request_host and destination_port dimensions to the requests_total metric emitted by both gateways and sidecars in the inbound and outbound direction, use the following:

$ cat <<EOF > ./custom_metrics.yaml
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: namespace-metrics
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - match:
        metric: REQUEST_COUNT
      tagOverrides:
        destination_port:
          value: "string(destination.port)"
        request_host:
          value: "request.host"
EOF
$ kubectl apply -f custom_metrics.yaml

Verify the results

Send traffic to the mesh. For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your web browser or issue the following command:

$ curl "http://$GATEWAY_URL/productpage"

$GATEWAY_URL is the value set in the Bookinfo example.

Use the following command to verify that Istio generates the data for your new or modified dimensions:

$ kubectl exec "$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')" -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus' | grep istio_requests_total

For example, in the output, locate the metric istio_requests_total and verify it contains your new dimension.

It might take a short period of time for the proxies to start applying the config. If the metric is not received, you may retry sending requests after a short wait, and look for the metric again.

Use expressions for values

The values in the metric configuration are common expressions, which means you must double-quote strings in JSON, e.g. “‘string value’”. Unlike Mixer expression language, there is no support for the pipe (|) operator, but you can emulate it with the has or in operator, for example:

has(request.host) ? request.host : "unknown"

For more information, see Common Expression Language.

Istio exposes all standard Envoy attributes. Peer metadata is available as attributes upstream_peer for outbound and downstream_peer for inbound with the following fields:

For example, the expression for the peer app label to be used in an outbound configuration is upstream_peer.labels['app'].value.

Cleanup

To delete the Bookinfo sample application and its configuration, see Bookinfo cleanup.