Customizing Istio Metrics with Telemetry API

Telemetry API has been in Istio as a first-class API for quite sometime now. Previously, users had to configure metrics in the telemetry section of the Istio configuration.

This task shows you how to customize the metrics that Istio generates with Telemetry API.

Before you begin

Install Istio in your cluster and deploy an application.

Telemetry API can not work together with EnvoyFilter. For more details please checkout this issue.

  • Starting with Istio version 1.18, the Prometheus EnvoyFilter will not be installed by default, and instead meshConfig.defaultProviders is used to enable it. Telemetry API should be used to further customize the telemetry pipeline.

  • For versions of Istio before 1.18, you should install with the following IstioOperator configuration:

    1. apiVersion: install.istio.io/v1alpha1
    2. kind: IstioOperator
    3. spec:
    4. values:
    5. telemetry:
    6. enabled: true
    7. v2:
    8. enabled: false

Override metrics

The metrics section provides values for the metric dimensions as expressions, and allows you to remove or override the existing metric dimensions. You can modify the standard metric definitions using tags_to_remove or by re-defining a dimension.

  1. Remove grpc_response_status tags from REQUEST_COUNT metric

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4. name: remove-tags
    5. namespace: istio-system
    6. spec:
    7. metrics:
    8. - providers:
    9. - name: prometheus
    10. overrides:
    11. - match:
    12. mode: CLIENT_AND_SERVER
    13. metric: REQUEST_COUNT
    14. tagOverrides:
    15. grpc_response_status:
    16. operation: REMOVE
  2. Add custom tags for REQUEST_COUNT metric

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4. name: custom-tags
    5. namespace: istio-system
    6. spec:
    7. metrics:
    8. - overrides:
    9. - match:
    10. metric: REQUEST_COUNT
    11. mode: CLIENT
    12. tagOverrides:
    13. destination_x:
    14. value: upstream_peer.labels['app'].value
    15. - match:
    16. metric: REQUEST_COUNT
    17. mode: SERVER
    18. tagOverrides:
    19. source_x:
    20. value: downstream_peer.labels['app'].value
    21. providers:
    22. - name: prometheus

Disable metrics

  1. Disable all metrics by following configuration:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4. name: remove-all-metrics
    5. namespace: istio-system
    6. spec:
    7. metrics:
    8. - providers:
    9. - name: prometheus
    10. overrides:
    11. - disabled: true
    12. match:
    13. mode: CLIENT_AND_SERVER
    14. metric: ALL_METRICS
  2. Disable REQUEST_COUNT metrics by following configuration:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4. name: remove-request-count
    5. namespace: istio-system
    6. spec:
    7. metrics:
    8. - providers:
    9. - name: prometheus
    10. overrides:
    11. - disabled: true
    12. match:
    13. mode: CLIENT_AND_SERVER
    14. metric: REQUEST_COUNT
  3. Disable REQUEST_COUNT metrics for client by following configuration:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4. name: remove-client
    5. namespace: istio-system
    6. spec:
    7. metrics:
    8. - providers:
    9. - name: prometheus
    10. overrides:
    11. - disabled: true
    12. match:
    13. mode: CLIENT
    14. metric: REQUEST_COUNT
  4. Disable REQUEST_COUNT metrics for server by following configuration:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4. name: remove-server
    5. namespace: istio-system
    6. spec:
    7. metrics:
    8. - providers:
    9. - name: prometheus
    10. overrides:
    11. - disabled: true
    12. match:
    13. mode: SERVER
    14. metric: REQUEST_COUNT

Verify the results

Send traffic to the mesh. For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your web browser or issue the following command:

  1. $ curl "http://$GATEWAY_URL/productpage"

$GATEWAY_URL is the value set in the Bookinfo example.

Use the following command to verify that Istio generates the data for your new or modified dimensions:

  1. $ istioctl x es "$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')" -oprom | grep istio_requests_total | grep -v TYPE |grep -v 'reporter="destination"'
  1. $ istioctl x es "$(kubectl get pod -l app=details -o jsonpath='{.items[0].metadata.name}')" -oprom | grep istio_requests_total

For example, in the output, locate the metric istio_requests_total and verify it contains your new dimension.

It might take a short period of time for the proxies to start applying the config. If the metric is not received, you may retry sending requests after a short wait, and look for the metric again.