Customizing Istio Metrics

Customizing Istio Metrics

This task shows you how to customize the metrics that Istio generates.

Istio generates telemetry that various dashboards consume to help you visualize your mesh. For example, dashboards that support Istio include:

By default, Istio defines and generates a set of standard metrics (e.g. requests_total), but you can also customize them and create new metrics.

Custom statistics configuration

Istio uses the Envoy proxy to generate metrics and provides its configuration in the EnvoyFilter at manifests/charts/istio-control/istio-discovery/templates/telemetryv2_1.6.yaml.

Configuring custom statistics involves two sections of the EnvoyFilter: definitions and metrics. The definitions section supports creating new metrics by name, the expected value expression, and the metric type (counter, gauge, and histogram). The metrics section provides values for the metric dimensions as expressions, and allows you to remove or override the existing metric dimensions. You can modify the standard metric definitions using tags_to_remove or by re-defining a dimension.

For more information, see Stats Config reference.

Before you begin

Install Istio in your cluster and deploy an application. Alternatively, you can set up custom statistics as part of the Istio installation.

Enable custom metrics

Edit the the EnvoyFilter to add or modify dimensions and metrics. Then, add annotations to all the Istio-enabled pods to extract the new or modified dimensions.

Find the stats-filter-1.6 EnvoyFilter resource from the istio-system namespace, using the following command:

$ kubectl -n istio-system get envoyfilter | grep ^stats-filter-1.6
stats-filter-1.6                    2d

Create a local file system copy of the EnvoyFilter configuration, using the following command:

$ kubectl -n istio-system get envoyfilter stats-filter-1.6 -o yaml > stats-filter-1.6.yaml

Open stats-filter-1.6.yaml with a text editor and locate the envoy.wasm.stats extension configuration. The default configuration is in the configuration section and looks like this example:
```
{
"debug": "false",
"stat_prefix": "istio"
}
```
Edit stats-filter-1.6.yaml and modify the configuration section for each instance of the extension configuration. For example, to add destination_port and request_host dimensions to the standard requests_total metric, change the configuration section to look like the following. Istio automatically prefixes all metric names with istio_, so omit the prefix from the name field in the metric specification.
```
{
    "debug": "false",
    "stat_prefix": "istio",
    "metrics": [
        {
            "name": "requests_total",
            "dimensions": {
                "destination_port": "string(destination.port)",
                "request_host": "request.host"
            }
        }
    ]
}
```
Save stats-filter-1.6.yaml and then apply the configuration using the following command:
```
$ kubectl -n istio-system apply -f stats-filter-1.6.yaml
```
Apply the following annotation to all injected pods with the list of the dimensions to extract into a Prometheus time series using the following command:

This step is needed only if your dimensions are not already in DefaultStatTags list
```
apiVersion: extensions/v1beta1
kind: Deployment
spec:
template:
    metadata:
    annotations:
        sidecar.istio.io/extraStatTags: destination_port,request_host
```

Verify the results

Use the following command to verify that Istio generates the data for your new or modified dimensions:

$ kubectl exec pod-name -c istio-proxy -- curl 'localhost:15000/stats/prometheus' | grep istio

For example, in the output, locate the metric istio_requests_total and verify it contains your new dimension.

Use expressions for values

The values in the metric configuration are common expressions, which means you must double-quote strings in JSON, e.g. “‘string value’”. Unlike Mixer expression language, there is no support for the pipe (|) operator, but you can emulate it with the has or in operator, for example:

has(request.host) ? request.host : "unknown"

For more information, see Common Expression Language.

Istio exposes all standard Envoy attributes. Additionally, you can use the following extra attributes.

Attribute	Type	Value
`listener_direction`	int64	Enumeration value for listener direction
`listener_metadata`	metadata	Per-listener metadata
`route_metadata`	metadata	Per-route metadata
`cluster_metadata`	metadata	Per-cluster metadata
`node`	node	Node description
`cluster_name`	string	Upstream cluster name
`route_name`	string	Route name
`filter_state`	map[string, bytes]	Per-filter state blob
`plugin_name`	string	Wasm extension name
`plugin_root_id`	string	Wasm root instance ID
`plugin_vm_id`	string	Wasm VM ID

For more information, see configuration reference.

Customizing Istio Metrics

Customizing Istio Metrics

Custom statistics configuration

Before you begin

Enable custom metrics

Verify the results

Use expressions for values

See also