Classifying Metrics Based on Request or Response

It’s useful to visualize telemetry based on the type of requests and responses handled by services in your mesh. For example, a bookseller tracks the number of times book reviews are requested. A book review request has this structure:

  1. GET /reviews/{review_id}

Counting the number of review requests must account for the unbounded element review_id. GET /reviews/1 followed by GET /reviews/2 should count as two requests to get reviews.

Istio lets you create classification rules using the AttributeGen plugin that groups requests into a fixed number of logical operations. For example, you can create an operation named GetReviews, which is a common way to identify operations using the Open API Spec operationId. This information is injected into request processing as istio_operationId attribute with value equal to GetReviews. You can use the attribute as a dimension in Istio standard metrics. Similarly, you can track metrics based on other operations like ListReviews and CreateReviews.

For more information, see the reference content.

Istio uses the Envoy proxy to generate metrics and provides its configuration in the EnvoyFilter at manifests/charts/istio-control/istio-discovery/templates/telemetryv2_1.10.yaml. As a result, writing classification rules involves adding attributes to the EnvoyFilter.

Classify metrics by request

You can classify requests based on their type, for example ListReview, GetReview, CreateReview.

  1. Create a file, for example attribute_gen_service.yaml, and save it with the following contents. This adds the istio.attributegen plugin to the EnvoyFilter. It also creates an attribute, istio_operationId and populates it with values for the categories to count as metrics.

    This configuration is service-specific since request paths are typically service-specific.

    1. apiVersion: networking.istio.io/v1alpha3
    2. kind: EnvoyFilter
    3. metadata:
    4.   name: istio-attributegen-filter
    5. spec:
    6.   workloadSelector:
    7.     labels:
    8.       app: reviews
    9.   configPatches:
    10.   - applyTo: HTTP_FILTER
    11.     match:
    12.       context: SIDECAR_INBOUND
    13.       proxy:
    14.         proxyVersion: '1\.9.*'
    15.       listener:
    16.         filterChain:
    17.           filter:
    18.             name: "envoy.http_connection_manager"
    19.             subFilter:
    20.               name: "istio.stats"
    21.     patch:
    22.       operation: INSERT_BEFORE
    23.       value:
    24.         name: istio.attributegen
    25.         typed_config:
    26.           "@type": type.googleapis.com/udpa.type.v1.TypedStruct
    27.           type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    28.           value:
    29.             config:
    30.               configuration:
    31.                 "@type": type.googleapis.com/google.protobuf.StringValue
    32.                 value: |
    33.                   {
    34.                     "attributes": [
    35.                       {
    36.                         "output_attribute": "istio_operationId",
    37.                         "match": [
    38.                           {
    39.                             "value": "ListReviews",
    40.                             "condition": "request.url_path == '/reviews' && request.method == 'GET'"
    41.                           },
    42.                           {
    43.                             "value": "GetReview",
    44.                             "condition": "request.url_path.matches('^/reviews/[[:alnum:]]*$') && request.method == 'GET'"
    45.                           },
    46.                           {
    47.                             "value": "CreateReview",
    48.                             "condition": "request.url_path == '/reviews/' && request.method == 'POST'"
    49.                           }
    50.                         ]
    51.                       }
    52.                     ]
    53.                   }
    54.               vm_config:
    55.                 runtime: envoy.wasm.runtime.null
    56.                 code:
    57.                   local: { inline_string: "envoy.wasm.attributegen" }

  2. Apply your changes using the following command:

    1. $ kubectl -n istio-system apply -f attribute_gen_service.yaml

  3. Find the stats-filter-1.10 EnvoyFilter resource from the istio-system namespace, using the following command:

    1. $ kubectl -n istio-system get envoyfilter | grep ^stats-filter-1.10
    2. stats-filter-1.10                    2d

  4. Create a local file system copy of the EnvoyFilter configuration, using the following command:

    1. $ kubectl -n istio-system get envoyfilter stats-filter-1.10 -o yaml > stats-filter-1.10.yaml

  5. Open stats-filter-1.10.yaml with a text editor and locate the name: istio.stats extension configuration. Update it to map request_operation dimension in the requests_total standard metric to istio_operationId attribute. The updated configuration file section should look like the following.

    1. name: istio.stats
    2. typed_config:
    3.   '@type': type.googleapis.com/udpa.type.v1.TypedStruct
    4.   type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    5.   value:
    6.     config:
    7.       configuration:
    8.         "@type": type.googleapis.com/google.protobuf.StringValue
    9.         value: |
    10.           {
    11.             "metrics": [
    12.              {
    13.                "name": "requests_total",
    14.                "dimensions": {
    15.                  "request_operation": "istio_operationId"
    16.                }
    17.              }]
    18.           }

  6. Save stats-filter-1.10.yaml and then apply the configuration using the following command:

    1. $ kubectl -n istio-system apply -f stats-filter-1.10.yaml

  7. Add the following configuration to the mesh config. This results in the addition of the request_operation as a new dimension to the istio_requests_total metric. Without it, a new metric with the name envoy_request_operation___somevalue___istio_requests_total is created.

    1. meshConfig:
    2.   defaultConfig:
    3.     extraStatTags:
    4.     - request_operation

  8. Generate metrics by sending traffic to your application.

  9. After the changes take effect, visit Prometheus and look for the new or changed dimensions, for example istio_requests_total.

Classify metrics by response

You can classify responses using a similar process as requests. Do note that the response_code dimension already exists by default. The example below will change how it is populated.

  1. Create a file, for example attribute_gen_service.yaml, and save it with the following contents. This adds the istio.attributegen plugin to the EnvoyFilter and generates the istio_responseClass attribute used by the stats plugin.

    This example classifies various responses, such as grouping all response codes in the 200 range as a 2xx dimension.

    1. apiVersion: networking.istio.io/v1alpha3
    2. kind: EnvoyFilter
    3. metadata:
    4.   name: istio-attributegen-filter
    5. spec:
    6.   workloadSelector:
    7.     labels:
    8.       app: productpage
    9.   configPatches:
    10.   - applyTo: HTTP_FILTER
    11.     match:
    12.       context: SIDECAR_INBOUND
    13.       proxy:
    14.         proxyVersion: '1\.9.*'
    15.       listener:
    16.         filterChain:
    17.           filter:
    18.             name: "envoy.http_connection_manager"
    19.             subFilter:
    20.               name: "istio.stats"
    21.     patch:
    22.       operation: INSERT_BEFORE
    23.       value:
    24.         name: istio.attributegen
    25.         typed_config:
    26.           "@type": type.googleapis.com/udpa.type.v1.TypedStruct
    27.           type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    28.           value:
    29.             config:
    30.               configuration:
    31.                 "@type": type.googleapis.com/google.protobuf.StringValue
    32.                 value: |
    33.                   {
    34.                     "attributes": [
    35.                       {
    36.                         "output_attribute": "istio_responseClass",
    37.                         "match": [
    38.                           {
    39.                             "value": "2xx",
    40.                             "condition": "response.code >= 200 && response.code <= 299"
    41.                           },
    42.                           {
    43.                             "value": "3xx",
    44.                             "condition": "response.code >= 300 && response.code <= 399"
    45.                           },
    46.                           {
    47.                             "value": "404",
    48.                             "condition": "response.code == 404"
    49.                           },
    50.                           {
    51.                             "value": "429",
    52.                             "condition": "response.code == 429"
    53.                           },
    54.                           {
    55.                             "value": "503",
    56.                             "condition": "response.code == 503"
    57.                           },
    58.                           {
    59.                             "value": "5xx",
    60.                             "condition": "response.code >= 500 && response.code <= 599"
    61.                           },
    62.                           {
    63.                             "value": "4xx",
    64.                             "condition": "response.code >= 400 && response.code <= 499"
    65.                           }
    66.                         ]
    67.                       }
    68.                     ]
    69.                   }
    70.               vm_config:
    71.                 runtime: envoy.wasm.runtime.null
    72.                 code:
    73.                   local: { inline_string: "envoy.wasm.attributegen" }

  2. Apply your changes using the following command:

    1. $ kubectl -n istio-system apply -f attribute_gen_service.yaml

  3. Find the stats-filter-1.10 EnvoyFilter resource from the istio-system namespace, using the following command:

    1. $ kubectl -n istio-system get envoyfilter | grep ^stats-filter-1.10
    2. stats-filter-1.10                    2d

  4. Create a local file system copy of the EnvoyFilter configuration, using the following command:

    1. $ kubectl -n istio-system get envoyfilter stats-filter-1.10 -o yaml > stats-filter-1.10.yaml

  5. Open stats-filter-1.10.yaml with a text editor and locate the name: istio.stats extension configuration. Update it to map response_code dimension in the requests_total standard metric to istio_responseClass attribute. The updated configuration file section should look like the following.

    1. name: istio.stats
    2. typed_config:
    3.   '@type': type.googleapis.com/udpa.type.v1.TypedStruct
    4.   type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    5.   value:
    6.     config:
    7.       configuration:
    8.         "@type": type.googleapis.com/google.protobuf.StringValue
    9.         value: |
    10.           {
    11.             "metrics": [
    12.              {
    13.                "name": "requests_total",
    14.                "dimensions": {
    15.                  "response_code": "istio_responseClass"
    16.                }
    17.              }]
    18.           }

  6. Save stats-filter-1.10.yaml and then apply the configuration using the following command:

    1. $ kubectl -n istio-system apply -f stats-filter-1.10.yaml

Verify the results

  1. Generate metrics by sending traffic to your application.

  2. Visit Prometheus and look for the new or changed dimensions, for example 2xx. Alternatively, use the following command to verify that Istio generates the data for your new dimension:

    1. $ kubectl exec pod-name -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus' | grep istio_

    In the output, locate the metric (e.g. istio_requests_total) and verify the presence of the new or changed dimension.

Troubleshooting

If classification does not occur as expected, check the following potential causes and resolutions.

Review the Envoy proxy logs for the pod that has the service on which you applied the configuration change. Check that there are no errors reported by the service in the Envoy proxy logs on the pod, (pod-name), where you configured classification by using the following command:

  1. $ kubectl logs pod-name -c istio-proxy | grep -e "Config Error" -e "envoy wasm"

Additionally, ensure that there are no Envoy proxy crashes by looking for signs of restarts in the output of the following command:

  1. $ kubectl get pods pod-name