MeshMetric

This policy uses new policy matching algorithm. Do not combine with Traffic Metrics.

Kuma facilitates consistent traffic metrics across all data plane proxies in your mesh.

You can define metrics configuration for a whole Mesh, and optionally tweak certain parts for individual data plane proxies. For example, you might need to override the default metrics port if it’s already in use on the specified machine.

Kuma provides full integration with Prometheus:

  • Each proxy can expose its metrics in Prometheus format.
  • Kuma exposes an API called the monitoring assignment service (MADS) which exposes proxies configured by MeshMetric.

Moreover, Kuma provides experimental integration with OpenTelemetry:

To collect metrics from Kuma, you need to expose metrics from proxies and applications.

In the rest of this page we assume you have already configured your observability tools to work with Kuma. If you haven’t already read the observability docs.

TargetRef support matrix

targetRefAllowed kinds
targetRef.kindMesh, MeshSubset, MeshService, MeshServiceSubset
targetRefAllowed kinds
targetRef.kindMesh, MeshGateway, MeshGateway with listener tags
targetRefAllowed kinds
targetRef.kindMesh, MeshSubset, MeshService, MeshServiceSubset

To learn more about the information in this table, see the matching docs.

Configuration

There are three main sections of the configuration: sidecar, applications, backends. The first two define how to scrape parts of the mesh (sidecar and underlying applications), the third one defines what to do with the data (in case of Prometheus instructs to scrape specific address, in case of OpenTelemetry defines where to push data).

In contrast to Traffic Metrics all configuration is dynamic and no restarts of the Data Plane Proxies are needed. You can define configuration refresh interval by using KUMA_DATAPLANE_RUNTIME_DYNAMIC_CONFIGURATION_REFRESH_INTERVAL env var or dataplaneRuntime.dynamicConfiguration.refreshInterval Helm value.

Sidecar

This part of the configuration applies to the data plane proxy scraping. In case you don’t want to retrieve all Envoy’s metrics, it’s possible to filter them.

Below are different methods of filtering. The order of the operations is as follows:

  1. Unused metrics
  2. Profiles
  3. Exclude
  4. Include

Unused metrics

By default, metrics that were not updated won’t be published. You can set the includeUnused flag that returns all metrics from Envoy.

Profiles

Profiles are predefined sets of metrics with manual include and exclude functionality. There are 3 sections:

  • appendProfiles - allows to combine multiple predefined profiles of metrics. Right now you can only define one profile but this might change it the future (e.g. there might be feature related profiles like “Fault injection profile” and “Circuit Breaker profile” so you can mix and match the ones that you need based on your features usage). Today only 3 profiles are available: All, Basic and None. All profile contains all metrics produced by Envoy. Basic profile contains all metrics needed by Kuma dashboards and golden 4 signals metrics. None profile removes all metrics
  • exclude - after profiles are applied you can manually exclude metrics on top of profile filtering.
  • include - after exclude is applied you can manually include metrics.

Examples

Include unused metrics of only Basic profile with manual exclude and include
  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: metrics-default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     sidecar:
  13.       includeUnused: true
  14.       profiles:
  15.         appendProfiles:
  16.         - name: Basic
  17.         exclude:
  18.         - type: Regex
  19.           match: envoy_cluster_external_upstream_rq_.*
  20.         include:
  21.         - type: Exact
  22.           match: envoy_cluster_default_total_match_count
  23.     backends:
  24.     - type: Prometheus
  25.       prometheus:
  26.         port: 5670
  27.         path: "/metrics"
  1. type: MeshMetric
  2. mesh: default
  3. name: metrics-default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     sidecar:
  9.       includeUnused: true
  10.       profiles:
  11.         appendProfiles:
  12.         - name: Basic
  13.         exclude:
  14.         - type: Regex
  15.           match: envoy_cluster_external_upstream_rq_.*
  16.         include:
  17.         - type: Exact
  18.           match: envoy_cluster_default_total_match_count
  19.     backends:
  20.     - type: Prometheus
  21.       prometheus:
  22.         port: 5670
  23.         path: "/metrics"
Include only manually defined metrics
  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: metrics-default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     sidecar:
  13.       profiles:
  14.         appendProfiles:
  15.         - name: None
  16.         include:
  17.         - type: Regex
  18.           match: envoy_cluster_external_upstream_rq_.*
  19.     backends:
  20.     - type: Prometheus
  21.       prometheus:
  22.         port: 5670
  23.         path: "/metrics"
  1. type: MeshMetric
  2. mesh: default
  3. name: metrics-default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     sidecar:
  9.       profiles:
  10.         appendProfiles:
  11.         - name: None
  12.         include:
  13.         - type: Regex
  14.           match: envoy_cluster_external_upstream_rq_.*
  15.     backends:
  16.     - type: Prometheus
  17.       prometheus:
  18.         port: 5670
  19.         path: "/metrics"
Include all metrics apart from one manually excluded
  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: metrics-default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     sidecar:
  13.       profiles:
  14.         appendProfiles:
  15.         - name: None
  16.         include:
  17.         - type: Regex
  18.           match: envoy_cluster_external_upstream_rq_.*
  19.     backends:
  20.     - type: Prometheus
  21.       prometheus:
  22.         port: 5670
  23.         path: "/metrics"
  1. type: MeshMetric
  2. mesh: default
  3. name: metrics-default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     sidecar:
  9.       profiles:
  10.         appendProfiles:
  11.         - name: None
  12.         include:
  13.         - type: Regex
  14.           match: envoy_cluster_external_upstream_rq_.*
  15.     backends:
  16.     - type: Prometheus
  17.       prometheus:
  18.         port: 5670
  19.         path: "/metrics"

Applications

Metrics exposed by the application need to be in Prometheus format for the Dataplane Proxy to be able to parse and expose them to either Prometheus or OpenTelemetry backend.

In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies. Kuma allows scraping Prometheus metrics from the applications endpoint running in the same Pod or VM. Later those metrics are aggregated and exposed at the same port/path as data plane proxy metrics. It is possible to configure it at the Mesh level, for all the applications in the Mesh, or just for specific applications.

Here are reasons where you’d want to use this feature:

  • Application metrics are labelled with your mesh parameters (tags, mesh, data plane name…), this means that in mixed Universal and Kubernetes mode metrics are reported with the same types of labels.
  • Both application and sidecar metrics are scraped at the same time. This makes sure they are coherent (with 2 different scrapers they can end up scraping at different intervals and make metrics harder to correlate).
  • If you disable passthrough and your mesh uses mTLS and Prometheus is outside the mesh this is the only way to retrieve these metrics as the app is completely hidden behind the sidecar.

Example section of the configuration:

  1. applications:
  2.   - name: "backend" # application name used for logging and to scope OpenTelemetry metrics (optional)
  3.     path: "/metrics/prometheus" # application metrics endpoint path
  4.     address: # optional custom address if the underlying application listens on a different address than the Data Plane Proxy
  5.     port: 8888 # port on which application is listening

Backends

Prometheus

  1. backends:
  2.   - type: Prometheus
  3.     prometheus: 
  4.       port: 5670
  5.       path: /metrics

This tells Kuma to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

The metrics endpoint is forwarded to the standard Envoy Prometheus metrics endpoint and supports the same query parameters. You can pass the filter query parameter to limit the results to metrics whose names match a given regular expression. By default, all available metrics are returned.

Secure metrics with TLS

Kuma allows configuring metrics endpoint with TLS.

  1. backends:
  2.   - type: Prometheus
  3.     prometheus: 
  4.       port: 5670
  5.       path: /metrics
  6.       tls:
  7.         mode: ProvidedTLS

In addition to the MeshMetric configuration, kuma-sidecar requires a provided certificate and key for its operation.

When the certificate and key are available within the container, kuma-sidecar needs the paths to provided files as the following environment variables:

  • KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH
  • KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH

It’s possible to use a ContainerPatch to add variables to kuma-sidecar:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: ContainerPatch
  3. metadata:
  4.   name: container-patch-1
  5.   namespace: kuma-system
  6. spec:
  7.   sidecarPatch:
  8.     - op: add
  9.       path: /env/-
  10.       value: '{
  11.           "name": "KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH",
  12.           "value": "/kuma/server.crt"
  13.         }'
  14.     - op: add
  15.       path: /env/-
  16.       value: '{
  17.           "name": "KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH",
  18.           "value": "/kuma/server.key"
  19.         }'

Please upload the certificate and the key to the machine, and then define the following environment variables with the correct paths:

  1. * `KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH`
  2. * `KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH`
activeMTLSBackend

We no longer support activeMTLSBackend, if you need to encrypt and authorize the metrics use Secure metrics with TLS with a combination of one of the authorization methods.

Running multiple prometheus deployments

If you need to run multiple instances of Prometheus and want to target different set of Data Plane Proxies you can do this by using Client ID setting on both MeshMetric (clientId) and Prometheus configuration (client_id).

Support for clientId was added in Prometheus version 2.50.0.

Example Prometheus configuration

Let’s assume we have two prometheus deployments main and secondary. We would like to use each of them to monitor different sets of data plane proxies, with different tags.

We can start with configuring each Prometheus deployments to use Kuma SD. Prometheus’s deployments will be differentiated by client_id parameter.

Main Prometheus config:

  1. scrape_configs:
  2.   - job_name: 'kuma-dataplanes'
  3.     # ...
  4.     kuma_sd_configs:
  5.     - server: http://kuma-control-plane.kuma-system:5676
  6.       refresh_interval: 60s # different from prometheus-secondary
  7.       client_id: "prometheus-main" # Kuma will use this to pick proper data plane proxies

Secondary Prometheus config:

  1. scrape_configs:
  2.   - job_name: 'kuma-dataplanes'
  3.     # ...
  4.     kuma_sd_configs:
  5.       - server: http://kuma-control-plane.kuma-system:5676
  6.         refresh_interval: 20s # different from prometheus-main
  7.         client_id: "prometheus-secondary"

Now we can configure first MeshMetric policy to pick data plane proxies with tag prometheus: main for main Prometheus discovery. clientId in policy should be the same as client_id in Prometheus configuration.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: prometheus-one
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     tags:
  12.       prometheus: main
  13.   default:
  14.     backends:
  15.     - type: Prometheus
  16.       prometheus:
  17.         clientId: prometheus-main
  18.         port: 5670
  19.         path: "/metrics"
  1. type: MeshMetric
  2. name: prometheus-one
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       prometheus: main
  9.   default:
  10.     backends:
  11.     - type: Prometheus
  12.       prometheus:
  13.         clientId: prometheus-main
  14.         port: 5670
  15.         path: "/metrics"

And policy for secondary Prometheus deployment that will pick dataplane proxies with tag prometheus: secondary.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: prometheus-two
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     tags:
  12.       prometheus: secondary
  13.   default:
  14.     backends:
  15.     - type: Prometheus
  16.       prometheus:
  17.         clientId: prometheus-secondary
  18.         port: 5670
  19.         path: "/metrics"
  1. type: MeshMetric
  2. name: prometheus-two
  3. mesh: default
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       prometheus: secondary
  9.   default:
  10.     backends:
  11.     - type: Prometheus
  12.       prometheus:
  13.         clientId: prometheus-secondary
  14.         port: 5670
  15.         path: "/metrics"

OpenTelemetry

  1. backends:
  2.   - type: OpenTelemetry
  3.     openTelemetry: 
  4.       endpoint: otel-collector.observability.svc:4317
  5.       refreshInterval: 60s

This configuration tells Kuma Dataplane Proxy to push metrics to OpenTelemetry collector. Dataplane Proxy will scrape metrics from Envoy and other applications in a Pod/VM and push them to configured OpenTelemetry collector, by default every 60 seconds (use refreshInterval to change it).

When you configure application scraping make sure to specify application.name to utilize OpenTelemetry scoping.

Pushing metrics from application to OpenTelemetry collector directly

Right now if you want to expose metrics from your application to OpenTelemetry collector you can access collector directly.

If you have disabled passthrough in your Mesh you need to configure ExternalService with you collector endpoint. Example ExternalService:

  1. apiVersion: kuma.io/v1alpha1
  2. kind: ExternalService
  3. mesh: default
  4. metadata:
  5.   name: otel-collector
  6. spec:
  7.   tags:
  8.     kuma.io/service: otel-collector-grpc
  9.     kuma.io/protocol: grpc
  10.   networking:
  11.     address: otel-collector.observability.svc.cluster.local:4317
  1. type: ExternalService
  2. mesh: default
  3. name: otel-collector
  4. tags:
  5.   kuma.io/service: otel-collector-grpc
  6.   kuma.io/protocol: grpc
  7. networking:
  8.   address: otel-collector.observability.svc.cluster.local:4317

Examples

With custom port, path, clientId, application aggregation and service override

The first policy defines a default MeshMetric policy for the default mesh. The second policy creates an override for workloads tagged with framework: example-web-framework. That web framework exposes metrics under /metrics/prometheus and port 8888.

  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: metrics-default
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: Mesh
  11.   default:
  12.     sidecar:
  13.       includeUnused: false
  14.     backends:
  15.     - type: Prometheus
  16.       prometheus:
  17.         clientId: main-backend
  18.         port: 5670
  19.         path: "/metrics"
  20.         tls:
  21.           mode: ProvidedTLS
  1. type: MeshMetric
  2. mesh: default
  3. name: metrics-default
  4. spec:
  5.   targetRef:
  6.     kind: Mesh
  7.   default:
  8.     sidecar:
  9.       includeUnused: false
  10.     backends:
  11.     - type: Prometheus
  12.       prometheus:
  13.         clientId: main-backend
  14.         port: 5670
  15.         path: "/metrics"
  16.         tls:
  17.           mode: ProvidedTLS
  1. apiVersion: kuma.io/v1alpha1
  2. kind: MeshMetric
  3. metadata:
  4.   name: metrics-for-mesh-service
  5.   namespace: kuma-system
  6.   labels:
  7.     kuma.io/mesh: default
  8. spec:
  9.   targetRef:
  10.     kind: MeshSubset
  11.     tags:
  12.       framework: example-web-framework
  13.   default:
  14.     applications:
  15.     - path: "/metrics/prometheus"
  16.       port: 8888
  1. type: MeshMetric
  2. mesh: default
  3. name: metrics-for-mesh-service
  4. spec:
  5.   targetRef:
  6.     kind: MeshSubset
  7.     tags:
  8.       framework: example-web-framework
  9.   default:
  10.     applications:
  11.     - path: "/metrics/prometheus"
  12.       port: 8888

All policy options