Configure metrics

Enable or disable Dapr metrics

By default, each Dapr system process emits Go runtime/process metrics and has their own Dapr metrics.

Prometheus endpoint

The Dapr sidecars exposes a Prometheus metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.

Configuring metrics using the CLI

The metrics application endpoint is enabled by default. You can disable it by passing the command line argument --enable-metrics=false.

The default metrics port is 9090. You can override this by passing the command line argument --metrics-port to Daprd.

Configuring metrics in Kubernetes

You can also enable/disable the metrics for a specific application by setting the dapr.io/enable-metrics: "false" annotation on your application deployment. With the metrics exporter disabled, daprd does not open the metrics listening port.

The following Kubernetes deployment example shows how metrics are explicitly enabled with the port specified as “9090”.

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nodeapp
  5.   labels:
  6.     app: node
  7. spec:
  8.   replicas: 1
  9.   selector:
  10.     matchLabels:
  11.       app: node
  12.   template:
  13.     metadata:
  14.       labels:
  15.         app: node
  16.       annotations:
  17.         dapr.io/enabled: "true"
  18.         dapr.io/app-id: "nodeapp"
  19.         dapr.io/app-port: "3000"
  20.         dapr.io/enable-metrics: "true"
  21.         dapr.io/metrics-port: "9090"
  22.     spec:
  23.       containers:
  24.       - name: node
  25.         image: dapriosamples/hello-k8s-node:latest
  26.         ports:
  27.         - containerPort: 3000
  28.         imagePullPolicy: Always

Configuring metrics using application configuration

You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars running in a specific namespace:

  • Use the metrics spec configuration.
  • Set enabled: false to disable the metrics in the Dapr runtime.
  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: tracing
  5.   namespace: default
  6. spec:
  7.   tracing:
  8.     samplingRate: "1"
  9.   metrics:
  10.     enabled: false

High cardinality metrics

Depending on your use case, some metrics emitted by Dapr might contain values that have a high cardinality. This might cause increased memory usage for the Dapr process/container and incur expensive egress costs in certain cloud environments. To mitigate this issue, you can set regular expressions for every metric exposed by the Dapr sidecar. See a list of all Dapr metrics.

The following example shows how to apply a regular expression for the label method in the metric dapr_runtime_service_invocation_req_sent_total:

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: daprConfig
  5. spec:
  6.   metric:
  7.       enabled: true
  8.       rules:
  9.       - name: dapr_runtime_service_invocation_req_sent_total
  10.         labels:
  11.         - name: method
  12.           regex:
  13.             "orders/": "orders/.+"

When this configuration is applied, a recorded metric with the method label of orders/a746dhsk293972nz will be replaced with orders/.

Watch the demo

Watch this video to walk through handling high cardinality metrics:

References

Last modified June 19, 2023: Merge pull request #3565 from dapr/aacrawfi/skip-secrets-close (b1763bf)