How-To: Observe metrics with Prometheus

Use Prometheus to collect time-series data relating to the execution of the Dapr runtime itself

Setup Prometheus Locally

To run Prometheus on your local machine, you can either install and run it as a process or run it as a Docker container.

Install

Note

You don’t need to install Prometheus if you plan to run it as a Docker container. Please refer to the Container instructions.

To install Prometheus, follow the steps outlined here for your OS.

Configure

Now you’ve installed Prometheus, you need to create a configuration.

Below is an example Prometheus configuration, save this to a file i.e. /tmp/prometheus.yml or C:\Temp\prometheus.yml

  1. global:
  2.   scrape_interval:     15s # By default, scrape targets every 15 seconds.
  4. # A scrape configuration containing exactly one endpoint to scrape:
  5. # Here it's Prometheus itself.
  6. scrape_configs:
  7.   - job_name: 'dapr'
  9.     # Override the global default and scrape targets from this job every 5 seconds.
  10.     scrape_interval: 5s
  12.     static_configs:
  13.       - targets: ['localhost:9090'] # Replace with Dapr metrics port if not default

Run as Process

Run Prometheus with your configuration to start it collecting metrics from the specified targets.

  1. ./prometheus --config.file=/tmp/prometheus.yml --web.listen-address=:8080

We change the port so it doesn’t conflict with Dapr’s own metrics endpoint.

If you are not currently running a Dapr application, the target will show as offline. In order to start collecting metrics you must start Dapr with the metrics port matching the one provided as the target in the configuration.

Once Prometheus is running, you’ll be able to visit its dashboard by visiting http://localhost:8080.

Run as Container

To run Prometheus as a Docker container on your local machine, first ensure you have Docker installed and running.

Then you can run Prometheus as a Docker container using:

  1. docker run \
  2.     --net=host \
  3.     -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \
  4.     prom/prometheus --config.file=/etc/prometheus/prometheus.yml --web.listen-address=:8080

--net=host ensures that the Prometheus instance will be able to connect to any Dapr instances running on the host machine. If you plan to run your Dapr apps in containers as well, you’ll need to run them on a shared Docker network and update the configuration with the correct target address.

Once Prometheus is running, you’ll be able to visit its dashboard by visiting http://localhost:8080.

Setup Prometheus on Kubernetes

Prerequisites

Install Prometheus

  1. First create namespace that can be used to deploy the Grafana and Prometheus monitoring tools
  1. kubectl create namespace dapr-monitoring
  1. Install Prometheus
  1. helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  2. helm repo update
  3. helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring

If you are Minikube user or want to disable persistent volume for development purposes, you can disable it by using the following command.

  1. helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring
  2.  --set alertmanager.persistence.enabled=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false
  1. Validation

Ensure Prometheus is running in your cluster.

  1. kubectl get pods -n dapr-monitoring
  3. NAME                                                READY   STATUS    RESTARTS   AGE
  4. dapr-prom-kube-state-metrics-9849d6cc6-t94p8        1/1     Running   0          4m58s
  5. dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8   2/2     Running   0          4m58s
  6. dapr-prom-prometheus-node-exporter-5jh8p            1/1     Running   0          4m58s
  7. dapr-prom-prometheus-node-exporter-88gbg            1/1     Running   0          4m58s
  8. dapr-prom-prometheus-node-exporter-bjp9f            1/1     Running   0          4m58s
  9. dapr-prom-prometheus-pushgateway-688665d597-h4xx2   1/1     Running   0          4m58s
  10. dapr-prom-prometheus-server-694fd8d7c-q5d59         2/2     Running   0          4m58s

Example

References

Last modified June 19, 2023: Merge pull request #3565 from dapr/aacrawfi/skip-secrets-close (b1763bf)