How-To: Set up Azure Monitor to search logs and collect metrics

Enable Dapr metrics and logs with Azure Monitor for Azure Kubernetes Service (AKS)

Prerequisites

Enable Prometheus metric scrape using config map

  1. Make sure that Azure Monitor Agents (AMA) are running.

    1. $ kubectl get pods -n kube-system
    2. NAME                                                  READY   STATUS    RESTARTS   AGE
    3. ...
    4. ama-logs-48kpv                                        2/2     Running   0          2d13h
    5. ama-logs-mx24c                                        2/2     Running   0          2d13h
    6. ama-logs-rs-f9bbb9898-vbt6k                           1/1     Running   0          30h
    7. ama-logs-sm2mz                                        2/2     Running   0          2d13h
    8. ama-logs-z7p4c                                        2/2     Running   0          2d13h
    9. ...

  2. Apply config map to enable Prometheus metrics endpoint scrape.

You can use azm-config-map.yaml to enable Prometheus metrics endpoint scrape.

If you installed Dapr to a different namespace, you need to change the monitor_kubernetes_pod_namespaces array values. For example:

  1. ...
  2.   prometheus-data-collection-settings: |-
  3.     [prometheus_data_collection_settings.cluster]
  4.         interval = "1m"
  5.         monitor_kubernetes_pods = true
  6.         monitor_kubernetes_pods_namespaces = ["dapr-system", "default"]
  7.     [prometheus_data_collection_settings.node]
  8.         interval = "1m"    
  9. ...

Apply config map:

  1. kubectl apply -f ./azm-config.map.yaml

Install Dapr with JSON formatted logs

  1. Install Dapr with enabling JSON-formatted logs.

    1. helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true

  2. Enable JSON formatted log in Dapr sidecar and add Prometheus annotations.

Note: The Azure Monitor Agents (AMA) only sends the metrics if the Prometheus annotations are set.

Add dapr.io/log-as-json: "true" annotation to your deployment yaml.

Example:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: pythonapp
  5.   namespace: default
  6.   labels:
  7.     app: python
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: python
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: python
  17.       annotations:
  18.         dapr.io/enabled: "true"
  19.         dapr.io/app-id: "pythonapp"
  20.         dapr.io/log-as-json: "true"
  21.         prometheus.io/scrape: "true"
  22.         prometheus.io/port: "9090"
  23.         prometheus.io/path: "/"
  25. ...

Search metrics and logs with Azure Monitor

  1. Go to Azure Monitor in the Azure portal.

  2. Search Dapr Logs.

Here is an example query, to parse JSON formatted logs and query logs from Dapr system processes.

  1. ContainerLog
  2. | extend parsed=parse_json(LogEntry)
  3. | project Time=todatetime(parsed['time']), app_id=parsed['app_id'], scope=parsed['scope'],level=parsed['level'], msg=parsed['msg'], type=parsed['type'], ver=parsed['ver'], instance=parsed['instance']
  4. | where level != ""
  5. | sort by Time
  1. Search Metrics.

This query, queries process_resident_memory_bytes Prometheus metrics for Dapr system processes and renders timecharts.

  1. InsightsMetrics
  2. | where Namespace == "prometheus" and Name == "process_resident_memory_bytes"
  3. | extend tags=parse_json(Tags)
  4. | project TimeGenerated, Name, Val, app=tostring(tags['app'])
  5. | summarize memInBytes=percentile(Val, 99) by bin(TimeGenerated, 1m), app
  6. | where app startswith "dapr-"
  7. | render timechart

References

Last modified June 19, 2023: Merge pull request #3565 from dapr/aacrawfi/skip-secrets-close (b1763bf)