Integrate OSM with your own Prometheus and Grafana stack

The following article shows you how to create an example bring your own (BYO) Prometheus and Grafana stack on your cluster and configure that stack for observability and monitoring of OSM. For an example using an automatic provisioning of a Prometheus and Grafana stack with OSM, see the Observability getting started guide.

IMPORTANT: The configuration created in this article should not be used in production environments. For production-grade deployments, see Prometheus Operator and Deploy Grafana in Kubernetes.

Prerequisites

  • Kubernetes cluster running Kubernetes v1.22.9 or greater.
  • OSM installed on the Kubernetes cluster.
  • kubectl installed and access to the cluster’s API server.
  • osm CLI installed.
  • helm CLI installed.

Deploy an example Prometheus instance

Use helm to deploy a Prometheus instance to your cluster in the default namespace.

  1. helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  2. helm repo update
  3. helm install stable prometheus-community/prometheus

The output of the helm install command contains the DNS name of the Prometheus server. For example:

  1. ...
  2. The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
  3. stable-prometheus-server.metrics.svc.cluster.local
  4. ...

Record this DNS name for use in a later step.

Configure Prometheus for OSM

Prometheus needs to be configured to scape the OSM endpoints and properly handle OSM’s labeling, relabelling, and endpoint configuration. This configuration also helps the OSM Grafana dashboards, which are configured in a later step, properly display the data scraped from OSM.

Use kubectl get configmap to verify the stable-prometheus-sever configmap has been created. For example:

  1. $ kubectl get configmap
  3. NAME                             DATA   AGE
  4. ...
  5. stable-prometheus-alertmanager   1      18m
  6. stable-prometheus-server         5      18m
  7. ...

Create update-prometheus-configmap.yaml with the following:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: stable-prometheus-server
  5. data:
  6.   prometheus.yml: |
  7.     global:
  8.       scrape_interval: 10s
  9.       scrape_timeout: 10s
  10.       evaluation_interval: 1m
  12.     scrape_configs:
  13.       - job_name: 'kubernetes-apiservers'
  14.         kubernetes_sd_configs:
  15.         - role: endpoints
  16.         scheme: https
  17.         tls_config:
  18.           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  19.           # TODO need to remove this when the CA and SAN match
  20.           insecure_skip_verify: true
  21.         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  22.         metric_relabel_configs:
  23.         - source_labels: [__name__]
  24.           regex: '(apiserver_watch_events_total|apiserver_admission_webhook_rejection_count)'
  25.           action: keep
  26.         relabel_configs:
  27.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  28.           action: keep
  29.           regex: default;kubernetes;https
  31.       - job_name: 'kubernetes-nodes'
  32.         scheme: https
  33.         tls_config:
  34.           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  35.         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         kubernetes_sd_configs:
  37.         - role: node
  38.         relabel_configs:
  39.         - action: labelmap
  40.           regex: __meta_kubernetes_node_label_(.+)
  41.         - target_label: __address__
  42.           replacement: kubernetes.default.svc:443
  43.         - source_labels: [__meta_kubernetes_node_name]
  44.           regex: (.+)
  45.           target_label: __metrics_path__
  46.           replacement: /api/v1/nodes/${1}/proxy/metrics
  48.       - job_name: 'kubernetes-pods'
  49.         kubernetes_sd_configs:
  50.         - role: pod
  51.         metric_relabel_configs:
  52.         - source_labels: [__name__]
  53.           regex: '(envoy_server_live|envoy_cluster_health_check_.*|envoy_cluster_upstream_rq_xx|envoy_cluster_upstream_cx_active|envoy_cluster_upstream_cx_tx_bytes_total|envoy_cluster_upstream_cx_rx_bytes_total|envoy_cluster_upstream_rq_total|envoy_cluster_upstream_cx_destroy_remote_with_active_rq|envoy_cluster_upstream_cx_connect_timeout|envoy_cluster_upstream_cx_destroy_local_with_active_rq|envoy_cluster_upstream_rq_pending_failure_eject|envoy_cluster_upstream_rq_pending_overflow|envoy_cluster_upstream_rq_timeout|envoy_cluster_upstream_rq_rx_reset|^osm.*)'
  54.           action: keep
  55.         relabel_configs:
  56.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  57.           action: keep
  58.           regex: true
  59.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
  60.           action: replace
  61.           target_label: __metrics_path__
  62.           regex: (.+)
  63.         - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  64.           action: replace
  65.           regex: ([^:]+)(?::\d+)?;(\d+)
  66.           replacement: $1:$2
  67.           target_label: __address__
  68.         - source_labels: [__meta_kubernetes_namespace]
  69.           action: replace
  70.           target_label: source_namespace
  71.         - source_labels: [__meta_kubernetes_pod_name]
  72.           action: replace
  73.           target_label: source_pod_name
  74.         - regex: '(__meta_kubernetes_pod_label_app)'
  75.           action: labelmap
  76.           replacement: source_service
  77.         - regex: '(__meta_kubernetes_pod_label_osm_envoy_uid|__meta_kubernetes_pod_label_pod_template_hash|__meta_kubernetes_pod_label_version)'
  78.           action: drop
  79.         # for non-ReplicaSets (DaemonSet, StatefulSet)
  80.         # __meta_kubernetes_pod_controller_kind=DaemonSet
  81.         # __meta_kubernetes_pod_controller_name=foo
  82.         # =>
  83.         # workload_kind=DaemonSet
  84.         # workload_name=foo
  85.         - source_labels: [__meta_kubernetes_pod_controller_kind]
  86.           action: replace
  87.           target_label: source_workload_kind
  88.         - source_labels: [__meta_kubernetes_pod_controller_name]
  89.           action: replace
  90.           target_label: source_workload_name
  91.         # for ReplicaSets
  92.         # __meta_kubernetes_pod_controller_kind=ReplicaSet
  93.         # __meta_kubernetes_pod_controller_name=foo-bar-123
  94.         # =>
  95.         # workload_kind=Deployment
  96.         # workload_name=foo-bar
  97.         # deplyment=foo
  98.         - source_labels: [__meta_kubernetes_pod_controller_kind]
  99.           action: replace
  100.           regex: ^ReplicaSet$
  101.           target_label: source_workload_kind
  102.           replacement: Deployment
  103.         - source_labels:
  104.           - __meta_kubernetes_pod_controller_kind
  105.           - __meta_kubernetes_pod_controller_name
  106.           action: replace
  107.           regex: ^ReplicaSet;(.*)-[^-]+$
  108.           target_label: source_workload_name
  110.       - job_name: 'smi-metrics'
  111.         kubernetes_sd_configs:
  112.         - role: pod
  113.         relabel_configs:
  114.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  115.           action: keep
  116.           regex: true
  117.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
  118.           action: replace
  119.           target_label: __metrics_path__
  120.           regex: (.+)
  121.         - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  122.           action: replace
  123.           regex: ([^:]+)(?::\d+)?;(\d+)
  124.           replacement: $1:$2
  125.           target_label: __address__
  126.         metric_relabel_configs:
  127.         - source_labels: [__name__]
  128.           regex: 'envoy_.*osm_request_(total|duration_ms_(bucket|count|sum))'
  129.           action: keep
  130.         - source_labels: [__name__]
  131.           action: replace
  132.           regex: envoy_response_code_(\d{3})_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  133.           target_label: response_code
  134.         - source_labels: [__name__]
  135.           action: replace
  136.           regex: envoy_response_code_\d{3}_source_namespace_(.*)_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  137.           target_label: source_namespace
  138.         - source_labels: [__name__]
  139.           action: replace
  140.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_(.*)_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  141.           target_label: source_kind
  142.         - source_labels: [__name__]
  143.           action: replace
  144.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_(.*)_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  145.           target_label: source_name
  146.         - source_labels: [__name__]
  147.           action: replace
  148.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_(.*)_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  149.           target_label: source_pod
  150.         - source_labels: [__name__]
  151.           action: replace
  152.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_(.*)_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  153.           target_label: destination_namespace
  154.         - source_labels: [__name__]
  155.           action: replace
  156.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_(.*)_destination_name_.*_destination_pod_.*_osm_request_total
  157.           target_label: destination_kind
  158.         - source_labels: [__name__]
  159.           action: replace
  160.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_(.*)_destination_pod_.*_osm_request_total
  161.           target_label: destination_name
  162.         - source_labels: [__name__]
  163.           action: replace
  164.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_(.*)_osm_request_total
  165.           target_label: destination_pod
  166.         - source_labels: [__name__]
  167.           action: replace
  168.           regex: .*(osm_request_total)
  169.           target_label: __name__
  171.         - source_labels: [__name__]
  172.           action: replace
  173.           regex: envoy_source_namespace_(.*)_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  174.           target_label: source_namespace
  175.         - source_labels: [__name__]
  176.           action: replace
  177.           regex: envoy_source_namespace_.*_source_kind_(.*)_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  178.           target_label: source_kind
  179.         - source_labels: [__name__]
  180.           action: replace
  181.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_(.*)_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  182.           target_label: source_name
  183.         - source_labels: [__name__]
  184.           action: replace
  185.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_(.*)_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  186.           target_label: source_pod
  187.         - source_labels: [__name__]
  188.           action: replace
  189.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_(.*)_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  190.           target_label: destination_namespace
  191.         - source_labels: [__name__]
  192.           action: replace
  193.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_(.*)_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  194.           target_label: destination_kind
  195.         - source_labels: [__name__]
  196.           action: replace
  197.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_(.*)_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  198.           target_label: destination_name
  199.         - source_labels: [__name__]
  200.           action: replace
  201.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_(.*)_osm_request_duration_ms_(bucket|sum|count)
  202.           target_label: destination_pod
  203.         - source_labels: [__name__]
  204.           action: replace
  205.           regex: .*(osm_request_duration_ms_(bucket|sum|count))
  206.           target_label: __name__
  208.       - job_name: 'kubernetes-cadvisor'
  209.         scheme: https
  210.         tls_config:
  211.           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  212.         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  213.         kubernetes_sd_configs:
  214.         - role: node
  215.         metric_relabel_configs:
  216.         - source_labels: [__name__]
  217.           regex: '(container_cpu_usage_seconds_total|container_memory_rss)'
  218.           action: keep
  219.         relabel_configs:
  220.         - action: labelmap
  221.           regex: __meta_kubernetes_node_label_(.+)
  222.         - target_label: __address__
  223.           replacement: kubernetes.default.svc:443
  224.         - source_labels: [__meta_kubernetes_node_name]
  225.           regex: (.+)
  226.           target_label: __metrics_path__
  227.           replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

Use kubectl apply to update the Prometheus server configmap.

  1. kubectl apply -f update-prometheus-configmap.yaml

Verify Prometheus is able to scrape the OSM mesh and API endpoints by using kubectl port-forward to forward the traffic between the Prometheus management application and your development computer.

  1. export POD_NAME=$(kubectl get pods -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  2. kubectl port-forward $POD_NAME 9090

Open a web browser to http://localhost:9090/targets to access the Prometheus management application and verify the endpoints are connected, up, and scrapping is running.

Integrate OSM with Prometheus and Grafana - 图1

Targets with specific relabeling config established by OSM should be “up”

Stop the port-forwarding command.

Deploying a Grafana Instance

Use helm to deploy a Grafana instance to your cluster in the default namespace.

  1. helm repo add grafana https://grafana.github.io/helm-charts
  2. helm repo update
  3. helm install grafana/grafana --generate-name

Use kubectl get secret to display the administrator password for Grafana.

  1. export SECRET_NAME=$(kubectl get secret -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}")
  2. kubectl get secret $SECRET_NAME -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Use kubectl port-forward to forward the traffic between the Grafana’s management application and your development computer.

  1. export POD_NAME=$(kubectl get pods -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}")
  2. kubectl port-forward $POD_NAME 3000

Open a web browser to http://localhost:3000 to access the Grafana’s management application. Use admin as the username and administrator password from the previous step. and verify the endpoints are connected, up, and scrapping is running.

From the management application:

  • Select Settings then Data Sources.
  • Select Add data source.
  • Find the Prometheus data source and select Select.
  • Enter the DNS name, for example stable-prometheus-server.default.svc.cluster.local, from the earlier step in URL.

Select Save and Test and confirm you see Data source is working.

Importing OSM Dashboards

OSM Dashboards are available through OSM GitHub repository, which can be imported as json blobs on the management application.

To import a dashboard:

  • Hover your cursor over the + and select Import.
  • Copy the JSON from the osm-mesh-envoy-details dashboard and paste it in Import via panel json.
  • Select Load.
  • Select Import.

Confirm you see a Mesh and Envoy Details dashboard created.