将 OSM 与自维护的 Prometheus 和 Grafana 集成

本文为你展示如何在集群中创建自行维护的 Prometheus 和 Grafana ，并为其配置以实现对 OSM 的可观测性和监控。有关使用 OSM 自动配置 Prometheus 和 Grafana 的示例，请参阅 Observability 入门指南。

重要提示：本文创建的配置不应在生产环境中使用。对于生产级部署，请参阅 Prometheus Operator 和 Deploy Grafana in Kubernetes。

先决条件

  • Kubernetes 集群运行版本 v1.22.9 或者更高。
  • 集群中已安装 OSM。
  • 已安装 kubectl 用于访问 API server 。
  • 已安装 osm 命令行工具
  • 已安装 helm 命令行工具

部署示例化的 Prometheus 实例

在 default 命名空间下，使用 helm 安装 Prometheus 实例。

  1. helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  2. helm repo update
  3. helm install stable prometheus-community/prometheus

helm install 命令的输出中包含了 Prometheus 服务端的 DNS 链接。例如：

  1. ...
  2. The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
  3. stable-prometheus-server.metrics.svc.cluster.local
  4. ...

记下 DNS 链接，后面步骤会用到。

为 OSM 配置 Prometheus

Prometheus 需要对 OSM 端点进行抓取配置，并对应地处理 OSM 的标记、重新标记和端点配置。此配置还有助于 OSM Grafana 仪表板正确显示从 OSM 抓取的数据（OSM Grafana 仪表板会在后续步骤中配置）

使用 kubectl get configmap 来验证 stable-prometheus-server configmap 是否已创建。 例如：

  1. $ kubectl get configmap
  3. NAME                             DATA   AGE
  4. ...
  5. stable-prometheus-alertmanager   1      18m
  6. stable-prometheus-server         5      18m
  7. ...

使用以下内容创建 update-prometheus-configmap.yaml：

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: stable-prometheus-server
  5. data:
  6.   prometheus.yml: |
  7.     global:
  8.       scrape_interval: 10s
  9.       scrape_timeout: 10s
  10.       evaluation_interval: 1m
  12.     scrape_configs:
  13.       - job_name: 'kubernetes-apiservers'
  14.         kubernetes_sd_configs:
  15.         - role: endpoints
  16.         scheme: https
  17.         tls_config:
  18.           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  19.           # TODO need to remove this when the CA and SAN match
  20.           insecure_skip_verify: true
  21.         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  22.         metric_relabel_configs:
  23.         - source_labels: [__name__]
  24.           regex: '(apiserver_watch_events_total|apiserver_admission_webhook_rejection_count)'
  25.           action: keep
  26.         relabel_configs:
  27.         - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  28.           action: keep
  29.           regex: default;kubernetes;https
  31.       - job_name: 'kubernetes-nodes'
  32.         scheme: https
  33.         tls_config:
  34.           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  35.         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  36.         kubernetes_sd_configs:
  37.         - role: node
  38.         relabel_configs:
  39.         - action: labelmap
  40.           regex: __meta_kubernetes_node_label_(.+)
  41.         - target_label: __address__
  42.           replacement: kubernetes.default.svc:443
  43.         - source_labels: [__meta_kubernetes_node_name]
  44.           regex: (.+)
  45.           target_label: __metrics_path__
  46.           replacement: /api/v1/nodes/${1}/proxy/metrics
  48.       - job_name: 'kubernetes-pods'
  49.         kubernetes_sd_configs:
  50.         - role: pod
  51.         metric_relabel_configs:
  52.         - source_labels: [__name__]
  53.           regex: '(envoy_server_live|envoy_cluster_health_check_.*|envoy_cluster_upstream_rq_xx|envoy_cluster_upstream_cx_active|envoy_cluster_upstream_cx_tx_bytes_total|envoy_cluster_upstream_cx_rx_bytes_total|envoy_cluster_upstream_rq_total|envoy_cluster_upstream_cx_destroy_remote_with_active_rq|envoy_cluster_upstream_cx_connect_timeout|envoy_cluster_upstream_cx_destroy_local_with_active_rq|envoy_cluster_upstream_rq_pending_failure_eject|envoy_cluster_upstream_rq_pending_overflow|envoy_cluster_upstream_rq_timeout|envoy_cluster_upstream_rq_rx_reset|^osm.*)'
  54.           action: keep
  55.         relabel_configs:
  56.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  57.           action: keep
  58.           regex: true
  59.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
  60.           action: replace
  61.           target_label: __metrics_path__
  62.           regex: (.+)
  63.         - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  64.           action: replace
  65.           regex: ([^:]+)(?::\d+)?;(\d+)
  66.           replacement: $1:$2
  67.           target_label: __address__
  68.         - source_labels: [__meta_kubernetes_namespace]
  69.           action: replace
  70.           target_label: source_namespace
  71.         - source_labels: [__meta_kubernetes_pod_name]
  72.           action: replace
  73.           target_label: source_pod_name
  74.         - regex: '(__meta_kubernetes_pod_label_app)'
  75.           action: labelmap
  76.           replacement: source_service
  77.         - regex: '(__meta_kubernetes_pod_label_osm_envoy_uid|__meta_kubernetes_pod_label_pod_template_hash|__meta_kubernetes_pod_label_version)'
  78.           action: drop
  79.         # for non-ReplicaSets (DaemonSet, StatefulSet)
  80.         # __meta_kubernetes_pod_controller_kind=DaemonSet
  81.         # __meta_kubernetes_pod_controller_name=foo
  82.         # =>
  83.         # workload_kind=DaemonSet
  84.         # workload_name=foo
  85.         - source_labels: [__meta_kubernetes_pod_controller_kind]
  86.           action: replace
  87.           target_label: source_workload_kind
  88.         - source_labels: [__meta_kubernetes_pod_controller_name]
  89.           action: replace
  90.           target_label: source_workload_name
  91.         # for ReplicaSets
  92.         # __meta_kubernetes_pod_controller_kind=ReplicaSet
  93.         # __meta_kubernetes_pod_controller_name=foo-bar-123
  94.         # =>
  95.         # workload_kind=Deployment
  96.         # workload_name=foo-bar
  97.         # deplyment=foo
  98.         - source_labels: [__meta_kubernetes_pod_controller_kind]
  99.           action: replace
  100.           regex: ^ReplicaSet$
  101.           target_label: source_workload_kind
  102.           replacement: Deployment
  103.         - source_labels:
  104.           - __meta_kubernetes_pod_controller_kind
  105.           - __meta_kubernetes_pod_controller_name
  106.           action: replace
  107.           regex: ^ReplicaSet;(.*)-[^-]+$
  108.           target_label: source_workload_name
  110.       - job_name: 'smi-metrics'
  111.         kubernetes_sd_configs:
  112.         - role: pod
  113.         relabel_configs:
  114.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  115.           action: keep
  116.           regex: true
  117.         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
  118.           action: replace
  119.           target_label: __metrics_path__
  120.           regex: (.+)
  121.         - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  122.           action: replace
  123.           regex: ([^:]+)(?::\d+)?;(\d+)
  124.           replacement: $1:$2
  125.           target_label: __address__
  126.         metric_relabel_configs:
  127.         - source_labels: [__name__]
  128.           regex: 'envoy_.*osm_request_(total|duration_ms_(bucket|count|sum))'
  129.           action: keep
  130.         - source_labels: [__name__]
  131.           action: replace
  132.           regex: envoy_response_code_(\d{3})_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  133.           target_label: response_code
  134.         - source_labels: [__name__]
  135.           action: replace
  136.           regex: envoy_response_code_\d{3}_source_namespace_(.*)_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  137.           target_label: source_namespace
  138.         - source_labels: [__name__]
  139.           action: replace
  140.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_(.*)_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  141.           target_label: source_kind
  142.         - source_labels: [__name__]
  143.           action: replace
  144.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_(.*)_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  145.           target_label: source_name
  146.         - source_labels: [__name__]
  147.           action: replace
  148.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_(.*)_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  149.           target_label: source_pod
  150.         - source_labels: [__name__]
  151.           action: replace
  152.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_(.*)_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_total
  153.           target_label: destination_namespace
  154.         - source_labels: [__name__]
  155.           action: replace
  156.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_(.*)_destination_name_.*_destination_pod_.*_osm_request_total
  157.           target_label: destination_kind
  158.         - source_labels: [__name__]
  159.           action: replace
  160.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_(.*)_destination_pod_.*_osm_request_total
  161.           target_label: destination_name
  162.         - source_labels: [__name__]
  163.           action: replace
  164.           regex: envoy_response_code_\d{3}_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_(.*)_osm_request_total
  165.           target_label: destination_pod
  166.         - source_labels: [__name__]
  167.           action: replace
  168.           regex: .*(osm_request_total)
  169.           target_label: __name__
  171.         - source_labels: [__name__]
  172.           action: replace
  173.           regex: envoy_source_namespace_(.*)_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  174.           target_label: source_namespace
  175.         - source_labels: [__name__]
  176.           action: replace
  177.           regex: envoy_source_namespace_.*_source_kind_(.*)_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  178.           target_label: source_kind
  179.         - source_labels: [__name__]
  180.           action: replace
  181.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_(.*)_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  182.           target_label: source_name
  183.         - source_labels: [__name__]
  184.           action: replace
  185.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_(.*)_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  186.           target_label: source_pod
  187.         - source_labels: [__name__]
  188.           action: replace
  189.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_(.*)_destination_kind_.*_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  190.           target_label: destination_namespace
  191.         - source_labels: [__name__]
  192.           action: replace
  193.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_(.*)_destination_name_.*_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  194.           target_label: destination_kind
  195.         - source_labels: [__name__]
  196.           action: replace
  197.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_(.*)_destination_pod_.*_osm_request_duration_ms_(bucket|sum|count)
  198.           target_label: destination_name
  199.         - source_labels: [__name__]
  200.           action: replace
  201.           regex: envoy_source_namespace_.*_source_kind_.*_source_name_.*_source_pod_.*_destination_namespace_.*_destination_kind_.*_destination_name_.*_destination_pod_(.*)_osm_request_duration_ms_(bucket|sum|count)
  202.           target_label: destination_pod
  203.         - source_labels: [__name__]
  204.           action: replace
  205.           regex: .*(osm_request_duration_ms_(bucket|sum|count))
  206.           target_label: __name__
  208.       - job_name: 'kubernetes-cadvisor'
  209.         scheme: https
  210.         tls_config:
  211.           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  212.         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  213.         kubernetes_sd_configs:
  214.         - role: node
  215.         metric_relabel_configs:
  216.         - source_labels: [__name__]
  217.           regex: '(container_cpu_usage_seconds_total|container_memory_rss)'
  218.           action: keep
  219.         relabel_configs:
  220.         - action: labelmap
  221.           regex: __meta_kubernetes_node_label_(.+)
  222.         - target_label: __address__
  223.           replacement: kubernetes.default.svc:443
  224.         - source_labels: [__meta_kubernetes_node_name]
  225.           regex: (.+)
  226.           target_label: __metrics_path__
  227.           replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

使用 kubectl apply 来更新 Prometheus 服务端 configmap。

  1. kubectl apply -f update-prometheus-configmap.yaml

通过使用 kubectl port-forward 来转发 Prometheus 管理应用程序和你的开发机器间的流量，来验证 Prometheus 是否能够来抓取 OSM 网格和 API 端点。

  1. export POD_NAME=$(kubectl get pods -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  2. kubectl port-forward $POD_NAME 9090

在 web 浏览器中打开 http://localhost:9090/targets 来访问 Prometheus 管理应用，并验证端点是否连接、启动和正在执行抓取。

将 OSM 与 Prometheus 和 Grafana 集成 - 图1

Targets with specific relabeling config established by OSM should be “up”

停止端口转发命令。

部署 Grafana 实例

在 default 命名空间下，使用 helm 安装 Grafana 实例。

  1. helm repo add grafana https://grafana.github.io/helm-charts
  2. helm repo update
  3. helm install grafana/grafana --generate-name

使用 kubectl get secret 来显示 Grafana 的管理员密码。

  1. export SECRET_NAME=$(kubectl get secret -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}")
  2. kubectl get secret $SECRET_NAME -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

使用 kubectl port-forward 来转发 Grafana 管理应用和你的开发机器间的流量。

  1. export POD_NAME=$(kubectl get pods -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}")
  2. kubectl port-forward $POD_NAME 3000

在 web 浏览器中打开 http://localhost:3000 访问 Grafana 的管理程序。使用 admin 作为用户名和前面获取到的管理员密码。验证端点已连接、启动和正在执行抓取。

在管理程序中：

  • 选择 Settings 然后 Data Sources。
  • 选择 Add Data source。
  • 找到 Prometheus 数据源并点击 Select。
  • 输入 DNS 名字，例如在前面拿到的 stable-prometheus-server.default.svc.cluster.local。

选择 Save and Test 并确保看到 Data source is working。

Importing OSM Dashboards

OSM Dashboards 可通过 OSM GitHub 存储库 获得，可以在管理应用程序上以 json blobs 方式导入。

要导入仪表盘：

  • 鼠标移放在 + 上并点击 Import
  • 从 osm-mesh-envoy-details dashboard 复制 JSON 内容并拷贝到 Import via panel json。
  • 选择 Load。
  • 选择 Import。

确保看到 Mesh and Envoy Details 仪表盘创建。