可观察性最佳实践

使用 Prometheus 进行生产规模的监控

使用 Istio 以及 Prometheus 进行生产规模的监控时推荐的方式是使用分层联邦并且结合一组记录规则。

尽管安装 Istio 不会默认部署 Prometheus，入门指导中 Option 1: Quick Start 的部署按照 Prometheus 集成指导安装了 Prometheus。 此 Prometheus 部署刻意地配置了很短的保留窗口 (6小时)。此快速入门 Prometheus 部署同时也配置为从网格上运行的每一个 Envoy 代理上收集指标，同时通过一组有关它们的源的标签( instance，pod, 和 namespace)来扩充指标。

Architecture for production monitoring of Istio using Prometheus.

Production-scale Istio monitoring with Istio

通过记录规则进行负载等级的聚合

为了聚合统计实例以及 pod 级别的指标起来，需要用以下的记录规则更新默认 Prometheus 配置：

Plain Prometheus Rules Prometheus Operator Rules CRD

  1. groups:
  2. - name: "istio.recording-rules"
  3.   interval: 5s
  4.   rules:
  5.   - record: "workload:istio_requests_total"
  6.     expr: |
  7.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_requests_total)
  9.   - record: "workload:istio_request_duration_milliseconds_count"
  10.     expr: |
  11.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_count)
  13.   - record: "workload:istio_request_duration_milliseconds_sum"
  14.     expr: |
  15.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_sum)
  17.   - record: "workload:istio_request_duration_milliseconds_bucket"
  18.     expr: |
  19.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_bucket)
  21.   - record: "workload:istio_request_bytes_count"
  22.     expr: |
  23.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_count)
  25.   - record: "workload:istio_request_bytes_sum"
  26.     expr: |
  27.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_sum)
  29.   - record: "workload:istio_request_bytes_bucket"
  30.     expr: |
  31.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_bucket)
  33.   - record: "workload:istio_response_bytes_count"
  34.     expr: |
  35.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_count)
  37.   - record: "workload:istio_response_bytes_sum"
  38.     expr: |
  39.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_sum)
  41.   - record: "workload:istio_response_bytes_bucket"
  42.     expr: |
  43.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_bucket)
  45.   - record: "workload:istio_tcp_connections_opened_total"
  46.     expr: |
  47.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_connections_opened_total)
  49.   - record: "workload:istio_tcp_connections_closed_total"
  50.     expr: |
  51.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_connections_closed_total)
  53.   - record: "workload:istio_tcp_sent_bytes_total_count"
  54.     expr: |
  55.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total_count)
  57.   - record: "workload:istio_tcp_sent_bytes_total_sum"
  58.     expr: |
  59.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total_sum)
  61.   - record: "workload:istio_tcp_sent_bytes_total_bucket"
  62.     expr: |
  63.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total_bucket)
  65.   - record: "workload:istio_tcp_received_bytes_total_count"
  66.     expr: |
  67.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total_count)
  69.   - record: "workload:istio_tcp_received_bytes_total_sum"
  70.     expr: |
  71.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total_sum)
  73.   - record: "workload:istio_tcp_received_bytes_total_bucket"
  74.     expr: |
  75.       sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total_bucket)
  1. apiVersion: monitoring.coreos.com/v1
  2. kind: PrometheusRule
  3. metadata:
  4.   name: istio-metrics-aggregation
  5.   labels:
  6.     app.kubernetes.io/name: istio-prometheus
  7. spec:
  8.   groups:
  9.   - name: "istio.metricsAggregation-rules"
  10.     interval: 5s
  11.     rules:
  12.     - record: "workload:istio_requests_total"
  13.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_requests_total)"
  15.     - record: "workload:istio_request_duration_milliseconds_count"
  16.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_count)"
  17.     - record: "workload:istio_request_duration_milliseconds_sum"
  18.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_sum)"
  19.     - record: "workload:istio_request_duration_milliseconds_bucket"
  20.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_duration_milliseconds_bucket)"
  22.     - record: "workload:istio_request_bytes_count"
  23.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_count)"
  24.     - record: "workload:istio_request_bytes_sum"
  25.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_sum)"
  26.     - record: "workload:istio_request_bytes_bucket"
  27.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_request_bytes_bucket)"
  29.     - record: "workload:istio_response_bytes_count"
  30.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_count)"
  31.     - record: "workload:istio_response_bytes_sum"
  32.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_sum)"
  33.     - record: "workload:istio_response_bytes_bucket"
  34.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_response_bytes_bucket)"
  36.     - record: "workload:istio_tcp_connections_opened_total"
  37.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_connections_opened_total)"
  38.     - record: "workload:istio_tcp_connections_closed_total"
  39.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_connections_closed_total)"
  41.     - record: "workload:istio_tcp_sent_bytes_total_count"
  42.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total_count)"
  43.     - record: "workload:istio_tcp_sent_bytes_total_sum"
  44.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total_sum)"
  45.     - record: "workload:istio_tcp_sent_bytes_total_bucket"
  46.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_sent_bytes_total_bucket)"
  48.     - record: "workload:istio_tcp_received_bytes_total_count"
  49.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total_count)"
  50.     - record: "workload:istio_tcp_received_bytes_total_sum"
  51.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total_sum)"
  52.     - record: "workload:istio_tcp_received_bytes_total_bucket"
  53.       expr: "sum without(instance, kubernetes_namespace, kubernetes_pod_name) (istio_tcp_received_bytes_total_bucket)"

以上的记录规则只是同聚合得到 pods 以及实例级别的指标。这仍然完整的保留了 Istio 标准指标中的全部项，包括全部的 Istio 维度。尽管这有助于通过联邦控制指标维度，您可能仍想进一步优化记录规则来匹配您现有的仪表盘，告警，以及特定的引用。

如需要更多关于如何配置您的记录规则。请参考使用记录规则优化指标收集。

使用负载级别的聚合指标进行联邦

为了建立 Prometheus 联邦，请修改您的 Prometheus 生产部署配置来抓取 Istio Prometheus 联邦终端的指标数据。

将以下的 Job 添加到配置中：

  1. - job_name: 'istio-prometheus'
  2.   honor_labels: true
  3.   metrics_path: '/federate'
  4.   kubernetes_sd_configs:
  5.   - role: pod
  6.     namespaces:
  7.       names: ['istio-system']
  8.   metric_relabel_configs:
  9.   - source_labels: [__name__]
  10.     regex: 'workload:(.*)'
  11.     target_label: __name__
  12.     action: replace
  13.   params:
  14.     'match[]':
  15.     - '{__name__=~"workload:(.*)"}'
  16.     - '{__name__=~"pilot(.*)"}'

如果您使用的是 Prometheus Operator，请使用以下的配置：

  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: istio-federation
  5.   labels:
  6.     app.kubernetes.io/name: istio-prometheus
  7. spec:
  8.   namespaceSelector:
  9.     matchNames:
  10.     - istio-system
  11.   selector:
  12.     matchLabels:
  13.       app: prometheus
  14.   endpoints:
  15.   - interval: 30s
  16.     scrapeTimeout: 30s
  17.     params:
  18.       'match[]':
  19.       - '{__name__=~"workload:(.*)"}'
  20.       - '{__name__=~"pilot(.*)"}'
  21.     path: /federate
  22.     targetPort: 9090
  23.     honorLabels: true
  24.     metricRelabelings:
  25.     - sourceLabels: ["__name__"]
  26.       regex: 'workload:(.*)'
  27.       targetLabel: "__name__"
  28.       action: replace

联邦配置的关键是首先匹配通过 Istio 部署的 Prometheus 中收集 Istio 标准指标的 job。并且将收集到的指标重命名，方法为去除负载等级记录规则命名前缀 (workload:)。 这使得现有的仪表盘以及引用能够无缝地针对生产用 Prometheus 继续工作 (并且不在指向 Istio 实例)。

您可以在设置联邦时包含额外的指标(例如 envoy, go 等)。

控制面指标也被生产用 Prometheus 收集并联邦。

Optimizing metrics collection with recording rules

除了使用记录规则在 pod 和实例等级聚合，您也许想要使用记录规则为您现有的仪表盘以及告警专门生成聚合指标。这方面针对收集的优化可以很大的节约您 Prometheus 生产实例的资源消耗，同时加速了引用性能。

例如，假设一个监控仪表盘使用以下 Prometheus 引用：

  • 请求速率在过去 1 分钟的平均值，并按照目的服务以及命名空间聚合

    1. sum(irate(istio_requests_total{reporter="source"}[1m]))
    2. by (
    3.     destination_canonical_service,
    4.     destination_workload_namespace
    5. )

  • P95 客户端延迟在过去 1 分钟的平均值，并按照来源，目的服务以及命名空间聚合

    1. histogram_quantile(0.95,
    2.   sum(irate(istio_request_duration_milliseconds_bucket{reporter="source"}[1m]))
    3.   by (
    4.     destination_canonical_service,
    5.     destination_workload_namespace,
    6.     source_canonical_service,
    7.     source_workload_namespace,
    8.     le
    9.   )
    10. )

以下记录规则可以加至 Istio Prometheus 配置中，使用 istio 前缀来使得联邦更容易识别这些指标。

  1. groups:
  2. - name: "istio.recording-rules"
  3.   interval: 5s
  4.   rules:
  5.   - record: "istio:istio_requests:by_destination_service:rate1m"
  6.     expr: |
  7.       sum(irate(istio_requests_total{reporter="destination"}[1m]))
  8.       by (
  9.         destination_canonical_service,
  10.         destination_workload_namespace
  11.       )
  12.   - record: "istio:istio_request_duration_milliseconds_bucket:p95:rate1m"
  13.     expr: |
  14.       histogram_quantile(0.95,
  15.         sum(irate(istio_request_duration_milliseconds_bucket{reporter="source"}[1m]))
  16.         by (
  17.           destination_canonical_service,
  18.           destination_workload_namespace,
  19.           source_canonical_service,
  20.           source_workload_namespace,
  21.           le
  22.         )
  23.       )

Prometheus 生产实例可以从 Istio 实例那里得到的信息更新联邦：

  • 匹配字句 {__name__=~"istio:(.*)"}

  • 重新将指标标签为： regex: "istio:(.*)"

原始引用被替代为：

  • istio_requests:by_destination_service:rate1m

  • avg(istio_request_duration_milliseconds_bucket:p95:rate1m)

更详细的关于 AutoTrader 上生产环境指标收集优化的文章提供了更丰富的例子来描述如何直接对引用聚合从而赋能仪表盘以及告警。