08-4. 部署 kube-prometheus 插架

kube-prometheus 是一整套监控解决方案，它使用 Prometheus 采集集群指标，Grafana 做展示，包含如下组件：

  • The Prometheus Operator
  • Highly available Prometheus
  • Highly available Alertmanager
  • Prometheus node-exporter
  • Prometheus Adapter for Kubernetes Metrics APIs （k8s-prometheus-adapter）
  • kube-state-metrics
  • Grafana

其中 k8s-prometheus-adapter 使用 Prometheus 实现了 metrics.k8s.io 和 custom.metrics.k8s.io API，所以不需要再部署 metrics-server。 如果要单独部署 metrics-server，请参考：C.metrics-server插件.md

如果没有特殊指明，本文档的所有操作均在 zhangjun-k8s-01 节点上执行；

下载和安装

  1. cd /opt/k8s/work
  2. git clone https://github.com/coreos/kube-prometheus.git
  3. cd kube-prometheus/
  4. sed -i -e 's_quay.io_quay.azk8s.cn_' manifests/*.yaml manifests/setup/*.yaml # 使用微软的 Registry
  5. kubectl apply -f manifests/setup # 安装 prometheus-operator
  6. kubectl apply -f manifests/ # 安装 promethes metric adapter

查看运行状态

  1. $ kubectl get pods -n monitoring
  2. NAME                                   READY   STATUS    RESTARTS   AGE
  3. alertmanager-main-0                    2/2     Running       0          63s
  4. alertmanager-main-1                    2/2     Running       0          63s
  5. alertmanager-main-2                    2/2     Running       0          63s
  6. grafana-76b8d59b9b-nd6gk               1/1     Running       0          11m
  7. kube-state-metrics-67b7c5dc78-sktzg    3/3     Running       0          73s
  8. node-exporter-prsvf                    2/2     Running       0          34s
  9. node-exporter-qdh6n                    2/2     Running       0          71s
  10. node-exporter-z6h4z                    2/2     Running       0          69s
  11. prometheus-adapter-5f46ccd66d-bbsns    1/1     Running       0          73s
  12. prometheus-k8s-0                       3/3     Running       1          53s
  13. prometheus-k8s-1                       3/3     Running       1          53s
  14. prometheus-operator-6d8b95b467-htx56   1/1     Running       0          74s
  1. $ kubectl top pods -n monitoring
  2. NAME                                  CPU(cores)   MEMORY(bytes)   
  3. alertmanager-main-0                    0m           18Mi            
  4. alertmanager-main-1                    2m           20Mi            
  5. alertmanager-main-2                    0m           19Mi            
  6. grafana-76b8d59b9b-nd6gk               4m           49Mi            
  7. kube-state-metrics-67b7c5dc78-sktzg    11m          29Mi            
  8. kube-state-metrics-959876458-cjtr5     9m           37Mi            
  9. node-exporter-prsvf                    4m           11Mi            
  10. node-exporter-qdh6n                    1m           20Mi            
  11. node-exporter-z6h4z                    5m           11Mi            
  12. prometheus-adapter-5f46ccd66d-bbsns    0m           17Mi            
  13. prometheus-k8s-0                       15m          190Mi           
  14. prometheus-k8s-1                       6m           199Mi           
  15. prometheus-operator-6d8b95b467-htx56   0m           20Mi

访问 Prometheus UI

启动服务代理：

  1. $ 
  2. $ kubectl port-forward --address 0.0.0.0 pod/prometheus-k8s-0 -n monitoring 9090:9090
  3. Forwarding from 0.0.0.0:9090 -> 9090
  • port-forward 依赖 socat。

浏览器访问：http://172.27.138.251:9090/new/graph?g0.expr=&g0.tab=1&g0.stacked=0&g0.range_input=1h

prometheus

访问 Grafana UI

启动代理：

  1. $ kubectl port-forward --address 0.0.0.0 svc/grafana -n monitoring 3000:3000 
  2. Forwarding from 0.0.0.0:3000 -> 3000

浏览器访问：http://172.27.138.251:3000/

用 admin/admin 登录： grafana_login

然后，就可以看到各种预定义的 dashboard 了： grafana_dashboard