收集metrics

在本文档中，介绍如何以Deployment或者Daemonset的方式部署grafana-agent到您的k8s集群中，抓取宿主机上kubelet和cAdvisor的metrics指标，并把抓取到的数据，以remote_write的方式推送到Nightingale.

通过本文档，我们预期达成以下目标：

  1. 部署grafana-agent到您的K8s集群中；
  2. 配置grafana-agent抓取kubelet和cAdvisor的metrics；

K8s是开源的容器编排系统，自动化管理容器的部署、扩缩容等工作。K8s默认会暴露Node和控制面的若干metrics接口，这些接口兼容Prometheus的metrics规范。我们可以部署grafana-agent来收集Node的cAdvisor和kubelet metrics，并以remote_write的方式发送到Nightingale.

前置依赖

  1. 一个开启RBAC（role-based access control）的Kubernetes集群；
  2. 安装并配置好了kubectl命令行工具；

步骤一：创建 ServiceAcount、ClusterRole、ClusterRoleBinding

  1. export NAMESPACE=default
  2. MANIFEST_URL=https://raw.githubusercontent.com/flashcatcloud/fc-agent/fc-release/etc/k8s/agent-bare.yaml
  4. curl -fsSL $MANIFEST_URL | envsubst |  kubectl apply -f -

步骤二：创建ConfigMap，配置grafana-agent

  1. export NAMESPACE=default
  2. export CLUSTER_NAME=kubernetes
  3. export FC_REMOTE_WRITE_URL=http://10.206.0.16:8480/insert/0/prometheus/api/v1/write
  4. #export FC_REMOTE_WRITE_URL=https://n9e-server:19000/prometheus/v1/write
  5. #export FC_REMOTE_WRITE_USERNAME=fc_laiwei
  6. #export FC_REMOTE_WRITE_PASSWORD=fc_laiweisecret
  8. cat <<EOF |
  9. kind: ConfigMap
  10. metadata:
  11.   name: grafana-agent
  12. apiVersion: v1
  13. data:
  14.   agent.yaml: |
  15.     server:
  16.       http_listen_port: 12345
  17.     metrics:
  18.       wal_directory: /tmp/grafana-agent-wal
  19.       global:
  20.         scrape_interval: 15s
  21.         scrape_timeout: 10s
  22.         external_labels:
  23.           cluster: ${CLUSTER_NAME}
  24.       configs:
  25.       - name: integrations
  26.         remote_write:
  27.         - url: ${FC_REMOTE_WRITE_URL}
  28.           basic_auth:
  29.             username: ${FC_REMOTE_WRITE_USERNAME}
  30.             password: ${FC_REMOTE_WRITE_PASSWORD}
  31.         scrape_configs:
  32.         - job_name: integrations/kubernetes/cadvisor
  33.           bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  34.           kubernetes_sd_configs:
  35.             - role: node
  36.           metric_relabel_configs:
  37.             - action: drop
  38.               regex: container_([a-z_]+);
  39.               source_labels:
  40.                 - __name__
  41.                 - image
  42.             - action: drop
  43.               regex: container_(network_tcp_usage_total|network_udp_usage_total|tasks_state|cpu_load_average_10s)
  44.               source_labels:
  45.                 - __name__
  46.           relabel_configs:
  47.             - replacement: kubernetes.default.svc:443
  48.               target_label: __address__
  49.             - regex: (.+)
  50.               replacement: /api/v1/nodes/\$1/proxy/metrics/cadvisor
  51.               source_labels:
  52.                 - __meta_kubernetes_node_name
  53.               target_label: __metrics_path__
  54.           scheme: https
  55.           tls_config:
  56.               ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  57.               insecure_skip_verify: false
  58.               server_name: kubernetes
  59.         - job_name: integrations/kubernetes/kubelet
  60.           bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  61.           kubernetes_sd_configs:
  62.             - role: node
  63.           relabel_configs:
  64.             - replacement: kubernetes.default.svc:443
  65.               target_label: __address__
  66.             - regex: (.+)
  67.               replacement: /api/v1/nodes/\$1/proxy/metrics
  68.               source_labels:
  69.                 - __meta_kubernetes_node_name
  70.               target_label: __metrics_path__
  71.           scheme: https
  72.           tls_config:
  73.               ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  74.               insecure_skip_verify: false
  75.               server_name: kubernetes
  76. EOF
  78. envsubst | kubectl apply -n $NAMESPACE -f -

步骤三：在K8s中创建grafana-agent实例

Daemonset

对于采集 node_exporter/ kubelet/ cAdvisor等指标，每个节点上只运行一个grafana-agent实例的情况，推荐以daemonset运行

  1. export NAMESPACE=default
  2. MANIFEST_URL=https://raw.githubusercontent.com/flashcatcloud/fc-agent/fc-release/etc/k8s/agent-daemonset.yaml
  4. curl -fsSL $MANIFEST_URL | envsubst |  kubectl apply -f -

Deployment

对于采集MySQLd_Exporter等需要运行多个grafana-agent实例的情况，推荐以deployment运行。

  1. export NAMESPACE=default
  2. MANIFEST_URL=https://raw.githubusercontent.com/flashcatcloud/fc-agent/fc-release/etc/k8s/agent-deployment.yaml
  4. curl -fsSL $MANIFEST_URL | envsubst |  kubectl apply -f -

如何重建grafana-agent

Daemonset

  1. kubectl rollout restart daemonset/grafana-agent

Deployment

  1. kubectl rollout restart deployment/grafana-agent

至此，我们已经完成了在K8s中部署grafana-agent并收集metrics，进一步，我们还可以配置grafana-agent来建立起完整的kubernetes指标监控体系。