09-4.部署 metrics-server 插件

注意：如果没有特殊指明，本文档的所有操作均在 m7-autocv-gpu01 节点上执行。

创建 metrics-server 使用的证书

创建 metrics-server 证书签名请求:

  1. cat > metrics-server-csr.json <<EOF
  2. {
  3.   "CN": "aggregator",
  4.   "hosts": [],
  5.   "key": {
  6.     "algo": "rsa",
  7.     "size": 2048
  8.   },
  9.   "names": [
  10.     {
  11.       "C": "CN",
  12.       "ST": "BeiJing",
  13.       "L": "BeiJing",
  14.       "O": "k8s",
  15.       "OU": "4Paradigm"
  16.     }
  17.   ]
  18. }
  19. EOF
  • 注意： CN 名称为 aggregator，需要与 kube-apiserver 的 —requestheader-allowed-names 参数配置一致；

生成 metrics-server 证书和私钥：

  1. cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  2.   -ca-key=/etc/kubernetes/cert/ca-key.pem  \
  3.   -config=/etc/kubernetes/cert/ca-config.json  \
  4.   -profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server

将生成的证书和私钥文件拷贝到 kube-apiserver 节点：

  1. source /opt/k8s/bin/environment.sh
  2. for node_ip in ${NODE_IPS[@]}
  3.   do
  4.     echo ">>> ${node_ip}"
  5.     scp metrics-server*.pem root@${node_ip}:/etc/kubernetes/cert/
  6.   done

修改 kubernetes 控制平面组件的配置以支持 metrics-server

kube-apiserver

添加如下配置参数：

  1. --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem
  2. --requestheader-allowed-names=""
  3. --requestheader-extra-headers-prefix="X-Remote-Extra-"
  4. --requestheader-group-headers=X-Remote-Group
  5. --requestheader-username-headers=X-Remote-User
  6. --proxy-client-cert-file=/etc/kubernetes/cert/metrics-server.pem
  7. --proxy-client-key-file=/etc/kubernetes/cert/metrics-server-key.pem
  8. --runtime-config=api/all=true
  • --requestheader-XXX、--proxy-client-XXX 是 kube-apiserver 的 aggregator layer 相关的配置参数，metrics-server & HPA 需要使用；
  • --requestheader-client-ca-file：用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用；
  • 如果 —requestheader-allowed-names 不为空，则—proxy-client-cert-file 证书的 CN 必须位于 allowed-names 中，默认为 aggregator;

如果 kube-apiserver 机器没有运行 kube-proxy，则还需要添加 --enable-aggregator-routing=true 参数；

关于 --requestheader-XXX 相关参数，参考：

注意：requestheader-client-ca-file 指定的 CA 证书，必须具有 client auth and server auth；

kube-controllr-manager

添加如下配置参数(从 v1.12 开始，该选项默认为 true，不需要再添加)：

—horizontal-pod-autoscaler-use-rest-clients=true

用于配置 HPA 控制器使用 REST 客户端获取 metrics 数据。

整体架构

k8s-hpa.png

修改插件配置文件配置文件

metrics-server 插件位于 kubernetes 的 cluster/addons/metrics-server/ 目录下。

修改 metrics-server-deployment 文件：

  1. $ cp metrics-server-deployment.yaml{,.orig}
  2. $ diff metrics-server-deployment.yaml.orig metrics-server-deployment.yaml
  3. 51c51
  4. <         image: mirrorgooglecontainers/metrics-server-amd64:v0.2.1
  5. ---
  6. >         image: k8s.gcr.io/metrics-server-amd64:v0.2.1
  7. 54c54
  8. <         - --source=kubernetes.summary_api:''
  9. ---
  10. >         - --source=kubernetes.summary_api:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250
  11. 60c60
  12. <         image: siriuszg/addon-resizer:1.8.1
  13. ---
  14. >         image: k8s.gcr.io/addon-resizer:1.8.1
  • metrics-server 的参数格式与 heapster 类似。由于 kubelet 只在 10250 监听 https 请求，故添加相关参数；

授予 kube-system:metrics-server ServiceAccount 访问 kubelet API 的权限：

  1. $ cat auth-kubelet.yaml
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRoleBinding
  4. metadata:
  5.   name: metrics-server:system:kubelet-api-admin
  6.   labels:
  7.     kubernetes.io/cluster-service: "true"
  8.     addonmanager.kubernetes.io/mode: Reconcile
  9. roleRef:
  10.   apiGroup: rbac.authorization.k8s.io
  11.   kind: ClusterRole
  12.   name: system:kubelet-api-admin
  13. subjects:
  14. - kind: ServiceAccount
  15.   name: metrics-server
  16.   namespace: kube-system
  • 新建一个 ClusterRoleBindings 定义文件，授予相关权限；

创建 metrics-server

  1. $ pwd
  2. /opt/k8s/kubernetes/cluster/addons/metrics-server
  3. $ ls -l *.yaml
  4. -rw-rw-r-- 1 k8s k8s  398 Jun  5 07:17 auth-delegator.yaml
  5. -rw-rw-r-- 1 k8s k8s  404 Jun 16 18:02 auth-kubelet.yaml
  6. -rw-rw-r-- 1 k8s k8s  419 Jun  5 07:17 auth-reader.yaml
  7. -rw-rw-r-- 1 k8s k8s  393 Jun  5 07:17 metrics-apiservice.yaml
  8. -rw-rw-r-- 1 k8s k8s 2640 Jun 16 17:54 metrics-server-deployment.yaml
  9. -rw-rw-r-- 1 k8s k8s  336 Jun  5 07:17 metrics-server-service.yaml
  10. -rw-rw-r-- 1 k8s k8s  801 Jun  5 07:17 resource-reader.yaml
  11. $ kubectl create -f .

查看运行情况

  1. $ kubectl get pods -n kube-system |grep metrics-server
  2. metrics-server-v0.2.1-7486f5bd67-v95q2   2/2       Running   0          45s
  4. $ kubectl get svc -n kube-system|grep metrics-server
  5. metrics-server         ClusterIP   10.254.115.120   <none>        443/TCP         1m

查看 metrcs-server 输出的 metrics

metrics-server 输出的 APIs：https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/resource-metrics-api.md

  1. 通过 kube-apiserver 或 kubectl proxy 访问：

    https://172.27.128.150:6443/apis/metrics.k8s.io/v1beta1/nodes https://172.27.128.150:6443/apis/metrics.k8s.io/v1beta1/nodes/ https://172.27.128.150:6443/apis/metrics.k8s.io/v1beta1/pods https://172.27.128.150:6443/apis/metrics.k8s.io/v1beta1/namespace//pods/

  2. 直接使用 kubectl 命令访问：

    kubectl get —raw apis/metrics.k8s.io/v1beta1/nodes kubectl get —raw apis/metrics.k8s.io/v1beta1/pods kubectl get —raw apis/metrics.k8s.io/v1beta1/nodes/ kubectl get —raw apis/metrics.k8s.io/v1beta1/namespace//pods/

  1. $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1" | jq .
  2. {
  3.   "kind": "APIResourceList",
  4.   "apiVersion": "v1",
  5.   "groupVersion": "metrics.k8s.io/v1beta1",
  6.   "resources": [
  7.     {
  8.       "name": "nodes",
  9.       "singularName": "",
  10.       "namespaced": false,
  11.       "kind": "NodeMetrics",
  12.       "verbs": [
  13.         "get",
  14.         "list"
  15.       ]
  16.     },
  17.     {
  18.       "name": "pods",
  19.       "singularName": "",
  20.       "namespaced": true,
  21.       "kind": "PodMetrics",
  22.       "verbs": [
  23.         "get",
  24.         "list"
  25.       ]
  26.     }
  27.   ]
  28. }
  30. $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .
  31. {
  32.   "kind": "NodeMetricsList",
  33.   "apiVersion": "metrics.k8s.io/v1beta1",
  34.   "metadata": {
  35.     "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  36.   },
  37.   "items": [
  38.     {
  39.       "metadata": {
  40.         "name": "m7-autocv-gpu03",
  41.         "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/m7-autocv-gpu03",
  42.         "creationTimestamp": "2018-06-16T10:24:03Z"
  43.       },
  44.       "timestamp": "2018-06-16T10:23:00Z",
  45.       "window": "1m0s",
  46.       "usage": {
  47.         "cpu": "133m",
  48.         "memory": "1115728Ki"
  49.       }
  50.     },
  51.     {
  52.       "metadata": {
  53.         "name": "m7-autocv-gpu01",
  54.         "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/m7-autocv-gpu01",
  55.         "creationTimestamp": "2018-06-16T10:24:03Z"
  56.       },
  57.       "timestamp": "2018-06-16T10:23:00Z",
  58.       "window": "1m0s",
  59.       "usage": {
  60.         "cpu": "221m",
  61.         "memory": "6799908Ki"
  62.       }
  63.     },
  64.     {
  65.       "metadata": {
  66.         "name": "m7-autocv-gpu02",
  67.         "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/m7-autocv-gpu02",
  68.         "creationTimestamp": "2018-06-16T10:24:03Z"
  69.       },
  70.       "timestamp": "2018-06-16T10:23:00Z",
  71.       "window": "1m0s",
  72.       "usage": {
  73.         "cpu": "76m",
  74.         "memory": "1130180Ki"
  75.       }
  76.     }
  77.   ]
  78. }
  • /apis/metrics.k8s.io/v1beta1/nodes 和 /apis/metrics.k8s.io/v1beta1/pods 返回的 usage 包含 CPU 和 Memory；

参考

  1. https://kubernetes.feisky.xyz/zh/addons/metrics.html
  2. metrics-server RBAC：https://github.com/kubernetes-incubator/metrics-server/issues/40
  3. metrics-server 参数：https://github.com/kubernetes-incubator/metrics-server/issues/25
  4. https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/