以 Daemonset 方式部署 kube-proxy

kube-proxy 可以用二进制部署，也可以用 kubelet 的静态 Pod 部署，但最简单使用 DaemonSet 部署。直接使用 ServiceAccount 的 token 认证，不需要签发证书，也就不用担心证书过期问题。

先在终端设置下面的变量:

  1. APISERVER="https://10.200.16.79:6443"
  2. CLUSTER_CIDR="10.10.0.0/16"
  • APISERVER 替换为 apiserver 对外暴露的访问地址。有同学想问为什么不直接用集群内的访问地址(kubernetes.default 或对应的 CLUSTER IP)，这是一个鸡生蛋还是蛋生鸡的问题，CLSUTER IP 本身就是由 kube-proxy 来生成 iptables 或 ipvs 规则转发 Service 对应 Endpoint 的 Pod IP，kube-proxy 刚启动还没有生成这些转发规则，生成规则的前提是 kube-proxy 需要访问 apiserver 获取 Service 与 Endpoint，而由于还没有转发规则，kube-proxy 访问 apiserver 的 CLUSTER IP 的请求无法被转发到 apiserver。
  • CLUSTER_CIDR 替换为集群 Pod IP 的 CIDR 范围，这个在部署 kube-controller-manager 时也设置过

为 kube-proxy 创建 RBAC 权限和配置文件:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   name: kube-proxy
  6.   namespace: kube-system
  8. ---
  10. apiVersion: rbac.authorization.k8s.io/v1
  11. kind: ClusterRoleBinding
  12. metadata:
  13.   name: system:kube-proxy
  14. roleRef:
  15.   apiGroup: rbac.authorization.k8s.io
  16.   kind: ClusterRole
  17.   name: system:node-proxier
  18. subjects:
  19. - kind: ServiceAccount
  20.   name: kube-proxy
  21.   namespace: kube-system
  23. ---
  25. kind: ConfigMap
  26. apiVersion: v1
  27. metadata:
  28.   name: kube-proxy
  29.   namespace: kube-system
  30.   labels:
  31.     app: kube-proxy
  32. data:
  33.   kubeconfig.conf: |-
  34.     apiVersion: v1
  35.     kind: Config
  36.     clusters:
  37.     - cluster:
  38.         certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  39.         server: ${APISERVER}
  40.       name: default
  41.     contexts:
  42.     - context:
  43.         cluster: default
  44.         namespace: default
  45.         user: default
  46.       name: default
  47.     current-context: default
  48.     users:
  49.     - name: default
  50.       user:
  51.         tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  52.   config.conf: |-
  53.     apiVersion: kubeproxy.config.k8s.io/v1alpha1
  54.     kind: KubeProxyConfiguration
  55.     bindAddress: 0.0.0.0
  56.     clientConnection:
  57.       acceptContentTypes: ""
  58.       burst: 10
  59.       contentType: application/vnd.kubernetes.protobuf
  60.       kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  61.       qps: 5
  62.     # 集群中 Pod IP 的 CIDR 范围
  63.     clusterCIDR: ${CLUSTER_CIDR}
  64.     configSyncPeriod: 15m0s
  65.     conntrack:
  66.       # 每个核心最大能跟踪的NAT连接数，默认32768
  67.       maxPerCore: 32768
  68.       min: 131072
  69.       tcpCloseWaitTimeout: 1h0m0s
  70.       tcpEstablishedTimeout: 24h0m0s
  71.     enableProfiling: false
  72.     healthzBindAddress: 0.0.0.0:10256
  73.     iptables:
  74.       # SNAT 所有 Service 的 CLUSTER IP
  75.       masqueradeAll: false
  76.       masqueradeBit: 14
  77.       minSyncPeriod: 0s
  78.       syncPeriod: 30s
  79.     ipvs:
  80.       minSyncPeriod: 0s
  81.       # ipvs 调度类型，默认是 rr，支持的所有类型:
  82.       # rr: round-robin
  83.       # lc: least connection
  84.       # dh: destination hashing
  85.       # sh: source hashing
  86.       # sed: shortest expected delay
  87.       # nq: never queue
  88.       scheduler: rr
  89.       syncPeriod: 30s
  90.     metricsBindAddress: 0.0.0.0:10249
  91.     # 使用 ipvs 模式转发 service
  92.     mode: ipvs
  93.     # 设置 kube-proxy 进程的 oom-score-adj 值，范围 [-1000,1000]
  94.     # 值越低越不容易被杀死，这里设置为 —999 防止发生系统OOM时将 kube-proxy 杀死
  95.     oomScoreAdj: -999
  96. EOF

在终端设置下面的变量:

  1. ARCH="amd64"
  2. VERSION="v1.16.1"
  • VERSION 是 K8S 版本
  • ARCH 是节点的 cpu 架构，大多数用的 amd64，即 x86_64。其它常见的还有: arm64, arm, ppc64le, s390x，如果你的集群有不同 cpu 架构的节点，可以分别指定 ARCH 部署多个 daemonset (每个节点不会有多个 kube-proxy，nodeSelector 会根据 cpu 架构来选中节点)

使用 hostNetwork 以 Daemonset 方式部署 kube-proxy 到每个节点:

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: apps/v1
  3. kind: DaemonSet
  4. metadata:
  5.   labels:
  6.     k8s-app: kube-proxy-ds-${ARCH}
  7.   name: kube-proxy-ds-${ARCH}
  8.   namespace: kube-system
  9. spec:
  10.   selector:
  11.     matchLabels:
  12.       k8s-app: kube-proxy-ds-${ARCH}
  13.   updateStrategy:
  14.     type: RollingUpdate
  15.   template:
  16.     metadata:
  17.       labels:
  18.         k8s-app: kube-proxy-ds-${ARCH}
  19.     spec:
  20.       priorityClassName: system-node-critical
  21.       containers:
  22.       - name: kube-proxy
  23.         image: k8s.gcr.io/kube-proxy-${ARCH}:${VERSION}
  24.         imagePullPolicy: IfNotPresent
  25.         command:
  26.         - /usr/local/bin/kube-proxy
  27.         - --config=/var/lib/kube-proxy/config.conf
  28.         - --hostname-override=\$(NODE_NAME)
  29.         securityContext:
  30.           privileged: true
  31.         volumeMounts:
  32.         - mountPath: /var/lib/kube-proxy
  33.           name: kube-proxy
  34.         - mountPath: /run/xtables.lock
  35.           name: xtables-lock
  36.           readOnly: false
  37.         - mountPath: /lib/modules
  38.           name: lib-modules
  39.           readOnly: true
  40.         env:
  41.           - name: NODE_NAME
  42.             valueFrom:
  43.               fieldRef:
  44.                 fieldPath: spec.nodeName
  45.       hostNetwork: true
  46.       serviceAccountName: kube-proxy
  47.       volumes:
  48.       - name: kube-proxy
  49.         configMap:
  50.           name: kube-proxy
  51.       - name: xtables-lock
  52.         hostPath:
  53.           path: /run/xtables.lock
  54.           type: FileOrCreate
  55.       - name: lib-modules
  56.         hostPath:
  57.           path: /lib/modules
  58.       tolerations:
  59.       - key: CriticalAddonsOnly
  60.         operator: Exists
  61.       - operator: Exists
  62.       nodeSelector:
  63.         beta.kubernetes.io/arch: ${ARCH}
  64. EOF