OpenYurt 安装前置条件

1.背景说明

OpenYurt为适应边端环境,需要用户对K8S做一些调整,如:CoreDNS,KubeProxy等。

2. CoreDNS调整

一般场景下,CoreDNS是以Deployment形式部署,在边端场景下,域名解析请求无法跨NodePool,所以CoreDNS需要以Daemonset或者YurtAppDaemon形式部署,以实现将hostname解析为tunnelserver地址。

2.1 CoreDNS 配置修改

修改kube-system namespace下的ConfigMap coredns,增加如下内容:

  1. hosts /etc/edge/tunnel-nodes { # 增加hosts插件
  2. reload 300ms
  3. fallthrough
  4. }

修改后效果如下:

  1. apiVersion: v1
  2. data:
  3. Corefile: |
  4. .:53 {
  5. errors
  6. log . {
  7. class denial success
  8. }
  9. health {
  10. lameduck 5s
  11. }
  12. ready
  13. hosts /etc/edge/tunnel-nodes { # 增加hosts插件
  14. reload 300ms
  15. fallthrough
  16. }
  17. kubernetes cluster.local in-addr.arpa ip6.arpa {
  18. pods insecure
  19. fallthrough in-addr.arpa ip6.arpa
  20. ttl 30
  21. }
  22. prometheus :9153
  23. forward . /etc/resolv.conf {
  24. max_concurrent 1000
  25. }
  26. cache 30
  27. loop
  28. reload
  29. loadbalance
  30. }
  31. kind: ConfigMap
  32. metadata:
  33. name: coredns
  34. namespace: kube-system

2.2 CoreDNS 支持服务拓扑

增加annotation,利用openyurt的机制实现边缘服务选择。

  1. # 利用openyurt实现endpoint过滤
  2. kubectl annotate svc kube-dns -n kube-system openyurt.io/topologyKeys='openyurt.io/nodepool'

修改后效果:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4. annotations:
  5. openyurt.io/topologyKeys: openyurt.io/nodepool
  6. prometheus.io/port: "9153"
  7. prometheus.io/scrape: "true"
  8. creationTimestamp: "2022-02-14T10:13:37Z"
  9. labels:
  10. k8s-app: kube-dns
  11. kubernetes.io/cluster-service: "true"
  12. kubernetes.io/name: KubeDNS
  13. name: kube-dns
  14. namespace: kube-system
  15. resourceVersion: "65474309"
  16. selfLink: /api/v1/namespaces/kube-system/services/kube-dns
  17. uid: ee23195f-44c3-4c70-99e2-aff4d5cf0ae1
  18. spec:
  19. clusterIP: 10.254.0.10
  20. ports:
  21. - name: dns
  22. port: 53
  23. protocol: UDP
  24. targetPort: 53
  25. - name: dns-tcp
  26. port: 53
  27. protocol: TCP
  28. targetPort: 53
  29. - name: metrics
  30. port: 9153
  31. protocol: TCP
  32. targetPort: 9153
  33. selector:
  34. k8s-app: kube-dns
  35. sessionAffinity: None
  36. type: ClusterIP

2.2 CoreDNS DaemonSet部署

如果CoreDNS原本使用DaemonSet部署,可以手工进行如下调整:

1)可以调整CoreDNS的镜像为自己的版本;

2)需要挂载Volume ConfigMap yurt-tunnel-nodes

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata:
  4. labels:
  5. k8s-app: kube-dns
  6. name: coredns
  7. namespace: kube-system
  8. spec:
  9. selector:
  10. matchLabels:
  11. k8s-app: kube-dns
  12. template:
  13. metadata:
  14. labels:
  15. k8s-app: kube-dns
  16. spec:
  17. containers:
  18. - args:
  19. - -conf
  20. - /etc/coredns/Corefile
  21. image: registry.aliyuncs.com/google_containers/coredns:1.7.0
  22. livenessProbe:
  23. failureThreshold: 5
  24. httpGet:
  25. path: /health
  26. port: 8080
  27. scheme: HTTP
  28. initialDelaySeconds: 60
  29. periodSeconds: 10
  30. successThreshold: 1
  31. timeoutSeconds: 5
  32. name: coredns
  33. ports:
  34. - containerPort: 53
  35. name: dns
  36. protocol: UDP
  37. - containerPort: 53
  38. name: dns-tcp
  39. protocol: TCP
  40. - containerPort: 9153
  41. name: metrics
  42. protocol: TCP
  43. readinessProbe:
  44. failureThreshold: 3
  45. httpGet:
  46. path: /ready
  47. port: 8181
  48. scheme: HTTP
  49. periodSeconds: 10
  50. successThreshold: 1
  51. timeoutSeconds: 1
  52. resources:
  53. limits:
  54. memory: 170Mi
  55. requests:
  56. cpu: 100m
  57. memory: 70Mi
  58. securityContext:
  59. allowPrivilegeEscalation: false
  60. capabilities:
  61. add:
  62. - NET_BIND_SERVICE
  63. drop:
  64. - all
  65. readOnlyRootFilesystem: true
  66. volumeMounts:
  67. - mountPath: /etc/coredns
  68. name: config-volume
  69. readOnly: true
  70. - mountPath: /etc/edge
  71. name: hosts
  72. readOnly: true
  73. dnsPolicy: Default
  74. nodeSelector:
  75. kubernetes.io/os: linux
  76. priorityClassName: system-cluster-critical
  77. serviceAccount: coredns
  78. serviceAccountName: coredns
  79. tolerations:
  80. - operator: Exists
  81. - key: CriticalAddonsOnly
  82. operator: Exists
  83. - effect: NoSchedule
  84. key: node-role.kubernetes.io/master
  85. volumes:
  86. - configMap:
  87. defaultMode: 420
  88. items:
  89. - key: Corefile
  90. path: Corefile
  91. name: coredns
  92. name: config-volume
  93. - configMap:
  94. defaultMode: 420
  95. name: yurt-tunnel-nodes
  96. name: hosts

2.4 减少CoreDNS Deployment 副本数

如果k8s不是用Deployment部署,可以不进行操作。

  1. kubectl scale --replicas=0 deployment/coredns -n kube-system

3. KubeProxy调整

kubeadm部署的k8s集群会为KubeProxy生成kubeconfig配置,在不配置Service TopologyTopology Aware Hints 情况下,KubeProxy使用这个kubeconfig拿到的endpoints是全量的。

云边端场景下,边缘节点间很有可能无法互通,因此需要endpoints基于nodepool进行拓扑。直接将kube-proxy的kubeconfig配置删除,将apiserver请求经过yurthub即可解决服务拓扑问题。

KubeProxy支持流量拓扑

  1. kubectl edit cm -n kube-system kube-proxy

注释掉config.conf文件下的clientConnection.kubeconfig,修改完后效果如下:

  1. apiVersion: v1
  2. data:
  3. config.conf: |-
  4. apiVersion: kubeproxy.config.k8s.io/v1alpha1
  5. bindAddress: 0.0.0.0
  6. bindAddressHardFail: false
  7. clientConnection:
  8. acceptContentTypes: ""
  9. burst: 0
  10. contentType: ""
  11. #kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  12. qps: 0
  13. clusterCIDR: 100.64.0.0/10
  14. configSyncPeriod: 0s
  15. // 省略

重启KubeProxy Pod

为使上述配置生效,需要重启kubeproxy的pod,线上环境谨慎操作

  1. kubectl delete pod -n kube-system -l k8s-app=kube-proxy

KubeProxy功能验证

可以通过KubeProxy的日志进行验证是否修改成功,为防止日志过多,生产环境谨慎使用

  1. kubectl edit ds -n kube-system kube-proxy

在command后面追加参数--v=6,修改后效果:

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata:
  4. annotations:
  5. deprecated.daemonset.template.generation: "3"
  6. creationTimestamp: "2022-05-10T06:27:27Z"
  7. generation: 3
  8. labels:
  9. k8s-app: kube-proxy
  10. name: kube-proxy
  11. namespace: kube-system
  12. resourceVersion: "5377081"
  13. uid: 0f8eccdd-d26f-48f0-8401-8d762a630dc8
  14. spec:
  15. revisionHistoryLimit: 10
  16. selector:
  17. matchLabels:
  18. k8s-app: kube-proxy
  19. template:
  20. metadata:
  21. creationTimestamp: null
  22. labels:
  23. k8s-app: kube-proxy
  24. spec:
  25. containers:
  26. - command:
  27. - /usr/local/bin/kube-proxy
  28. - --config=/var/lib/kube-proxy/config.conf
  29. - --hostname-override=$(NODE_NAME)
  30. - --v=6

检查KubeProxy的Pod输出日志,如果apiserver地址是:169.254.2.1:10268代表修改成功。日志输出样例:

  1. I0521 02:57:01.986790 1 round_trippers.go:454] GET https://169.254.2.1:10268/api/v1/nodes/jd-sh-qianyi-test-02 200 OK in 12 milliseconds
  2. I0521 02:57:02.021682 1 round_trippers.go:454] POST https://169.254.2.1:10268/api/v1/namespaces/default/events 201 Created in 4 milliseconds