多节点高可用部署

本文提供 Kubernetes 的高可用部署和 KubeCube 的高可用部署方案，VIP 的实现需要用户自行提供

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="true"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="control-plane-master"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. MASTER_USER="root"
  49. # +optional
  50. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  51. # password for master user to access master node
  52. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

master2 节点加入 control-plane

在 master2 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 master2 加入 control-plane

master3 加入 control-plane 与此类似，仅需修改 LOCAL_IP为10.173.32.6

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="true"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="node-join-control-plane"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. MASTER_USER="root"
  49. # +optional
  50. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  51. # password for master user to access master node
  52. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群

worker2 和 worker3 加入集群的方式与之类似，仅需修改LOCAL_IP为本机 IP 即可

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="true"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="node-join-master"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. MASTER_USER="root"
  49. # +optional
  50. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  51. # password for master user to access master node
  52. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="false"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="control-plane-master"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. MASTER_USER="root"
  49. # +optional
  50. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  51. # password for master user to access master node
  52. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  • cube.conf

将kubecube_replicas设置为3，使得 KubeCube 使用 3 副本部署，并且由于podAntiAffinity，它们会运行在非controlPlane的节点上，并且每个节点仅运行单个副本

  1. # custom values for kubecube
  3. kubecube_replicas=3
  4. kubecube_args_logLevel="info"

最后修改 August 18, 2021 : fix problem in deploy doc (#24) (e9b4a3f)