多节点高可用部署

本文提供 Kubernetes 的高可用部署和 KubeCube 的高可用部署方案，VIP 的实现需要用户自行提供

v1.4.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力，当然，你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.4
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成，master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip，在此我们用任意 master 节点的 ip 代替
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install k8s
  5. INSTALL_KUBERNETES="true"
  7. # k8s cni, support now is calico only
  8. CNI="calico"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="control-plane-master"
  17. # zone has two choice
  18. # 1. "cn" : in mainland
  19. # 2. "others" : out of mainland
  20. ZONE="cn"
  22. # k8s version you want to install
  23. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  24. KUBERNETES_VERSION="1.23.5"
  26. # +optional
  27. # must be set when NODE_MODE="control-plane-master"
  28. # or "node-join-control-plane"
  29. CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns
  31. # +optional
  32. # KUBERNETES_BIND_ADDRESS generally is node_ip
  33. # can be set when NODE_MODE="master" ot "control-plane-master"
  34. # default value is $(hostname -I |awk '{print $1}')
  35. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  37. #######################################################################
  38. # member cluster config
  39. # used when INSTALL_KUBECUBE_MEMBER="true"
  40. #######################################################################
  42. # if install kubecube on member cluster
  43. INSTALL_KUBECUBE_MEMBER="false"
  45. # +optional
  46. # KUBECUBE_HOST must be set when as a member cluster to
  47. # join pivot cluster, the value is pivot node ip
  48. KUBECUBE_HOST=""
  50. # +optional
  51. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  52. # this value is the name of member cluster you
  53. # want to take over
  54. MEMBER_CLUSTER_NAME=""
  56. #######################################################################
  57. # ssh config
  58. # used when NODE_MODE="node-join-master" or node-join-control-plane
  59. #######################################################################
  61. # +optional
  62. # master ip means master node ip of cluster
  63. MASTER_IP=""
  65. # +optional
  66. # the user who can access master node, it can be empty
  67. SSH_USER="root"
  69. # +optional
  70. # the port specified to access master node, it can be empty
  71. SSH_PORT=22
  73. # +optional
  74. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  75. # password for master user to access master node
  76. ACCESS_PASSWORD=""
  78. # +optional
  79. # must be empty when ACCESS_PASSWORD set
  80. # ACCESS_PRIVATE_KEY for master user to access master node
  81. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  83. #######################################################################
  84. # offline config
  85. # used when offline install choose, must lift offline pkg first
  86. #######################################################################
  88. OFFLINE_INSTALL="false"
  90. OFFLINE_PKG_PATH=""
  92. #######################################################################
  93. # container runtime config
  94. # if value is docker, then use docker as container runtime
  95. # else if value is containerd, then use containerd as container runtime
  96. #######################################################################
  97. CONTAINER_RUNTIME="containerd"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.4
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群，worker2 和 worker3 加入集群的方式与之相同

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install k8s
  5. INSTALL_KUBERNETES="true"
  7. # k8s cni, support now is calico only
  8. CNI="calico"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="node-join-master"
  17. # zone has two choice
  18. # 1. "cn" : in mainland
  19. # 2. "others" : out of mainland
  20. ZONE="cn"
  22. # k8s version you want to install
  23. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  24. KUBERNETES_VERSION="1.23.5"
  26. # +optional
  27. # must be set when NODE_MODE="control-plane-master"
  28. # or "node-join-control-plane"
  29. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  31. # +optional
  32. # KUBERNETES_BIND_ADDRESS generally is node_ip
  33. # can be set when NODE_MODE="master" ot "control-plane-master"
  34. # default value is $(hostname -I |awk '{print $1}')
  35. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  37. #######################################################################
  38. # member cluster config
  39. # used when INSTALL_KUBECUBE_MEMBER="true"
  40. #######################################################################
  42. # if install kubecube on member cluster
  43. INSTALL_KUBECUBE_MEMBER="false"
  45. # +optional
  46. # KUBECUBE_HOST must be set when as a member cluster to
  47. # join pivot cluster, the value is pivot node ip
  48. KUBECUBE_HOST=""
  50. # +optional
  51. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  52. # this value is the name of member cluster you
  53. # want to take over
  54. MEMBER_CLUSTER_NAME=""
  56. #######################################################################
  57. # ssh config
  58. # used when NODE_MODE="node-join-master" or node-join-control-plane
  59. #######################################################################
  61. # +optional
  62. # master ip means master node ip of cluster
  63. MASTER_IP=""
  65. # +optional
  66. # the user who can access master node, it can be empty
  67. SSH_USER="root"
  69. # +optional
  70. # the port specified to access master node, it can be empty
  71. SSH_PORT=22
  73. # +optional
  74. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  75. # password for master user to access master node
  76. ACCESS_PASSWORD=""
  78. # +optional
  79. # must be empty when ACCESS_PASSWORD set
  80. # ACCESS_PRIVATE_KEY for master user to access master node
  81. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  83. #######################################################################
  84. # offline config
  85. # used when offline install choose, must lift offline pkg first
  86. #######################################################################
  88. OFFLINE_INSTALL="false"
  90. OFFLINE_PKG_PATH=""
  92. #######################################################################
  93. # container runtime config
  94. # if value is docker, then use docker as container runtime
  95. # else if value is containerd, then use containerd as container runtime
  96. #######################################################################
  97. CONTAINER_RUNTIME="docker"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.4
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  4. # if install k8s
  5. INSTALL_KUBERNETES="false"
  7. # k8s cni, support now is calico only
  8. CNI="calico"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="master"
  17. # zone has two choice
  18. # 1. "cn" : in mainland
  19. # 2. "others" : out of mainland
  20. ZONE="cn"
  22. # k8s version you want to install
  23. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  24. KUBERNETES_VERSION="1.23.5"
  26. # +optional
  27. # must be set when NODE_MODE="control-plane-master"
  28. # or "node-join-control-plane"
  29. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  31. # +optional
  32. # KUBERNETES_BIND_ADDRESS generally is node_ip
  33. # can be set when NODE_MODE="master" ot "control-plane-master"
  34. # default value is $(hostname -I |awk '{print $1}')
  35. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  37. #######################################################################
  38. # member cluster config
  39. # used when INSTALL_KUBECUBE_MEMBER="true"
  40. #######################################################################
  42. # if install kubecube on member cluster
  43. INSTALL_KUBECUBE_MEMBER="false"
  45. # +optional
  46. # KUBECUBE_HOST must be set when as a member cluster to
  47. # join pivot cluster, the value is pivot node ip
  48. KUBECUBE_HOST=""
  50. # +optional
  51. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  52. # this value is the name of member cluster you
  53. # want to take over
  54. MEMBER_CLUSTER_NAME=""
  56. #######################################################################
  57. # ssh config
  58. # used when NODE_MODE="node-join-master" or node-join-control-plane
  59. #######################################################################
  61. # +optional
  62. # master ip means master node ip of cluster
  63. MASTER_IP=""
  65. # +optional
  66. # the user who can access master node, it can be empty
  67. SSH_USER="root"
  69. # +optional
  70. # the port specified to access master node, it can be empty
  71. SSH_PORT=22
  73. # +optional
  74. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  75. # password for master user to access master node
  76. ACCESS_PASSWORD=""
  78. # +optional
  79. # must be empty when ACCESS_PASSWORD set
  80. # ACCESS_PRIVATE_KEY for master user to access master node
  81. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  83. #######################################################################
  84. # offline config
  85. # used when offline install choose, must lift offline pkg first
  86. #######################################################################
  88. OFFLINE_INSTALL="false"
  90. OFFLINE_PKG_PATH=""
  92. #######################################################################
  93. # container runtime config
  94. # if value is docker, then use docker as container runtime
  95. # else if value is containerd, then use containerd as container runtime
  96. #######################################################################
  97. CONTAINER_RUNTIME="docker"

v1.2.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力，当然，你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.2
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成，master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip，在此我们用任意 master 节点的 ip 代替
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install k8s
  5. INSTALL_KUBERNETES="true"
  7. # k8s cni, support now is calico only
  8. CNI="calico"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="control-plane-master"
  17. # zone has two choice
  18. # 1. "cn" : in mainland
  19. # 2. "others" : out of mainland
  20. ZONE="cn"
  22. # k8s version you want to install
  23. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  24. KUBERNETES_VERSION="1.23.5"
  26. # +optional
  27. # must be set when NODE_MODE="control-plane-master"
  28. # or "node-join-control-plane"
  29. CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns
  31. # +optional
  32. # KUBERNETES_BIND_ADDRESS generally is node_ip
  33. # can be set when NODE_MODE="master" ot "control-plane-master"
  34. # default value is $(hostname -I |awk '{print $1}')
  35. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  37. #######################################################################
  38. # member cluster config
  39. # used when INSTALL_KUBECUBE_MEMBER="true"
  40. #######################################################################
  42. # if install kubecube on member cluster
  43. INSTALL_KUBECUBE_MEMBER="false"
  45. # +optional
  46. # KUBECUBE_HOST must be set when as a member cluster to
  47. # join pivot cluster, the value is pivot node ip
  48. KUBECUBE_HOST=""
  50. # +optional
  51. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  52. # this value is the name of member cluster you
  53. # want to take over
  54. MEMBER_CLUSTER_NAME=""
  56. #######################################################################
  57. # ssh config
  58. # used when NODE_MODE="node-join-master" or node-join-control-plane
  59. #######################################################################
  61. # +optional
  62. # master ip means master node ip of cluster
  63. MASTER_IP=""
  65. # +optional
  66. # the user who can access master node, it can be empty
  67. SSH_USER="root"
  69. # +optional
  70. # the port specified to access master node, it can be empty
  71. SSH_PORT=22
  73. # +optional
  74. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  75. # password for master user to access master node
  76. ACCESS_PASSWORD=""
  78. # +optional
  79. # must be empty when ACCESS_PASSWORD set
  80. # ACCESS_PRIVATE_KEY for master user to access master node
  81. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  83. #######################################################################
  84. # offline config
  85. # used when offline install choose, must lift offline pkg first
  86. #######################################################################
  88. OFFLINE_INSTALL="false"
  90. OFFLINE_PKG_PATH=""
  92. #######################################################################
  93. # container runtime config
  94. # if value is docker, then use docker as container runtime
  95. # else if value is containerd, then use containerd as container runtime
  96. #######################################################################
  97. CONTAINER_RUNTIME="containerd"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.2
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群，worker2 和 worker3 加入集群的方式与之相同

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install k8s
  5. INSTALL_KUBERNETES="true"
  7. # k8s cni, support now is calico only
  8. CNI="calico"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="node-join-master"
  17. # zone has two choice
  18. # 1. "cn" : in mainland
  19. # 2. "others" : out of mainland
  20. ZONE="cn"
  22. # k8s version you want to install
  23. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  24. KUBERNETES_VERSION="1.23.5"
  26. # +optional
  27. # must be set when NODE_MODE="control-plane-master"
  28. # or "node-join-control-plane"
  29. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  31. # +optional
  32. # KUBERNETES_BIND_ADDRESS generally is node_ip
  33. # can be set when NODE_MODE="master" ot "control-plane-master"
  34. # default value is $(hostname -I |awk '{print $1}')
  35. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  37. #######################################################################
  38. # member cluster config
  39. # used when INSTALL_KUBECUBE_MEMBER="true"
  40. #######################################################################
  42. # if install kubecube on member cluster
  43. INSTALL_KUBECUBE_MEMBER="false"
  45. # +optional
  46. # KUBECUBE_HOST must be set when as a member cluster to
  47. # join pivot cluster, the value is pivot node ip
  48. KUBECUBE_HOST=""
  50. # +optional
  51. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  52. # this value is the name of member cluster you
  53. # want to take over
  54. MEMBER_CLUSTER_NAME=""
  56. #######################################################################
  57. # ssh config
  58. # used when NODE_MODE="node-join-master" or node-join-control-plane
  59. #######################################################################
  61. # +optional
  62. # master ip means master node ip of cluster
  63. MASTER_IP=""
  65. # +optional
  66. # the user who can access master node, it can be empty
  67. SSH_USER="root"
  69. # +optional
  70. # the port specified to access master node, it can be empty
  71. SSH_PORT=22
  73. # +optional
  74. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  75. # password for master user to access master node
  76. ACCESS_PASSWORD=""
  78. # +optional
  79. # must be empty when ACCESS_PASSWORD set
  80. # ACCESS_PRIVATE_KEY for master user to access master node
  81. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  83. #######################################################################
  84. # offline config
  85. # used when offline install choose, must lift offline pkg first
  86. #######################################################################
  88. OFFLINE_INSTALL="false"
  90. OFFLINE_PKG_PATH=""
  92. #######################################################################
  93. # container runtime config
  94. # if value is docker, then use docker as container runtime
  95. # else if value is containerd, then use containerd as container runtime
  96. #######################################################################
  97. CONTAINER_RUNTIME="containerd"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.2
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  4. # if install k8s
  5. INSTALL_KUBERNETES="false"
  7. # k8s cni, support now is calico only
  8. CNI="calico"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="master"
  17. # zone has two choice
  18. # 1. "cn" : in mainland
  19. # 2. "others" : out of mainland
  20. ZONE="cn"
  22. # k8s version you want to install
  23. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  24. KUBERNETES_VERSION="1.23.5"
  26. # +optional
  27. # must be set when NODE_MODE="control-plane-master"
  28. # or "node-join-control-plane"
  29. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  31. # +optional
  32. # KUBERNETES_BIND_ADDRESS generally is node_ip
  33. # can be set when NODE_MODE="master" ot "control-plane-master"
  34. # default value is $(hostname -I |awk '{print $1}')
  35. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  37. #######################################################################
  38. # member cluster config
  39. # used when INSTALL_KUBECUBE_MEMBER="true"
  40. #######################################################################
  42. # if install kubecube on member cluster
  43. INSTALL_KUBECUBE_MEMBER="false"
  45. # +optional
  46. # KUBECUBE_HOST must be set when as a member cluster to
  47. # join pivot cluster, the value is pivot node ip
  48. KUBECUBE_HOST=""
  50. # +optional
  51. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  52. # this value is the name of member cluster you
  53. # want to take over
  54. MEMBER_CLUSTER_NAME=""
  56. #######################################################################
  57. # ssh config
  58. # used when NODE_MODE="node-join-master" or node-join-control-plane
  59. #######################################################################
  61. # +optional
  62. # master ip means master node ip of cluster
  63. MASTER_IP=""
  65. # +optional
  66. # the user who can access master node, it can be empty
  67. SSH_USER="root"
  69. # +optional
  70. # the port specified to access master node, it can be empty
  71. SSH_PORT=22
  73. # +optional
  74. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  75. # password for master user to access master node
  76. ACCESS_PASSWORD=""
  78. # +optional
  79. # must be empty when ACCESS_PASSWORD set
  80. # ACCESS_PRIVATE_KEY for master user to access master node
  81. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  83. #######################################################################
  84. # offline config
  85. # used when offline install choose, must lift offline pkg first
  86. #######################################################################
  88. OFFLINE_INSTALL="false"
  90. OFFLINE_PKG_PATH=""
  92. #######################################################################
  93. # container runtime config
  94. # if value is docker, then use docker as container runtime
  95. # else if value is containerd, then use containerd as container runtime
  96. #######################################################################
  97. CONTAINER_RUNTIME="containerd"

v1.1.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力，当然，你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.1
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成，master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip，在此我们用任意 master 节点的 ip 代替
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install k8s
  5. INSTALL_KUBERNETES="true"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="control-plane-master"
  14. # zone has two choice
  15. # 1. "cn" : in mainland
  16. # 2. "others" : out of mainland
  17. ZONE="cn"
  19. # k8s version you want to install
  20. # support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
  21. KUBERNETES_VERSION="1.20.9"
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns
  28. #######################################################################
  29. # member cluster config
  30. # used when INSTALL_KUBECUBE_MEMBER="true"
  31. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  36. # +optional
  37. # KUBECUBE_HOST must be set when as a member cluster to
  38. # join pivot cluster, the value is pivot node ip
  39. KUBECUBE_HOST=""
  41. # +optional
  42. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  43. # this value is the name of member cluster you
  44. # want to take over
  45. MEMBER_CLUSTER_NAME=""
  47. #######################################################################
  48. # ssh config
  49. # used when NODE_MODE="node-join-master" or node-join-control-plane
  50. #######################################################################
  52. # +optional
  53. # master ip means master node ip of cluster
  54. MASTER_IP=""
  56. # +optional
  57. # the user who can access master node, it can be empty
  58. SSH_USER="root"
  60. # +optional
  61. # the port specified to access master node, it can be empty
  62. SSH_PORT=22
  64. # +optional
  65. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  66. # password for master user to access master node
  67. ACCESS_PASSWORD=""
  69. # +optional
  70. # must be empty when ACCESS_PASSWORD set
  71. # ACCESS_PRIVATE_KEY for master user to access master node
  72. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  74. #######################################################################
  75. # offline config
  76. # used when offline install choose, must lift offline pkg first
  77. #######################################################################
  79. OFFLINE_INSTALL="false"
  81. OFFLINE_PKG_PATH=""

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.1
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群，worker2 和 worker3 加入集群的方式与之相同

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install k8s
  5. INSTALL_KUBERNETES="true"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="node-join-master"
  14. # zone has two choice
  15. # 1. "cn" : in mainland
  16. # 2. "others" : out of mainland
  17. ZONE="cn"
  19. # k8s version you want to install
  20. # support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
  21. KUBERNETES_VERSION="1.20.9"
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  28. #######################################################################
  29. # member cluster config
  30. # used when INSTALL_KUBECUBE_MEMBER="true"
  31. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  36. # +optional
  37. # KUBECUBE_HOST must be set when as a member cluster to
  38. # join pivot cluster, the value is pivot node ip
  39. KUBECUBE_HOST=""
  41. # +optional
  42. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  43. # this value is the name of member cluster you
  44. # want to take over
  45. MEMBER_CLUSTER_NAME=""
  47. #######################################################################
  48. # ssh config
  49. # used when NODE_MODE="node-join-master" or node-join-control-plane
  50. #######################################################################
  52. # +optional
  53. # master ip means master node ip of cluster
  54. MASTER_IP="10.173.32.4"
  56. # +optional
  57. # the user who can access master node, it can be empty
  58. SSH_USER="root"
  60. # +optional
  61. # the port specified to access master node, it can be empty
  62. SSH_PORT=22
  64. # +optional
  65. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  66. # password for master user to access master node
  67. ACCESS_PASSWORD=""
  69. # +optional
  70. # must be empty when ACCESS_PASSWORD set
  71. # ACCESS_PRIVATE_KEY for master user to access master node
  72. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  74. #######################################################################
  75. # offline config
  76. # used when offline install choose, must lift offline pkg first
  77. #######################################################################
  79. OFFLINE_INSTALL="false"
  81. OFFLINE_PKG_PATH=""

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.1
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  4. # if install k8s
  5. INSTALL_KUBERNETES="false"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="control-plane-master"
  14. # zone has two choice
  15. # 1. "cn" : in mainland
  16. # 2. "others" : out of mainland
  17. ZONE="cn"
  19. # k8s version you want to install
  20. # support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
  21. KUBERNETES_VERSION="1.20.9"
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  28. #######################################################################
  29. # member cluster config
  30. # used when INSTALL_KUBECUBE_MEMBER="true"
  31. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  36. # +optional
  37. # KUBECUBE_HOST must be set when as a member cluster to
  38. # join pivot cluster, the value is pivot node ip
  39. KUBECUBE_HOST=""
  41. # +optional
  42. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  43. # this value is the name of member cluster you
  44. # want to take over
  45. MEMBER_CLUSTER_NAME=""
  47. #######################################################################
  48. # ssh config
  49. # used when NODE_MODE="node-join-master" or node-join-control-plane
  50. #######################################################################
  52. # +optional
  53. # master ip means master node ip of cluster
  54. MASTER_IP="10.173.32.4"
  56. # +optional
  57. # the user who can access master node, it can be empty
  58. SSH_USER="root"
  60. # +optional
  61. # the port specified to access master node, it can be empty
  62. SSH_PORT=22
  64. # +optional
  65. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  66. # password for master user to access master node
  67. ACCESS_PASSWORD=""
  69. # +optional
  70. # must be empty when ACCESS_PASSWORD set
  71. # ACCESS_PRIVATE_KEY for master user to access master node
  72. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  74. #######################################################################
  75. # offline config
  76. # used when offline install choose, must lift offline pkg first
  77. #######################################################################
  79. OFFLINE_INSTALL="false"
  81. OFFLINE_PKG_PATH=""
  • cube.conf

将kubecube_replicas设置为3，使得 KubeCube 使用 3 副本部署，并且由于podAntiAffinity

  1. # custom values for kubecube
  3. kubecube_replicas=3
  4. kubecube_args_logLevel="info"

v1.0.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 Kubernetes 安装完成

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="true"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="control-plane-master"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. SSH_USER="root"
  49. # +optional
  50. # the port specified to access master node, it can be empty
  51. # when NODE_MODE="master" or "control-plane-master"
  52. SSH_PORT=22
  54. # +optional
  55. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  56. # password for master user to access master node
  57. ACCESS_PASSWORD=""
  59. # +optional
  60. # must be empty when ACCESS_PASSWORD set
  61. # ACCESS_PRIVATE_KEY for master user to access master node
  62. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

master2 节点加入 control-plane

在 master2 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 master2 加入 control-plane

master3 加入 control-plane 与此类似，仅需修改 LOCAL_IP为10.173.32.6

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="true"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="node-join-control-plane"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. SSH_USER="root"
  49. # +optional
  50. # the port specified to access master node, it can be empty
  51. # when NODE_MODE="master" or "control-plane-master"
  52. SSH_PORT=22
  54. # +optional
  55. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  56. # password for master user to access master node
  57. ACCESS_PASSWORD=""
  59. # +optional
  60. # must be empty when ACCESS_PASSWORD set
  61. # ACCESS_PRIVATE_KEY for master user to access master node
  62. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 worker1 加入集群

worker2 和 worker3 加入集群的方式与之类似，仅需修改LOCAL_IP为本机 IP 即可

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="true"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="node-join-master"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. SSH_USER="root"
  49. # +optional
  50. # the port specified to access master node, it can be empty
  51. # when NODE_MODE="master" or "control-plane-master"
  52. SSH_PORT=22
  54. # +optional
  55. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  56. # password for master user to access master node
  57. ACCESS_PASSWORD=""
  59. # +optional
  60. # must be empty when ACCESS_PASSWORD set
  61. # ACCESS_PRIVATE_KEY for master user to access master node
  62. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数，并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  4. # if install kubecube on member cluster
  5. INSTALL_KUBECUBE_MEMBER="false"
  7. # if install k8s
  8. INSTALL_KUBERNETES="false"
  10. # there are four node mode below:
  11. # "master" : node will be installed as a master of cluster
  12. # "node-join-master" : node will be install as a worker of cluster to join master
  13. # "control-plane-master" : node will be installed as a master to control plane of cluster
  14. # "node-join-control-plane" : node will be installed as a master to join control plane
  15. NODE_MODE="control-plane-master"
  17. # +optional
  18. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  19. # this value is the name of member cluster you
  20. # want to take over
  21. MEMBER_CLUSTER_NAME=""
  23. # +optional
  24. # must be set when NODE_MODE="control-plane-master"
  25. # or "node-join-control-plane"
  26. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  28. # master ip means master node ip of cluster
  29. MASTER_IP="10.173.32.4"
  31. # +optional
  32. # KUBECUBE_HOST must be set when as a member cluster to
  33. # join pivot cluster, the value is pivot node ip
  34. KUBECUBE_HOST=""
  36. # zone has two choice
  37. # 1. "cn" : in mainland
  38. # 2. "others" : out of mainland
  39. ZONE="cn"
  41. # k8s version you want to install
  42. KUBERNETES_VERSION="1.20.9"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. # when NODE_MODE="master" or "control-plane-master"
  47. SSH_USER="root"
  49. # +optional
  50. # the port specified to access master node, it can be empty
  51. # when NODE_MODE="master" or "control-plane-master"
  52. SSH_PORT=22
  54. # +optional
  55. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  56. # password for master user to access master node
  57. ACCESS_PASSWORD=""
  59. # +optional
  60. # must be empty when ACCESS_PASSWORD set
  61. # ACCESS_PRIVATE_KEY for master user to access master node
  62. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  • cube.conf

将kubecube_replicas设置为3，使得 KubeCube 使用 3 副本部署，并且由于podAntiAffinity，它们会运行在非controlPlane的节点上，并且每个节点仅运行单个副本

  1. # custom values for kubecube
  3. kubecube_replicas=3
  4. kubecube_args_logLevel="info"

最后修改 April 11, 2023 : Release v1.8 doc (#57) (cbc6022b)