多节点高可用部署

本文提供 Kubernetes 的高可用部署和 KubeCube 的高可用部署方案,VIP 的实现需要用户自行提供

v1.4.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力,当然,你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.4
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成,master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip,在此我们用任意 master 节点的 ip 代替
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install k8s
  4. INSTALL_KUBERNETES="true"
  5. # k8s cni, support now is calico only
  6. CNI="calico"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="control-plane-master"
  13. # zone has two choice
  14. # 1. "cn" : in mainland
  15. # 2. "others" : out of mainland
  16. ZONE="cn"
  17. # k8s version you want to install
  18. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  19. KUBERNETES_VERSION="1.23.5"
  20. # +optional
  21. # must be set when NODE_MODE="control-plane-master"
  22. # or "node-join-control-plane"
  23. CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns
  24. # +optional
  25. # KUBERNETES_BIND_ADDRESS generally is node_ip
  26. # can be set when NODE_MODE="master" ot "control-plane-master"
  27. # default value is $(hostname -I |awk '{print $1}')
  28. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  29. #######################################################################
  30. # member cluster config
  31. # used when INSTALL_KUBECUBE_MEMBER="true"
  32. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  35. # +optional
  36. # KUBECUBE_HOST must be set when as a member cluster to
  37. # join pivot cluster, the value is pivot node ip
  38. KUBECUBE_HOST=""
  39. # +optional
  40. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  41. # this value is the name of member cluster you
  42. # want to take over
  43. MEMBER_CLUSTER_NAME=""
  44. #######################################################################
  45. # ssh config
  46. # used when NODE_MODE="node-join-master" or node-join-control-plane
  47. #######################################################################
  48. # +optional
  49. # master ip means master node ip of cluster
  50. MASTER_IP=""
  51. # +optional
  52. # the user who can access master node, it can be empty
  53. SSH_USER="root"
  54. # +optional
  55. # the port specified to access master node, it can be empty
  56. SSH_PORT=22
  57. # +optional
  58. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  59. # password for master user to access master node
  60. ACCESS_PASSWORD=""
  61. # +optional
  62. # must be empty when ACCESS_PASSWORD set
  63. # ACCESS_PRIVATE_KEY for master user to access master node
  64. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  65. #######################################################################
  66. # offline config
  67. # used when offline install choose, must lift offline pkg first
  68. #######################################################################
  69. OFFLINE_INSTALL="false"
  70. OFFLINE_PKG_PATH=""
  71. #######################################################################
  72. # container runtime config
  73. # if value is docker, then use docker as container runtime
  74. # else if value is containerd, then use containerd as container runtime
  75. #######################################################################
  76. CONTAINER_RUNTIME="containerd"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.4
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群,worker2 和 worker3 加入集群的方式与之相同

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install k8s
  4. INSTALL_KUBERNETES="true"
  5. # k8s cni, support now is calico only
  6. CNI="calico"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="node-join-master"
  13. # zone has two choice
  14. # 1. "cn" : in mainland
  15. # 2. "others" : out of mainland
  16. ZONE="cn"
  17. # k8s version you want to install
  18. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  19. KUBERNETES_VERSION="1.23.5"
  20. # +optional
  21. # must be set when NODE_MODE="control-plane-master"
  22. # or "node-join-control-plane"
  23. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  24. # +optional
  25. # KUBERNETES_BIND_ADDRESS generally is node_ip
  26. # can be set when NODE_MODE="master" ot "control-plane-master"
  27. # default value is $(hostname -I |awk '{print $1}')
  28. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  29. #######################################################################
  30. # member cluster config
  31. # used when INSTALL_KUBECUBE_MEMBER="true"
  32. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  35. # +optional
  36. # KUBECUBE_HOST must be set when as a member cluster to
  37. # join pivot cluster, the value is pivot node ip
  38. KUBECUBE_HOST=""
  39. # +optional
  40. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  41. # this value is the name of member cluster you
  42. # want to take over
  43. MEMBER_CLUSTER_NAME=""
  44. #######################################################################
  45. # ssh config
  46. # used when NODE_MODE="node-join-master" or node-join-control-plane
  47. #######################################################################
  48. # +optional
  49. # master ip means master node ip of cluster
  50. MASTER_IP=""
  51. # +optional
  52. # the user who can access master node, it can be empty
  53. SSH_USER="root"
  54. # +optional
  55. # the port specified to access master node, it can be empty
  56. SSH_PORT=22
  57. # +optional
  58. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  59. # password for master user to access master node
  60. ACCESS_PASSWORD=""
  61. # +optional
  62. # must be empty when ACCESS_PASSWORD set
  63. # ACCESS_PRIVATE_KEY for master user to access master node
  64. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  65. #######################################################################
  66. # offline config
  67. # used when offline install choose, must lift offline pkg first
  68. #######################################################################
  69. OFFLINE_INSTALL="false"
  70. OFFLINE_PKG_PATH=""
  71. #######################################################################
  72. # container runtime config
  73. # if value is docker, then use docker as container runtime
  74. # else if value is containerd, then use containerd as container runtime
  75. #######################################################################
  76. CONTAINER_RUNTIME="docker"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.4
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  3. # if install k8s
  4. INSTALL_KUBERNETES="false"
  5. # k8s cni, support now is calico only
  6. CNI="calico"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="master"
  13. # zone has two choice
  14. # 1. "cn" : in mainland
  15. # 2. "others" : out of mainland
  16. ZONE="cn"
  17. # k8s version you want to install
  18. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  19. KUBERNETES_VERSION="1.23.5"
  20. # +optional
  21. # must be set when NODE_MODE="control-plane-master"
  22. # or "node-join-control-plane"
  23. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  24. # +optional
  25. # KUBERNETES_BIND_ADDRESS generally is node_ip
  26. # can be set when NODE_MODE="master" ot "control-plane-master"
  27. # default value is $(hostname -I |awk '{print $1}')
  28. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  29. #######################################################################
  30. # member cluster config
  31. # used when INSTALL_KUBECUBE_MEMBER="true"
  32. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  35. # +optional
  36. # KUBECUBE_HOST must be set when as a member cluster to
  37. # join pivot cluster, the value is pivot node ip
  38. KUBECUBE_HOST=""
  39. # +optional
  40. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  41. # this value is the name of member cluster you
  42. # want to take over
  43. MEMBER_CLUSTER_NAME=""
  44. #######################################################################
  45. # ssh config
  46. # used when NODE_MODE="node-join-master" or node-join-control-plane
  47. #######################################################################
  48. # +optional
  49. # master ip means master node ip of cluster
  50. MASTER_IP=""
  51. # +optional
  52. # the user who can access master node, it can be empty
  53. SSH_USER="root"
  54. # +optional
  55. # the port specified to access master node, it can be empty
  56. SSH_PORT=22
  57. # +optional
  58. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  59. # password for master user to access master node
  60. ACCESS_PASSWORD=""
  61. # +optional
  62. # must be empty when ACCESS_PASSWORD set
  63. # ACCESS_PRIVATE_KEY for master user to access master node
  64. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  65. #######################################################################
  66. # offline config
  67. # used when offline install choose, must lift offline pkg first
  68. #######################################################################
  69. OFFLINE_INSTALL="false"
  70. OFFLINE_PKG_PATH=""
  71. #######################################################################
  72. # container runtime config
  73. # if value is docker, then use docker as container runtime
  74. # else if value is containerd, then use containerd as container runtime
  75. #######################################################################
  76. CONTAINER_RUNTIME="docker"

v1.2.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力,当然,你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.2
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成,master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip,在此我们用任意 master 节点的 ip 代替
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install k8s
  4. INSTALL_KUBERNETES="true"
  5. # k8s cni, support now is calico only
  6. CNI="calico"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="control-plane-master"
  13. # zone has two choice
  14. # 1. "cn" : in mainland
  15. # 2. "others" : out of mainland
  16. ZONE="cn"
  17. # k8s version you want to install
  18. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  19. KUBERNETES_VERSION="1.23.5"
  20. # +optional
  21. # must be set when NODE_MODE="control-plane-master"
  22. # or "node-join-control-plane"
  23. CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns
  24. # +optional
  25. # KUBERNETES_BIND_ADDRESS generally is node_ip
  26. # can be set when NODE_MODE="master" ot "control-plane-master"
  27. # default value is $(hostname -I |awk '{print $1}')
  28. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  29. #######################################################################
  30. # member cluster config
  31. # used when INSTALL_KUBECUBE_MEMBER="true"
  32. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  35. # +optional
  36. # KUBECUBE_HOST must be set when as a member cluster to
  37. # join pivot cluster, the value is pivot node ip
  38. KUBECUBE_HOST=""
  39. # +optional
  40. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  41. # this value is the name of member cluster you
  42. # want to take over
  43. MEMBER_CLUSTER_NAME=""
  44. #######################################################################
  45. # ssh config
  46. # used when NODE_MODE="node-join-master" or node-join-control-plane
  47. #######################################################################
  48. # +optional
  49. # master ip means master node ip of cluster
  50. MASTER_IP=""
  51. # +optional
  52. # the user who can access master node, it can be empty
  53. SSH_USER="root"
  54. # +optional
  55. # the port specified to access master node, it can be empty
  56. SSH_PORT=22
  57. # +optional
  58. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  59. # password for master user to access master node
  60. ACCESS_PASSWORD=""
  61. # +optional
  62. # must be empty when ACCESS_PASSWORD set
  63. # ACCESS_PRIVATE_KEY for master user to access master node
  64. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  65. #######################################################################
  66. # offline config
  67. # used when offline install choose, must lift offline pkg first
  68. #######################################################################
  69. OFFLINE_INSTALL="false"
  70. OFFLINE_PKG_PATH=""
  71. #######################################################################
  72. # container runtime config
  73. # if value is docker, then use docker as container runtime
  74. # else if value is containerd, then use containerd as container runtime
  75. #######################################################################
  76. CONTAINER_RUNTIME="containerd"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.2
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群,worker2 和 worker3 加入集群的方式与之相同

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install k8s
  4. INSTALL_KUBERNETES="true"
  5. # k8s cni, support now is calico only
  6. CNI="calico"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="node-join-master"
  13. # zone has two choice
  14. # 1. "cn" : in mainland
  15. # 2. "others" : out of mainland
  16. ZONE="cn"
  17. # k8s version you want to install
  18. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  19. KUBERNETES_VERSION="1.23.5"
  20. # +optional
  21. # must be set when NODE_MODE="control-plane-master"
  22. # or "node-join-control-plane"
  23. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  24. # +optional
  25. # KUBERNETES_BIND_ADDRESS generally is node_ip
  26. # can be set when NODE_MODE="master" ot "control-plane-master"
  27. # default value is $(hostname -I |awk '{print $1}')
  28. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  29. #######################################################################
  30. # member cluster config
  31. # used when INSTALL_KUBECUBE_MEMBER="true"
  32. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  35. # +optional
  36. # KUBECUBE_HOST must be set when as a member cluster to
  37. # join pivot cluster, the value is pivot node ip
  38. KUBECUBE_HOST=""
  39. # +optional
  40. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  41. # this value is the name of member cluster you
  42. # want to take over
  43. MEMBER_CLUSTER_NAME=""
  44. #######################################################################
  45. # ssh config
  46. # used when NODE_MODE="node-join-master" or node-join-control-plane
  47. #######################################################################
  48. # +optional
  49. # master ip means master node ip of cluster
  50. MASTER_IP=""
  51. # +optional
  52. # the user who can access master node, it can be empty
  53. SSH_USER="root"
  54. # +optional
  55. # the port specified to access master node, it can be empty
  56. SSH_PORT=22
  57. # +optional
  58. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  59. # password for master user to access master node
  60. ACCESS_PASSWORD=""
  61. # +optional
  62. # must be empty when ACCESS_PASSWORD set
  63. # ACCESS_PRIVATE_KEY for master user to access master node
  64. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  65. #######################################################################
  66. # offline config
  67. # used when offline install choose, must lift offline pkg first
  68. #######################################################################
  69. OFFLINE_INSTALL="false"
  70. OFFLINE_PKG_PATH=""
  71. #######################################################################
  72. # container runtime config
  73. # if value is docker, then use docker as container runtime
  74. # else if value is containerd, then use containerd as container runtime
  75. #######################################################################
  76. CONTAINER_RUNTIME="containerd"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.2
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  3. # if install k8s
  4. INSTALL_KUBERNETES="false"
  5. # k8s cni, support now is calico only
  6. CNI="calico"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="master"
  13. # zone has two choice
  14. # 1. "cn" : in mainland
  15. # 2. "others" : out of mainland
  16. ZONE="cn"
  17. # k8s version you want to install
  18. # support now is: 1.19.13, 1.20.9, 1.21.2, 1.22.2, 1.23.5
  19. KUBERNETES_VERSION="1.23.5"
  20. # +optional
  21. # must be set when NODE_MODE="control-plane-master"
  22. # or "node-join-control-plane"
  23. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  24. # +optional
  25. # KUBERNETES_BIND_ADDRESS generally is node_ip
  26. # can be set when NODE_MODE="master" ot "control-plane-master"
  27. # default value is $(hostname -I |awk '{print $1}')
  28. KUBERNETES_BIND_ADDRESS="" #{node_ip}
  29. #######################################################################
  30. # member cluster config
  31. # used when INSTALL_KUBECUBE_MEMBER="true"
  32. #######################################################################
  33. # if install kubecube on member cluster
  34. INSTALL_KUBECUBE_MEMBER="false"
  35. # +optional
  36. # KUBECUBE_HOST must be set when as a member cluster to
  37. # join pivot cluster, the value is pivot node ip
  38. KUBECUBE_HOST=""
  39. # +optional
  40. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  41. # this value is the name of member cluster you
  42. # want to take over
  43. MEMBER_CLUSTER_NAME=""
  44. #######################################################################
  45. # ssh config
  46. # used when NODE_MODE="node-join-master" or node-join-control-plane
  47. #######################################################################
  48. # +optional
  49. # master ip means master node ip of cluster
  50. MASTER_IP=""
  51. # +optional
  52. # the user who can access master node, it can be empty
  53. SSH_USER="root"
  54. # +optional
  55. # the port specified to access master node, it can be empty
  56. SSH_PORT=22
  57. # +optional
  58. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  59. # password for master user to access master node
  60. ACCESS_PASSWORD=""
  61. # +optional
  62. # must be empty when ACCESS_PASSWORD set
  63. # ACCESS_PRIVATE_KEY for master user to access master node
  64. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  65. #######################################################################
  66. # offline config
  67. # used when offline install choose, must lift offline pkg first
  68. #######################################################################
  69. OFFLINE_INSTALL="false"
  70. OFFLINE_PKG_PATH=""
  71. #######################################################################
  72. # container runtime config
  73. # if value is docker, then use docker as container runtime
  74. # else if value is containerd, then use containerd as container runtime
  75. #######################################################################
  76. CONTAINER_RUNTIME="containerd"

v1.1.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

KubeCube 部署脚本提供部署高可用 k8s 的能力,当然,你也可以使用其他工具搭建高可用的 k8s 集群

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.1
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成,master2 和 master3 加入 control-plane 的方式与之相同

  • CONTROL_PLANE_ENDPOINT 为高可用 k8s-apiserver 的 vip,在此我们用任意 master 节点的 ip 代替
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install k8s
  4. INSTALL_KUBERNETES="true"
  5. # there are four node mode below:
  6. # "master" : node will be installed as a master of cluster
  7. # "node-join-master" : node will be install as a worker of cluster to join master
  8. # "control-plane-master" : node will be installed as a master to control plane of cluster
  9. # "node-join-control-plane" : node will be installed as a master to join control plane
  10. NODE_MODE="control-plane-master"
  11. # zone has two choice
  12. # 1. "cn" : in mainland
  13. # 2. "others" : out of mainland
  14. ZONE="cn"
  15. # k8s version you want to install
  16. # support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
  17. KUBERNETES_VERSION="1.20.9"
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="10.173.32.4" #{ip}:{port} , dns
  22. #######################################################################
  23. # member cluster config
  24. # used when INSTALL_KUBECUBE_MEMBER="true"
  25. #######################################################################
  26. # if install kubecube on member cluster
  27. INSTALL_KUBECUBE_MEMBER="false"
  28. # +optional
  29. # KUBECUBE_HOST must be set when as a member cluster to
  30. # join pivot cluster, the value is pivot node ip
  31. KUBECUBE_HOST=""
  32. # +optional
  33. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  34. # this value is the name of member cluster you
  35. # want to take over
  36. MEMBER_CLUSTER_NAME=""
  37. #######################################################################
  38. # ssh config
  39. # used when NODE_MODE="node-join-master" or node-join-control-plane
  40. #######################################################################
  41. # +optional
  42. # master ip means master node ip of cluster
  43. MASTER_IP=""
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. SSH_USER="root"
  47. # +optional
  48. # the port specified to access master node, it can be empty
  49. SSH_PORT=22
  50. # +optional
  51. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  52. # password for master user to access master node
  53. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  58. #######################################################################
  59. # offline config
  60. # used when offline install choose, must lift offline pkg first
  61. #######################################################################
  62. OFFLINE_INSTALL="false"
  63. OFFLINE_PKG_PATH=""

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.1
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群,worker2 和 worker3 加入集群的方式与之相同

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install k8s
  4. INSTALL_KUBERNETES="true"
  5. # there are four node mode below:
  6. # "master" : node will be installed as a master of cluster
  7. # "node-join-master" : node will be install as a worker of cluster to join master
  8. # "control-plane-master" : node will be installed as a master to control plane of cluster
  9. # "node-join-control-plane" : node will be installed as a master to join control plane
  10. NODE_MODE="node-join-master"
  11. # zone has two choice
  12. # 1. "cn" : in mainland
  13. # 2. "others" : out of mainland
  14. ZONE="cn"
  15. # k8s version you want to install
  16. # support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
  17. KUBERNETES_VERSION="1.20.9"
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  22. #######################################################################
  23. # member cluster config
  24. # used when INSTALL_KUBECUBE_MEMBER="true"
  25. #######################################################################
  26. # if install kubecube on member cluster
  27. INSTALL_KUBECUBE_MEMBER="false"
  28. # +optional
  29. # KUBECUBE_HOST must be set when as a member cluster to
  30. # join pivot cluster, the value is pivot node ip
  31. KUBECUBE_HOST=""
  32. # +optional
  33. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  34. # this value is the name of member cluster you
  35. # want to take over
  36. MEMBER_CLUSTER_NAME=""
  37. #######################################################################
  38. # ssh config
  39. # used when NODE_MODE="node-join-master" or node-join-control-plane
  40. #######################################################################
  41. # +optional
  42. # master ip means master node ip of cluster
  43. MASTER_IP="10.173.32.4"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. SSH_USER="root"
  47. # +optional
  48. # the port specified to access master node, it can be empty
  49. SSH_PORT=22
  50. # +optional
  51. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  52. # password for master user to access master node
  53. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  58. #######################################################################
  59. # offline config
  60. # used when offline install choose, must lift offline pkg first
  61. #######################################################################
  62. OFFLINE_INSTALL="false"
  63. OFFLINE_PKG_PATH=""

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.1
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  3. # if install k8s
  4. INSTALL_KUBERNETES="false"
  5. # there are four node mode below:
  6. # "master" : node will be installed as a master of cluster
  7. # "node-join-master" : node will be install as a worker of cluster to join master
  8. # "control-plane-master" : node will be installed as a master to control plane of cluster
  9. # "node-join-control-plane" : node will be installed as a master to join control plane
  10. NODE_MODE="control-plane-master"
  11. # zone has two choice
  12. # 1. "cn" : in mainland
  13. # 2. "others" : out of mainland
  14. ZONE="cn"
  15. # k8s version you want to install
  16. # support now is: 1.20.9, 1.19.13, 1.18.20, 1.21.2
  17. KUBERNETES_VERSION="1.20.9"
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  22. #######################################################################
  23. # member cluster config
  24. # used when INSTALL_KUBECUBE_MEMBER="true"
  25. #######################################################################
  26. # if install kubecube on member cluster
  27. INSTALL_KUBECUBE_MEMBER="false"
  28. # +optional
  29. # KUBECUBE_HOST must be set when as a member cluster to
  30. # join pivot cluster, the value is pivot node ip
  31. KUBECUBE_HOST=""
  32. # +optional
  33. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  34. # this value is the name of member cluster you
  35. # want to take over
  36. MEMBER_CLUSTER_NAME=""
  37. #######################################################################
  38. # ssh config
  39. # used when NODE_MODE="node-join-master" or node-join-control-plane
  40. #######################################################################
  41. # +optional
  42. # master ip means master node ip of cluster
  43. MASTER_IP="10.173.32.4"
  44. # +optional
  45. # the user who can access master node, it can be empty
  46. SSH_USER="root"
  47. # +optional
  48. # the port specified to access master node, it can be empty
  49. SSH_PORT=22
  50. # +optional
  51. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  52. # password for master user to access master node
  53. ACCESS_PASSWORD=""
  54. # +optional
  55. # must be empty when ACCESS_PASSWORD set
  56. # ACCESS_PRIVATE_KEY for master user to access master node
  57. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  58. #######################################################################
  59. # offline config
  60. # used when offline install choose, must lift offline pkg first
  61. #######################################################################
  62. OFFLINE_INSTALL="false"
  63. OFFLINE_PKG_PATH=""
  • cube.conf

kubecube_replicas设置为3,使得 KubeCube 使用 3 副本部署,并且由于podAntiAffinity

  1. # custom values for kubecube
  2. kubecube_replicas=3
  3. kubecube_args_logLevel="info"

v1.0.x

主机规划

IP 地址主机名角色
10.173.32.2lb1Keepalived & HAproxy
10.173.32.3lb2Keepalived & HAproxy
10.173.32.4master1master, etcd
10.173.32.5master2master, etcd
10.173.32.6master3master, etcd
10.173.32.7worker1worker
10.173.32.8worker2worker
10.173.32.9worker3worker
10.173.32.10vip 地址

⚠️master2、master3、worker1、worker2、worker3 需要能够通过密钥或者密码 ssh 访问 master1

部署高可用 Kubernetes

开始安装

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 Kubernetes 安装完成

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install kubecube on member cluster
  4. INSTALL_KUBECUBE_MEMBER="false"
  5. # if install k8s
  6. INSTALL_KUBERNETES="true"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="control-plane-master"
  13. # +optional
  14. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  15. # this value is the name of member cluster you
  16. # want to take over
  17. MEMBER_CLUSTER_NAME=""
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns
  22. # master ip means master node ip of cluster
  23. MASTER_IP="10.173.32.4"
  24. # +optional
  25. # KUBECUBE_HOST must be set when as a member cluster to
  26. # join pivot cluster, the value is pivot node ip
  27. KUBECUBE_HOST=""
  28. # zone has two choice
  29. # 1. "cn" : in mainland
  30. # 2. "others" : out of mainland
  31. ZONE="cn"
  32. # k8s version you want to install
  33. KUBERNETES_VERSION="1.20.9"
  34. # +optional
  35. # the user who can access master node, it can be empty
  36. # when NODE_MODE="master" or "control-plane-master"
  37. SSH_USER="root"
  38. # +optional
  39. # the port specified to access master node, it can be empty
  40. # when NODE_MODE="master" or "control-plane-master"
  41. SSH_PORT=22
  42. # +optional
  43. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  44. # password for master user to access master node
  45. ACCESS_PASSWORD=""
  46. # +optional
  47. # must be empty when ACCESS_PASSWORD set
  48. # ACCESS_PRIVATE_KEY for master user to access master node
  49. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

master2 节点加入 control-plane

在 master2 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 master2 加入 control-plane

master3 加入 control-plane 与此类似,仅需修改 LOCAL_IP10.173.32.6

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install kubecube on member cluster
  4. INSTALL_KUBECUBE_MEMBER="false"
  5. # if install k8s
  6. INSTALL_KUBERNETES="true"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="node-join-control-plane"
  13. # +optional
  14. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  15. # this value is the name of member cluster you
  16. # want to take over
  17. MEMBER_CLUSTER_NAME=""
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="10.173.32.10" #{ip}:{port} , dns
  22. # master ip means master node ip of cluster
  23. MASTER_IP="10.173.32.4"
  24. # +optional
  25. # KUBECUBE_HOST must be set when as a member cluster to
  26. # join pivot cluster, the value is pivot node ip
  27. KUBECUBE_HOST=""
  28. # zone has two choice
  29. # 1. "cn" : in mainland
  30. # 2. "others" : out of mainland
  31. ZONE="cn"
  32. # k8s version you want to install
  33. KUBERNETES_VERSION="1.20.9"
  34. # +optional
  35. # the user who can access master node, it can be empty
  36. # when NODE_MODE="master" or "control-plane-master"
  37. SSH_USER="root"
  38. # +optional
  39. # the port specified to access master node, it can be empty
  40. # when NODE_MODE="master" or "control-plane-master"
  41. SSH_PORT=22
  42. # +optional
  43. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  44. # password for master user to access master node
  45. ACCESS_PASSWORD=""
  46. # +optional
  47. # must be empty when ACCESS_PASSWORD set
  48. # ACCESS_PRIVATE_KEY for master user to access master node
  49. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

worker1 作为工作节点加入集群

在 worker1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 worker1 加入集群

worker2 和 worker3 加入集群的方式与之类似,仅需修改LOCAL_IP为本机 IP 即可

  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="false"
  3. # if install kubecube on member cluster
  4. INSTALL_KUBECUBE_MEMBER="false"
  5. # if install k8s
  6. INSTALL_KUBERNETES="true"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="node-join-master"
  13. # +optional
  14. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  15. # this value is the name of member cluster you
  16. # want to take over
  17. MEMBER_CLUSTER_NAME=""
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  22. # master ip means master node ip of cluster
  23. MASTER_IP="10.173.32.4"
  24. # +optional
  25. # KUBECUBE_HOST must be set when as a member cluster to
  26. # join pivot cluster, the value is pivot node ip
  27. KUBECUBE_HOST=""
  28. # zone has two choice
  29. # 1. "cn" : in mainland
  30. # 2. "others" : out of mainland
  31. ZONE="cn"
  32. # k8s version you want to install
  33. KUBERNETES_VERSION="1.20.9"
  34. # +optional
  35. # the user who can access master node, it can be empty
  36. # when NODE_MODE="master" or "control-plane-master"
  37. SSH_USER="root"
  38. # +optional
  39. # the port specified to access master node, it can be empty
  40. # when NODE_MODE="master" or "control-plane-master"
  41. SSH_PORT=22
  42. # +optional
  43. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  44. # password for master user to access master node
  45. ACCESS_PASSWORD=""
  46. # +optional
  47. # must be empty when ACCESS_PASSWORD set
  48. # ACCESS_PRIVATE_KEY for master user to access master node
  49. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"

部署高可用 KubeCube

在 master1 上执行部署脚本

  1. KUBECUBE_VERSION=v1.0
  1. export CUSTOMIZE="true";curl -fsSL https://kubecube.nos-eastchina1.126.net/kubecube-installer/${KUBECUBE_VERSION}/entry.sh | bash

设置脚本参数,并按照提示继续运行安装脚本并等待 KubeCube 部署完成

  • install.conf
  1. # if install kubecube on pivot cluster
  2. INSTALL_KUBECUBE_PIVOT="true"
  3. # if install kubecube on member cluster
  4. INSTALL_KUBECUBE_MEMBER="false"
  5. # if install k8s
  6. INSTALL_KUBERNETES="false"
  7. # there are four node mode below:
  8. # "master" : node will be installed as a master of cluster
  9. # "node-join-master" : node will be install as a worker of cluster to join master
  10. # "control-plane-master" : node will be installed as a master to control plane of cluster
  11. # "node-join-control-plane" : node will be installed as a master to join control plane
  12. NODE_MODE="control-plane-master"
  13. # +optional
  14. # must be set when INSTALL_KUBECUBE_MEMBER="true"
  15. # this value is the name of member cluster you
  16. # want to take over
  17. MEMBER_CLUSTER_NAME=""
  18. # +optional
  19. # must be set when NODE_MODE="control-plane-master"
  20. # or "node-join-control-plane"
  21. CONTROL_PLANE_ENDPOINT="" #{ip}:{port} , dns
  22. # master ip means master node ip of cluster
  23. MASTER_IP="10.173.32.4"
  24. # +optional
  25. # KUBECUBE_HOST must be set when as a member cluster to
  26. # join pivot cluster, the value is pivot node ip
  27. KUBECUBE_HOST=""
  28. # zone has two choice
  29. # 1. "cn" : in mainland
  30. # 2. "others" : out of mainland
  31. ZONE="cn"
  32. # k8s version you want to install
  33. KUBERNETES_VERSION="1.20.9"
  34. # +optional
  35. # the user who can access master node, it can be empty
  36. # when NODE_MODE="master" or "control-plane-master"
  37. SSH_USER="root"
  38. # +optional
  39. # the port specified to access master node, it can be empty
  40. # when NODE_MODE="master" or "control-plane-master"
  41. SSH_PORT=22
  42. # +optional
  43. # must be empty when ACCESS_PRIVATE_KEY_PATH set
  44. # password for master user to access master node
  45. ACCESS_PASSWORD=""
  46. # +optional
  47. # must be empty when ACCESS_PASSWORD set
  48. # ACCESS_PRIVATE_KEY for master user to access master node
  49. ACCESS_PRIVATE_KEY_PATH="/root/.ssh/id_rsa"
  • cube.conf

kubecube_replicas设置为3,使得 KubeCube 使用 3 副本部署,并且由于podAntiAffinity,它们会运行在非controlPlane的节点上,并且每个节点仅运行单个副本

  1. # custom values for kubecube
  2. kubecube_replicas=3
  3. kubecube_args_logLevel="info"

最后修改 April 11, 2023 : Release v1.8 doc (#57) (cbc6022b)