KubeSphere 在华为云 ECS 高可用实例

由于对于生产环境，我们需要考虑集群的高可用性。教您部署如何在华为云 ECS 实例服务快速部署一套高可用的生产环境 Kubernetes 服务需要做到高可用，需要保证 kube-apiserver 的 HA ，推荐华为云负载均衡器服务.

前提条件

  • 请遵循该指南，确保您已经知道如何将 KubeSphere 与多节点集群一起安装。有关用于安装的 config.yaml 文件的详细信息。本教程重点介绍配置华为云负载均衡器服务高可用安装。
  • 考虑到数据的持久性，对于生产环境，我们不建议您使用存储OpenEBS，建议 NFS、GlusterFS、Ceph 等存储(需要提前准备)。文章为了进行开发和测试，集成了 OpenEBS 将 LocalPV 设置为默认的存储服务。
  • SSH 可以互相访问所有节点。
  • 所有节点的时间同步。

创建主机

本示例创建 6 台 Ubuntu 18.04 server 64bit 的云服务器，每台配置为 4 核 8 GB

主机IP主机名称角色
192.168.1.10master1master1, etcd
192.168.1.11master2master2, etcd
192.168.1.12master3master3, etcd
192.168.1.13node1node
192.168.1.14node2node
192.168.1.15node3node

注意:机器有限，所以把 etcd 放入 master，在生产环境建议单独部署 etcd，提高稳定性

华为云负载均衡器部署

创建 VPC

进入到华为云控制， 在左侧列表选择’虚拟私有云’， 选择’创建虚拟私有云’ 创建VPC,配置如下图

1-1-创建VPC

创建安全组

在 访问控制→ 安全组下，创建一个安全组，设置入方向的规则参考如下：

2-1-创建安全组

提示：后端服务器的安全组规则必须放行 100.125.0.0/16 网段，否则会导致健康检查异常，详见 后端服务器配置安全组 。此外，还应放行 192.168.1.0/24 （主机之间的网络需全放行）。

创建主机

3-1-选择主机配置 在网络配置中，网络选择第一步创建的 VPC 和子网。在安全组中，选择上一步创建的安全组。 3-2-选择网络配置

创建负载均衡器

在左侧栏选择 ‘弹性负载均衡器’,进入后选择 购买弹性负载均衡器

以下健康检查结果在部署后才会显示正常,目前状态为异常

内网LB 配置

为所有master 节点 添加后端监听器 ,监听端口为 6443

4-1-配置内网LB

4-2-配置内网LB

外网LB 配置

若集群需要配置公网访问，则需要为外网负载均衡器配置一个公网 IP为 所有节点 添加后端监听器，监听端口为 80(测试使用 30880 端口,此处 80 端口也需要在安全组中开放)。

4-3-配置外网LB

4-4-配置外网LB

后面配置文件 config.yaml 需要配置在前面创建的 SLB 分配的地址（VIP）

  1.      controlPlaneEndpoint:
  2.          domain: lb.kubesphere.local
  3.          address: "192.168.1.8"
  4.          port: 6443

获取安装程序可执行文件

下载可执行安装程序 kk 至一台目标机器：

从 GitHub Release Page 下载 KubeKey 或直接使用以下命令。

  1. curl -sfL https://get-kk.kubesphere.io | VERSION=v2.2.1 sh -

先执行以下命令以确保您从正确的区域下载 KubeKey。

  1. export KKZONE=cn

执行以下命令下载 KubeKey。

  1. curl -sfL https://get-kk.kubesphere.io | VERSION=v2.2.1 sh -

备注

在您下载 KubeKey 后，如果您将其传至新的机器，且访问 Googleapis 同样受限，在您执行以下步骤之前请务必再次执行 export KKZONE=cn 命令。

备注

执行以上命令会下载最新版 KubeKey (v2.2.1)，您可以修改命令中的版本号下载指定版本。

为 kk 添加可执行权限：

  1. chmod +x kk

提示

您可以使用高级安装来控制自定义参数或创建多节点集群。具体来说，通过指定配置文件来创建集群。

使用 kubekey 部署

在当前位置创建配置文件 master-HA.yaml：

  1. ./kk create config --with-kubesphere v3.3.0 --with-kubernetes v1.22.10 -f master-HA.yaml

提示：默认是 Kubernetes 1.17.9，这些 Kubernetes 版本也与 KubeSphere 同时进行过充分的测试： v1.15.12, v1.16.13, v1.17.9 (default), v1.18.6，您可以根据需要指定版本。

集群配置调整

目前当前集群开启了全量的组件,文末也提供了自定义的方法.可默认为 false：

  1. apiVersion: kubekey.kubesphere.io/v1alpha1
  2. kind: Cluster
  3. metadata:
  4.   name: master-HA
  5. spec:
  6.   hosts:
  7.   - {name: master1, address: 192.168.1.10, internalAddress: 192.168.1.10, password: yourpassword} # Assume that the default port for SSH is 22, otherwise add the port number after the IP address as above
  8.   - {name: master2, address: 192.168.1.11, internalAddress: 192.168.1.11, password: yourpassword} # Assume that the default port for SSH is 22, otherwise add the port number after the IP address as above
  9.   - {name: master3, address: 192.168.1.12, internalAddress: 192.168.1.12, password: yourpassword} # Assume that the default port for SSH is 22, otherwise add the port number after the IP address as above
  10.   - {name: node1, address:  192.168.1.13, internalAddress: 192.168.1.13, password: yourpassword} # Assume that the default port for SSH is 22, otherwise add the port number after the IP address as above
  11.   - {name: node2, address: 192.168.1.14, internalAddress: 192.168.1.14, password: yourpassword} # Assume that the default port for SSH is 22SSH is 22, otherwise add the port number after the IP address as above
  12.   - {name: node3, address: 192.168.1.15, internalAddress: 192.168.1.15, password: yourpassword} # Assume that the default port for SSH is 22, otherwise add the port number after the IP address as above
  13.   roleGroups:
  14.     etcd:
  15.      - master[1:3]
  16.     control-plane:
  17.      - master[1:3]
  18.     worker:
  19.      - node[1:3]
  20.   controlPlaneEndpoint:
  21.     domain: lb.kubesphere.local
  22.     address: "192.168.1.8"
  23.     port: 6443
  24.   kubernetes:
  25.     version: v1.17.9
  26.     imageRepo: kubesphere
  27.     clusterName: cluster.local
  28.     masqueradeAll: false  # masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode. [Default: false]
  29.     maxPods: 110  # maxPods is the number of pods that can run on this Kubelet. [Default: 110]
  30.     nodeCidrMaskSize: 24  # internal network node size allocation. This is the size allocated to each node on your network. [Default: 24]
  31.     proxyMode: ipvs  # mode specifies which proxy mode to use. [Default: ipvs]
  32.   network:
  33.     plugin: calico
  34.     calico:
  35.       ipipMode: Always  # IPIP Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, vxlanMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Always]
  36.       vxlanMode: Never  # VXLAN Mode to use for the IPv4 POOL created at start up. If set to a value other than Never, ipipMode should be set to "Never". [Always | CrossSubnet | Never] [Default: Never]
  37.       vethMTU: 1440  # The maximum transmission unit (MTU) setting determines the largest packet size that can be transmitted through your network. [Default: 1440]
  38.     kubePodsCIDR: 10.233.64.0/18
  39.     kubeServiceCIDR: 10.233.0.0/18
  40.   registry:
  41.     registryMirrors: ["https://*.mirror.aliyuncs.com"] # # input your registryMirrors
  42.     insecureRegistries: []
  43.     privateRegistry: ""
  44.   storage:
  45.     defaultStorageClass: localVolume
  46.     localVolume:
  47.       storageClassName: local
  49. ---
  50. apiVersion: installer.kubesphere.io/v1alpha1
  51. kind: ClusterConfiguration
  52. metadata:
  53.   name: ks-installer
  54.   namespace: kubesphere-system
  55.   labels:
  56.     version: v3.3.0
  57. spec:
  58.   local_registry: ""
  59.   persistence:
  60.     storageClass: ""
  61.   authentication:
  62.     jwtSecret: ""
  63.   etcd:
  64.     monitoring: true        # Whether to install etcd monitoring dashboard
  65.     endpointIps: 192.168.1.10,192.168.1.11,192.168.1.12  # etcd cluster endpointIps
  66.     port: 2379              # etcd port
  67.     tlsEnable: true
  68.   common:
  69.     mysqlVolumeSize: 20Gi # MySQL PVC size
  70.     minioVolumeSize: 20Gi # Minio PVC size
  71.     etcdVolumeSize: 20Gi  # etcd PVC size
  72.     openldapVolumeSize: 2Gi   # openldap PVC size
  73.     redisVolumSize: 2Gi # Redis PVC size
  74.     es:  # Storage backend for logging, tracing, events and auditing.
  75.       elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
  76.       elasticsearchDataReplicas: 1     # total number of data nodes
  77.       elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes
  78.       elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes
  79.       logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
  80.       elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log
  81.       # externalElasticsearchUrl:
  82.       # externalElasticsearchPort:
  83.   console:
  84.     enableMultiLogin: false  # enable/disable multiple sing on, it allows a user can be used by different users at the same time.
  85.     port: 30880
  86.   alerting:                # Whether to install KubeSphere alerting system. It enables Users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.
  87.     enabled: true
  88.   auditing:                # Whether to install KubeSphere audit log system. It provides a security-relevant chronological set of records，recording the sequence of activities happened in platform, initiated by different tenants.
  89.     enabled: true         
  90.   devops:                  # Whether to install KubeSphere DevOps System. It provides out-of-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image
  91.     enabled: true
  92.     jenkinsMemoryLim: 2Gi      # Jenkins memory limit
  93.     jenkinsMemoryReq: 1500Mi   # Jenkins memory request
  94.     jenkinsVolumeSize: 8Gi     # Jenkins volume size
  95.     jenkinsJavaOpts_Xms: 512m  # The following three fields are JVM parameters
  96.     jenkinsJavaOpts_Xmx: 512m
  97.     jenkinsJavaOpts_MaxRAM: 2g
  98.   events:                  # Whether to install KubeSphere events system. It provides a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.
  99.     enabled: true
  100.   logging:                 # Whether to install KubeSphere logging system. Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.
  101.     enabled: true
  102.     logsidecarReplicas: 2
  103.   metrics_server:                    # Whether to install metrics-server. IT enables HPA (Horizontal Pod Autoscaler).
  104.     enabled: true
  105.   monitoring:                        #
  106.     prometheusReplicas: 1            # Prometheus replicas are responsible for monitoring different segments of data source and provide high availability as well.
  107.     prometheusMemoryRequest: 400Mi   # Prometheus request memory
  108.     prometheusVolumeSize: 20Gi       # Prometheus PVC size
  109.     alertmanagerReplicas: 1          # AlertManager Replicas
  110.   multicluster:
  111.     clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the role of host or member cluster
  112.   networkpolicy:       # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).
  113.     enabled: true     
  114.   notification:        # It supports notification management in multi-tenant Kubernetes clusters. It allows you to set AlertManager as its sender, and receivers include Email, Wechat Work, and Slack.
  115.     enabled: true
  116.   openpitrix:          # Whether to install KubeSphere App Store. It provides an application store for Helm-based applications, and offer application lifecycle management
  117.     enabled: true
  118.   servicemesh:         # Whether to install KubeSphere Service Mesh (Istio-based). It provides fine-grained traffic management, observability and tracing, and offer visualization for traffic topology
  119.     enabled: true

持久化存储配置

如本文开头的前提条件所说，对于生产环境，我们建议您准备持久性存储，可参考以下说明进行配置。若搭建开发和测试，您可以直接使用默认集成的 OpenEBS 准备 LocalPV，则可以跳过这小节。

备注

如果您有已有存储服务端，例如华为云可使用 弹性文件存储（SFS） 来作为存储服务。继续编辑上述 config-sample.yaml 文件，找到 [addons] 字段，这里支持定义任何持久化存储的插件或客户端，如 CSI、NFS Client、Ceph、GlusterFS，您可以根据您自己的持久化存储服务类型，并参考 持久化存储服务 中对应的示例 YAML 文件进行设置。

执行命令创建集群

  1. # 指定配置文件创建集群
  2. ./kk create cluster --with-kubesphere v3.3.0 -f master-HA.yaml
  4. # 查看 KubeSphere 安装日志  -- 直到出现控制台的访问地址和登录帐户
  5. kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
  1. #####################################################
  2. ###              Welcome to KubeSphere!           ###
  3. #####################################################
  5. Console: http://192.168.1.10:30880
  6. Account: admin
  7. Password: [email protected]
  9. NOTES：
  10.   1. After you log into the console, please check the
  11.      monitoring status of service components in
  12.      the "Cluster Management". If any service is not
  13.      ready, please wait patiently until all components
  14.      are up and running.
  15.   2. Please change the default password after login.
  17. #####################################################
  18. https://kubesphere.io             2020-08-28 01:25:54
  19. #####################################################

访问公网 IP + Port 为部署后的使用情况，使用默认帐户密码 (admin/[[email protected]](https://kubesphere.io/cdn-cgi/l/email-protection))，文章组件安装为最大化，登录点击平台管理>集群管理可看到下图安装组件列表和机器情况。

如何自定义开启可插拔组件

点击集群管理 > 定制资源定义，在过滤条件框输入 ClusterConfiguration。 点击 ClusterConfiguration 详情，对 ks-installer 编辑保存退出即可，组件描述介绍：文档说明。