Deploy KubeSphere on Bare Metal

Introduction

In addition to the deployment on cloud, KubeSphere can also be installed on bare metal. As the virtualization layer is removed, the infrastructure overhead is drastically reduced, which brings more compute and storage resources to app deployments. As a result, hardware efficiency is improved. Refer to the example below of how to deploy KubeSphere on bare metal.

Prerequisites

  • Please make sure that you already know how to install KubeSphere with a multi-node cluster based on the tutorial Multi-node Installation.
  • Server and network redundancy in your environment.
  • Considering data persistence, for a production environment, it is recommended you prepare persistent storage and create a StorageClass in advance. For development and testing, you can use the integrated OpenEBS to provision LocalPV as the storage service directly.

Prepare Linux Hosts

This tutorial uses 3 physical machines of DELL 620 Intel (R) Xeon (R) CPU E5-2640 v2 @ 2.00GHz (32G memory), on which CentOS Linux release 7.6.1810 (Core) will be installed for the minimal deployment of KubeSphere.

CentOS Installation

Download and install the image first. Make sure you allocate at least 200 GB to the root directory where it stores docker images (you can skip this if you are installing KubeSphere for testing).

For more information about the supported systems, see System Requirements.

Here is a list of the three hosts for your reference.

Host IPHost NameRole
192.168.60.152master1master1, etcd
192.168.60.153worker1worker
192.168.60.154worker2worker

NIC Setting

  1. Clear NIC configurations.
  1. ifdown em1
  1. ifdown em2
  1. rm -rf /etc/sysconfig/network-scripts/ifcfg-em1
  1. rm -rf /etc/sysconfig/network-scripts/ifcfg-em2
  1. Create the NIC bonding.
  1. nmcli con add type bond con-name bond0 ifname bond0 mode 802.3ad ip4 192.168.60.152/24 gw4 192.168.60.254
  1. Set the bonding mode.
  1. nmcli con mod id bond0 bond.options mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer2+3
  1. Bind the physical NIC.
  1. nmcli con add type bond-slave ifname em1 con-name em1 master bond0
  1. nmcli con add type bond-slave ifname em2 con-name em2 master bond0
  1. Change the NIC mode.
  1. vi /etc/sysconfig/network-scripts/ifcfg-bond0
  2. BOOTPROTO=static
  1. Restart Network Manager.
  1. systemctl restart NetworkManager
  1. nmcli con # Display NIC information
  1. Change the host name and DNS.
  1. hostnamectl set-hostname worker-1
  1. vim /etc/resolv.conf

Time Setting

  1. Synchronize time.
  1. yum install -y chrony
  1. systemctl enable chronyd
  1. systemctl start chronyd
  1. timedatectl set-ntp true
  1. Set the time zone.
  1. timedatectl set-timezone Asia/Shanghai
  1. Check if the ntp-server is available.
  1. chronyc activity -v

Firewall Setting

Execute the following commands to stop and disable the FirewallD service:

  1. iptables -F
  1. systemctl status firewalld
  1. systemctl stop firewalld
  1. systemctl disable firewalld

Package Update and Dependencies

Execute the following commands to update system packages and install dependencies.

  1. yum update
  1. yum install openssl openssl-devel
  1. yum install socat
  1. yum install epel-release
  1. yum install conntrack-tools
  1. yum install wget # This tool will be used later to download KubeKey.

Note

You may not need to install all the dependencies depending on the Kubernetes version to be installed. For more information, see Dependency Requirements.

Download KubeKey

Kubekey is the next-gen installer which provides an easy, fast and flexible way to install Kubernetes and KubeSphere.

Follow the step below to download KubeKey.

Download KubeKey using the following command:

  1. wget -c https://kubesphere.io/download/kubekey-v1.0.0-linux-amd64.tar.gz -O - | tar -xz

Download KubeKey from GitHub Release Page or use the following command directly.

  1. wget https://github.com/kubesphere/kubekey/releases/download/v1.0.0/kubekey-v1.0.0-linux-amd64.tar.gz -O - | tar -xz

Make kk executable:

  1. chmod +x kk

Create a Multi-node Cluster

With KubeKey, you can install Kubernetes and KubeSphere together. You have the option to create a multi-node cluster by customizing parameters in the configuration file.

Create a Kubernetes cluster with KubeSphere installed (e.g. --with-kubesphere v3.0.0):

  1. ./kk create config --with-kubernetes v1.17.9 --with-kubesphere v3.0.0

Note

The following Kubernetes versions have been fully tested with KubeSphere: v1.15.12, v1.16.13, v1.17.9 (default) and v1.18.6.

A default file config-sample.yaml will be created. Modify it according to your environment.

  1. vi config-sample.yaml
  1. apiVersion: kubekey.kubesphere.io/v1alpha1
  2. kind: Cluster
  3. metadata:
  4.   name: config-sample
  5. spec:
  6.   hosts:
  7.   - {name: master1, address: 192.168.60.152, internalAddress: 192.168.60.152, user: root, password: [email protected]}
  8.   - {name: worker1, address: 192.168.60.153, internalAddress: 192.168.60.153, user: root, password: [email protected]}
  9.   - {name: worker2, address: 192.168.60.154, internalAddress: 192.168.60.154, user: root, password: [email protected]}
  10.   roleGroups:
  11.     etcd:
  12.     - master1
  13.     master:
  14.     - master1
  15.     worker:
  16.     - worker1
  17.     - worker2
  18.   controlPlaneEndpoint:
  19.     domain: lb.kubesphere.local
  20.     address: ""                    
  21.     port: "6443"

Create a cluster using the configuration file you customized above:

  1. ./kk create cluster -f config-sample.yaml

Verify the Multi-node Installation

After the installation finishes, you can inspect the logs of installation by executing the command below:

  1. kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

If you can see the welcome log return, it means the installation is successful. Your cluster is up and running.

  1. **************************************************
  2. #####################################################
  3. ###              Welcome to KubeSphere!           ###
  4. #####################################################
  5. Console: http://192.168.60.152:30880
  6. Account: admin
  7. Password: [email protected]
  8. NOTES：
  9.   1. After logging into the console, please check the
  10.      monitoring status of service components in
  11.      the "Cluster Management". If any service is not
  12.      ready, please wait patiently until all components
  13.      are ready.
  14.   2. Please modify the default password after login.
  15. #####################################################
  16. https://kubesphere.io             20xx-xx-xx xx:xx:xx
  17. #####################################################

Log in the Console

You will be able to use default account and password admin/[[email protected]](https://kubesphere.io/cdn-cgi/l/email-protection) to log in the console http://{$IP}:30880 to take a tour of KubeSphere. Please change the default password after login.

Enable Pluggable Components (Optional)

The example above demonstrates the process of a default minimal installation. To enable other components in KubeSphere, see Enable Pluggable Components for more details.

System Improvements

  • Update your system.
  1. yum update
  • Add the required options to the kernel boot arguments:
  1. sudo /sbin/grubby --update-kernel=ALL --args='cgroup_enable=memory cgroup.memory=nokmem swapaccount=1'
  • Enable the overlay2 kernel module.
  1. echo "overlay2" | sudo tee -a /etc/modules-load.d/overlay.conf
  • Refresh the dynamically generated grub2 configuration.
  1. sudo grub2-set-default 0
  • Adjust kernel parameters and make the change effective.
  1. cat <<EOF | sudo tee -a /etc/sysctl.conf
  2. vm.max_map_count = 262144
  3. fs.may_detach_mounts = 1
  4. net.ipv4.ip_forward = 1
  5. vm.swappiness=1
  6. kernel.pid_max =1000000
  7. fs.inotify.max_user_instances=524288
  8. EOF
  9. sudo sysctl -p
  • Adjust system limits.
  1. vim /etc/security/limits.conf
  2. *                soft    nofile         1024000
  3. *                hard    nofile         1024000
  4. *                soft    memlock        unlimited
  5. *                hard    memlock        unlimited
  6. root             soft    nofile         1024000
  7. root             hard    nofile         1024000
  8. root             soft    memlock        unlimited
  • Remove the previous limit configuration.
  1. sudo rm /etc/security/limits.d/20-nproc.conf
  • Root the system.
  1. reboot