Creating a multi-node cluster

As k0s binary has everything it needs packaged into a single binary, it makes it super easy to spin up Kubernetes clusters.

Prerequisites

Install k0s as documented in the installation instructions

Bootstrapping a controller node

Create a configuration file:

$ k0s default-config > k0s.yaml

If you wish to modify some of the settings, please check out the configuration documentation.

$ k0s install controller 
INFO[2021-02-25 15:34:59] Installing k0s service
$ systemctl start k0scontroller

k0s process will act as a “supervisor” for all of the control plane components. In a few seconds you’ll have the control plane up-and-running.

Create a join token

To be able to join workers into the cluster we need a token. The token embeds information with which we can enable mutual trust between the worker and controller(s) and allow the node to join the cluster as worker.

To get a token run the following on one of the existing controller nodes:

k0s token create --role=worker

This will output a long token string, which we will then use to add a worker to the cluster. For enhanced security, we can also set an expiration time for the token by using:

$ k0s token create --role=worker --expiry=100h > token-file

Adding Workers to a Cluster

To join the worker we need to run k0s in worker mode with the token from the previous step:

$ k0s install worker --token-file /path/to/token/file

That’s it, really.

Tokens

The tokens are actually base64 encoded kubeconfigs.

Why: - well defined structure - can be used directly as bootstrap auth configs for kubelet - embeds CA info for mutual trust

The actual bearer token embedded in the kubeconfig is a bootstrap token. For controller join token and for worker join token we use different usage attributes so we can make sure we can validate the token role on the controller side.

Adding a Controller Node

To add new controller nodes to the cluster, you must be using either etcd or an external data store (MySQL or Postgres) via kine. Please pay extra attention to the HA Configuration section in the configuration documentation, and make sure this configuration is identical for all controller nodes.

To create a join token for the new controller, run the following on an existing controller node:

$ k0s token create --role=controller --expiry=1h > token-file

On the new controller, run:

$ sudo k0s install controller --token-file /path/to/token/file

Adding a Cluster User

To add a user to cluster, use the kubeconfig create command. This will output a kubeconfig for the user, which can be used for authentication.

On the controller, run the following to generate a kubeconfig for a user:

$ k0s kubeconfig create [username]

Enabling Access to Cluster Resources

To allow the user access to the cluster, the user needs to be created with the system:masters group:

$ k0s kubeconfig create --groups "system:masters" testUser > k0s.config

Create a roleBinding to grant the user access to the resources:

$ k0s kubectl create clusterrolebinding --kubeconfig k0s.config testUser-admin-binding --clusterrole=admin --user=testUser

Service and Log Setup

k0s install sub-command was created as a helper command to allow users to easily install k0s as a service. For more information, read here.

Configuring an HA Control Plane

The following pre-requisites are required in order to configure an HA control plane:

Requirements

Load Balancer

A load balancer with a single external address should be configured as the IP gateway for the controllers. The load balancer should allow traffic to each controller on the following ports:

6443
8132
8133
9443

Cluster configuration

On each controller node, a k0s.yaml configuration file should be configured. The following options need to match on each node, otherwise the control plane components will end up in very unknown states:

network
storage: Needless to say, one cannot create a clustered controlplane with each node only storing data locally on SQLite.
externalAddress

Full configuration file refrence

Enabling Shell Completion

The k0s completion script for Bash, zsh, fish and powershell can be generated with the command k0s completion < shell >. Sourcing the completion script in your shell enables k0s autocompletion.

Bash

echo 'source <(k0s completion bash)' >>~/.bashrc

# To load completions for each session, execute once:
$ k0s completion bash > /etc/bash_completion.d/k0s

Zsh

If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once:

$ echo "autoload -U compinit; compinit" >> ~/.zshrc

# To load completions for each session, execute once:
$ k0s completion zsh > "${fpath[1]}/_k0s"

You will need to start a new shell for this setup to take effect.

Fish

$ k0s completion fish | source

# To load completions for each session, execute once:
$ k0s completion fish > ~/.config/fish/completions/k0s.fish

Multi-Node Cluster