Control Plane High Availability

You can create high availability for the control plane by distributing the control plane across multiple nodes and installing a load balancer on top. Etcd can be colocated with the controller nodes (default in k0s) to achieve highly available datastore at the same time.

k0s high availability

Note: In this context even 2 node controlplane is considered HA even though it’s not really HA from etcd point of view. The same requirement for LB still applies.

Network considerations

You should plan to allocate the control plane nodes into different zones. This will avoid failures in case one zone fails.

For etcd high availability it’s recommended to configure 3 or 5 controller nodes. For more information, refer to the etcd documentation.

Load Balancer

Control plane high availability requires a tcp load balancer, which acts as a single point of contact to access the controllers. The load balancer needs to allow and route traffic to each controller through the following ports:

  • 6443 (for Kubernetes API)
  • 8132 (for Konnectivity)
  • 9443 (for controller join API)

The load balancer can be implemented in many different ways and k0s doesn’t have any additional requirements. You can use for example HAProxy, NGINX or your cloud provider’s load balancer.

Example configuration: HAProxy

Add the following lines to the end of the haproxy.cfg:

  1. frontend kubeAPI
  2.     bind :6443
  3.     mode tcp
  4.     default_backend kubeAPI_backend
  5. frontend konnectivity
  6.     bind :8132
  7.     mode tcp
  8.     default_backend konnectivity_backend
  9. frontend controllerJoinAPI
  10.     bind :9443
  11.     mode tcp
  12.     default_backend controllerJoinAPI_backend
  14. backend kubeAPI_backend
  15.     mode tcp
  16.     server k0s-controller1 <ip-address1>:6443 check check-ssl verify none
  17.     server k0s-controller2 <ip-address2>:6443 check check-ssl verify none
  18.     server k0s-controller3 <ip-address3>:6443 check check-ssl verify none
  19. backend konnectivity_backend
  20.     mode tcp
  21.     server k0s-controller1 <ip-address1>:8132 check check-ssl verify none
  22.     server k0s-controller2 <ip-address2>:8132 check check-ssl verify none
  23.     server k0s-controller3 <ip-address3>:8132 check check-ssl verify none
  24. backend controllerJoinAPI_backend
  25.     mode tcp
  26.     server k0s-controller1 <ip-address1>:9443 check check-ssl verify none
  27.     server k0s-controller2 <ip-address2>:9443 check check-ssl verify none
  28.     server k0s-controller3 <ip-address3>:9443 check check-ssl verify none
  30. listen stats
  31.    bind *:9000
  32.    mode http
  33.    stats enable
  34.    stats uri /

The last block “listen stats” is optional, but can be helpful. It enables HAProxy statistics with a separate dashboard to monitor for example the health of each backend server. You can access it using a web browser:

  1. http://<ip-addr>:9000

Restart HAProxy to apply the configuration changes.

k0s configuration

First and foremost, all controllers should utilize the same CA certificates and SA key pair:

  1. /var/lib/k0s/pki/ca.key
  2. /var/lib/k0s/pki/ca.crt
  3. /var/lib/k0s/pki/sa.key
  4. /var/lib/k0s/pki/sa.pub
  5. /var/lib/k0s/pki/etcd/ca.key
  6. /var/lib/k0s/pki/etcd/ca.crt

To generate these certificates, you have two options: either generate them manually using the instructions for installing custom CA certificates, and then share them between controller nodes, or use k0sctl to generate and share them automatically.

The second important aspect is: the load balancer address must be configured to k0s either by using k0s.yaml or by using k0sctl to automatically deploy all controllers with the same configuration:

Configuration using k0s.yaml (for each controller)

Note to update your load balancer’s public ip address into two places.

  1. spec:
  2.   api:
  3.     externalAddress: <load balancer public ip address>
  4.     sans:
  5.     - <load balancer public ip address>

Configuration using k0sctl.yaml (for k0sctl)

Add the following lines to the end of the k0sctl.yaml. Note to update your load balancer’s public ip address into two places.

  1.   k0s:
  2.     config:
  3.       spec:
  4.         api:
  5.           externalAddress: <load balancer public ip address>
  6.           sans:
  7.           - <load balancer public ip address>

For greater detail about k0s configuration, refer to the Full configuration file reference.