Self-managed Kubernetes in Google Compute Engine (GCE)

Big picture

Use Calico with a self-managed Kubernetes cluster in Google Compute Engine (GCE).

Value

Managing your own Kubernetes cluster (as opposed to using a managed-Kubernetes service like GKE) gives you the most flexibility in configuring Calico and Kubernetes. Calico combines flexible networking capabilities with “run-anywhere” security enforcement to provide a solution with native Linux kernel performance and true cloud-native scalability.

Concepts

kubeadm is a cluster management tool that is used to install Kubernetes.

Before you begin…

Install and configure the Google Cloud CLI tools

How to

There are many ways to install and manage Kubernetes in GCE. Using kubeadm is a good default choice for most people, as it gives you access to all of Calico’s flexible and powerful networking features. However, there are other options that may work better for your environment.

kubeadm for Calico networking and network policy

Create cloud resources

You will need at least one VM to serve as a control plane node and one or more worker nodes. (It is possible to have control plane nodes also act as workers. This is not recommended in most cases and not covered by this guide.) See requirements for specific OS requirements for these VMs.

The following worked example creates a single control node and three workers on a dedicated virtual private network (VPC). Adjust the example as needed for your requirements. Consider a dedicated infrastructure management tool like Terraform for managing cloud resources. (This example is adapted from Kubernetes the Hard Way.)

Create the VPC

  1. gcloud compute networks create example-k8s --subnet-mode custom

Create the k8s-nodes subnet in the example-k8s VPC network:

  1. gcloud compute networks subnets create k8s-nodes \
  2.   --network example-k8s \
  3.   --range 10.240.0.0/24

Create a firewall rule that allows internal communication across TCP, UDP, ICMP and IP in IP (used for the Calico overlay):

  1. gcloud compute firewall-rules create example-k8s-allow-internal \
  2.   --allow tcp,udp,icmp,ipip \
  3.   --network example-k8s \
  4.   --source-ranges 10.240.0.0/24

Create a firewall rule that allows external SSH, ICMP, and HTTPS:

  1. gcloud compute firewall-rules create example-k8s-allow-external \
  2.   --allow tcp:22,tcp:6443,icmp \
  3.   --network example-k8s \
  4.   --source-ranges 0.0.0.0/0

Create the controller VM:

  1. gcloud compute instances create controller \
  2.     --async \
  3.     --boot-disk-size 200GB \
  4.     --can-ip-forward \
  5.     --image-family ubuntu-1804-lts \
  6.     --image-project ubuntu-os-cloud \
  7.     --machine-type n1-standard-2 \
  8.     --private-network-ip 10.240.0.11 \
  9.     --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
  10.     --subnet k8s-nodes \
  11.     --zone us-central1-f \
  12.     --tags example-k8s,controller

Create three worker VMs

  1. for i in 0 1 2; do
  2.   gcloud compute instances create worker-${i} \
  3.     --async \
  4.     --boot-disk-size 200GB \
  5.     --can-ip-forward \
  6.     --image-family ubuntu-1804-lts \
  7.     --image-project ubuntu-os-cloud \
  8.     --machine-type n1-standard-2 \
  9.     --private-network-ip 10.240.0.2${i} \
  10.     --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
  11.     --subnet k8s-nodes \
  12.     --zone us-central1-f \
  13.     --tags example-k8s,worker
  14. done

Install Docker on the controller VM and each worker VM. On each VM run:

  1. sudo apt update
  2. sudo apt install -y docker.io
  3. sudo systemctl enable docker.service
  4. sudo apt install -y apt-transport-https curl

Install Kubernetes and create the cluster

Install kubeadm, kubelet, and kubectl on each node (see kubeadm docs for more details).

  1. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
  2. cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
  3. deb https://apt.kubernetes.io/ kubernetes-xenial main
  4. EOF
  5. sudo apt-get update
  6. sudo apt-get install -y kubelet kubeadm kubectl
  7. sudo apt-mark hold kubelet kubeadm kubectl

Create the controller node of a new cluster. On the controller VM, execute:

  1. sudo kubeadm init --pod-network-cidr 192.168.0.0/16

To set up kubectl for the ubuntu user, run:

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

The final line of the kubeadm init output contains the command for joining your workers to the controller. Run this on each worker, prepending sudo to run it as root. It will look something like this:

  1. sudo kubeadm join 10.240.0.11:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

On the controller, verify that all nodes have joined

  1. kubectl get nodes

which should output something similar to:

  1. NAME         STATUS     ROLES    AGE     VERSION
  2. controller   NotReady   master   5m49s   v1.17.2
  3. worker-0     NotReady   <none>   3m38s   v1.17.2
  4. worker-1     NotReady   <none>   3m7s    v1.17.2
  5. worker-2     NotReady   <none>   5s      v1.17.2

Install Calico

On the controller, install Calico using the operator:

  1. kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml

Download the custom resources necessary to configure Calico

  1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/custom-resources.yaml -O

If you wish to customize the Calico install, customize the downloaded custom-resources.yaml manifest. Then create the manifest to install Calico.

  1. kubectl create -f custom-resources.yaml

The geeky details of what you get:

PolicyIPAMCNIOverlayRoutingDatastore

Other tools and options

Terraform

You may have noticed that the bulk of the above instructions are about provisioning the Google Cloud resources for the cluster and installing Kubernetes. Terraform is a tool for automating infrastructure provisioning using declarative configurations. You can also go as far as automating the install of Docker, kubeadm, and Kubernetes using Terraform “provisioners.” See the Terraform documentation for more details.

Kubespray

Kubespray is a tool for provisioning and managing Kubernetes clusters with support for multiple clouds including Google Compute Engine. Calico is the default networking provider, or you can set the kube_network_plugin variable to calico. See the Kubespray docs for more details. See the Kubespray docs for more details.

Next steps

Required

Recommended