Install Calico for Windows

Big picture

Install Calico for Windows on Kubernetes clusters. The standard installation for Calico for Windows requires more time and expertise to configure. If you need to get started quickly, we recommend the Quickstart.

Value

Extend your Kubernetes deployment to Windows environments.

Before you begin

Required

  • Install and configure calicoctl

  • Linux and Windows nodes meet requirements

  • If using Calico networking, copy the kubeconfig file (used by kubelet) to each Windows node to the file, c:\k\config.

  • Download Calico for Windows and Kubernetes binaries to each Windows nodes to prepare for install:

    On each of your Windows nodes, download and run Calico for Windows installation scripts:

    1. Invoke-WebRequest https://github.com/projectcalico/calico/releases/download/v3.25.0/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1
    2. c:\install-calico-windows.ps1 -DownloadOnly yes -KubeVersion <your Kubernetes version>

    cd into C:\CalicoWindows, you will see the calico-node.exe binary, install scripts, and other files.

How to

Because the Kubernetes and Calico control components do not run on Windows yet, a hybrid Linux/Windows cluster is required. First you create a Linux cluster for Calico components, then you join Windows nodes to the Linux cluster.

The geeky details of what you get by default:

PolicyIPAMCNIOverlayRoutingDatastore

Kubernetes

  1. Create a Linux cluster
  2. Ensure pods run on the correct nodes
  3. Prepare Windows nodes to join the Linux cluster

Calico

  1. Install Calico on Linux control and worker nodes
  2. Install Calico and Kubernetes on Windows nodes

Create a Linux cluster

There are many ways to create a Linux Kubernetes cluster. We regularly test Calico for Windows with kubeadm.

Ensure pods run on the correct nodes

A primary issue of running a hybrid Kubernetes cluster is that many Kubernetes manifests do not specify a node selector to restrict where their pods can run. For example, kubeadm installs kube-proxy (Kubernetes per-host NAT daemon) using a DaemonSet that does not include a node selector. This means that the kube-proxy pod, which only supports Linux, will be scheduled to both Linux and Windows nodes. Services/pods that should run only on Linux nodes (such as the kube-proxy DaemonSet) should be started with a node selector to avoid attempting to schedule them to Windows nodes.

To get around this for kube-proxy:

  1. Use kubectl to retrieve the DaemonSet.

    1.  kubectl get ds kube-proxy -n kube-system -o yaml > kube-proxy.yaml

  2. Modify the kube-proxy.yaml file to include a node selector that selects only Linux nodes:

    1. spec:
    2.   template:
    3.   ...
    4.     spec:
    5.       nodeSelector:
    6.         kubernetes.io/os: linux
    7.       containers:

  3. Apply the updated manifest.

    1. kubectl apply -f kube-proxy.yaml

A similar change may be needed for other Kubernetes services (such as kube-dns or core-dns).

Prepare Windows nodes to join the Linux cluster

On each Windows node, follow the steps below to configure kubelet and kube-proxy service.

Step 1: Configure kubelet

kubelet must be configured to use CNI networking by setting the following command line arguments, depending on the installed container runtime.

For Docker:

  • --network-plugin=cni
  • --cni-bin-dir=<directory for CNI binaries>
  • --cni-conf-dir=<directory for CNI configuration>

For containerd:

  • --container-runtime=remote
  • --container-runtime-endpoint=npipe:////.//pipe//containerd-containerd

The CNI bin and conf dir settings are required by the Calico installer to install the CNI binaries and configuration file.

Install Calico for Windows - 图1note

Among other parameters, the containerd configuration file includes options to configure the CNI bin and conf dirs.

The following kubelet settings are also important:

  • --hostname-override can be set to $(hostname) to match Calico’s default. kubelet and Calico must agree on the host/nodename; if your network environment results in hostnames that vary over time you should set the hostname override to a static value per host and update Calico’s nodename accordingly.

  • --node-ip should be used to explicitly set the IP that kubelet reports to the API server for the node. We recommend setting this to the host’s main network adapter’s IP since we’ve seen kubelet incorrectly use an IP assigned to a HNS bridge device rather than the host’s network adapter.

  • Because of a Windows networking limitation, if using Calico IPAM, —max-pods should be set to, at most, the IPAM block size of the IP pool in use minus 4:

    IP pool block sizeMax pods
    /n2^/32-n^ - 4
    /24252
    /25124
    /26 (default)60
    /2728
    /2812
    /294
    /30 or aboveCannot be used

In addition, it’s important that kubelet is started after the vSwitch has been created, which happens when Calico initializes the dataplane. Otherwise, kubelet can be disconnected for the API server when the vSwitch is created.

AWS users: If using the AWS cloud provider, you should add the following argument to the kubelet:

--hostname-override=<aws instance private DNS name> (and set the Calico nodename variable to match). In addition, you should add KubernetesCluster=<cluster-name> as a tag when creating your Windows instance.

As a quickstart, the Calico package includes a sample script at C:\CalicoWindows\kubernetes\kubelet-service.ps1 that:

  • Waits for Calico to initialise the vSwitch
  • Starts kubelet with
    • If containerd service is running, the following flags are set:
      • --container-runtime set to remote
      • --container-runtime-endpoint set to npipe:////.//pipe//containerd-containerd
    • Otherwise, the following flags are set for Docker:
      • --network-plugin set to cni
      • --cni-bin-dir set to c:\k\cni
      • --cni-conf-dir set to c:\k\cni\config
      • --pod-infra-container-image set to kubeletwin/pause
    • --kubeconfig set to the path of node kubeconfig file
    • --hostname-override set to match Calico’s nodename
    • --node-ip set to the IP of the default vEthernet device
    • --cluster-dns set to the IPs of the dns name servers

See the README in the same directory for more details. Feel free to modify the script to adjust other kubelet parameters.

Install Calico for Windows - 图2note

The script will pause at the first stage until Calico is installed by following the instructions in the next section.

Step 2: Configure kube-proxy

kube-proxy must be configured as follows:

  • With the correct HNS network name used by the active CNI plugin. kube-proxy reads the HNS network name from an environment variable KUBE_NETWORK
    • With default configuration, Calico uses network name “Calico”
  • For VXLAN, with the source VIP for the pod subnet allocated to the node. This is the IP that kube-proxy uses when it does SNAT for a NodePort. For Calico, the source VIP should be the second IP address in the subnet chosen for the host. For example, if Calico chooses an IP block 10.0.0.0/26 then the source VIP should be 10.0.0.2. The script below will automatically wait for the block to be chosen and configure kube-proxy accordingly.
  • For Calico policy to function correctly with Kubernetes services, the WinDSR feature gate must be enabled. This requires Windows Server build 17763.1432 or greater and Kubernetes v1.14 or greater. Calico will automatically enable the WinDSR feature gate if kubernetes services are managed by Calico for Windows.

kube-proxy should be started via a script that waits for the Calico HNS network to be provisioned. The Calico package contains a suitable script for use with Calico networking at C:\CalicoWindows\kubernetes\kube-proxy-service.ps1. The script:

  • Waits for Calico to initialise the vSwitch.
  • Calculates the correct source VIP for the local subnet.
  • Starts kube-proxy with the correct feature gates and hostname to work with Calico.

See the README in the same directory for more details. Feel free to modify the script to adjust other kube-proxy parameters.

The script will pause at the first stage until Calico is installed by following the instructions in the next section.

Install Calico on Linux control and worker nodes

If using Calico BGP networking

  1. Disable the default Calico IP-in-IP networking (which is not compatible with Windows), by modifying the Calico manifest, and setting the CALICO_IPV4POOL_IPIP environment variable to “Never” before applying the manifest.

    If you do apply the manifest with the incorrect value, changing the manifest and re-applying will have no effect. To adjust the already-created IP pool:

    1. calicoctl get ippool -o yaml > ippool.yaml

    Then, modify ippool.yaml by setting the ipipMode to Never and then apply the updated manifest:

    1. calicoctl apply -f ippool.yaml

If using Calico VXLAN networking

  1. Modify VXLAN as described in Customize the manifests guide. Note the following:

    • Windows can support only a single type of IP pool so it is important that you use only a single VXLAN IP pool in this mode.
    • Windows supports only VXLAN on port 4789 and VSID >=4096. Calico’s default (on Linux and Windows) is to use port 4789 and VSID 4096.

  2. Apply the manifest using calicoctl, and verify that you have a single pool with VXLANMODE Always.

    1. $ calicoctl get ippool -o wide

  3. For Linux control nodes using Calico networking, strict affinity must be set to true. This is required to prevent Linux nodes from borrowing IP addresses from Windows nodes:

    1. kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'

Install Calico and Kubernetes on Windows nodes

Follow the steps below on each Windows node to install Kubernetes and Calico:

If using Calico BGP

Install the RemoteAccess service using the following PowerShell commands:

  1. Install-WindowsFeature RemoteAccess
  2. Install-WindowsFeature RSAT-RemoteAccess-PowerShell
  3. Install-WindowsFeature Routing

Then restart the computer:

  1. Restart-Computer -Force

before running:

  1. Install-RemoteAccess -VpnType RoutingOnly

Sometimes the remote access service fails to start automatically after install. To make sure it is running, execute the following command:

  1. Start-Service RemoteAccess

  1. If using a non-Calico network plugin for networking, install and verify it now.

  2. Edit the install configuration file, config.ps1 as follows:

    Set this variable…To…
    $env:KUBENETWORKCNI plugin you plan to use. For Calico, set the variable to Calico.*
    $env:CALICO_NETWORKING_BACKENDwindows-bgp vxlan or none (if using a non-Calico CNI plugin).
    $env:CNI variablesLocation of your Kubernetes installation.
    $env:K8SSERVICE_CIDRYour Kubernetes service cluster IP CIDR.
    $env:CALICO_DATASTORE_TYPECalico datastore you want to use.
    $env:KUBECONFIGLocation of the kubeconfig file Calico should use to access the Kubernetes API server. To set up a secure kubeconfig with the correct permissions for Calico for Windows, see Create a kubeconfig for Calico for Windows.
    $env:ETCD parametersetcd3 datastore parameters. Note: Because of a limitation of the Windows dataplane, a Kubernetes service ClusterIP cannot be used for the etcd endpoint (the host compartment cannot reach Kubernetes services).
    $env:NODENAMEHostname used by kubelet. The default uses the node’s hostname. Note: If you are using the sample kubelet start-up script from the Calico package, kubelet is started with a hostname override that forces it to use this value.
    For AWS to work properly, kubelet should use the node’s internal domain name for the AWS integration.

  3. Run the installer.

    • Change directory to the location that you unpacked the archive. For example:
  1. cd C:\CalicoWindows
  • Run the install script:
  1. .\install-calico.ps1

Install Calico for Windows - 图3note

The installer initializes the Windows vSwitch, which can cause a short connectivity outage as the networking stack is reconfigured. After running that command, you may need to:

  • Reconnect to your remote desktop session.
  • Restart kubelet and kube-proxy if they were already running.
  • If you haven’t started kubelet and kube-proxy already, you should do so now. The quickstart scripts provided in the Calico package provide an easy way to do this. Calico requires kubelet to be running to complete its per-node configuration (since Kubelet creates the Kubernetes Node resource).

Install Calico for Windows - 图4note

After you run the installer, do not move the directory because the service registration refers to the path of the directory.

  1. Verify that the Calico services are running.

    1. Get-Service -Name CalicoNode
    2. Get-Service -Name CalicoFelix

Next steps