Install using Operator

Big picture

Install Calico for Windows on your Kubernetes cluster using the Tigera Operator.

Concepts

Calico for Windows is a hybrid implementation that requires a Linux cluster for Calico components and Linux workloads, and Windows nodes for Windows workloads.

Before you begin

Review the Linux requirements and the Calico for Windows requirements.

Before beginning, setup a Calico cluster on Linux nodes and provision Windows machines.

How to

Configure strict affinity for clusters using Calico networking

For Linux control nodes using Calico networking, strict affinity must be set to true. This is required to prevent Linux nodes from borrowing IP addresses from Windows nodes:

  1. kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'

Install using Operator - 图1note

If the above command failed to find ipamconfigurations resource, you need to install the Calico API server. Please refer to installing the Calico API server.

Install Calico for Windows using the operator

With Kubernetes v1.22, there is a new Windows container type called HostProcess containers (HPC) that can run directly on the host with access to the host network namespace, storage, and devices. With this feature, Calico for Windows can now be installed and managed using Kubernetes resources such as Daemonsets and ConfigMaps, instead of needing to configure and install Calico for Windows manually on each node. Using this installation method, the Calico for Windows services are no longer registered on the host. Instead, the services are run directly within HostProcess containers.

HPC requirements

In addition to the Calico for Windows requirements, this installation method has additional requirements:

  • Kubernetes v1.22+
  • Enable HostProcess containers support. For Kubernetes v1.22, see here. For Kubernetes v1.23+, HostProcess containers are enabled by default.
  • containerd v1.6+
  • The Windows nodes have joined the cluster.

To install containerd on the Windows node and configure the containerd service:

  1. Invoke-WebRequest https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/Install-Containerd.ps1 -OutFile c:\Install-Containerd.ps1
  2. c:\Install-Containerd.ps1 -ContainerDVersion 1.6.22 -skipHypervisorSupportCheck -CNIConfigPath "c:/etc/cni/net.d" -CNIBinPath "c:/opt/cni/bin"

If you have an existing Calico for Windows installation using the manual method, your Windows nodes may have already joined the cluster.

To join a Windows node to a cluster provisioned with kubeadm:

  • Install kubeadm and kubelet binaries and install the kubelet service
  1. Invoke-WebRequest https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/PrepareNode.ps1 -OutFile c:\PrepareNode.ps1
  2. c:\PrepareNode.ps1 -KubernetesVersion v1.26.6
  • Run kubeadm on a control plane host and copy the join command
  1. kubeadm token create --print-join-command
  • Edit the join command by updating the kubeadm.exe path to c:\k\kubeadm.exe.
  • If using a Kubernetes version earlier than v1.25, also append --cri-socket "npipe:////./pipe/containerd-containerd" to the join command. An example join command:
  1. c:\k\kubeadm.exe join 172.16.101.139:6443 --token v8w2jt.jmc45acn85dbll1e --discovery-token-ca-cert-hash sha256:d0b7040a704d8deb805ba1f29f56bbc7cea8af6aafa78137a9338a62831739b4 --cri-socket "npipe:////./pipe/containerd-containerd"
  • Run the join command on the Windows node. Shortly after it completes successfully, the Windows node will appear in kubectl get nodes. The new node’s status will be NotReady since the Calico CNI has not yet been installed.

Migrate from Calico for Windows manual installation to operator installation

The HPC installation in this guide is operator-based and will automatically migrate your Windows nodes that have Calico for Windows installed using the manual installation method. This installation process will uninstall any existing Calico for Windows services and overwrite the Calico for Windows installation files with those included in the calico/cni-windows and calico/node-windows images. If kubelet was installed using C:\CalicoWindows\kubernetes\install-kube-services.ps1, this service will remain installed. If the kube-proxy service was also installed using C:\CalicoWindows\kubernetes\install-kube-services.ps1, it will be stopped and uninstalled by the calico-node-windows HPC initContainers. It is recommended to run kube-proxy-windows as hostprocess container images using the manifest provided in the kubernetes-sigs sig-windows-tools repository.

Install using Operator - 图2note

Before proceeding, take note of the configuration parameters in C:\CalicoWindows\config.ps1. These configuration parameters will be needed during the install.

Operator installation

  • Kubernetes VXLAN
  • Kubernetes BGP

  1. Ensure that BGP is disabled.

    • If you installed Calico using the manifest, BGP is already disabled.
    • If you installed Calico using the operator, run this command:
    1. kubectl patch installation default --type=merge -p '{"spec": {"calicoNetwork": {"bgp": "Disabled"}}}'

  2. Get the cluster’s Kubernetes API server host and port, which will be used to update the Calico for Windows config map. The API server host and port is required so that the Calico for Windows installation script can create a kubeconfig file for Calico services. If your Windows nodes already have Calico for Windows installed manually, skip this step. The installation script will use the API server host and port from your node’s existing kubeconfig file if the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables are not provided in the calico-windows-config ConfigMap.

    First, make a note of the address of the API server:

    • If you have a single API server with a static IP address, you can use its IP address and port. The IP can be found by running:

      1. kubectl get endpoints kubernetes -o wide

      The output should look like the following, with a single IP address and port under “ENDPOINTS”:

      1. NAME         ENDPOINTS             AGE
      2. kubernetes   172.16.101.157:6443   40m

      If there are multiple entries under “ENDPOINTS”, then your cluster must have more than one API server. In this case, use the appropriate load balancing option below for your cluster.

    • If using DNS load balancing (as used by kops), use the FQDN and port of the API server api.internal.<clustername>.

    • If you have multiple API servers with a load balancer in front, you should use the IP and port of the load balancer.

    Install using Operator - 图3tip

    If your cluster uses a ConfigMap to configure kube-proxy you can find the “right” way to reach the API server by examining the config map. For example:

    1. kubectl get configmap -n kube-system kube-proxy -o yaml | grep server
    2. server: https://d881b853ae312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com

    In this case, the server is d881b853aea312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com and the port is 443 (the standard HTTPS port).

  3. Create the kubernetes-services-endpoint ConfigMap with the Kubernetes API server host and port (discovered in the previous step) used to create a kubeconfig file for Calico services.

    1. kubectl apply -f - << EOF
    2. kind: ConfigMap
    3. apiVersion: v1
    4. metadata:
    5.   name: kubernetes-services-endpoint
    6.   namespace: tigera-operator
    7. data:
    8.   KUBERNETES_SERVICE_HOST: "${APISERVER_ADDR}"
    9.   KUBERNETES_SERVICE_PORT: "${APISERVER_PORT}"
    10. EOF

  4. Get the Kubernetes service clusterIP range configured in your cluster. This must match the service-cluster-ip-range used by kube-apiserver.

  5. Add the Kubernetes service CIDR (discovered in the previous step) enable Calico for Windows on the Tigera operator installation resource.

    For example, with a Kubernetes service clusterIP range of 10.96.0.0/12:

    1. kubectl patch installation default --type merge --patch='{"spec": {"serviceCIDRs": ["10.96.0.0/12"], "calicoNetwork": {"windowsDataplane": "HNS"}}}'

  6. Install kube-proxy on Windows nodes.

    Depending on your platform, you may already have kube-proxy running on your Windows nodes. If kube-proxy is already running on your Windows nodes, skip this step. If kube-proxy is not running, you must install and run kube-proxy on each of the Windows nodes in your cluster. Note: The manifest provided in the kubernetes-sigs sig-windows-tools repository depends on the kubeconfig provided by the kube-proxy ConfigMap in the kube-system namespace.

    You must replace KUBE_PROXY_VERSION with your cluster’s Kubernetes version in kube-proxy.yml to ensure the daemonset uses a kube-proxy Windows image that is compatible with your Kubernetes cluster. Use a command like the following to retrieve the YAML file, replace the version, and apply it:

    1. curl -L  https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml | sed "s/KUBE_PROXY_VERSION/<YOUR_KUBERNETES_VERSION>/g" | kubectl apply -f -

  7. Monitor the installation.

    The Calico for Windows HPC installation has 2 initContainers: uninstall-calico, which deals with removing previous manually installed Calico for Windows services, if any and install-cni, which installs needed CNI binaries and configuration, when using Calico CNI.

    1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c uninstall-calico
    2. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c install-cni

    After these initContainers finish their execution, installation is complete. Next, theCalico for Windows services are started in separate containers:

    1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c node
    2. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c felix

    The calico-node-windows pods will be ready after their containers finish initializing.

  8. Enable BGP service on the Windows nodes. Install the RemoteAccess service using the following Powershell commands:

    1. Install-WindowsFeature RemoteAccess
    2. Install-WindowsFeature RSAT-RemoteAccess-PowerShell
    3. Install-WindowsFeature Routing

    Then restart the computer:

    1. Restart-Computer -Force

    before running:

    1. Install-RemoteAccess -VpnType RoutingOnly

    Sometimes the remote access service fails to start automatically after install. To make sure it is running, run the following command:

    1. Start-Service RemoteAccess

  9. Get the cluster’s Kubernetes API server host and port, which will be used to update the Calico for Windows config map. The API server host and port is required so that the Calico for Windows installation script can create a kubeconfig file for Calico services. If your Windows nodes already have Calico for Windows installed manually, skip this step. The installation script will use the API server host and port from your node’s existing kubeconfig file if the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables are not provided in the calico-windows-config ConfigMap.

    First, make a note of the address of the API server:

    • If you have a single API server with a static IP address, you can use its IP address and port. The IP can be found by running:

      1. kubectl get endpoints kubernetes -o wide

      The output should look like the following, with a single IP address and port under “ENDPOINTS”:

      1. NAME         ENDPOINTS             AGE
      2. kubernetes   172.16.101.157:6443   40m

      If there are multiple entries under “ENDPOINTS”, then your cluster must have more than one API server. In this case, use the appropriate load balancing option below for your cluster.

    • If using DNS load balancing (as used by kops), use the FQDN and port of the API server api.internal.<clustername>.

    • If you have multiple API servers with a load balancer in front, you should use the IP and port of the load balancer.

    Install using Operator - 图4tip

    If your cluster uses a ConfigMap to configure kube-proxy you can find the “right” way to reach the API server by examining the config map. For example:

    1. kubectl get configmap -n kube-system kube-proxy -o yaml | grep server
    2. server: https://d881b853ae312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com

    In this case, the server is d881b853aea312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com and the port is 443 (the standard HTTPS port).

  10. Create the kubernetes-services-endpoint ConfigMap with the Kubernetes API server host and port (discovered in the previous step) used to create a kubeconfig file for Calico services.

    1. kubectl apply -f - << EOF
    2. kind: ConfigMap
    3. apiVersion: v1
    4. metadata:
    5.   name: kubernetes-services-endpoint
    6.   namespace: tigera-operator
    7. data:
    8.   KUBERNETES_SERVICE_HOST: "${APISERVER_ADDR}"
    9.   KUBERNETES_SERVICE_PORT: "${APISERVER_PORT}"
    10. EOF

  11. Get the Kubernetes service clusterIP range configured in your cluster. This must match the service-cluster-ip-range used by kube-apiserver.

  12. Add the Kubernetes service CIDR (discovered in the previous step) enable Calico for Windows on the Tigera operator installation resource.

    For example, with a Kubernetes service clusterIP range of 10.96.0.0/12:

    1. kubectl patch installation default --type merge --patch='{"spec": {"serviceCIDRs": ["10.96.0.0/12"], "calicoNetwork": {"windowsDataplane": "HNS"}}}'

  13. Install kube-proxy on Windows nodes.

    Depending on your platform, you may already have kube-proxy running on your Windows nodes. If kube-proxy is already running on your Windows nodes, skip this step. If kube-proxy is not running, you must install and run kube-proxy on each of the Windows nodes in your cluster. Note: The manifest provided in the kubernetes-sigs sig-windows-tools repository depends on the kubeconfig provided by the kube-proxy ConfigMap in the kube-system namespace.

    You must replace KUBE_PROXY_VERSION with your cluster’s Kubernetes version in kube-proxy.yml to ensure the daemonset uses a kube-proxy Windows image that is compatible with your Kubernetes cluster. Use a command like the following to retrieve the YAML file, replace the version, and apply it:

    1. curl -L  https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml | sed "s/KUBE_PROXY_VERSION/<YOUR_KUBERNETES_VERSION>/g" | kubectl apply -f -

  14. Monitor the installation.

    The Calico for Windows HPC installation has 2 initContainers: uninstall-calico, which deals with removing previous manually installed Calico for Windows services, if any and install-cni, which installs needed CNI binaries and configuration, when using Calico CNI.

    1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c uninstall-calico
    2. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c install-cni

    After these initContainers finish their execution, installation is complete. Next, theCalico for Windows services are started in separate containers:

    1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c node
    2. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c felix
    3. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c confd

    The calico-node-windows pods will be ready after their containers finish initializing.

Congratulations! You now have a Kubernetes cluster with Calico for Windows and a Linux control node.

Next steps

You can now use the Calico Linux-based docs site for your documentation. Before you continue, review the Limitations and known issues to understand the features (and sections of documentation) that do not apply to Windows.