Upgrade Calico on Kubernetes

About upgrading Calico

This page describes how to upgrade to v3.27 from Calico v3.0 or later. The procedure varies by datastore type and install method.

If you are using Calico in etcd mode on a Kubernetes cluster, we recommend upgrading to the Kubernetes API datastore as discussed here.

If you have installed Calico using the calico.yaml manifest, we recommend upgrading to the Calico operator, as discussed here.

Upgrade Calico on Kubernetes - 图1note

Do not use older versions of calicoctl after the upgrade. This may result in unexpected behavior and data.

Host Endpoints

Upgrade Calico on Kubernetes - 图2caution

If your cluster has host endpoints with interfaceName: * you must prepare your cluster before upgrading. Failure to do so will result in an outage.

In versions of Calico prior to v3.14, all-interfaces host endpoints (host endpoints with interfaceName: *) only supported pre-DNAT policy. The default behavior of all-interfaces host endpoints, in the absence of any policy, was to allow all traffic.

Beginning from v3.14, all-interfaces host endpoints support normal policy in addition to pre-DNAT policy. The support for normal policy includes a change in default behavior for all-interfaces host endpoints: in the absence of policy the default behavior is to drop traffic. This default behavior is consistent with “named” host endpoints (which specify a named interface such as “eth0”); named host endpoints drop traffic in the absence of policy.

Before upgrading to v3.27, you must ensure that global network policies are in place that select existing all-interfaces host endpoints and explicitly allow existing traffic flows. As a starting point, you can create an allow-all policy that selects existing all-interfaces host endpoints. First, we’ll add a label to the existing host endpoints. Get a list of the nodes that have an all-interfaces host endpoint:

  1. calicoctl get hep -owide | grep | awk '"print $1"'

With the names of the all-interfaces host endpoints, we can label each host endpoint with a new label (for example, host-endpoint-upgrade: “”):

  1. calicoctl get hep -owide | grep '*' | awk '"print $1"' \| xargs -I {} kubectl exec -i -n kube-system calicoctl -- /calicoctl label hostendpoint {} host-endpoint-upgrade=

Now that the nodes with an all-interfaces host endpoint are labeled with host-endpoint-upgrade, we can create a policy to log and allow all traffic going into or out of the host endpoints temporarily:

  1. cat > allow-all-upgrade.yaml <<EOF
  2. apiVersion: projectcalico.org/v3
  3. kind: GlobalNetworkPolicy
  4. metadata:
  5.   name: allow-all-upgrade
  6. spec:
  7.   selector: has(host-endpoint-upgrade)
  8.   types:
  9.   - Ingress
  10.   - Egress
  11.   ingress:
  12.   - action: Log
  13.   - action: Allow
  14.   egress:
  15.   - action: Log
  16.   - action: Allow
  17. EOF

Apply the policy:

  1. calicoctl apply -f - < allow-all-upgrade.yaml

After applying this policy, all-interfaces host endpoints will log and allow all traffic through them. This policy will allow all traffic not accounted for by other policies. After upgrading, please review syslog logs for traffic going through the host endpoints and update the policy as needed to secure traffic to the host endpoints.

Upgrading an installation that was installed using helm

Prior to release v3.23, the Calico helm chart itself deployed the tigera-operator namespace and required that the helm release was installed in the default namespace. Newer releases properly defer creation of the tigera-operator namespace to the user and allow installation of the chart into the tigera-operator namespace.

When upgrading from a version of Calico v3.22 or lower to a version of Calico v3.23 or greater, you must complete the following steps to migrate ownership of the helm resources to the new chart location.

Upgrade from Calico versions prior to v3.23.0

  1. Patch existing resources so that the new chart can assume ownership.

    1. kubectl patch installation default --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
    2. kubectl patch apiserver default --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
    3. kubectl patch podsecuritypolicy tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
    4. kubectl patch -n tigera-operator deployment tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
    5. kubectl patch -n tigera-operator serviceaccount tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
    6. kubectl patch clusterrole tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
    7. kubectl patch clusterrolebinding tigera-operator tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'

  2. Apply the v3.27 CRDs:

    1. kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/operator-crds.yaml

  3. Install the helm chart in the tigera-operator namespace.

    1. helm install calico projectcalico/tigera-operator --version v3.27.4 --namespace tigera-operator

  4. Once the install has succeeded, you can delete any old releases in the default namespace.

    1. kubectl delete secret -n default -l name=calico,owner=helm --dry-run

Upgrade Calico on Kubernetes - 图3note

The above command uses —dry-run to avoid making changes to your cluster. We recommend reviewing the output and then re-running the command without —dry-run to commit to the changes.

All other upgrades

  1. Apply the v3.27 CRDs:

    1. kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/operator-crds.yaml

  2. Run the helm upgrade:

    1. helm upgrade calico projectcalico/tigera-operator

Upgrading an installation that uses the operator

  1. Download the v3.27 operator manifest.

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/tigera-operator.yaml -O

  2. Use the following command to initiate an upgrade.

    1. kubectl apply --server-side --force-conflicts -f tigera-operator.yaml

Upgrading an installation that uses manifests and the Kubernetes API datastore

  1. Download the v3.27 manifest that corresponds to your original installation method.

    Calico for policy and networking

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/calico.yaml -o upgrade.yaml

    Calico for policy and flannel for networking

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/canal.yaml -o upgrade.yaml

    Calico for policy (advanced)

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/calico-policy-only.yaml -o upgrade.yaml

    Upgrade Calico on Kubernetes - 图4note

    If you manually modified the manifest, you must manually apply the same changes to the downloaded manifest.

  2. Use the following command to initiate a rolling update.

    1. kubectl apply --server-side --force-conflicts -f upgrade.yaml

  3. Watch the status of the upgrade as follows.

    1. watch kubectl get pods -n kube-system

    Verify that the status of all Calico pods indicate Running.

    1. calico-node-hvvg8     2/2   Running   0    3m
    2. calico-node-vm8kh     2/2   Running   0    3m
    3. calico-node-w92wk     2/2   Running   0    3m

  4. Remove any existing calicoctl instances, install the new calicoctl and configure it to connect to your datastore.

  5. Use the following command to check the Calico version number.

    1. calicoctl version

    It should return a Cluster Version of v3.27.x.

  6. If you have enable application layer policy, follow the instructions below to complete your upgrade. Skip this if you are not using Istio with Calico.

  7. If you were upgrading from a version of Calico prior to v3.14 and followed the pre-upgrade steps for host endpoints above, review traffic logs from the temporary policy, add any global network policies needed to allow traffic, and delete the temporary network policy allow-all-upgrade.

  8. Congratulations! You have upgraded to Calico v3.27.

Upgrading an installation that uses an etcd datastore

  1. Download the v3.27 manifest that corresponds to your original installation method.

    Calico for policy and networking

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/calico-etcd.yaml -o upgrade.yaml

    Calico for policy and flannel for networking

    1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/canal-etcd.yaml -o upgrade.yaml

    Upgrade Calico on Kubernetes - 图5note

    You must manually apply the changes you made to the manifest during installation to the downloaded v3.27 manifest. At a minimum, you must set the etcd_endpoints value.

  2. Use the following command to initiate a rolling update.

    1. kubectl apply --server-side --force-conflicts -f upgrade.yaml

  3. Watch the status of the upgrade as follows.

    1. watch kubectl get pods -n kube-system

    Verify that the status of all Calico pods indicate Running.

    1. calico-kube-controllers-6d4b9d6b5b-wlkfj   1/1       Running   0          3m
    2. calico-node-hvvg8                          1/2       Running   0          3m
    3. calico-node-vm8kh                          1/2       Running   0          3m
    4. calico-node-w92wk                          1/2       Running   0          3m

    Upgrade Calico on Kubernetes - 图6tip

    The calico-node pods will report 1/2 in the READY column, as shown.

  4. Remove any existing calicoctl instances, install the new calicoctl and configure it to connect to your datastore.

  5. Use the following command to check the Calico version number.

    1. calicoctl version

    It should return a Cluster Version of v3.27.

  6. If you have enabled application layer policy, follow the instructions below to complete your upgrade. Skip this if you are not using Istio with Calico.

  7. If you were upgrading from a version of Calico prior to v3.14 and followed the pre-upgrade steps for host endpoints above, review traffic logs from the temporary policy, add any global network policies needed to allow traffic, and delete the temporary network policy allow-all-upgrade.

  8. Congratulations! You have upgraded to Calico v3.27.

Upgrading if you have Application Layer Policy enabled

Dikastes is versioned the same as the rest of Calico, but an upgraded calico-node will still be able to work with a downlevel Dikastes so that you will not lose data plane connectivity during the upgrade. Once calico-node is upgraded, you can begin redeploying your service pods with the updated version of Dikastes.

If you have enabled application layer policy, take the following steps to upgrade the Dikastes sidecars running in your application pods. Skip these steps if you are not using Istio with Calico.

  1. Update the Istio sidecar injector template to use the new version of Dikastes. Replace <your Istio version> below with the full version string of your Istio install, for example 1.4.2.

    1. kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.4/manifests/alp/istio-inject-configmap-<your Istio version>.yaml

  2. Once the new template is in place, newly created pods use the upgraded version of Dikastes. Perform a rolling update of each of your service deployments to get them on the new version of Dikastes.

Migrating to auto host endpoints

Upgrade Calico on Kubernetes - 图7caution

Auto host endpoints have an allow-all profile attached which allows all traffic in the absence of network policy. This may result in unexpected behavior and data.

In order to migrate existing all-interfaces host endpoints to Calico-managed auto host endpoints:

  1. Add any labels on existing all-interfaces host endpoints to their corresponding Kubernetes nodes. Calico manages labels on automatic host endpoints by syncing labels from their nodes. Any labels on existing all-interfaces host endpoints should be added to their respective nodes. For example, if your existing all-interface host endpoint for node node1 has the label environment: dev, then you must add that same label to its node:

    1. kubectl label node node1 environment=dev

  2. Enable auto host endpoints by following the enable automatic host endpoints how-to guide. Note that automatic host endpoints are created with a profile attached that allows all traffic in the absence of network policy.

    1. calicoctl patch kubecontrollersconfiguration default --patch ={"spec": {"controllers": {"node": {"hostEndpoint": {"autoCreate": "Enabled"}}}}}

  3. Delete old all-interfaces host endpoints. You can distinguish host endpoints managed by Calico from others in several ways. First, automatic host endpoints have the label projectcalico.org/created-by: calico-kube-controllers. Secondly, automatic host endpoints’ name have the suffix -auto-hep.

    1. calicoctl delete hostendpoint <old_hostendpoint_name>