YurtStaticSet

Background

Static pod is a special type of pod in Kubernetes, which is managed directly by Kubelet. Static pods are often used in cloud-edge collaboration scenarios, such as in some AI-related applications. In OpenYurt, the core component, YurtHub, is deployed using static pod. Static pods are typically created through configuration files located in /etc/kubernetes/manifests directory, and upgrades are performed by manually replacing or modifying these configuration files. During this process, Kubelet directly handles the creation and deletion of Static pods. However, due to the large quantity and dispersed nature of edge devices, deploying and upgrading Static pods manually in cloud-edge collaboration scenarios can lead to significant operational burdens and risks of mistakes. Therefore, OpenYurt has introduced a new Custom Resource Definition (CRD), YurtStaticSet, to enhance the management of Static pods. It provides capabilities such as rolling update and Over-The-Air (OTA) upgrade through a custom controller.

Configuration

  1. apiVersion: apps.openyurt.io/v1alpha1
  2. kind: YurtStaticSet
  3. metadata:
  4. # ···
  5. spec:
  6. # static pod configuration file name
  7. staticPodManifest: xxx
  8. # Upgrade strategy, supporting AdvancedRollingUpdate and OTA upgrade modes
  9. upgradeStrategy:
  10. type: AdvancedRollingUpdate
  11. # For AdvancedRollingUpdate upgrade mode, set the maximum unavailable count during rolling update, default is 10%
  12. # maxUnavailable: 3
  13. # static pod template
  14. template:
  15. # ···

Usage

1)Deploy OpenYurt

The yurt-static-set controller is integrated within Yurt-Manager component. Before using, OpenYurt needs to be installed and deployed. You can refer to Deploy OpenYurt for detailed operations.

2)Create static pod

YurtStaticSet Operator does not manage the initialization of the static pod, which must be done manually or via yurtadm tool. As an example, this guide creates a cluster with three worker nodes using Kind, and manually deploys a static pod on each node.

  1. cat > nginx.yaml << EOF
  2. apiVersion: v1
  3. kind: Pod
  4. metadata:
  5. name: nginx
  6. spec:
  7. containers:
  8. - name: web
  9. image: nginx:1.19.1
  10. EOF

3) Deploy YurtStaticSet CR

YurtStaticSet resource corresponds to static pods via its namespace/name. Therefore, we create a CR instance with namespace:default, name:nginx to take over these static pods.

  1. cat <<EOF | kubectl apply -f -
  2. apiVersion: apps.openyurt.io/v1alpha1
  3. kind: YurtStaticSet
  4. metadata:
  5. name: nginx
  6. spec:
  7. staticPodManifest: nginx
  8. upgradeStrategy:
  9. type: AdvancedRollingUpdate
  10. maxUnavailable: 3
  11. template:
  12. metadata:
  13. name: nginx
  14. spec:
  15. containers:
  16. - name: web
  17. image: nginx:1.19.1
  18. EOF

4) Upgrade

We can easily manage static pods with YurtStaticSet resource, including upgrading static pods. YurtStaticSet supports two upgrade modes, AdvancedRollingUpdate and OTA. Simply, AdvancedRollingUpdate mode offers rolling update that skips not-ready nodes; OTA mode allows users to control the upgrade process. Detailed introductions of the two modes can be found in DaemonSet Upgrade Model.

AdvancedRollingUpdate Upgrade

  • Check the static pods in the cluster before upgrade
  1. $ kubectl get pods | grep nginx
  2. default nginx-openyurt-e2e-test-worker 1/1 Running 0 3h4m 10.244.2.3 openyurt-e2e-test-worker
  3. default nginx-openyurt-e2e-test-worker2 1/1 Running 0 3h4m 10.244.1.2 openyurt-e2e-test-worker2
  4. default nginx-openyurt-e2e-test-worker3 1/1 Running 0 3h5m 10.244.3.3 openyurt-e2e-test-worker3
  5. $ kubectl describe pods nginx-openyurt-e2e-test-worker
  6. ···
  7. Containers:
  8. web:
  9. ···
  10. # At this time, the version of nginx pod is 1.19.1
  11. Image: nginx:1.19.1
  12. ···
  13. ···
  • Modify YurtStaticSet spec, upgrading the container image from nginx:1.19.1 to nginx:1.19.2
  1. apiVersion: apps.openyurt.io/v1alpha1
  2. kind: YurtStaticSet
  3. metadata:
  4. name: nginx
  5. spec:
  6. ···
  7. spec:
  8. containers:
  9. - name: web
  10. image: nginx:1.19.2
  • Check the resource status, we can see that all three static pods have been upgraded. TOTAL represents how many static pods YurtStaticSet nginx matches in this cluster. READY represents the number of ready pods, and UPGRADED indicates how many pods have been upgraded to the latest version.
  1. $ kubectl get yurtstaticsets nginx
  2. NAME AGE TOTAL READY UPGRADED
  3. nginx 4m20s 3 3 3
  • Check the corresponding static pods in the cluster
  1. $ kubectl describe pods nginx-openyurt-e2e-test-worker
  2. ···
  3. Containers:
  4. web:
  5. ···
  6. # At this point, the version of the nginx pod has been upgraded to 1.19.2
  7. Image: nginx:1.19.2
  8. ···
  9. ···

OTA Upgrade

  1. OTA Upgrade API

    YurtHub provides two OTA upgrade-related REST APIs.

    • GET /pods This interface can be used to obtain information about the pods on the node.
    • POST /openyurt.io/v1/namespaces/{ns}/pods/{podname}/upgrade This interface allows users to specify the upgrade of a particular static Pod. The path parameters ns and podname represent the namespace and name of the Pod respectively.
  2. OTA Upgrade Process

    • In OTA upgrades, the upgrade status is represented by the PodNeedUpgrade condition field in pod status. When the value is true, it means there is upgradable version available. Otherwise, it cannot be upgraded.
    • Static pod upgrades can be achieved by actively calling the upgrade API interface above.