CloneSet lifecycle — Insert Customized Logic Into Pod Lifecycle

Due to various historical reasons or objective factors, the existing system architectures of some companies are not compatible with Kubernetes very well. For example, some companies use another service registration and discovery center, which is totally independent with Kubernetes, instead of Kubernetes Service. Under this architecture, users will encounter many problems if they want to manage their infrastructure and applications with Kubernetes.

For example, whenever Kubernetes successfully creates a Pod, users should register the Pod with the service discovery center so that it can provide services to the outside. Accordingly, when users want to offline a Pod, they usually need to delete it first in the service discovery center before they can gracefully offline the Pod, otherwise it may lead to some traffic routing problems.

However, in Kubernetes, if replicas or template field of workloads was changed, the workload controller will immediately create/delete/update the pod. No time is provided for users to do customized operations. Hence, it is difficult for users to manage the life cycle of the pods in this scenarios.

why

Openkruise CloneSet Lifecycle feature provides such a set of highly-expandable mechanism which allows users do refined and customized management to pod life cycle. Specifically, CloneSet provide some hooks at several important time spots in the Pod life cycle, so that users can insert their customized actions via these hooks. For example, CloneSet allow users to unregister a pod with the service discovery center before the pod upgrade, and then register the pod after the pod upgrade complete.

In the following, we will explain this feature in a specific scenario to help you further understand it.

Scenario hypothesis

We assume the following scenario:

  • Users do not use Kubernetes Service, and their service discovery system is totally independent of Kubernetes;
  • Users use CloneSet as the Kubernetes workload to manage their applications.

And we make the following reasonable assumptions about the needs of users:

  • When the Pod is created：
    • Register the Pod IP with service discovery center when the pod is available；
  • When the Pod is upgraded ：
    • Before the upgrade, Unregister the Pod IP from the service discovery center；
    • After the upgrade, Register the Pod IP with service discovery center when the pod is available；
  • When the Pod is deleted：
    • Before the deletion, Unregister the Pod IP from the service discovery center；

Based on the above assumptions, we will describe in detail how to use CloneSet Lifecycle to write a simple operator to implement the user-defined pod life cycle management.

Principle

CloneSet Lifecycle defines the pod life cycle via the following 5 states:

  • Normal: normal state;
  • PreparingUpdate: The CloneSet is preparing to update the pod. The pod upgrade will be blocked, so as to wait for users to execute hook and complete some preprocessing operations before upgrading;
  • Updating: the pod is being upgraded in inPlace way;
  • Updated: the pod upgrade is completed.
  • PreparingDelete: the CloneSet is preparing to delete the pod. The pod deletion will be blocked, so as to wait for users to execute hook and do some preprocessing operations before deletion;

The transition logic among the above five states is controlled by a state machine, which is explained in detail in [CloneSet official document] (https://openkruise.io/docs/user-manuals/cloneset/#lifecycle-hook). Users can select one or more of their concerns, implement an independent operator to manage the pod life cycle states, control the life cycle of pod, and insert customized logic at the time spots they are concerned about.

CloneSet Lifecycle Configuration

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: CloneSet
  3. metadata:
  4.   namespace: demo
  5.   name: cloneset-lifecycle-demo
  6. spec:
  7.   replicas: 2
  8.   ##########################################################################
  9.   ## Lifecycle configuration
  10.   lifecycle:
  11.     inPlaceUpdate:
  12.       labelsHandler:
  13.         ## define the label that:
  14.         ##    1. block inPlace update pod operation for cloneset controller
  15.         ##    2. inform your operator to execute inPlace update hook
  16.         example.com/unready-blocker-inplace: "true"
  17.     preDelete:
  18.       labelsHandler:
  19.         ## define the label that:
  20.         ##    1. block deletion pod operation for cloneset controller
  21.         ##    2. inform your operator to execute preDelete hook
  22.         example.com/unready-blocker-delete: "true"
  23.   ##########################################################################
  24.   selector:
  25.     matchLabels:
  26.       app: nginx
  27.   template:
  28.     metadata:
  29.       labels:
  30.         app: nginx
  31.         ## this label is useful to judge whether this pod is newly-created.
  32.         example.com/newly-create: "true"
  33.         ## corresponding to the spec.lifecycle.inPlaceUpdate.labelsHandler.example.com/unready-blocker-inplace
  34.         example.com/unready-blocker-inplace: "true"
  35.         ## corresponding to the spec.lifecycle.preDelete.labelsHandler.example.com/unready-blocker-inplace
  36.         example.com/unready-blocker-delete: "true"
  37.     spec:
  38.       containers:
  39.         - name: main
  40.           image: nginx:latest
  41.           imagePullPolicy: Always
  42.   updateStrategy:
  43.     maxUnavailable: 20%
  44.     type: InPlaceIfPossible

Lifecycle Operator

In the OpenKruise repository, we give a complete CloneSet Lifecycle Operator code example. Therefore, we will not repeat some code details in this article, but focus on explaining some key logic of the operation under the hypothetical scenario. If you need to refer to the complete code, you can go directly to see this repository.

Operator initialization

It is recommended to use Kubebuilder to build the operator. Please refer to Kubebuilder official document for the specific steps.

The directory structure of an initialized operator is similar to the following:

  1. $ tree
  2. .
  3. ├── Dockerfile
  4. ├── LICENSE
  5. ├── Makefile
  6. ├── PROJECT
  7. ├── README.md
  8. ├── config
  9. │   └── ....
  10. ├── controllers
  11. │   └── lifecyclehook
  12. │       └── lifecyclehook_controller.go
  13. ├── go.mod
  14. ├── go.sum
  15. ├── hack
  16. │   └── ....
  17. └── main.go

Controller Logic

Lifecycle Hook is implemented in Reconcile(req ctrl.Request) (ctrl.Result, error) function in lifecyclehook_controller.go file. In our hypothetical scenarios, we implemented the pod life cycle management logic through the following codes:

  1. const (
  2.     deleteHookLabel  = "example.com/unready-blocker-delete"
  3.     inPlaceHookLabel = "example.com/unready-blocker-inplace"
  4.     newlyCreateLabel = "example.com/newly-create"
  5. )
  7. func (r *SampleReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
  9.     ... ...
  11.     switchLabel := func(pod *v1.Pod, key, value string) error {
  12.         body := fmt.Sprintf(`{"metadata":{"labels":{"%s":"%s"}}}`, key, value)
  13.         if err := r.Patch(context.TODO(), pod, client.RawPatch(types.StrategicMergePatchType, []byte(body))); err != nil {
  14.             return err
  15.         }
  16.         return nil
  17.     }
  20.     /*
  21.         Pod Lifecycle Hook Logic
  22.     */
  23.     switch {
  24.     // handle newly-created pod
  25.     case IsNewlyCreateHooked(pod):
  26.         // register this pod to your service discovery center
  27.         if err := postRegistry(pod); err != nil {
  28.             return reconcile.Result{}, err
  29.         }
  30.         if err := switchLabel(pod, newlyCreateLabel, "false"); err != nil {
  31.             return reconcile.Result{}, err
  32.         }
  34.     // handle the pod which is preparing to inplace update
  35.     case IsPreUpdateHooked(pod):
  36.         // let the service discover center fail over this pod 
  37.         if err := postFailOver(pod); err != nil {
  38.             return reconcile.Result{}, err
  39.         }
  40.         if err := switchLabel(pod, inPlaceHookLabel, "false"); err != nil {
  41.             return reconcile.Result{}, err
  42.         }
  44.     // handle the pod which is updated completely 
  45.     case IsUpdatedHooked(pod):
  46.         // register this pod again to your service discovery center 
  47.         if err := postRegistry(pod); err != nil {
  48.             return reconcile.Result{}, err
  49.         }
  50.         if err := switchLabel(pod, inPlaceHookLabel, "true"); err != nil {
  51.             return reconcile.Result{}, err
  52.         }
  54.     // handle the pod which is preparing to delete  
  55.     case IsPreDeleteHooked(pod):
  56.         // just unregister this pod from your service discovery center
  57.         if err := postUnregister(pod); err != nil {
  58.             return reconcile.Result{}, err
  59.         }
  60.         if err := switchLabel(pod, deleteHookLabel, "false"); err != nil {
  61.             return reconcile.Result{}, err
  62.         }
  63.     }
  65.     return ctrl.Result{}, nil
  66. }
  68. func IsNewlyCreateHooked(pod *v1.Pod) bool {
  69.     return kruiseappspub.LifecycleStateType(pod.Labels[kruiseappspub.LifecycleStateKey]) == kruiseappspub.LifecycleStateNormal && pod.Labels[newlyCreateLabel] == "true" && IsPodReady(pod)
  70. }
  72. func IsPreUpdateHooked(pod *v1.Pod) bool {
  73.     return kruiseappspub.LifecycleStateType(pod.Labels[kruiseappspub.LifecycleStateKey]) == kruiseappspub.LifecycleStatePreparingUpdate && pod.Labels[inPlaceHookLabel] == "true"
  74. }
  76. func IsUpdatedHooked(pod *v1.Pod) bool {
  77.     return kruiseappspub.LifecycleStateType(pod.Labels[kruiseappspub.LifecycleStateKey]) == kruiseappspub.LifecycleStateUpdated && pod.Labels[inPlaceHookLabel] == "false" && IsPodReady(pod)
  78. }
  80. func IsPreDeleteHooked(pod *v1.Pod) bool {
  81.     return kruiseappspub.LifecycleStateType(pod.Labels[kruiseappspub.LifecycleStateKey]) == kruiseappspub.LifecycleStatePreparingDelete && pod.Labels[DeleteHookLabel] == "true"
  82. }

The four switch-cases correspond to the four key time spots: after creation, pre-update, updated and pre-delete. Users can implement the corresponding hook according to their needs. In our hypothetical scenario, the implementation of the above hooks are as follows:

  • postRegistry(pod *v1.Pod) : send a http/https request to the service discovery center to register the Pod;
  • postFailOver(pod *v1.Pod) : send a http/https request to the service discovery center to fail over the Pod;
  • postUnregiste(pod *v1.Pod): send a http/https request to the service discovery center to unregister the Pod;

Operator 部署

When the code of the operator is perfect, the operator needs to be built and deployed to the target cluster. You can refer to the Kubebuilder official document to do that.

After the deployment of the operator is completed, the operator will continuously watch the state of pods in the cluster and automatically execute the above hook at the key time spot of pod life cycle, to bridge the gap between Kubernetes and the service discovery center.