Advanced DaemonSet

This controller enhances the rolling update workflow of Kubernetes DaemonSet controller in large-scale scenarios, such as support for image pre-download, in-place upgrade, etc.

If you don’t know much about the Kubernetes DaemonSet, we strongly recommend you read its documents before learning Advanced DaemonSet.

Note that Advanced DaemonSet extends the same CRD schema of default DaemonSet with newly added fields. The CRD kind name is still DaemonSet. This is done on purpose so that user can easily migrate workload to the Advanced DaemonSet from the default DaemonSet. For example, one may simply replace the value of apiVersion in the DaemonSet yaml file from apps/v1 to apps.kruise.io/v1alpha1 after installing Kruise manager.

  1. -  apiVersion: apps/v1
  2. +  apiVersion: apps.kruise.io/v1alpha1
  3.    kind: DaemonSet
  4.    metadata:
  5.      name: sample-ds
  6.    spec:
  7.      #...

Enhanced strategies

These new fields have been added into RollingUpdateDaemonSet:

  1. const (
  2. +    // StandardRollingUpdateType replace the old daemons by new ones using rolling update i.e replace them on each node one after the other.
  3. +    // this is the default type for RollingUpdate.
  4. +    StandardRollingUpdateType RollingUpdateType = "Standard"
  6. +    // InplaceRollingUpdateType update container image without killing the pod if possible.
  7. +    InplaceRollingUpdateType RollingUpdateType = "InPlaceIfPossible"
  8. )
  10. // Spec to control the desired behavior of daemon set rolling update.
  11. type RollingUpdateDaemonSet struct {
  12. +    // Type is to specify which kind of rollingUpdate.
  13. +    Type RollingUpdateType `json:"rollingUpdateType,omitempty" protobuf:"bytes,1,opt,name=rollingUpdateType"`
  15.     // ...
  16.     MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty" protobuf:"bytes,2,opt,name=maxUnavailable"`
  18.     // ...
  19.     MaxSurge *intstr.IntOrString `json:"maxSurge,omitempty" protobuf:"bytes,7,opt,name=maxSurge"`
  21. +    // A label query over nodes that are managed by the daemon set RollingUpdate.
  22. +    // Must match in order to be controlled.
  23. +    // It must match the node's labels.
  24. +    Selector *metav1.LabelSelector `json:"selector,omitempty" protobuf:"bytes,3,opt,name=selector"`
  26. +    // The number of DaemonSet pods remained to be old version.
  27. +    // Default value is 0.
  28. +    // Maximum value is status.DesiredNumberScheduled, which means no pod will be updated.
  29. +    // +optional
  30. +    Partition *int32 `json:"partition,omitempty" protobuf:"varint,4,opt,name=partition"`
  32. +    // Indicates that the daemon set is paused and will not be processed by the
  33. +    // daemon set controller.
  34. +    // +optional
  35. +    Paused *bool `json:"paused,omitempty" protobuf:"varint,5,opt,name=paused"`
  36. }

Type for rolling update

Advanced DaemonSet has a rollingUpdateType field in spec.updateStrategy.rollingUpdate which controls the way to rolling update.

  • Standard (default): controller will update daemon Pods by recreating them. It is the same behavior as upstream DaemonSet. You can use maxUnavailable or maxSurge to control order of recreating old and new pods.
  • InPlaceIfPossible: controller will try to in-place update Pod instead of recreating them if possible. You may need to read the concept doc for more details of in-place update. Note that in this type, you can only use maxUnavailable without maxSurge.
  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. spec:
  4.   # ...
  5.   updateStrategy:
  6.     type: RollingUpdate
  7.     rollingUpdate:
  8.       rollingUpdateType: Standard

Selector for rolling update

It helps users to update Pods on specific nodes whose labels could be matched with the selector.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. spec:
  4.   # ...
  5.   updateStrategy:
  6.     type: RollingUpdate
  7.     rollingUpdate:
  8.       selector:
  9.         matchLabels:
  10.           nodeType: canary

Partition for rolling update and scaling up

This strategy defines rules for calculating the priority of updating pods. Partition is the number of DaemonSet pods that should be remained to be old version.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. spec:
  4.   # ...
  5.   updateStrategy:
  6.     type: RollingUpdate
  7.     rollingUpdate:
  8.       partition: 10

And if you put daemonset.kruise.io/progressive-create-pod: "true" annotation into Advanced DaemonSet, the partition will also control the number of pods to be created when scaling up.

Paused for rolling update

paused indicates that Pods updating is paused, controller will not update Pods but just maintain the number of replicas.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. spec:
  4.   # ...
  5.   updateStrategy:
  6.     rollingUpdate:
  7.       paused: true

Pre-download image for update

FEATURE STATE: Kruise v1.3.0

If you have enabled the PreDownloadImageForDaemonSetUpdate feature-gate during Kruise installation or upgrade, DaemonSet controller will automatically pre-download the image you want to update to the nodes of all old Pods. It is quite useful to accelerate the progress of applications upgrade.

The parallelism of each new image pre-downloading by DaemonSet is 1, which means the image is downloaded on nodes one by one. You can change the parallelism using apps.kruise.io/image-predownload-parallelism annotation on DaemonSet according to the capability of image registry, for registries with more bandwidth and P2P image downloading ability, a larger parallelism can speed up the pre-download process.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. metadata:
  4.   annotations:
  5.     apps.kruise.io/image-predownload-parallelism: "10"

Lifecycle hook

FEATURE STATE: Kruise v1.1.0

This is similar to Lifecycle hook of CloneSet.

Now Advanced DaemonSet only supports PreDelete hook, which means it allows users to do something (for example check node resources) before Pod deleting.

  1. type LifecycleStateType string
  3. // Lifecycle contains the hooks for Pod lifecycle.
  4. type Lifecycle struct {
  5.     // PreDelete is the hook before Pod to be deleted.
  6.     PreDelete *LifecycleHook `json:"preDelete,omitempty"`
  7. }
  9. type LifecycleHook struct {
  10.     LabelsHandler     map[string]string `json:"labelsHandler,omitempty"`
  11.     FinalizersHandler []string          `json:"finalizersHandler,omitempty"`
  13.     /**********************  FEATURE STATE: 1.2.0 ************************/
  14.     // MarkPodNotReady = true means:
  15.     // - Pod will be set to 'NotReady' at preparingDelete/preparingUpdate state.
  16.     // - Pod will be restored to 'Ready' at Updated state if it was set to 'NotReady' at preparingUpdate state.
  17.     // Default to false.
  18.     MarkPodNotReady bool `json:"markPodNotReady,omitempty"`
  19.     /*********************************************************************/
  20. }

Examples:

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. spec:
  5.   # define with label
  6.   lifecycle:
  7.     preDelete:
  8.       labelsHandler:
  9.         example.io/block-deleting: "true"
  • When Advanced DaemonSet delete a Pod (including scale in and recreate update):
    • Delete it directly if no lifecycle hook definition or Pod not matched preDelete hook
    • Otherwise, Advanced DaemonSet will firstly update Pod to PreparingDelete state and wait for user controller to remove the label/finalizer and Pod not matched preDelete hook
  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   labels:
  5.     example.io/block-deleting: "true"                   # the pod is hooked by PreDelete hook label
  6.     lifecycle.apps.kruise.io/state: PreparingDelete     # so we update it to `PreparingDelete` state and wait for user controller to do something and remove the label

MarkPodNotReady

FEATURE STATE: Kruise v1.2.0

  1.   lifecycle:
  2.     preDelete:
  3.       markPodNotReady: true
  4.       finalizersHandler:
  5.       - example.io/unready-blocker

If you set markPodNotReady=true for preDelete, Kruise will try to set KruisePodReady condition to False when Pods enter PreparingDelete lifecycle state, and Pods will be NotReady, but containers still Running.

One can use this markPodNotReady feature to drain service traffic before terminating containers.

Note: this feature only works when pod has KruisePodReady ReadinessGate.

Example for user controller logic

Same as yaml example above, we should firstly define example.io/block-deleting label in template and lifecycle of Advanced DaemonSet.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: DaemonSet
  3. spec:
  4.   template:
  5.     metadata:
  6.       labels:
  7.         example.io/block-deleting: "true"
  8.   # ...
  9.   lifecycle:
  10.     preDelete:
  11.       labelsHandler:
  12.         example.io/block-deleting: "true"

User controller logic:

  • For Pod in PreparingDelete, check if its Node existing, do something (for example reserve resources) and then remove the label.