WorkloadSpread

FEATURE STATE: Kruise v0.10.0

WorkloadSpread can distribute Pods of workload to different types of Node according to some polices, which empowers single workload the abilities for multi-domain deployment and elastic deployment.

Some common policies include:

  • fault toleration spread (for example, spread evenly among hosts, az, etc)
  • spread according to the specified ratio (for example, deploy Pod to several specified az according to the proportion)
  • subset management with priority, such as
    • deploy Pods to ecs first, and then deploy to eci when its resources are insufficient.
    • deploy a fixed number of Pods to ecs first, and the rest Pods are deployed to eci.
  • subset management with customization, such as
    • control how many pods in a workload are deployed in different cpu arch
    • enable pods in different cpu arch to have different resource requirements

The feature of WorkloadSpread is similar with UnitedDeployment in OpenKruise community. Each WorkloadSpread defines multi-domain called subset. Each domain may provide the limit to run the replicas number of pods called maxReplicas. WorkloadSpread injects the domain configuration into the Pod by Webhook, and it also controls the order of scale in and scale out.

Kruise with version lower than 1.3.0 supports CloneSet, Deployment, ReplicaSet.

Since Kruise 1.3.0, WorkloadSpread supports StatefulSet.

In particular, for StatefulSet, WorkloadSpread supports manage its subsets only when scale up. The order of scale down is still controlled by StatefulSet controller. The subset management of StatefulSet is based on ordinals of Pods, and more details can be found here.

Since Kruise 1.5.0, WorkloadSpread supports customized workloads that have scale sub-resource.

Demo

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: WorkloadSpread
  3. metadata:
  4. name: workloadspread-demo
  5. spec:
  6. targetRef:
  7. apiVersion: apps/v1 | apps.kruise.io/v1alpha1
  8. kind: Deployment | CloneSet
  9. name: workload-xxx
  10. subsets:
  11. - name: subset-a
  12. requiredNodeSelectorTerm:
  13. matchExpressions:
  14. - key: topology.kubernetes.io/zone
  15. operator: In
  16. values:
  17. - zone-a
  18. preferredNodeSelectorTerms:
  19. - weight: 1
  20. preference:
  21. matchExpressions:
  22. - key: another-node-label-key
  23. operator: In
  24. values:
  25. - another-node-label-value
  26. maxReplicas: 3
  27. tolerations: [ ]
  28. patch:
  29. metadata:
  30. labels:
  31. xxx-specific-label: xxx
  32. - name: subset-b
  33. requiredNodeSelectorTerm:
  34. matchExpressions:
  35. - key: topology.kubernetes.io/zone
  36. operator: In
  37. values:
  38. - zone-b
  39. scheduleStrategy:
  40. type: Adaptive | Fixed
  41. adaptive:
  42. rescheduleCriticalSeconds: 30

targetRef: specify the target workload. Can not be mutated,and one workload can only correspond to one WorkloadSpread.

subsets

subsets consists of multiple domain called subset, and each topology has different configuration.

sub-fields

  • name: the name of subset, it is distinct in a WorkloadSpread, which represents a topology.

  • maxReplicas:the replicas limit of subset, and must be Integer and >= 0. There is no replicas limit while the maxReplicas is nil.

    Don’t support percentage type in current version.

  • requiredNodeSelectorTerm: match zone hardly。

  • preferredNodeSelectorTerms: match zone softly。

CautionrequiredNodeSelectorTerm corresponds the requiredDuringSchedulingIgnoredDuringExecution of nodeAffinity. preferredNodeSelectorTerms corresponds the preferredDuringSchedulingIgnoredDuringExecution of nodeAffinity.

  • tolerations: the tolerations of Pod in subset.
  1. tolerations:
  2. - key: "key1"
  3. operator: "Equal"
  4. value: "value1"
  5. effect: "NoSchedule"
  • patch: customize the Pod configuration of subset, such as Annotations, Labels, Env.

Example:

  1. # patch pod with a topology label:
  2. patch:
  3. metadata:
  4. labels:
  5. topology.application.deploy/zone: "zone-a"
  1. # patch pod container resources:
  2. patch:
  3. spec:
  4. containers:
  5. - name: main
  6. resources:
  7. limit:
  8. cpu: "2"
  9. memory: 800Mi
  1. # patch pod container env with a zone name:
  2. patch:
  3. spec:
  4. containers:
  5. - name: main
  6. env:
  7. - name: K8S_AZ_NAME
  8. value: zone-a

Schedule strategy

WorkloadSpread provides two kind strategies, the default strategy is Fixed.

  1. scheduleStrategy:
  2. type: Adaptive | Fixed
  3. adaptive:
  4. rescheduleCriticalSeconds: 30
  • Fixed:

    Workload is strictly spread according to the definition of the subset.

  • Adaptive:

    Reschedule: Kruise will check the unschedulable Pods of subset. If it exceeds the defined duration, the failed Pods will be rescheduled to the other subset.

Requirements

WorkloadSpread defaults to be disabled. You have to configure the feature-gate WorkloadSpread when install or upgrade Kruise:

  1. $ helm install kruise https://... --set featureGates="WorkloadSpread=true"

Pod Webhook

WorkloadSpread uses webhook to inject fault domain rules.

If the PodWebhook feature-gate is set to false, WorkloadSpread will also be disabled.

deletion-cost feature

CloneSet has supported deletion-cost feature in the latest versions.

The other native workload need kubernetes version >= 1.21. (In 1.21, users need to enable PodDeletionCost feature-gate, and since 1.22 it will be enabled by default)

Scale order:

The workload managed by WorkloadSpread will scale according to the defined order in spec.subsets.

The order of subset in spec.subsets can be changed, which can adjust the scale order of workload.

Scale out

  • The Pods are scheduled in the subset order defined in the spec.subsets. It will be scheduled in the next subset while the replica number reaches the maxReplicas of subset

Scale in

  • When the replica number of the subset is greater than the maxReplicas, the extra Pods will be removed in a high priority.
  • According to the subset order in the spec.subsets, the Pods of the subset at the back are deleted before the Pods at the front.
  1. # subset-a subset-b subset-c
  2. # maxReplicas 10 10 nil
  3. # pods number 10 10 10
  4. # deletion order: c -> b -> a
  5. # subset-a subset-b subset-c
  6. # maxReplicas 10 10 nil
  7. # pods number 20 20 20
  8. # deletion order: b -> a -> c

Use WorkloadSpread with customized workload

If you want to use WorkloadSpread with custom workloads, which is disabled by default, some additional configuration is required. This section uses the Rollout Workload from the Argo community as an example to demonstrate how to integrate it with WorkloadSpread.

Configure the custom workload watch whitelist

First, you need to add the custom workload to the WorkloadSpread_Watch_Custom_Workload_WhiteList to ensure it can be read and understood by WorkloadSpread.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4. name: kruise-configuration
  5. namespace: kruise-system
  6. data:
  7. "WorkloadSpread_Watch_Custom_Workload_WhiteList": |
  8. {
  9. "workloads": [
  10. {
  11. "Group": "argoproj.io",
  12. "Kind": "Rollout",
  13. "replicasPath": "spec.replicas",
  14. }
  15. ]
  16. }

The specific configuration items are explained as follows:

  • Group: ApiGroup of the customized workload.
  • Kind: Kind of the customized workload.
  • subResources: SubResources of the customized workload, including Group and Kind. For example: Deployment’s ReplicaSet. This field is optional, and can be left as empty slice if no sub-workload is used for the customized workload.
  • replicasPath: Resource path to the replicas in the resource. For example: spec.replicas.

Authorize kruise-manager

To use WorkloadSpread with custom workloads, you need to grant the kruise-manager service account read permissions for the respective resources.

Caution: The WorkloadSpread Webhook does not set a deletion cost for Pods created by custom workloads, so it cannot ensure the scaling-down order of custom workloads.

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRole
  3. metadata:
  4. name: kruise-rollouts-access
  5. rules:
  6. - apiGroups: [ "argoproj.io" ]
  7. resources: [ "rollouts" ]
  8. verbs: [ "get" ]
  9. ---
  10. apiVersion: rbac.authorization.k8s.io/v1
  11. kind: ClusterRoleBinding
  12. metadata:
  13. name: kruise-rollouts-access-binding
  14. subjects:
  15. - kind: ServiceAccount
  16. name: kruise-manager
  17. namespace: kruise-system
  18. roleRef:
  19. kind: ClusterRole
  20. name: kruise-rollouts-access
  21. apiGroup: rbac.authorization.k8s.io

Reference the custom workload in WorkloadSpread

Once the configuration is complete, the custom workload can be referenced in the targetRef field of WorkloadSpread.

  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: WorkloadSpread
  3. metadata:
  4. name: workloadspread-demo
  5. spec:
  6. targetRef:
  7. apiVersion: argoproj.io/v1alpha1
  8. kind: Rollout
  9. name: rollouts-demo
  10. subsets:
  11. ...

feature-gates

WorkloadSpread feature is turned off by default, if you want to turn it on set feature-gates WorkloadSpread.

  1. $ helm install kruise https://... --set featureGates="WorkloadSpread=true"

Example

Elastic deployment

zone-a(ACK) holds 100 Pods, zone-b(ECI) as an elastic zone holds additional Pods.

  1. Create a WorkloadSpread instance.
  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: WorkloadSpread
  3. metadata:
  4. name: ws-demo
  5. namespace: deploy
  6. spec:
  7. targetRef: # workload in the same namespace
  8. apiVersion: apps.kruise.io/v1alpha1
  9. kind: CloneSet
  10. name: workload-xxx
  11. subsets:
  12. - name: ACK # zone ACK
  13. requiredNodeSelectorTerm:
  14. matchExpressions:
  15. - key: topology.kubernetes.io/zone
  16. operator: In
  17. values:
  18. - ack
  19. maxReplicas: 100
  20. patch: # inject label.
  21. metadata:
  22. labels:
  23. deploy/zone: ack
  24. - name: ECI # zone ECI
  25. requiredNodeSelectorTerm:
  26. matchExpressions:
  27. - key: topology.kubernetes.io/zone
  28. operator: In
  29. values:
  30. - eci
  31. patch:
  32. metadata:
  33. labels:
  34. deploy/zone: eci
  1. Creat a corresponding workload, the number of replicas ca be adjusted freely.

Effect

  • When the number of replicas <= 100, the Pods are scheduled in ACK zone.
  • When the number of replicas > 100, the 100 Pods are in ACK zone, the extra Pods are scheduled in ECI zone.
  • The Pods in ECI elastic zone are removed first when scaling in.

Multi-domain deployment

Deploy 100 Pods to two zone(zone-a, zone-b) separately.

  1. Create a WorkloadSpread instance.
  1. apiVersion: apps.kruise.io/v1alpha1
  2. kind: WorkloadSpread
  3. metadata:
  4. name: ws-demo
  5. namespace: deploy
  6. spec:
  7. targetRef:
  8. apiVersion: apps.kruise.io/v1alpha1
  9. kind: CloneSet
  10. name: workload-xxx
  11. subsets:
  12. - name: subset-a
  13. requiredNodeSelectorTerm:
  14. matchExpressions:
  15. - key: topology.kubernetes.io/zone
  16. operator: In
  17. values:
  18. - zone-a
  19. maxReplicas: 100
  20. patch:
  21. metadata:
  22. labels:
  23. deploy/zone: zone-a
  24. - name: subset-b
  25. requiredNodeSelectorTerm:
  26. matchExpressions:
  27. - key: topology.kubernetes.io/zone
  28. operator: In
  29. values:
  30. - zone-b
  31. maxReplicas: 100
  32. patch:
  33. metadata:
  34. labels:
  35. deploy/zone: zone-b
  1. Creat a corresponding workload with a 200 replicas, or perform a rolling update on an existing workload.

  2. If the spread of zone needs to be changed, first adjust the maxReplicas of subset, and then change the replicas of workload.