Resource Propagating

The PropagationPolicy and ClusterPropagationPolicy APIs are provided to propagate resources. For the differences between the two APIs, please see here.

Here, we use PropagationPolicy as an example to describe how to propagate resources.

Before you start

Install Karmada and prepare the karmadactl command-line tool.

Note: We need to point kubectl to <karmada-apiserver.config> instead of the member cluster in advance.

Deploy a simple multi-cluster Deployment

Create a PropagationPolicy object

You can propagate a Deployment by creating a PropagationPolicy object defined in a YAML file. For example, this YAML file describes a Deployment object named nginx under default namespace need to be propagated to member1 cluster:

  1. # propagationpolicy.yaml
  2. apiVersion: policy.karmada.io/v1alpha1
  3. kind: PropagationPolicy
  4. metadata:
  5.   name: example-policy # The default namespace is `default`.
  6. spec:
  7.   resourceSelectors:
  8.     - apiVersion: apps/v1
  9.       kind: Deployment
  10.       name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
  11.   placement:
  12.     clusterAffinity:
  13.       clusterNames:
  14.         - member1
  1. Create a propagationPolicy based on the YAML file:
  1. kubectl apply -f propagationpolicy.yaml
  1. Create a Deployment nginx resource:
  1. kubectl create deployment nginx --image nginx

Note: The resource exists only as a template in karmada. After being propagated to a member cluster, the behavior of the resource is the same as that of a single kubernetes cluster.

Note: Resources and PropagationPolicy are created in no sequence.

  1. Display information of the deployment:
  1. karmadactl get deployment

The output is similar to this:

  1. NAME    CLUSTER   READY   UP-TO-DATE   AVAILABLE   AGE   ADOPTION
  2. nginx   member1   1/1     1            1           52s   Y
  1. List the pods created by the deployment:
  1. karmadactl get pod -l app=nginx

The output is similar to this:

  1. NAME                     CLUSTER   READY   STATUS    RESTARTS   AGE
  2. nginx-6799fc88d8-s7vv9   member1   1/1     Running   0          52s

Update PropagationPolicy

You can update the propagationPolicy by applying a new YAML file. This YAML file propagates the Deployment to the member2 cluster.

  1. # propagationpolicy-update.yaml
  2. apiVersion: policy.karmada.io/v1alpha1
  3. kind: PropagationPolicy
  4. metadata:
  5.   name: example-policy
  6. spec:
  7.   resourceSelectors:
  8.     - apiVersion: apps/v1
  9.       kind: Deployment
  10.       name: nginx
  11.   placement:
  12.     clusterAffinity:
  13.       clusterNames: # Modify the selected cluster to propagate the Deployment.
  14.         - member2
  1. Apply the new YAML file:
  1. kubectl apply -f propagationpolicy-update.yaml
  1. Display information of the deployment (the output is similar to this):
  1. NAME    CLUSTER   READY   UP-TO-DATE   AVAILABLE   AGE   ADOPTION
  2. nginx   member2   1/1     1            1           5s    Y
  1. List the pods of the deployment (the output is similar to this):
  1. NAME                     CLUSTER   READY   STATUS    RESTARTS   AGE
  2. nginx-6799fc88d8-8t8cc   member2   1/1     Running   0          17s

Update Deployment

You can update the deployment template. The changes will be automatically synchronized to the member clusters.

  1. Update deployment replicas to 2.
  2. Display information of the deployment (the output is similar to this):
  1. NAME    CLUSTER   READY   UP-TO-DATE   AVAILABLE   AGE     ADOPTION
  2. nginx   member2   2/2     2            2           7m59s   Y
  1. List the pods of the deployment (the output is similar to this):
  1. NAME                     CLUSTER   READY   STATUS    RESTARTS   AGE
  2. nginx-6799fc88d8-8t8cc   member2   1/1     Running   0          8m12s
  3. nginx-6799fc88d8-zpl4j   member2   1/1     Running   0          17s

Delete a propagationPolicy

Delete the propagationPolicy by name:

  1. kubectl delete propagationpolicy example-policy

Deleting a propagationPolicy does not delete deployments propagated to member clusters. You need to delete deployments in the karmada control-plane:

  1. kubectl delete deployment nginx

Deploy deployment into a specified set of target clusters

.spec.placement.clusterAffinity field of PropagationPolicy represents scheduling restrictions on a certain set of clusters, without which any cluster can be scheduling candidate.

It has four fields to set:

  • LabelSelector
  • FieldSelector
  • ClusterNames
  • ExcludeClusters

LabelSelector

LabelSelector is a filter to select member clusters by labels. It uses *metav1.LabelSelector type. If it is non-nil and non-empty, only the clusters matching this filter will be selected.

PropagationPolicy can be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: test-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     clusterAffinity:
  9.       labelSelector:
  10.         matchLabels:
  11.           location: us
  12.     #...

PropagationPolicy can also be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: test-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     clusterAffinity:
  9.       labelSelector:
  10.         matchExpressions:
  11.         - key: location
  12.           operator: In
  13.           values:
  14.           - us
  15.     #...

For a description of matchLabels and matchExpressions, you can refer to Resources that support set-based requirements.

FieldSelector

FieldSelector is a filter to select member clusters by fields. If it is non-nil and non-empty, only the clusters matching this filter will be selected.

PropagationPolicy can be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     clusterAffinity:
  9.       fieldSelector:
  10.         matchExpressions:
  11.         - key: provider
  12.           operator: In
  13.           values:
  14.           - huaweicloud
  15.         - key: region
  16.           operator: NotIn
  17.           values:
  18.           - cn-south-1
  19.     #...

If multiple matchExpressions are specified in the fieldSelector, the cluster must match all matchExpressions.

The key in matchExpressions now supports three values: provider, region, and zone, which correspond to the .spec.provider, .spec.region, and .spec.zone fields of the Cluster object, respectively.

The operator in matchExpressions now supports In and NotIn.

ClusterNames

Users can set the ClusterNames field to specify the selected clusters.

PropagationPolicy can be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     clusterAffinity:
  9.       clusterNames:
  10.         - member1
  11.         - member2
  12.     #...

ExcludeClusters

Users can set the ExcludeClusters fields to specify the clusters to be ignored.

PropagationPolicy can be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.  name: nginx-propagation
  5. spec:
  6.  #...
  7.  placement:
  8.   clusterAffinity:
  9.    exclude:
  10.     - member1
  11.     - member3
  12.   #...

Multiple cluster affinity groups

Users can set the ClusterAffinities field and declare multiple cluster groups in PropagationPolicy. The scheduler will evaluate these groups one by one in the order they appear in the spec and the group that does not satisfy scheduling restrictions will be ignored, which means that each cluster in this group will not be selected unless it also belongs to the next group (a cluster could belong to multiple groups).

If none of the groups satisfy the scheduling restrictions, the scheduling fails, which means no cluster will be selected.

Note:

  1. ClusterAffinities can not co-exist with ClusterAffinity.
  2. If both ClusterAffinity and ClusterAffinities are not set, any cluster can be scheduling candidate.

Potential use case 1: The private clusters in the local data center could be the main group, and the managed clusters provided by cluster providers could be the secondary group. So that the Karmada scheduler would prefer to schedule workloads to the main group and the second group will only be considered in case of the main group not satisfying restrictions (like, lack of resources).

PropagationPolicy can be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.  name: test-propagation
  5. spec:
  6.  #...
  7.  placement:
  8.   clusterAffinities:
  9.    - affinityName: local-clusters
  10.      clusterNames:
  11.       - local-member1
  12.       - local-member2
  13.    - affinityName: cloud-clusters
  14.      clusterNames:
  15.       - public-cloud-member1
  16.       - public-cloud-member2
  17.   #...

Potential use case 2: For the disaster recovery scenario, the clusters could be organized to primary and backup groups, the workloads would be scheduled to primary clusters firstly, and when primary clusters fail (like data center power off), Karmada scheduler could migrate workloads to the backup clusters.

PropagationPolicy can be configured as follows:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.  name: test-propagation
  5. spec:
  6.  #...
  7.  placement:
  8.   clusterAffinities:
  9.    - affinityName: primary-clusters
  10.      clusterNames:
  11.       - member1
  12.    - affinityName: backup-clusters
  13.      clusterNames:
  14.       - member1
  15.       - member2
  16.   #...

For more detailed design information, please refer to Multiple scheduling group.

Schedule based on Taints and Tolerations

.spec.placement.clusterTolerations field of PropagationPolicy represents the tolerations. Like Kubernetes, tolerations need to be used in conjunction with taints on the clusters. After setting one or more taints on the cluster, workloads cannot be scheduled or run on these clusters unless the policy explicitly states that these taints are tolerated. Karmada currently supports taints whose effects are NoSchedule and NoExecute.

You can karmadactl taint to taint a cluster:

  1. # Update cluster 'foo' with a taint with key 'dedicated' and value 'special-user' and effect 'NoSchedule'
  2. # If a taint with that key and effect already exists, its value is replaced as specified
  3. karmadactl taint clusters foo dedicated=special-user:NoSchedule

In order to schedule to the above cluster, you need to declare the following in the Policy:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     clusterTolerations:
  9.     - key: dedicated
  10.       value: special-user
  11.       Effect: NoSchedule

NoExecute taints are also used in Multi-cluster Failover. See details here.

Multi region HA support

By leveraging the spread-by-region constraint, users are able to deploy workloads across regions, e.g. people may want their workloads always running on different regions for HA purposes.

To enable multi region deployment, you should use the command below to customize the region of clusters.

  1. $ kubectl --kubeconfig ~/.kube/karmada.config --context karmada-apiserver edit cluster/member1
  3. ...
  4. spec:
  5.   apiEndpoint: https://172.18.0.4:6443
  6.   id: 257b5c81-dfae-4ae5-bc7c-6eaed9ed6a39
  7.   impersonatorSecretRef:
  8.     name: member1-impersonator
  9.     namespace: karmada-cluster
  10.   region: test
  11. ...

Then you need to restrict the maximum and minimum number of cluster groups to be selected. Assume there are two regions, you may want to deploy the workload in one cluster per region. You can refer to:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     replicaScheduling:
  9.      replicaSchedulingType: Duplicated  
  10.     spreadConstraints:
  11.       - spreadByField: region
  12.         maxGroups: 2
  13.         minGroups: 2
  14.       - spreadByField: cluster
  15.         maxGroups: 1
  16.         minGroups: 1

Resource Propagating - 图1note

If the replica division preference is StaticWeightList, the declaration specified by spread constraints will be ignored. If one of spread constraints are using SpreadByField, the SpreadByFieldCluster must be included. For example, when using SpreadByFieldRegion to specify region groups, at the meantime, you must use SpreadByFieldCluster to specify how many clusters should be selected.

Multiple strategies of replica Scheduling

.spec.placement.replicaScheduling represents the scheduling policy on dealing with the number of replicas when propagating resources that have replicas in spec (e.g. deployments, statefulsets and CRDs which can be interpreted by Customizing Resource Interpreter) to member clusters.

It has two replicaSchedulingTypes which determine how the replicas are scheduled when Karmada propagates a resource:

  • Duplicated: duplicate the same replicas to each candidate member cluster from resources.
  • Divided: divide replicas into parts according to numbers of valid candidate member clusters, and exact replicas for each cluster are determined by ReplicaDivisionPreference.

ReplicaDivisionPreference determines how replicas are divided when ReplicaSchedulingType is Divided.

  • Aggregated: divide replicas into clusters as few as possible, while respecting clusters’ resource availabilities during the division. See details in Schedule based on Cluster Resource Modeling.
  • Weighted: divide replicas by weight according to WeightPreference. There are two kinds of WeightPreference to set. StaticWeightList statically allocates replicas to target clusters based on weight. Target clusters can be selected by ClusterAffinity. DynamicWeight specifies the factor to generate the dynamic weight list. If specified, StaticWeightList will be ignored. Karmada currently supports the factor AvailableReplicas.

Following, we have two simple examples:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     replicaScheduling:
  9.       replicaDivisionPreference: Weighted
  10.       replicaSchedulingType: Divided
  11.       weightPreference:
  12.         staticWeightList:
  13.           - targetCluster:
  14.               clusterNames:
  15.                 - member1
  16.             weight: 1
  17.           - targetCluster:
  18.               clusterNames:
  19.                 - member2
  20.             weight: 1

Replicas will be evenly propagated to member1 and member2.

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-propagation
  5. spec:
  6.   #...
  7.   placement:
  8.     replicaScheduling:
  9.       replicaDivisionPreference: Weighted
  10.       replicaSchedulingType: Divided
  11.       weightPreference:
  12.         dynamicWeight: AvailableReplicas

Replicas will be propagated based on how many replicas can fit in the member clusters. Let’s say for example that the scheduler has selected 3 clusters (A/B/C) and must divide 12 replicas to them. Based on cluster resource modeling, we get that the max number of replicas on A, B, C is 6, 12, 18 respectively. Therefore, the clusters’ weights will be A:B:C -> 6:12:18 (equal to 1:2:3). At last, the assignment would be “A: 2, B: 4, C: 6”.

Resource Propagating - 图2note

If ReplicaDivisionPreference is set to Weighted and WeightPreference is not set, the default strategy is to weight all clusters averagely.

Configure PropagationPolicy/ClusterPropagationPolicy priority

If a PropagationPolicy and a ClusterPropagationPolicy match the workload, Karmada will select the PropagationPolicy. If multiple PropagationPolicies match the workload, Karmada will select the one with the highest priority. A PropagationPolicy supports implicit and explicit priorities. The same goes for ClusterPropagationPolicy.

The following takes PropagationPolicy as an example.

Configure explicit priority

The spec.priority in a PropagationPolicy represents the explicit priority. A greater value means a higher priority.

Note: If not specified, defaults to 0.

Assume there are multiple policies:

  1. # highexplicitpriority.yaml
  2. apiVersion: policy.karmada.io/v1alpha1
  3. kind: PropagationPolicy
  4. metadata:
  5.   name: propagation-high-explicit-priority
  6. spec:
  7.   resourceSelectors:
  8.     - apiVersion: apps/v1
  9.       kind: Deployment
  10.       labelSelector:
  11.         matchLabels:
  12.           app: nginx
  13.   priority: 2
  14.   placement:
  15.     clusterAffinity:
  16.       clusterNames:
  17.         - member1
  18. ---
  19. # lowexplicitpriority.yaml
  20. apiVersion: policy.karmada.io/v1alpha1
  21. kind: PropagationPolicy
  22. metadata:
  23.   name: propagation-low-explicit-priority
  24. spec:
  25.   resourceSelectors:
  26.     - apiVersion: apps/v1
  27.       kind: Deployment
  28.       labelSelector:
  29.         matchLabels:
  30.           app: nginx
  31.   priority: 1
  32.   placement:
  33.     clusterAffinity:
  34.       clusterNames:
  35.         - member2
  36. ---
  37. # defaultexplicitpriority.yaml
  38. apiVersion: policy.karmada.io/v1alpha1
  39. kind: PropagationPolicy
  40. metadata:
  41.   name: propagation-low-explicit-priority
  42. spec:
  43.   resourceSelectors:
  44.     - apiVersion: apps/v1
  45.       kind: Deployment
  46.       labelSelector:
  47.         matchLabels:
  48.           app: nginx
  49.   placement:
  50.     clusterAffinity:
  51.       clusterNames:
  52.         - member3

The nginx deployment in default namespace will be propagated to cluster member1.

Configure implicit priority

The spec.resourceSelectors in a PropagationPolicy represents the implicit priority. The priority order (from low to high) is as follows:

  • The PropagationPolicy resourceSelector whose name and labelSelector are empty matches the workload.
  • The labelSelector of PropagationPolicy resourceSelector matches the workload.
  • The name of PropagationPolicy resourceSelector matches the workload.

Assume there are multiple policies:

  1. # emptymatch.yaml
  2. apiVersion: policy.karmada.io/v1alpha1
  3. kind: PropagationPolicy
  4. metadata:
  5.   name: propagation-emptymatch
  6. spec:
  7.   resourceSelectors:
  8.     - apiVersion: apps/v1
  9.       kind: Deployment
  10.   placement:
  11.     clusterAffinity:
  12.       clusterNames:
  13.         - member1
  14. ---
  15. # labelselectormatch.yaml
  16. apiVersion: policy.karmada.io/v1alpha1
  17. kind: PropagationPolicy
  18. metadata:
  19.   name: propagation-labelselectormatch
  20. spec:
  21.   resourceSelectors:
  22.     - apiVersion: apps/v1
  23.       kind: Deployment
  24.       labelSelector:
  25.         matchLabels:
  26.           app: nginx
  27.   placement:
  28.     clusterAffinity:
  29.       clusterNames:
  30.         - member2
  31. ---
  32. # namematch.yaml
  33. apiVersion: policy.karmada.io/v1alpha1
  34. kind: PropagationPolicy
  35. metadata:
  36.   name: propagation-namematch
  37. spec:
  38.   resourceSelectors:
  39.     - apiVersion: apps/v1
  40.       kind: Deployment
  41.       name: nginx
  42.   placement:
  43.     clusterAffinity:
  44.       clusterNames:
  45.         - member3

The nginx deployment in default namespace will be propagated to cluster member3.

Choose from same-priority PropagationPolicies

If multiple PropagationPolicies with the same explicit priority match the workload, the one with the highest implicit priority will be selected.

Assume there are multiple policies:

  1. # explicit-labelselectormatch.yaml
  2. apiVersion: policy.karmada.io/v1alpha1
  3. kind: PropagationPolicy
  4. metadata:
  5.   name: propagation-explicit-labelselectormatch
  6. spec:
  7.   resourceSelectors:
  8.     - apiVersion: apps/v1
  9.       kind: Deployment
  10.       labelSelector:
  11.         matchLabels:
  12.           app: nginx
  13.   priority: 3
  14.   placement:
  15.     clusterAffinity:
  16.       clusterNames:
  17.         - member1
  18. ---
  19. # explicit-namematch.yaml
  20. apiVersion: policy.karmada.io/v1alpha1
  21. kind: PropagationPolicy
  22. metadata:
  23.   name: propagation-explicit-namematch
  24. spec:
  25.   resourceSelectors:
  26.     - apiVersion: apps/v1
  27.       kind: Deployment
  28.       name: nginx
  29.   priority: 3
  30.   placement:
  31.     clusterAffinity:
  32.       clusterNames:
  33.         - member3

The nginx deployment in default namespace will be propagated to cluster member3.

If both explicit and implicit priorities are the same, Karmada applies the PropagationPolicy in an ascending alphabetical order, for example, choosing xxx-a-xxx instead of xxx-b-xxx.

Assume there are multiple policies:

  1. # higher-alphabetical.yaml
  2. apiVersion: policy.karmada.io/v1alpha1
  3. kind: PropagationPolicy
  4. metadata:
  5.   name: propagation-b-higher-alphabetical
  6. spec:
  7.   resourceSelectors:
  8.     - apiVersion: apps/v1
  9.       kind: Deployment
  10.       name: nginx
  11.   priority: 3
  12.   placement:
  13.     clusterAffinity:
  14.       clusterNames:
  15.         - member1
  16. ---
  17. # lower-alphabetical.yaml
  18. apiVersion: policy.karmada.io/v1alpha1
  19. kind: PropagationPolicy
  20. metadata:
  21.   name: propagation-a-lower-alphabetical
  22. spec:
  23.   resourceSelectors:
  24.     - apiVersion: apps/v1
  25.       kind: Deployment
  26.       name: nginx
  27.   priority: 3
  28.   placement:
  29.     clusterAffinity:
  30.       clusterNames:
  31.         - member2

The nginx deployment in default namespace will be propagated to cluster member2.

Suspend and resume of resource propagation

The .spec.suspension field in the PropagationPolicy and ClusterPropagationPolicy allows suspending and resuming resource propagation to one or more clusters.

Suspend resource propagation to all clusters

To suspend resource propagation work to all member clusters, use the following configuration:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.  name: nginx-propagation
  5. spec:
  6.  #...
  7.  placement:
  8.   clusterAffinity:
  9.    exclude:
  10.     - member1
  11.     - member3
  12.   #...
  13.   suspension:
  14.     dispatching: true

Updates to the nginx deployment in the Karmada control plane will not be synchronized to any member clusters.

Suspend resource propagation to specific clusters

To suspend resource propagation to individual member clusters, specify the cluster names in the .spec.suspension.clusterNames field:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.  name: nginx-propagation
  5. spec:
  6.  #...
  7.  placement:
  8.   clusterAffinity:
  9.    exclude:
  10.     - member1
  11.     - member3
  12.   #...
  13.   suspension:
  14.    dispatchingOnClusters:
  15.     clusterNames:
  16.      - member3 # Modify the cluster name here

Updates to the nginx deployment in the Karmada control plane will not be synchronized to the member3 cluster but will be synchronized to all other clusters.

Resume resource propagation

To resume dispatching work, simply remove the .spec.suspension configuration:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.  name: nginx-propagation
  5. spec:
  6.  #...
  7.  placement:
  8.   clusterAffinity:
  9.    exclude:
  10.     - member1
  11.     - member3
  12.   #...

The state of the nginx deployment in the Karmada control plane will be synchronized to all member clusters, and any subsequent updates will be synchronized as well.