Resource Migration

Objectives

Assuming you have a single kubernetes cluster which already has many native resource installed, furthermore, you want to migrate the existing resource to Karmada and then achieve multi-cluster management.

So, this section will guide you to cover:

  • Migrate all the existing resource from original cluster to Karmada based on resource granularity.
  • Apply higher priority PropagationPolicy to meet more propagate demands based on application granularity.

Prerequisites

Karmada with multi cluster has been installed

Step 1: Run the command

  1. $ git clone https://github.com/karmada-io/karmada
  2. $ cd karmada
  3. $ hack/local-up-karmada.sh
  4. $ export KUBECONFIG=~/.kube/karmada.config:~/.kube/members.config

Note:

Before guide started, we should install at least three kubernetes clusters, one is for Karmada control plane, the other two for member clusters. For convenience, we use hack/local-up-karmada.sh script to quickly prepare the above clusters.

After the above command executed, you will see Karmada control plane installed with multi member clusters.

Enable PropagationPolicyPreemption in karmada-controller-manager

Step 2: Run the command

  1. $ kubectl --context karmada-host get deploy karmada-controller-manager -n karmada-system -o yaml | sed '/- --failover-eviction-timeout=30s/{n;s/- --v=4/- --feature-gates=PropagationPolicyPreemption=true\n        &/g}' | kubectl --context karmada-host replace -f -

Note:

The feature PropagationPolicy Priority and Preemption was introduced in v1.7, and it is controlled by the feature gate PropagationPolicyPreemption which is disabled by default.

You can just execute the above one command to enable this feature gate. Or, if you want to use a more cautious approach, you can do like this:

  1. execute kubectl --context karmada-host edit deploy karmada-controller-manager -n karmada-system
  2. check if feature gate --feature-gates=PropagationPolicyPreemption=true is existed in spec.template.spec.containers[0].command field.
  3. If not, you shall add --feature-gates=PropagationPolicyPreemption=true into that field.

Preset resource in a member cluster

To simulate resources already exist in member cluster, we deploy some simple Deployments and Services to member1 cluster.

Step 3: Write the code

Create new file /tmp/deployments-and-services.yaml and copy text below to it:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   selector:
  7.     matchLabels:
  8.       app: nginx
  9.   replicas: 2
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.         - name: nginx
  17.           image: nginx:latest
  18.           ports:
  19.             - containerPort: 80
  20. ---
  21. apiVersion: v1
  22. kind: Service
  23. metadata:
  24.   name: nginx-svc
  25. spec:
  26.   selector:
  27.     app: nginx
  28.   type: NodePort
  29.   ports:
  30.   - port: 80
  31.     nodePort: 30000
  32.     targetPort: 80

Step 4: Run the command

  1. $ kubectl --context member1 apply -f /tmp/deployments-and-services.yaml
  2. deployment.apps/nginx-deploy created
  3. service/nginx-svc created
  4. deployment.apps/hello-deploy created
  5. service/hello-svc created

Thus, we can use member1 as the cluster with existing resources, while member2 as a bare cluster.

Tutorials

Migrate all the resources to Karmada

Step 1: Run the command

  1. $ kubectl --context karmada-apiserver apply -f /tmp/deployments-and-services.yaml
  2. deployment.apps/nginx-deploy created
  3. service/nginx-svc created
  4. deployment.apps/hello-deploy created
  5. service/hello-svc created

Note：

Same Deployments and Services should be deployed to Karmada control plane as ResourceTemplate.

Step 2: Write the code

Create new file /tmp/pp-for-migrating-deployments-and-services.yaml and copy text below to it:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: migrate-pp
  5. spec:
  6.   conflictResolution: Overwrite
  7.   placement:
  8.     clusterAffinity:
  9.       clusterNames:
  10.       - member1
  11.   priority: 0
  12.   resourceSelectors:
  13.   - apiVersion: apps/v1
  14.     kind: Deployment
  15.   - apiVersion: v1
  16.     kind: Service
  17.   schedulerName: default-scheduler

Note:

You should pay attention to two fields：

  • spec.conflictResolution: Overwrite：the value must be Overwrite.
  • spec.resourceSelectors：selecting resources to migrate, you can define your custom ResourceSelector.

Step 3: Run the command

Apply this PropagationPolicy to Karmada control plane.

  1. $ kubectl --context karmada-apiserver apply -f /tmp/pp-for-migrating-deployments-and-services.yaml
  2. propagationpolicy.policy.karmada.io/migrate-pp created

Step 4: Verification

  1. $ kubectl --context karmada-apiserver get deploy
  2. NAME           READY   UP-TO-DATE   AVAILABLE   AGE
  3. nginx-deploy   2/2     2            2           38s
  4. $ kubectl --context karmada-apiserver get rb
  5. NAME                      SCHEDULED   FULLYAPPLIED   AGE
  6. nginx-deploy-deployment   True        True           13s
  7. nginx-svc-service         True        True           13s

You shall see the Deployments in Karmada are all ready and the aggregatedStatus of ResourceBinding is applied, which means the existing resources in member1 cluster has been taken over by Karmada.

So far, you have finished the migration, isn’t it so easy?

Apply higher priority PropagationPolicy

Step 5: Write the code

Create new file /tmp/pp-for-nginx-app.yaml and copy text below to it:

  1. apiVersion: policy.karmada.io/v1alpha1
  2. kind: PropagationPolicy
  3. metadata:
  4.   name: nginx-pp
  5. spec:
  6.   conflictResolution: Overwrite
  7.   placement:
  8.     clusterAffinity:
  9.       clusterNames:
  10.       - member1
  11.       - member2                   ## propagate to more clusters other than member1
  12.   priority: 10                    ## priority greater than above PropagationPolicy (10 > 0)
  13.   preemption: Always              ## preemption should equal to Always
  14.   resourceSelectors:
  15.   - apiVersion: apps/v1
  16.     kind: Deployment
  17.     name: nginx-deploy
  18.   - apiVersion: v1
  19.     kind: Service
  20.     name: nginx-svc
  21.   schedulerName: default-scheduler

Step 6: Run the command

Apply this higher priority PropagationPolicy to Karmada control plane.

  1. $ kubectl --context karmada-apiserver apply -f /tmp/pp-for-nginx-app.yaml
  2. propagationpolicy.policy.karmada.io/nginx-pp created

Step 7: Verification

  1. $ kubectl --context member2 get deploy -o wide 
  2. NAME           READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES         SELECTOR
  3. nginx-deploy   2/2     2            2           5m24s   nginx        nginx:latest   app=nginx
  4. $ kubectl --context member2 get svc -o wide   
  5. NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE   SELECTOR
  6. nginx-svc    NodePort    10.13.161.255   <none>        80:30000/TCP   54s   app=nginx
  7. ...

As you see, you shall find nginx application related resource are all propagated to member2 cluster, which means the higher priority PropagationPolicy does work.