Determine cluster failures

Karmada supports both Push and Pull modes to manage member clusters. More details about cluster registration please refer to Cluster Registration.

For clusters there are two forms of heartbeats:

  • updates to the .status of a Cluster(include both Push and Pull mode).
  • Lease objects within the karmada-cluster namespace in karmada control plane. Each Pull mode cluster has an associated Lease object.

Collect cluster status

For Push mode clusters, the clusterStatus controller in karmada control plane will continually collect cluster’s status for a configured interval. For Pull mode clusters, the karmada-agent is responsible for creating and updating the .status of clusters with configured interval.

The interval for .status updates to Cluster can be configured via --cluster-status-update-frequency flag(default is 10 seconds).

Cluster’s Ready condition might be set to the False with following conditions:

  • cluster is unreachable for a period of configured interval.
  • cluster’s health endpoint responded without ok persists for a period of configured interval.

The above two intervals can be configured via --cluster-failure-threshold flag(default is 30 seconds).

Update cluster Lease object

Karmada will create a Lease object and a lease controller for each Pull mode cluster when clusters are joined.

Each lease controller is responsible for updating the related Leases. The lease renewing time can be configured via --cluster-lease-duration and --cluster-lease-renew-interval-fraction flags(default is 10 seconds).

Lease’s updating process is independent with cluster’s status updating process, since cluster’s .status field is maintained by clusterStatus controller.

The cluster controller in Karmada control plane would check the state of each pull mode cluster every --cluster-monitor-period period(default is 5 seconds).

The cluster’s Ready condition would be changed to Unknown when cluster controller has not heard from the cluster in the last --cluster-monitor-grace-period(default is 40 seconds).

Check cluster status

You can use kubectl to check a Cluster’s status and other details:

  1. kubectl describe cluster <cluster-name>

The Ready condition in Status field indicates the cluster is healthy and ready to accept workloads. It will be set to False if the cluster is not healthy and is not accepting workloads, and Unknown if the cluster controller has not heard from the cluster in the last cluster-monitor-grace-period.

The following example describes an unhealthy cluster:

unfold me to see the yaml

  1. kubectl describe cluster member1
  3. Name:         member1
  4. Namespace:    
  5. Labels:       <none>
  6. Annotations:  <none>
  7. API Version:  cluster.karmada.io/v1alpha1
  8. Kind:         Cluster
  9. Metadata:
  10.   Creation Timestamp:  2021-12-29T08:49:35Z
  11.   Finalizers:
  12.     karmada.io/cluster-controller
  13.   Resource Version:  152047
  14.   UID:               53c133ab-264e-4e8e-ab63-a21611f7fae8
  15. Spec:
  16.   API Endpoint:  https://172.23.0.7:6443
  17.   Impersonator Secret Ref:
  18.     Name:       member1-impersonator
  19.     Namespace:  karmada-cluster
  20.   Secret Ref:
  21.     Name:       member1
  22.     Namespace:  karmada-cluster
  23.   Sync Mode:    Push
  24. Status:
  25.   Conditions:
  26.     Last Transition Time:  2021-12-31T03:36:08Z
  27.     Message:               cluster is not reachable
  28.     Reason:                ClusterNotReachable
  29.     Status:                False
  30.     Type:                  Ready
  31. Events:                    <none>