Customizing Resource Interpreter

Resource Interpreter Framework

In the progress of propagating a resource from karmada-apiserver to member clusters, Karmada needs to know the resource definition. Take Propagating Deployment as an example, at the phase of building ResourceBinding, the karmada-controller-manager will parse the replicas from the deployment object.

For Kubernetes native resources, Karmada knows how to parse them, but for custom resources defined by CRD(or extended by something like aggregated-apiserver), as lack of the knowledge of the resource structure, they can only be treated as normal resources. Therefore, the advanced scheduling algorithms cannot be used for them.

The Resource Interpreter Framework is designed for interpreting resource structure. It consists of built-in and customized interpreters:

  • built-in interpreter: used for common Kubernetes native or well-known extended resources.
  • customized interpreter: interprets custom resources or overrides the built-in interpreters.

Note: The major difference between built-in and customized interpreters is that the built-in interpreter is implemented and maintained by Karmada community and will be built into Karmada components, such as karmada-controller-manager. On the contrary, the customized interpreter is implemented and maintained by users. It should be registered to Karmada as an Interpreter Webhook or declarative configuration (see Customized Interpreter for more details).

Interpreter Operations

When interpreting resources, we often get multiple pieces of information extracted. The Interpreter Operations defines the interpreter request type, and the Resource Interpreter Framework provides services for each operation type.

For all operations designed by Resource Interpreter Framework, please refer to Interpreter Operations.

Note: Not all the designed operations are supported (see below for supported operations).

Note: At most one interpreter will be consulted to when interpreting a resource with specific interpreter operation and the customized interpreter has higher priority than built-in interpreter if they are both interpreting the same resource. For example, the built-in interpreter serves InterpretReplica for Deployment with version apps/v1. If there is a customized interpreter registered to Karmada for interpreting the same resource, the customized interpreter wins and the built-in interpreter will be ignored.

Built-in Interpreter

For the common Kubernetes native or well-known extended resources, the interpreter operations are built-in, which means the users usually don’t need to implement customized interpreters. If you want more resources to be built-in, please feel free to file an issue to let us know your user case.

The built-in interpreter now supports following interpreter operations:

InterpretReplica

Supported resources:

  • Deployment(apps/v1)
  • StatefulSet(apps/v1)
  • Job(batch/v1)
  • Pod(v1)

ReviseReplica

Supported resources:

  • Deployment(apps/v1)
  • StatefulSet(apps/v1)
  • Job(batch/v1)

Retain

Supported resources:

  • Pod(v1)
  • Service(v1)
  • ServiceAccount(v1)
  • PersistentVolumeClaim(v1)
  • PersistentVolume(V1)
  • Job(batch/v1)

AggregateStatus

Supported resources:

  • Deployment(apps/v1)
  • Service(v1)
  • Ingress(networking.k8s.io/v1)
  • Job(batch/v1)
  • CronJob(batch/v1)
  • DaemonSet(apps/v1)
  • StatefulSet(apps/v1)
  • Pod(v1)
  • PersistentVolume(V1)
  • PersistentVolumeClaim(v1)
  • PodDisruptionBudget(policy/v1)

InterpretStatus

Supported resources:

  • Deployment(apps/v1)
  • Service(v1)
  • Ingress(networking.k8s.io/v1)
  • Job(batch/v1)
  • DaemonSet(apps/v1)
  • StatefulSet(apps/v1)
  • PodDisruptionBudget(policy/v1)

InterpretDependency

Supported resources:

  • Deployment(apps/v1)
  • Job(batch/v1)
  • CronJob(batch/v1)
  • Pod(v1)
  • DaemonSet(apps/v1)
  • StatefulSet(apps/v1)
  • Ingress(networking.k8s.io/v1)

InterpretHealth

Supported resources:

  • Deployment(apps/v1)
  • StatefulSet(apps/v1)
  • ReplicaSet(apps/v1)
  • DaemonSet(apps/v1)
  • Service(v1)
  • Ingress(networking.k8s.io/v1)
  • PersistentVolumeClaim(v1)
  • PodDisruptionBudget(policy/v1)
  • Pod(v1)

Customized Interpreter

The customized interpreter is implemented and maintained by users, it can be extended in two ways, either by defining declarative configuration files or by running as webhook at runtime.

Note: Decalrative configuration has a higher priority than webhook.

Built-in Resource Declarative Configuration

Karmada bundles some popular and open-sourced resources so that users can save the effort to customize them. The configurable interpreter now supports following interpreter operations:

InterpretReplica

Supported resources:

  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)

ReviseReplica

Supported resources:

  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)

Retain

Supported resources:

  • BroadcastJob(apps.kruise.io/v1alpha1)
  • Workflow(argoproj.io/v1alpha1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

AggregateStatus

Supported resources:

  • AdvancedCronJob(apps.kruise.io/v1alpha1)
  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • ClusterPolicy(kyverno.io/v1)
  • Policy(kyverno.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

InterpretStatus

Supported resources:

  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • ClusterPolicy(kyverno.io/v1)
  • Policy(kyverno.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

InterpretDependency

Supported resources:

  • AdvancedCronJob(apps.kruise.io/v1alpha1)
  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

InterpretHealth

Supported resources:

  • AdvancedCronJob(apps.kruise.io/v1alpha1)
  • AdvancedDaemonSet(apps.kruise.io/v1alpha1)
  • BroadcastJob(apps.kruise.io/v1alpha1)
  • CloneSet(apps.kruise.io/v1alpha1)
  • AdvancedStatefulSet(apps.kruise.io/v1beta1)
  • Workflow(argoproj.io/v1alpha1)
  • HelmRelease(helm.toolkit.fluxcd.io/v2beta1)
  • Kustomization(kustomize.toolkit.fluxcd.io/v1)
  • ClusterPolicy(kyverno.io/v1)
  • Policy(kyverno.io/v1)
  • GitRepository(source.toolkit.fluxcd.io/v1)
  • Bucket(source.toolkit.fluxcd.io/v1beta2)
  • HelmChart(source.toolkit.fluxcd.io/v1beta2)
  • HelmRepository(source.toolkit.fluxcd.io/v1beta2)
  • OCIRepository(source.toolkit.fluxcd.io/v1beta2)

Declarative Configuration

What are interpreter declarative configuration?

Users can quickly customize resource interpreters for both Kubernetes resources and CR resources by the rules declaraed in the ResourceInterpreterCustomization API specification.

Write with configuration

You can configure resource interpretation rules by creating or updating ResourceInterpreterCustomization resource, the newest version supports the definition of lua scripts in the ResourceInterpreterCustomization. You can learn how to define the lua script in the API definition, take retention as an example.

Below we provide a yaml writing example of the ResourceInterpreterCustomization resource:

resource-interpreter-customization.yaml

  1. apiVersion: config.karmada.io/v1alpha1
  2. kind: ResourceInterpreterCustomization
  3. metadata:
  4.   name: declarative-configuration-example
  5. spec:
  6.   target:
  7.     apiVersion: apps/v1
  8.     kind: Deployment
  9.   customizations:
  10.     replicaResource:
  11.       luaScript: >
  12.         local kube = require("kube")
  13.         function GetReplicas(obj)
  14.           replica = obj.spec.replicas
  15.           requirement = kube.accuratePodRequirements(obj.spec.template)
  16.           return replica, requirement
  17.         end
  18.     replicaRevision:
  19.       luaScript: >
  20.         function ReviseReplica(obj, desiredReplica)
  21.           obj.spec.replicas = desiredReplica
  22.           return obj
  23.         end
  24.     retention:
  25.       luaScript: >
  26.         function Retain(desiredObj, observedObj)
  27.           desiredObj.spec.paused = observedObj.spec.paused
  28.           return desiredObj
  29.         end
  30.     statusAggregation:
  31.       luaScript: >
  32.         function AggregateStatus(desiredObj, statusItems)
  33.           if statusItems == nil then
  34.             return desiredObj
  35.           end
  36.           if desiredObj.status == nil then
  37.             desiredObj.status = {}
  38.           end
  39.           replicas = 0
  40.           for i = 1, #statusItems do
  41.             if statusItems[i].status ~= nil and statusItems[i].status.replicas ~= nil then
  42.               replicas = replicas + statusItems[i].status.replicas
  43.             end
  44.           end
  45.           desiredObj.status.replicas = replicas
  46.           return desiredObj
  47.         end
  48.     statusReflection:
  49.       luaScript: >
  50.         function ReflectStatus (observedObj)
  51.           return observedObj.status
  52.         end
  53.     healthInterpretation:
  54.       luaScript: >
  55.         function InterpretHealth(observedObj)
  56.           return observedObj.status.readyReplicas == observedObj.spec.replicas
  57.         end
  58.     dependencyInterpretation:
  59.       luaScript: >
  60.         function GetDependencies(desiredObj)
  61.           dependentSas = {}
  62.           refs = {}
  63.           if desiredObj.spec.template.spec.serviceAccountName ~= nil and desiredObj.spec.template.spec.serviceAccountName ~= 'default' then
  64.             dependentSas[desiredObj.spec.template.spec.serviceAccountName] = true
  65.           end
  66.           local idx = 1
  67.           for key, value in pairs(dependentSas) do
  68.             dependObj = {}
  69.             dependObj.apiVersion = 'v1'
  70.             dependObj.kind = 'ServiceAccount'
  71.             dependObj.name = key
  72.             dependObj.namespace = desiredObj.metadata.namespace
  73.             refs[idx] = dependObj
  74.             idx = idx + 1
  75.           end
  76.           return refs
  77.         end

Verify the configuration

Users can use the karmadactl interpret command to verify the ResourceInterpreterCustomization configuration before applying them to the system. Some examples are provided to help users better understand how this interpreter can be used, please refer to examples.

Webhook

What are interpreter webhooks?

Interpreter webhooks are HTTP callbacks that receive interpret requests and do something with them.

Write an interpreter webhook server

Please refer to the implementation of the Example of Customize Interpreter that is validated in Karmada E2E test. The webhook handles the ResourceInterpreterRequest request sent by the Karmada components (such as karmada-controller-manager), and sends back its decision as an ResourceInterpreterResponse.

Deploy the admission webhook service

The Example of Customize Interpreter is deployed in the host cluster for E2E and exposed by a service as the front-end of the webhook server.

You may also deploy your webhooks outside the cluster. You will need to update your webhook configurations accordingly.

Configure webhook on the fly

You can configure what resources and supported operations are subject to what interpreter webhook via ResourceInterpreterWebhookConfiguration.

The following is an example ResourceInterpreterWebhookConfiguration:

  1. apiVersion: config.karmada.io/v1alpha1
  2. kind: ResourceInterpreterWebhookConfiguration
  3. metadata:
  4.   name: examples
  5. webhooks:
  6.   - name: workloads.example.com
  7.     rules:
  8.       - operations: [ "InterpretReplica","ReviseReplica","Retain","AggregateStatus" ]
  9.         apiGroups: [ "workload.example.io" ]
  10.         apiVersions: [ "v1alpha1" ]
  11.         kinds: [ "Workload" ]
  12.     clientConfig:
  13.       url: https://karmada-interpreter-webhook-example.karmada-system.svc:443/interpreter-workload
  14.       caBundle: {{caBundle}}
  15.     interpreterContextVersions: [ "v1alpha1" ]
  16.     timeoutSeconds: 3

You can config more than one webhook in a ResourceInterpreterWebhookConfiguration, each webhook serves at least one operation.

Write the ResourceInterpreterCustomization

You can learn how to write the ResourceInterpreterCustomization to customize your resource.

First, we introduce the kube library functions. Then, we introduce how to write ResourceInterpreterCustomization using kyverno.io/v1/ClusterPolicy as an example.

build-in functions of luavm

The rules declared in the ResourceInterpreterCustomization API specification define interpreter operations. These operations are written by lua and called via luavm. Users can use luavm’s built-in functions when writing interpreter operations.

In kubeLibrary, there are two functions that are useful for writing interpreter operations. accuratePodRequirements is useful to write ReplicaResource operation and getPodDependenciesis useful to write DependencyInterpretation operation.

The accuratePodRequirements function accurates total resource requirements for pod. Its argument is PodTemplateSpec and its return value is ReplicaRequirements. PodTemplateSpec describes the data a pod should have when created from a template, and ReplicaRequirements represents the requirements required by each replica.

The getPodDependencies function gets total dependencies from podTemplate and namespace. Its arguments are PodTemplateSpec and namespace. Its return value is dependencies. PodTemplateSpec describes the data a pod should have when created from a template. namespace is the namespace of customized resource. And dependencies are the resources on which the customized resources depend.

ReplicaResource

ReplicaResource describes the rules for Karmada to discover the resource’s replica as well as resource requirements. It would be useful for those CRD resources that declare workload types like Deployment.

A Kyverno ClusterPolicy is a collection of rules, which doesn’t have fields like .spec.replicas or .spec.template.spec.nodeSelector. So there is no need to implement ReplicaResource for ClusterPolicy.

ReplicaRevision

ReplicaRevision describes the rules for Karmada to revise the resource’s replica. It would be useful for those CRD resources that declare workload types like Deployment.

A Kyverno ClusterPolicy is a collection of rules, which doesn’t have field like .spec.replicas. So there is no need to implement ReplicaRevision for ClusterPolicy.

Retention

Retention describes the desired behavior that Karmada should react on the changes made by member cluster components. This avoids system running into a meaningless loop that Karmada resource controller and the member cluster component continually applying opposite values of a field.

A Kyverno ClusterPolicy is a collection of rules, which is usually not changed by member cluster components. So there is no need to implement Retention for ClusterPolicy.

StatusAggregation

StatusAggregation describes the rules for Karmada to aggregate status collected from member clusters to resource template.

A Kyverno ClusterPolicy is a collection of rules. Here we define the status aggregation rules for ClusterPolicy.

StatusAggregation-Defined-In-ResourceInterpreterCustomization

  1. statusAggregation:
  2.   luaScript: >
  3.     function AggregateStatus(desiredObj, statusItems)
  4.       if statusItems == nil then
  5.         return desiredObj
  6.       end
  7.       desiredObj.status = {}
  8.       desiredObj.status.conditions = {}
  9.       rulecount = {}
  10.       rulecount.validate = 0
  11.       rulecount.generate = 0
  12.       rulecount.mutate = 0
  13.       rulecount.verifyimages = 0
  14.       conditions = {}
  15.       local conditionsIndex = 1
  16.       for i = 1, #statusItems do
  17.         if statusItems[i].status ~= nil and statusItems[i].status.autogen ~= nil then
  18.           desiredObj.status.autogen = statusItems[i].status.autogen
  19.         end
  20.         if statusItems[i].status ~= nil and statusItems[i].status.ready ~= nil then
  21.           desiredObj.status.ready = statusItems[i].status.ready
  22.         end                        
  23.         if statusItems[i].status ~= nil and statusItems[i].status.rulecount ~= nil then
  24.           rulecount.validate = rulecount.validate + statusItems[i].status.rulecount.validate
  25.           rulecount.generate = rulecount.generate + statusItems[i].status.rulecount.generate
  26.           rulecount.mutate = rulecount.mutate + statusItems[i].status.rulecount.mutate
  27.           rulecount.verifyimages = rulecount.verifyimages + statusItems[i].status.rulecount.verifyimages
  28.         end
  29.         if statusItems[i].status ~= nil and statusItems[i].status.conditions ~= nil then
  30.           for conditionIndex = 1, #statusItems[i].status.conditions do
  31.             statusItems[i].status.conditions[conditionIndex].message = statusItems[i].clusterName..'='..statusItems[i].status.conditions[conditionIndex].message
  32.             hasCondition = false
  33.             for index = 1, #conditions do
  34.               if conditions[index].type == statusItems[i].status.conditions[conditionIndex].type and conditions[index].status == statusItems[i].status.conditions[conditionIndex].status and conditions[index].reason == statusItems[i].status.conditions[conditionIndex].reason then
  35.                 conditions[index].message = conditions[index].message..', '..statusItems[i].status.conditions[conditionIndex].message
  36.                 hasCondition = true
  37.                 break
  38.               end
  39.             end
  40.             if not hasCondition then
  41.               conditions[conditionsIndex] = statusItems[i].status.conditions[conditionIndex]
  42.               conditionsIndex = conditionsIndex + 1                  
  43.             end
  44.           end
  45.         end
  46.       end
  47.       desiredObj.status.rulecount = rulecount
  48.       desiredObj.status.conditions = conditions
  49.       return desiredObj
  50.     end

StatusReflection

StatusReflection describes the rules for Karmada to pick the resource’s status.

A Kyverno ClusterPolicy is a collection of rules, whose .status contains policy runtime data. StatusReflection determines which fields Karmada collect from the member clusters. Here we pick some fields from resource in member cluster.

StatusReflection-Defined-In-ResourceInterpreterCustomization

  1. statusReflection:
  2.   luaScript: >
  3.     function ReflectStatus (observedObj)
  4.       status = {}
  5.       if observedObj == nil or observedObj.status == nil then 
  6.         return status
  7.       end
  8.       status.ready = observedObj.status.ready
  9.       status.conditions = observedObj.status.conditions
  10.       status.autogen = observedObj.status.autogen
  11.       status.rulecount = observedObj.status.rulecount
  12.       return status
  13.     end

HealthInterpretation

HealthInterpretation describes the health assessment rules by which Karmada can assess the health state of the resource type.

A Kyverno ClusterPolicy is a collection of rules. We determine whether the ClusterPolicy in member cluster is healthy by defining the health assessment rules.

HealthInterpretation-Defined-In-ResourceInterpreterCustomization

  1. healthInterpretation:
  2.   luaScript: >
  3.     function InterpretHealth(observedObj)
  4.       if observedObj.status ~= nil and observedObj.status.ready ~= nil then
  5.         return observedObj.status.ready
  6.       end
  7.       if observedObj.status ~= nil and observedObj.status.conditions ~= nil then
  8.         for conditionIndex = 1, #observedObj.status.conditions do
  9.           if observedObj.status.conditions[conditionIndex].type == 'Ready' and observedObj.status.conditions[conditionIndex].status == 'True' and observedObj.status.conditions[conditionIndex].reason == 'Succeeded' then
  10.             return true
  11.           end
  12.         end
  13.       end
  14.       return false
  15.     end

DependencyInterpretation

DependencyInterpretation describes the rules for Karmada to analyze the dependent resources.

A Kyverno ClusterPolicy is a collection of rules, which doesn’t depend on other resources. So there is no need to implement DependencyInterpretation for ClusterPolicy.

Important Notes

Use the Retain Interpreter to Resolve Control Conflict between the Control Plane and Member Clusters

Issue: Retain is a user-customizable interpreter in Karmada that resolves control conflicts when both the Karmada control plane and member clusters have control over member cluster resources. A typical scenario is when the replicas of a member cluster’s Deployment is controlled by both the control plane’s resource template and the member cluster’s HPA. This leads to an abnormal state of the member cluster’s Deployment due to repeated modifications of the replicas by both entities.

Solution:

  • Implement the corresponding Retain Interpreter for your workload type resources to decide when to respond to modifications from the control plane’s resource template and when to respond to modifications from the member cluster’s HPA. Currently, Karmada has only implemented the Retain interpreter for Deployment resources. The specific implementation is as follows: if the resource template has the label resource template.karmada.io/retain-replicas, it will be controlled by the member cluster’s HPA; otherwise, it will be controlled by the control plane’s resource template (if the hpaReplicasSyncer controller is explicitly enabled, Karmada can automatically label the Deployment enabled with HPA with this label). If you need to resolve this conflict for other resources or custom CRD resources, you can refer to the Retain solution for Deployments.
  • If you want a more elegant and comprehensive solution to the above problem, we recommend replacing HPA with FederatedHPA.