Field Merge Semantics

Dealing with Field Merges

TL;DR

  • Fields set and deleted from Resource Config are merged into Resources by Apply
  • If a Resource already exists, Apply updates the Resources by merging the local Resource Config into the remote Resources
  • Fields removed from the Resource Config will be deleted from the remote Resource

Merging Fields

Advanced Section

This chapter contains advanced material that readers may want to skip and come back to later.

When are fields merged?

This page describes how Resource Config is merged with Resources or other Resource Config. This may occur when:

  • Applying Resource Config updates to the live Resources in the cluster
  • Defining Patches in the kustomization.yaml which are overlayed on resources and bases

Applying Resource Config Updates

Rather than replacing the Resource with the new Resource Config, Apply will merge the new Resource Config into the live Resource. This retains values which may be set by the control plane - such as replicas values set by auto scalers

Defining Patches

patches are sparse Resource Config which contain a subset of fields that override values defined in other Resource Config with the same Group/Version/Kind/Namespace/Name. This is used to alter values defined on Resource Config without having to fork it.

Motivation (Apply)

This page describes the semantics for merging Resource Config.

Ownership of Resource fields are shared between declarative Resource Config authored by human users, and values set by Controllers running in the cluster. Some fields, such as the status and clusterIp fields, are owned exclusively by Controllers. Fields, such as the name and namespace fields, are owned exclusively by the human user managing the Resource.

Other fields, such as replicas, may be owned by either human users, the apiserver or Controllers. For example, replicas may be explicitly set by a user, implicitly set to a default value by the apiserver, or continuously adjusted by a Controller such as and HorizontalPodAutoscaler.

Last Applied Resource Config

When Apply creates or updates a Resource, it writes the Resource Config it Applied to an annotation on the Resource. This allows it to compare the last Resource Config it Applied to the current Resource Config and identify fields that have been deleted.

  1. # deployment.yaml (Resource Config)
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: nginx-deployment
  6. spec:
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx
  17.         image: nginx:1.7.9
  1. # Original Resource
  2. Doesn't Exist
  1. # Applied Resource
  2. kind: Deployment
  3. metadata:
  4.   annotations:
  5.     # ...
  6.     # This is the deployment.yaml Resource Config written as an annotation on the object
  7.     # It was written by kubectl apply when the object was created
  8.     kubectl.kubernetes.io/last-applied-configuration: |
  9.       {"apiVersion":"apps/v1","kind":"Deployment",
  10.       "metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
  11.       "spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
  12.       "spec":{"containers":[{"image":"nginx:1.7.9","name":"nginx"}]}}}}      
  13.   # ...
  14. spec:
  15.   # ...
  16. status:
  17.   # ...

Merging Resources

Following are the merge semantics for Resources:

Adding Fields:

  • Fields present in the Resource Config that are missing from the Resource will be added to the Resource.
  • Fields will be added to the Last Applied Resource Config
  1. # deployment.yaml (Resource Config)
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   # ...
  6.   name: nginx-deployment
  7. spec:
  8.   # ...
  9.   minReadySeconds: 3
  1. # Original Resource
  2. kind: Deployment
  3. metadata:
  4.   # ...
  5.   name: nginx-deployment
  6. spec:
  7.   # ...
  8. status:
  9.   # ...
  1. # Applied Resource
  2. kind: Deployment
  3. metadata:
  4.   # ...
  5.   name: nginx-deployment
  6. spec:
  7.   # ...
  8.   minReadySeconds: 3
  9. status:
  10.   # ...

Updating Fields

  • Fields present in the Resource Config that are also present in the Resource will be merged recursively until a primitive field is updated, or a field is added / deleted.
  • Fields will be updated in the Last Applied Resource Config
  1. # deployment.yaml (Resource Config)
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   # ...
  6.   name: nginx-deployment
  7. spec:
  8.   # ...
  9.   replicas: 2
  1. # Original Resource
  2. kind: Deployment
  3. metadata:
  4.   # ...
  5.   name: nginx-deployment
  6. spec:
  7.   # ...
  8.   # could be defaulted or set by Resource Config
  9.   replicas: 1
  10. status:
  11.   # ...
  1. # Applied Resource
  2. kind: Deployment
  3. metadata:
  4.   # ...
  5.   name: nginx-deployment
  6. spec:
  7.   # ...
  8.   # updated
  9.   replicas: 2
  10. status:
  11.   # ...

Deleting Fields

  • Fields present in the Last Applied Resource Config that have been removed from the Resource Config will be deleted from the Resource.
  • Fields set to null in the Resource Config that are present in the Resource Config will be deleted from the Resource.
  • Fields will be removed from the Last Applied Resource Config
  1. # deployment.yaml (Resource Config)
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   # ...
  6.   name: nginx-deployment
  7. spec:
  8.   # ...
  1. # Original Resource
  2. kind: Deployment
  3. metadata:
  4.   # ...
  5.   name: nginx-deployment
  6.   # Containers replicas and minReadySeconds
  7.   kubectl.kubernetes.io/last-applied-configuration: |
  8.             {"apiVersion":"apps/v1","kind":"Deployment", "spec":{"replicas": "2", "minReadySeconds": "3", ...}, "metadata": {...}}
  9. spec:
  10.   # ...
  11.   minReadySeconds: 3
  12.   replicas: 2
  13. status:
  14.   # ...
  1. # Applied Resource
  2. kind: Deployment
  3. metadata:
  4.   # ...
  5.   name: nginx-deployment
  6.   kubectl.kubernetes.io/last-applied-configuration: |
  7.             {"apiVersion":"apps/v1","kind":"Deployment", "spec":{...}, "metadata": {...}}
  8. spec:
  9.   # ...
  10.   # deleted and then defaulted, but not in Last Applied
  11.   replicas: 1
  12.   # minReadySeconds deleted
  13. status:
  14.   # ...

Removing Fields from Resource Config

Simply removing a field from the Resource Config will not transfer the ownership to the cluster. Instead it will delete the field from the Resource. If a field is set in the Resource Config and the user wants to give up ownership (e.g. removing replicas from the Resource Config and using and autoscaler), the user must first remove it from the last Applied Resource Config stored by the cluster.

This can be performed using kubectl apply edit-last-applied to delete the replicas field from the Last Applied Resource Config, and then deleting it from the Resource Config.

Field Merge Semantics

Merging Primitives

Primitive fields are merged by replacing the current value with the new value.

Field Creation: Add the primitive field

Field Update: Change the primitive field value

Field Deletion: Delete the primitive field

Field in Resource ConfigField in ResourceField in Last AppliedAction
YesYes-Set live to the Resource Config value.
YesNo-Set live to the Resource Config value.
No-YesRemove from Resource.
No-NoDo nothing.

Merging Objects

Objects fields are updated by merging the sub-fields recursively (by field name) until a primitive field is found or the field is added / deleted.

Field Creation: Add the object field

Field Update: Recursively compare object sub-field values and merge them

Field Deletion: Delete the object field

Merge Table: For each field merge Resource Config and Resource values with the same name

Field in Resource ConfigField in ResourceField in Last AppliedAction
YesYes-Recursively merge the Resource Config and Resource values.
YesNo-Set live to the Resource Config value.
No-YesRemove field from Resource.
No-NoDo nothing.

Merging Maps

Map fields are updated by merging the elements (by key) until a primitive field is found or the value is added / deleted.

Field Creation: Add the map field

Field Update: Recursively compare map values by key and merge them

Field Deletion: Delete the map field

Merge Table: For each map element merge Resource Config and Resource values with the same key

Key in Resource ConfigKey in ResourceKey in Last AppliedAction
YesYes-Recursively merge the Resource Config and Resource values.
YesNo-Set live to the Resource Config value.
No-YesRemove map element from Resource.
No-NoDo nothing.

Merging Lists of Primitives

Lists of primitives will be merged if they have a patch strategy: merge on the field otherwise they will be replaced. Finalizer list example

Merge Strategy:

  • Merged primitive lists behave like ordered sets
  • Replace primitive lists are replaced when merged

Ordering: Uses the ordering specified in the Resource Config. Elements not specified in the Resource Config do not have ordering guarantees with respect to the elements in the Resource Config.

Merge Table: For each list element merge Resource Config and Resource element with the same value

Element in Resource ConfigElement in ResourceElement in Last AppliedAction
YesYes-Do nothing
YesNo-Add to list.
No-YesRemove from list.
No-NoDo nothing.

This merge strategy uses the patch merge key to identify container elements in a list and merge them. The patch merge key is defined in the Kubernetes API on the field.

  1. # Last Applied
  2. args: ["a", "b"]
  1. # Resource Config (Local)
  2. args: ["a", "c"]
  1. # Resource (Live)
  2. args: ["a", "b", "d"]
  1. # Applied Resource
  2. args: ["a", "c", "d"]

Merging Lists of Objects

Merge Strategy: Lists of primitives may be merged or replaced. Lists are merged if the list has a patch strategy of merge and a patch merge key on the list field. Container list example.

Merge Key: The patch merge key is used to identify same elements in a list. Unlike map elements (keyed by key) and object fields (keyed by field name), lists don’t have a built-in merge identity for elements (index does not define identity). Instead an object field is used as a synthetic key/value for merging elements. This fields is the patch merge key. List elements with the same patch merge key will be merged when lists are merged.

Ordering: Uses the ordering specified in the Resource Config. Elements not specified in the Resource Config do not have ordering guarantees.

Merge Table: For each list element merge Resource Config and Resource element where the elements have the same value for the patch merge key

Element in Resource ConfigElement in ResourceElement in Last AppliedAction
Yes--Recursively merge the Resource Config and Resource values.
YesNo-Add to list.
No-YesRemove from list.
No-NoDo nothing.

This merge strategy uses the patch merge key to identify container elements in a list and merge them. The patch merge key is defined in the Kubernetes API on the field.

  1. # Last Applied Resource Config
  2. containers:
  3. - name: nginx          # key: nginx
  4.   image: nginx:1.10
  5. - name: nginx-helper-a # key: nginx-helper-a; will be deleted in result
  6.   image: helper:1.3
  7. - name: nginx-helper-b # key: nginx-helper-b; will be retained
  8.   image: helper:1.3
  1. # Resource Config (Local)
  2. containers:
  3. - name: nginx
  4.   image: nginx:1.10
  5. - name: nginx-helper-b
  6.   image: helper:1.3
  7. - name: nginx-helper-c # key: nginx-helper-c; will be added in result
  8.   image: helper:1.3
  1. # Resource (Live)
  2. containers:
  3. - name: nginx
  4.   image: nginx:1.10
  5. - name: nginx-helper-a
  6.   image: helper:1.3
  7. - name: nginx-helper-b
  8.   image: helper:1.3
  9.   args: ["run"] # Field will be retained
  10. - name: nginx-helper-d # key: nginx-helper-d; will be retained
  11.   image: helper:1.3
  1. # Applied Resource
  2. containers:
  3. - name: nginx
  4.   image: nginx:1.10
  5.   # Element nginx-helper-a was Deleted
  6. - name: nginx-helper-b
  7.   image: helper:1.3
  8.   # Field was Ignored
  9.   args: ["run"]
  10.   # Element was Added
  11. - name: nginx-helper-c
  12.   image: helper:1.3
  13.   # Element was Ignored
  14. - name: nginx-helper-d
  15.   image: helper:1.3

Edit and Set

While kubectl edit and kubectl set ignore the Last Applied Resource Config, Apply will change any values in the Resource Config set by either kubectl edit or kubectl set. To ignore values set by kubectl edit or kubectl set:

  • Use kubectl apply edit-last-applied to remove the value from the Last Applied (if it is present)
  • Remove the field from the Resource Config

This is the same technique for retaining values set by cluster components such as autoscalers.