Kubelet Configuration Directory Merging

When using the kubelet’s --config-dir flag to specify a drop-in directory for configuration, there is some specific behavior on how different types are merged.

Here are some examples of how different data types behave during configuration merging:

Structure Fields

There are two types of structure fields in a YAML structure: singular (or a scalar type) and embedded (structures that contain scalar types). The configuration merging process handles the overriding of singular and embedded struct fields to create a resulting kubelet configuration.

For instance, you may want a baseline kubelet configuration for all nodes, but you may want to customize the address and authorization fields. This can be done as follows:

Main kubelet configuration file contents:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. port: 20250
  4. authorization:
  5.   mode: Webhook
  6.   webhook:
  7.     cacheAuthorizedTTL: "5m"
  8.     cacheUnauthorizedTTL: "30s"
  9. serializeImagePulls: false
  10. address: "192.168.0.1"

Contents of a file in --config-dir directory:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. authorization:
  4.   mode: AlwaysAllow
  5.   webhook:
  6.     cacheAuthorizedTTL: "8m"
  7.     cacheUnauthorizedTTL: "45s"
  8. address: "192.168.0.8"

The resulting configuration will be as follows:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. port: 20250
  4. serializeImagePulls: false
  5. authorization:
  6.   mode: AlwaysAllow
  7.   webhook:
  8.     cacheAuthorizedTTL: "8m"
  9.     cacheUnauthorizedTTL: "45s"
  10. address: "192.168.0.8"

Lists

You can overide the slices/lists values of the kubelet configuration. However, the entire list gets overridden during the merging process. For example, you can override the clusterDNS list as follows:

Main kubelet configuration file contents:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. port: 20250
  4. serializeImagePulls: false
  5. clusterDNS:
  6.   - "192.168.0.9"
  7.   - "192.168.0.8"

Contents of a file in --config-dir directory:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. clusterDNS:
  4.   - "192.168.0.2"
  5.   - "192.168.0.3"
  6.   - "192.168.0.5"

The resulting configuration will be as follows:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. port: 20250
  4. serializeImagePulls: false
  5. clusterDNS:
  6.   - "192.168.0.2"
  7.   - "192.168.0.3"
  8.   - "192.168.0.5"

Maps, including Nested Structures

Individual fields in maps, regardless of their value types (boolean, string, etc.), can be selectively overridden. However, for map[string][]string, the entire list associated with a specific field gets overridden. Let’s understand this better with an example, particularly on fields like featureGates and staticPodURLHeader:

Main kubelet configuration file contents:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. port: 20250
  4. serializeImagePulls: false
  5. featureGates:
  6.   AllAlpha: false
  7.   MemoryQoS: true
  8. staticPodURLHeader:
  9.   kubelet-api-support:
  10.   - "Authorization: 234APSDFA"
  11.   - "X-Custom-Header: 123"
  12.   custom-static-pod:
  13.   - "Authorization: 223EWRWER"
  14.   - "X-Custom-Header: 456"

Contents of a file in --config-dir directory:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. featureGates:
  4.   MemoryQoS: false
  5.   KubeletTracing: true
  6.   DynamicResourceAllocation: true
  7. staticPodURLHeader:
  8.   custom-static-pod:
  9.   - "Authorization: 223EWRWER"
  10.   - "X-Custom-Header: 345"

The resulting configuration will be as follows:

  1. apiVersion: kubelet.config.k8s.io/v1beta1
  2. kind: KubeletConfiguration
  3. port: 20250
  4. serializeImagePulls: false
  5. featureGates:
  6.   AllAlpha: false
  7.   MemoryQoS: false
  8.   KubeletTracing: true
  9.   DynamicResourceAllocation: true
  10. staticPodURLHeader:
  11.   kubelet-api-support:
  12.   - "Authorization: 234APSDFA"
  13.   - "X-Custom-Header: 123"
  14.   custom-static-pod:
  15.   - "Authorization: 223EWRWER"
  16.   - "X-Custom-Header: 345"