kubeadm Configuration (v1beta2)

Overview

Package v1beta2 defines the v1beta2 version of the kubeadm configuration file format. This version improves on the v1beta1 format by fixing some minor issues and adding a few new fields.

A list of changes since v1beta1:

• “certificateKey” field is added to InitConfiguration and JoinConfiguration.
• “ignorePreflightErrors” field is added to the NodeRegistrationOptions.
• The JSON “omitempty” tag is used in a more places where appropriate.
• The JSON “omitempty” tag of the “taints” field (inside NodeRegistrationOptions) is removed.

See the Kubernetes 1.15 changelog for further details.

Migration from old kubeadm config versions

Please convert your v1beta1 configuration files to v1beta2 using the “kubeadm config migrate” command of kubeadm v1.15.x (conversion from older releases of kubeadm config files requires older release of kubeadm as well e.g.

• kubeadm v1.11 should be used to migrate v1alpha1 to v1alpha2; kubeadm v1.12 should be used to translate v1alpha2 to v1alpha3;
• kubeadm v1.13 or v1.14 should be used to translate v1alpha3 to v1beta1)

Nevertheless, kubeadm v1.15.x will support reading from v1beta1 version of the kubeadm config file format.

Basics

The preferred way to configure kubeadm is to pass an YAML configuration file with the --config option. Some of the configuration options defined in the kubeadm config file are also available as command line flags, but only the most common/simple use case are supported with this approach.

A kubeadm config file could contain multiple configuration types separated using three dashes (---).

kubeadm supports the following configuration types:

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration

To print the defaults for “init” and “join” actions use the following commands:

kubeadm config print init-defaults
kubeadm config print join-defaults

The list of configuration types that must be included in a configuration file depends by the action you are performing (init or join) and by the configuration options you are going to use (defaults or advanced customization).

If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults provided by kubeadm includes also enforcing consistency of values across components when required (e.g. --cluster-cidr flag on controller manager and clusterCIDR on kube-proxy).

Users are always allowed to override default values, with the only exception of a small subset of setting with relevance for security (e.g. enforce authorization-mode Node and RBAC on API server)

If the user provides a configuration types that is not expected for the action you are performing, kubeadm will ignore those types and print a warning.

Kubeadm init configuration types

When executing kubeadm init with the --config option, the following configuration types could be used: InitConfiguration, ClusterConfiguration, KubeProxyConfiguration, KubeletConfiguration, but only one between InitConfiguration and ClusterConfiguration is mandatory.

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
  ...
nodeRegistration:
  ...

The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init are the configuration of the bootstrap token and all the setting which are specific to the node where kubeadm is executed, including:

• nodeRegistration, that holds fields that relate to registering the new node to the cluster; use it to customize the node name, the CRI socket to use or any other settings that should apply to this node only (e.g. the node ip).

• apiServer, that represents the endpoint of the instance of the API server to be deployed on this node; use it e.g. to customize the API server advertise address.

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
    ...
etcd:
    ...
apiServer:
  extraArgs:
    ...
  extraVolumes:
    ...
...

The ClusterConfiguration type should be used to configure cluster-wide settings, including settings for:

• Networking, that holds configuration for the networking topology of the cluster; use it e.g. to customize pod subnet or services subnet.

• Etcd configurations; use it e.g. to customize the local etcd or to configure the API server for using an external etcd cluster.

• kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane components by adding customized setting or overriding kubeadm default settings.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
  ...

The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or https://godoc.org/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration for kube proxy official documentation.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
  ...

The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.

See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration for kubelet official documentation.

Here is a fully populated example of a single YAML file containing multiple configuration types to be used during a kubeadm init run.

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
  - token: "9a08jv.c0izixklcxtmnze7"
    description: "kubeadm bootstrap token"
    ttl: "24h"
  - token: "783bde.3f89s0fje9f38fhf"
    description: "another bootstrap token"
    usages:
      - authentication
      - signing
    groups:
      - system:bootstrappers:kubeadm:default-node-token
nodeRegistration:
  name: "ec2-10-100-0-1"
  criSocket: "/var/run/dockershim.sock"
  taints:
    - key: "kubeadmNode"
      value: "someValue"
      effect: "NoSchedule"
  kubeletExtraArgs:
    v: 4
  ignorePreflightErrors:
    - IsPrivilegedUser
localAPIEndpoint:
  advertiseAddress: "10.100.0.1"
  bindPort: 6443
certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
etcd:
  # one of local or external
  local:
    imageRepository: "k8s.gcr.io"
    imageTag: "3.2.24"
    dataDir: "/var/lib/etcd"
    extraArgs:
      listen-client-urls: "http://10.100.0.1:2379"
    serverCertSANs:
      -  "ec2-10-100-0-1.compute-1.amazonaws.com"
    peerCertSANs:
      - "10.100.0.1"
  # external:
  #   endpoints:
  #     - "10.100.0.1:2379"
  #     - "10.100.0.2:2379"
  #   caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
  #   certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
  #   keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.244.0.0/24"
  dnsDomain: "cluster.local"
kubernetesVersion: "v1.12.0"
controlPlaneEndpoint: "10.100.0.1:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
  extraVolumes:
    - name: "some-volume"
      hostPath: "/etc/some-path"
      mountPath: "/etc/some-pod-path"
      readOnly: false
      pathType: File
  certSANs:
    - "10.100.1.1"
    - "ec2-10-100-0-1.compute-1.amazonaws.com"
  timeoutForControlPlane: 4m0s
controllerManager:
  extraArgs:
    "node-cidr-mask-size": "20"
  extraVolumes:
    - name: "some-volume"
      hostPath: "/etc/some-path"
      mountPath: "/etc/some-pod-path"
      readOnly: false
      pathType: File
scheduler:
  extraArgs:
    address: "10.100.0.1"
  extraVolumes:
    - name: "some-volume"
      hostPath: "/etc/some-path"
      mountPath: "/etc/some-pod-path"
      readOnly: false
      pathType: File
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"
useHyperKubeImage: false
clusterName: "example-cluster"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# kubelet specific options here
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
# kube-proxy specific options here

Kubeadm join configuration types

When executing kubeadm join with the --config option, the JoinConfiguration type should be provided.

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
  ...

The JoinConfiguration type should be used to configure runtime settings, that in case of kubeadm join are the discovery method used for accessing the cluster info and all the setting which are specific to the node where kubeadm is executed, including:

• NodeRegistration, that holds fields that relate to registering the new node to the cluster; use it to customize the node name, the CRI socket to use or any other settings that should apply to this node only (e.g. the node IP).

• APIEndpoint, that represents the endpoint of the instance of the API server to be eventually deployed on this node.

Resource Types

• ClusterConfiguration
• ClusterStatus
• InitConfiguration
• JoinConfiguration

ClusterConfiguration

ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster

ClusterStatus

ClusterStatus contains the cluster status. The ClusterStatus will be stored in the kubeadm-config ConfigMap in the cluster, and then updated by kubeadm when additional control plane instance joins or leaves the cluster.

InitConfiguration

InitConfiguration contains a list of elements that is specific “kubeadm init”-only runtime information.

JoinConfiguration

JoinConfiguration contains elements describing a particular node.

APIEndpoint

Appears in:

• ClusterStatus

• InitConfiguration

• JoinControlPlane

APIEndpoint struct contains elements of API server instance deployed on a node.

APIServer

Appears in:

• ClusterConfiguration

APIServer holds settings necessary for API server deployments in the cluster.

BootstrapToken

Appears in:

• InitConfiguration

BootstrapToken describes one bootstrap token, stored as a Secret in the cluster

BootstrapTokenDiscovery

Appears in:

• Discovery

BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery

BootstrapTokenString

Appears in:

• BootstrapToken

BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used for both validation of the practically of the API server from a joining node’s point of view and as an authentication method for the node in the bootstrap phase of “kubeadm join”. This token is and should be short-lived

ControlPlaneComponent

Appears in:

• ClusterConfiguration

• APIServer

ControlPlaneComponent holds settings common to control plane component of the cluster

DNS

Appears in:

• ClusterConfiguration

DNS defines the DNS addon that should be used in the cluster

DNSAddOnType

(Alias of string)

Appears in:

• DNS

DNSAddOnType defines string identifying DNS add-on types.

Discovery

Appears in:

• JoinConfiguration

Discovery specifies the options for the kubelet to use during the TLS Bootstrap process

Etcd

Appears in:

• ClusterConfiguration

Etcd contains elements describing Etcd configuration.

ExternalEtcd

Appears in:

• Etcd

ExternalEtcd describes an external etcd cluster. Kubeadm has no knowledge of where certificate files live and they must be supplied.

FileDiscovery

Appears in:

• Discovery

FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information

HostPathMount

Appears in:

• ControlPlaneComponent

HostPathMount contains elements describing volumes that are mounted from the host.

ImageMeta

Appears in:

• DNS

• LocalEtcd

ImageMeta allows to customize the image used for components that are not originated from the Kubernetes/Kubernetes release process

JoinControlPlane

Appears in:

• JoinConfiguration

JoinControlPlane contains elements describing an additional control plane instance to be deployed on the joining node.

LocalEtcd

Appears in:

• Etcd

LocalEtcd describes that kubeadm should run an etcd cluster locally.

Networking

Appears in:

• ClusterConfiguration

Networking contains elements describing cluster’s networking configuration

NodeRegistrationOptions

Appears in:

• InitConfiguration

• JoinConfiguration

NodeRegistrationOptions holds fields that relate to registering a new control-plane or node to the cluster, either via “kubeadm init” or “kubeadm join”.