K3s Cluster Configuration Reference

This section covers the configuration options that are available in Rancher for a new or existing K3s Kubernetes cluster.

Overview

You can configure the Kubernetes options one of two ways:

  • Rancher UI: Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
  • Cluster Config File: Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a K3s config file. Using a config file allows you to set any of the options available in an K3s installation.

Editing Clusters in the Rancher UI

The Rancher UI provides two ways to edit a cluster:

  1. With a form.
  2. With YAML.

Editing Clusters with a Form

The form covers the most frequently needed options for clusters.

To edit your cluster,

  1. Click ☰ > Cluster Management.
  2. Go to the cluster you want to configure and click ⋮ > Edit Config.

Editing Clusters in YAML

For a complete reference of configurable options for K3s clusters in YAML, see the K3s documentation.

To edit your cluster with YAML:

  1. Click ☰ > Cluster Management.
  2. Go to the cluster you want to configure and click ⋮ > Edit as YAML.
  3. Edit the RKE options under the rkeConfig directive.

Configuration Options in the Rancher UI

Machine Pool Configuration

This subsection covers generic machine pool configurations. For specific infrastructure provider configurations, refer to the following:

Pool Name

The name of the machine pool.

Machine Count

The number of machines in the pool.

Roles

Option to assign etcd, control plane, and worker roles to nodes.

Advanced

Auto Replace

The amount of time nodes can be unreachable before they are automatically deleted and replaced.

Drain Before Delete

Enables draining nodes by evicting all pods before the node is deleted.

Kubernetes Node Labels

Add labels to nodes to help with organization and object selection.

For details on label syntax requirements, see the Kubernetes documentation.

Taints

Add taints to nodes, to prevent pods from being scheduled to or executed on the nodes, unless the pods have matching tolerations.

Cluster Configuration

Basics

Kubernetes Version

The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on hyperkube.

For more detail, see Upgrading Kubernetes.

Pod Security Admission Configuration Template

The default pod security admission configuration template for the cluster.

Encrypt Secrets

Option to enable or disable secrets encryption. When enabled, secrets will be encrypted using a AES-CBC key. If disabled, any previously secrets will not be readable until encryption is enabled again. Refer to the K3s documentation for details.

Project Network Isolation

If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.

SELinux

Option to enable or disable SELinux support.

CoreDNS

By default, CoreDNS is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the K3s documentation for details..

Klipper Service LB

Option to enable or disable the Klipper service load balancer. Refer to the K3s documentation for details.

Traefik Ingress

Option to enable or disable the Traefik HTTP reverse proxy and load balancer. For more details and configuration options, see the K3s documentation.

Local Storage

Option to enable or disable local storage on the node(s).

Metrics Server

Option to enable or disable the metrics server. If enabled, ensure port 10250 is opened for inbound TCP traffic.

Add-On Config

Additional Kubernetes manifests, managed as a Add-on, to apply to the cluster on startup. Refer to the K3s documentation for details.

Agent Environment Vars

Option to set environment variables for K3s agents. The environment variables can be set using key value pairs. Refer to the K3 documentation for more details.

etcd

Automatic Snapshots

Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the K3s documentation.

Metrics

Option to choose whether to expose etcd metrics to the public or only within the cluster.

Networking

Cluster CIDR

IPv4/IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).

Service CIDR

IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).

Cluster DNS

IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).

Cluster Domain

Select the domain for the cluster. The default is cluster.local.

NodePort Service Port Range

Option to change the range of ports that can be used for NodePort services. The default is 30000-32767.

Truncate Hostnames

Option to truncate hostnames to 15 characters or less. You can only set this field during the initial creation of the cluster. You can’t enable or disable the 15 character limit after cluster creation.

This setting only affects machine-provisioned clusters. Since custom clusters set hostnames during their own node creation process, which occurs outside of Rancher, this field doesn’t restrict custom cluster hostname length.

Truncating hostnames in a cluster improves compatibility with Windows-based systems. Although Kubernetes allows hostnames up to 63 characters in length, systems that use NetBIOS restrict hostnames to 15 characters or less.

TLS Alternate Names

Add hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.

Authorized Cluster Endpoint

Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.

For more detail on how an authorized cluster endpoint works and why it is used, refer to the architecture section.

We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the recommended architecture section.

Registries

Select the image repository to pull Rancher images from. For more details and configuration options, see the K3s documentation.

Upgrade Strategy

Control Plane Concurrency

Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.

Worker Concurrency

Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.

Drain Nodes (Control Plane)

Option to remove all pods from the node prior to upgrading.

Drain Nodes (Worker Nodes)

Option to remove all pods from the node prior to upgrading.

Advanced

Option to set kubelet options for different nodes. For available options, refer to the Kubernetes documentation.

Cluster Config File Reference

Editing clusters in YAML allows you to set configurations that are already listed in Configuration Options in the Rancher UI, as well as set Rancher-specific parameters.

Example Cluster Config File Snippet

  1. apiVersion: provisioning.cattle.io/v1
  2. kind: Cluster
  3. spec:
  4.   cloudCredentialSecretName: cattle-global-data:cc-fllv6
  5.   clusterAgentDeploymentCustomization: {}
  6.   fleetAgentDeploymentCustomization: {}
  7.   kubernetesVersion: v1.26.7+k3s1
  8.   localClusterAuthEndpoint: {}
  9.   rkeConfig:
  10.     additionalManifest: ""
  11.     chartValues: {}
  12.     etcd:
  13.       snapshotRetention: 5
  14.       snapshotScheduleCron: 0 */5 * * *
  15.     machineGlobalConfig:
  16.       disable-apiserver: false
  17.       disable-cloud-controller: false
  18.       disable-controller-manager: false
  19.       disable-etcd: false
  20.       disable-kube-proxy: false
  21.       disable-network-policy: false
  22.       disable-scheduler: false
  23.       etcd-expose-metrics: false
  24.       kube-apiserver-arg:
  25.         - audit-policy-file=/etc/rancher/k3s/user-audit-policy.yaml
  26.         - audit-log-path=/etc/rancher/k3s/user-audit.logs
  27.       profile: null
  28.       secrets-encryption: false
  29.     machinePools:
  30.     - controlPlaneRole: true
  31.       etcdRole: true
  32.       machineConfigRef:
  33.         kind: Amazonec2Config
  34.         name: nc-test-pool1-pwl5h
  35.       name: pool1
  36.       quantity: 1
  37.       unhealthyNodeTimeout: 0s
  38.       workerRole: true
  39.     machineSelectorConfig:
  40.       - config:
  41.           docker: false
  42.           protect-kernel-defaults: false
  43.           selinux: false
  44.     machineSelectorFiles:
  45.       - fileSources:
  46.           - configMap:
  47.               name: ''
  48.             secret:
  49.               name: audit-policy
  50.               items:
  51.                 - key: audit-policy
  52.                   path: /etc/rancher/k3s/user-audit-policy.yaml
  53.         machineLabelSelector:
  54.           matchLabels:
  55.             rke.cattle.io/control-plane-role: 'true'
  56.     registries: {}
  57.     upgradeStrategy:
  58.       controlPlaneConcurrency: '1'
  59.       controlPlaneDrainOptions:
  60.         deleteEmptyDirData: true
  61.         disableEviction: false
  62.         enabled: false
  63.         force: false
  64.         gracePeriod: -1
  65.         ignoreDaemonSets: true
  66.         ignoreErrors: false
  67.         postDrainHooks: null
  68.         preDrainHooks: null
  69.         skipWaitForDeleteTimeoutSeconds: 0
  70.         timeout: 120
  71.       workerConcurrency: '1'
  72.       workerDrainOptions:
  73.         deleteEmptyDirData: true
  74.         disableEviction: false
  75.         enabled: false
  76.         force: false
  77.         gracePeriod: -1
  78.         ignoreDaemonSets: true
  79.         ignoreErrors: false
  80.         postDrainHooks: null
  81.         preDrainHooks: null
  82.         skipWaitForDeleteTimeoutSeconds: 0
  83.         timeout: 120

additionalManifest

Specify additional manifests to deliver to the control plane nodes.

The value is a String, and will be placed at the path /var/lib/rancher/k3s/server/manifests/rancher/addons.yaml on target nodes.

Example:

  1. additionalManifest: |-
  2.   apiVersion: v1
  3.   kind: Namespace
  4.   metadata:
  5.     name: name-xxxx

K3s Cluster Configuration Reference - 图1note

If you want to customize system charts, you should use the chartValues field as described below.

Alternatives, such as using a HelmChartConfig to customize the system charts via additionalManifest, can cause unexpected behavior, due to having multiple HelmChartConfigs for the same chart.

chartValues

Specify the values for the system charts installed by K3s.

For more information about how K3s manges packaged components, please refer to K3s documentation.

Example:

  1. chartValues:
  2.     chart-name:
  3.         key: value

machineGlobalConfig

Specify K3s configurations. Any configuration change made here will apply to every node. The configuration options available in the standalone version of k3s can be applied here.

Example:

  1. machineGlobalConfig:
  2.     etcd-arg:
  3.         - key1=value1
  4.         - key2=value2

To make it easier to put files on nodes beforehand, Rancher expects the following values to be included in the configuration, while K3s expects the values to be entered as file paths:

  • private-registry
  • flannel-conf

Rancher delivers the files to the path /var/lib/rancher/k3s/etc/config-files/<option> in target nodes, and sets the proper options in the K3s server.

Example:

  1. apiVersion: provisioning.cattle.io/v1
  2. kind: Cluster
  3. spec:
  4.   rkeConfig:
  5.     machineGlobalConfig:
  6.       private-registry: |
  7.         mirrors:
  8.           docker.io:
  9.             endpoint:
  10.               - "http://mycustomreg.com:5000"
  11.         configs:
  12.           "mycustomreg:5000":
  13.             auth:
  14.               username: xxxxxx # this is the registry username
  15.               password: xxxxxx # this is the registry password

machineSelectorConfig

machineSelectorConfig is the same as machineGlobalConfig except that a label selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.

Multiple config entries are allowed, each specifying their own machineLabelSelector. A user can specify matchExpressions, matchLabels, both, or neither. Omitting the machineLabelSelector section of this field has the same effect as putting the config in the machineGlobalConfig section.

Example:

  1. machineSelectorConfig
  2.   - config:
  3.       config-key: config-value
  4.     machineLabelSelector:
  5.       matchExpressions:
  6.         - key: example-key
  7.           operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
  8.           values:
  9.             - example-value1
  10.             - example-value2
  11.       matchLabels:
  12.         key1: value1
  13.         key2: value2

machineSelectorFiles

K3s Cluster Configuration Reference - 图2note

This feature is available in Rancher v2.7.2 and later.

Deliver files to nodes, so that the files can be in place before initiating K3s server or agent processes. The content of the file is retrieved from either a secret or a configmap. The target nodes are filtered by the machineLabelSelector.

Example :

  1. machineSelectorFiles:
  2.   - fileSources:
  3.       - secret:
  4.           items:
  5.             - key: example-key
  6.               path: path-to-put-the-file-on-nodes
  7.               permissions: 644 (optional)
  8.               hash: base64-encoded-hash-of-the-content (optional)
  9.           name: example-secret-name
  10.     machineLabelSelector:
  11.       matchExpressions:
  12.         - key: example-key
  13.           operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
  14.           values:
  15.             - example-value1
  16.             - example-value2
  17.       matchLabels:
  18.         key1: value1
  19.         key2: value2
  20.   - fileSources:
  21.       - configMap:
  22.           items:
  23.             - key: example-key
  24.               path: path-to-put-the-file-on-nodes
  25.               permissions: 644 (optional)
  26.               hash: base64-encoded-hash-of-the-content (optional)
  27.           name: example-configmap-name
  28.     machineLabelSelector:
  29.       matchExpressions:
  30.         - key: example-key
  31.           operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
  32.           values:
  33.             - example-value1
  34.             - example-value2
  35.       matchLabels:
  36.         key1: value1
  37.         key2: value2

The secret or configmap must meet the following requirements:

  1. It must be in the fleet-default namespace where the Cluster object exists.
  2. It must have the annotation rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2, which permits the target clusters to use it.

K3s Cluster Configuration Reference - 图3tip

Rancher Dashboard provides an easy-to-use form for creating the secret or configmap.

Example:

  1. apiVersion: v1
  2. data:
  3.   audit-policy: >-
  4.     IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE=
  5. kind: Secret
  6. metadata:
  7.   annotations:
  8.     rke.cattle.io/object-authorized-for-clusters: cluster1
  9.   name: name1
  10.   namespace: fleet-default