RKE2 Hardening Guide

This document provides prescriptive guidance for how to harden an RKE2 cluster intended for production, before provisioning it with Rancher. It outlines the configurations and controls required for Center for Information Security (CIS) Kubernetes benchmark controls.

RKE2 Hardening Guides - 图1note

This hardening guide describes how to secure the nodes in your cluster. We recommended that you follow this guide before you install Kubernetes.

This hardening guide is intended to be used for RKE2 clusters and is associated with the following versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:

Rancher VersionCIS Benchmark VersionKubernetes Version
Rancher v2.7Benchmark v1.23Kubernetes v1.23 up to v1.25

RKE2 Hardening Guides - 图2note

At the time of writing, the upstream CIS Kubernetes v1.25 benchmark is not yet available in Rancher. At this time Rancher is using the CIS v1.23 benchmark when scanning Kubernetes v1.25 clusters. Due to that, the CIS checks 5.2.2, 5.2.3, 5.2.5, 5.2.6, 5.2.7 and 5.2.8 might fail.

For more details on how to evaluate a hardened RKE2 cluster against the official CIS benchmark, refer to the RKE2 self-assessment guides for specific Kubernetes and CIS benchmark versions.

RKE2 passes a number of the Kubernetes CIS controls without modification, as it applies several security mitigations by default. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark:

  1. RKE2 will not modify the host operating system. Therefore, you, the operator, must make a few host-level modifications.
  2. Certain CIS controls for Network Policies and Pod Security Standards (or Pod Security Policies (PSP) on RKE2 versions prior to v1.25) will restrict the functionality of the cluster. You must opt into having RKE2 configure these for you. To help ensure these requirements are met, RKE2 can be started with the profile flag set to cis-1.23 for v1.25 and newer or cis-1.6 for v1.24 and older.

Host-level requirements

There are two areas of host-level requirements: kernel parameters and etcd process/directory configuration. These are outlined in this section.

Set kernel parameters

The following sysctl configuration is recommended for all nodes type in the cluster. Set the following parameters in /etc/sysctl.d/90-kubelet.conf:

  1. vm.panic_on_oom=0
  2. vm.overcommit_memory=1
  3. kernel.panic=10
  4. kernel.panic_on_oops=1

Run sudo sysctl -p /etc/sysctl.d/90-kubelet.conf to enable the settings.

Ensure etcd is configured properly

The CIS Benchmark requires that the etcd data directory be owned by the etcd user and group. This implicitly requires the etcd process run as the host-level etcd user. To achieve this, RKE2 takes several steps when started with a valid cis-1.xx profile:

  1. Check that the etcd user and group exists on the host. If they don’t, exit with an error.
  2. Create etcd’s data directory with etcd as the user and group owner.
  3. Ensure the etcd process is ran as the etcd user and group by setting the etcd static pod’s SecurityContext appropriately.

To meet the above requirements, you must:

Create the etcd user

On some Linux distributions, the useradd command will not create a group. The -U flag is included below to account for that. This flag tells useradd to create a group with the same name as the user.

  1. sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U

Kubernetes runtime requirements

The runtime requirements to pass the CIS Benchmark are centered around pod security, network policies and kernel parameters. Most of this is automatically handled by RKE2 when using a valid cis-1.xx profile, but some additional operator intervention is required. These are outlined in this section.

Ensure protect-kernel-defaults is set

This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet’s defaults.

Both protect-kernel-defaults and profile flags can be set in the RKE2 template configuration file. When the profile flag is set, RKE2 will set the flag to true if it is unset.

  1. spec:
  2.   rkeConfig:
  3.     machineSelectorConfig:
  4.       - config:
  5.           profile: # use cis-1.23 or cis-1.6
  6.           protect-kernel-defaults: true

PodSecurity

RKE2 always runs with some amount of pod security.

  • v1.25 and Newer
  • v1.24 and Older

On v1.25 and newer, Pod Security Admissions (PSAs) are used for pod security.

Below is the minimum necessary configuration needed for hardening RKE2 to pass CIS v1.23 hardened profile rke2-cis-1.23-hardened available in Rancher.

  1. spec:
  2.   defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted
  3.   rkeConfig:
  4.     machineSelectorConfig:
  5.       - config:
  6.           profile: cis-1.23

When both the defaultPodSecurityAdmissionConfigurationTemplateName and profile flags are set, Rancher and RKE2 does the following:

  1. Checks that host-level requirements have been met. If they haven’t, RKE2 will exit with a fatal error describing the unmet requirements.
  2. Applies network policies that allow the cluster to pass associated controls.
  3. Configures the Pod Security Admission Controller with the PSA configuration template rancher-restricted, to enforce restricted mode in all namespaces, except the ones in the template’s exemption list. These namespaces are exempted to allow system pods to run without restrictions, which is required for proper operation of the cluster.

RKE2 Hardening Guides - 图3note

If you intend to import an RKE cluster into Rancher, please consult the documentation for how to configure the PSA to exempt Rancher system namespaces.

On Kubernetes v1.24 and older, the PodSecurityPolicy admission controller is always enabled.

Below is the minimum necessary configuration needed for hardening RKE2 to pass CIS v1.23 hardened profile rke2-cis-1.23-hardened available in Rancher.

RKE2 Hardening Guides - 图4note

In the following example the profile is set to cis-1.6 which is the value defined in the upstream RKE2, but the cluster is actually configured to pass the CIS v1.23 hardened profile

  1. spec:
  2.   defaultPodSecurityPolicyTemplateName: restricted-noroot
  3.   rkeConfig:
  4.     machineSelectorConfig:
  5.       - config:
  6.           profile: cis-1.6

When both the defaultPodSecurityPolicyTemplateName and profile flags are set, Rancher and RKE2 does the following:

  1. Checks that host-level requirements have been met. If they haven’t, RKE2 will exit with a fatal error describing the unmet requirements.
  2. Applies network policies that allow the cluster to pass associated controls.
  3. Configures runtime pod security policies that allow the cluster to pass associated controls.

RKE2 Hardening Guides - 图5note

The Kubernetes control plane components and critical additions such as CNI, DNS, and Ingress are ran as pods in the kube-system namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.

NetworkPolicies

When ran with a valid cis-1.xx profile, RKE2 will put NetworkPolicies in place that passes the CIS Benchmark for Kubernetes’ built-in namespaces. These namespaces are: kube-system, kube-public, kube-node-lease, and default.

The NetworkPolicy used will only allow pods within the same namespace to talk to each other. The notable exception to this is that it allows DNS requests to be resolved.

RKE2 Hardening Guides - 图6note

Operators must manage network policies as normal for additional namespaces that are created.

Configure default service account

Set automountServiceAccountToken to false for default service accounts

Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.

For each namespace including default and kube-system on a standard RKE2 install, the default service account must include this value:

  1. automountServiceAccountToken: false

For namespaces created by the cluster operator, the following script and configuration file can be used to configure the default service account.

The configuration bellow must be saved to a file called account_update.yaml.

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: default
  5. automountServiceAccountToken: false

Create a bash script file called account_update.sh. Be sure to sudo chmod +x account_update.sh so the script has execute permissions.

  1. #!/bin/bash -e
  3. for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do
  4.   echo -n "Patching namespace $namespace - "
  5.   kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
  6. done

Execute this script to apply the account_update.yaml configuration to default service account in all namespaces.

API Server audit configuration

CIS requirements 1.2.19 to 1.2.22 are related to configuring audit logs for the API Server. When RKE2 is started with the profile flag set, it will automatically configure hardened --audit-log- parameters in the API Server to pass those CIS checks.

RKE2’s default audit policy is configured to not log requests in the API Server. This is done to allow cluster operators flexibility to customize an audit policy that suits their auditing requirements and needs, as these are specific to each users’ environment and policies.

A default audit policy is created by RKE2 when started with the profile flag set. The policy is defined in /etc/rancher/rke2/audit-policy.yaml.

  1. apiVersion: audit.k8s.io/v1
  2. kind: Policy
  3. metadata:
  4.   creationTimestamp: null
  5. rules:
  6. - level: None

Reference Hardened RKE2 Template Configuration

The reference template configuration is used in Rancher to create a hardened RKE2 custom cluster. This reference does not include other required cluster configuration directives which will vary depending on your environment.

  • v1.25 and Newer
  • v1.24 and Older
  1. apiVersion: provisioning.cattle.io/v1
  2. kind: Cluster
  3. metadata:
  4.   name: # Define cluster name
  5. spec:
  6.   defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted
  7.   kubernetesVersion: # Define RKE2 version
  8.   rkeConfig:
  9.     machineSelectorConfig:
  10.       - config:
  11.           profile: cis-1.23
  12.           protect-kernel-defaults: true
  1. apiVersion: provisioning.cattle.io/v1
  2. kind: Cluster
  3. metadata:
  4.   name: # Define cluster name
  5. spec:
  6.   defaultPodSecurityPolicyTemplateName: restricted-noroot
  7.   kubernetesVersion: # Define RKE2 version
  8.   rkeConfig:
  9.     machineSelectorConfig:
  10.       - config:
  11.           profile: cis-1.6
  12.           protect-kernel-defaults: true

Conclusion

If you have followed this guide, your RKE2 custom cluster provisioned by Rancher will be configured to pass the CIS Kubernetes Benchmark. You can review our RKE2 self-assessment guides to understand how we verified each of the benchmarks and how you can do the same on your cluster.