CIS Hardening Guide

This document provides prescriptive guidance for hardening a production installation of K3s. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Internet Security (CIS).

K3s has a number of security mitigations applied and turned on by default and will pass a number of the Kubernetes CIS controls without modification. There are some notable exceptions to this that require manual intervention to fully comply with the CIS Benchmark:

  1. K3s will not modify the host operating system. Any host-level modifications will need to be done manually.
  2. Certain CIS policy controls for NetworkPolicies and PodSecurityStandards (PodSecurityPolicies on v1.24 and older) will restrict the functionality of the cluster. You must opt into having K3s configure these by adding the appropriate options (enabling of admission plugins) to your command-line flags or configuration file as well as manually applying appropriate policies. Further details are presented in the sections below.

The first section (1.1) of the CIS Benchmark concerns itself primarily with pod manifest permissions and ownership. K3s doesn’t utilize these for the core components since everything is packaged into a single binary.

Host-level Requirements

There are two areas of host-level requirements: kernel parameters and etcd process/directory configuration. These are outlined in this section.

Ensure protect-kernel-defaults is set

This is a kubelet flag that will cause the kubelet to exit if the required kernel parameters are unset or are set to values that are different from the kubelet’s defaults.

Note: protect-kernel-defaults is exposed as a top-level flag for K3s.

Set kernel parameters

Create a file called /etc/sysctl.d/90-kubelet.conf and add the snippet below. Then run sysctl -p /etc/sysctl.d/90-kubelet.conf.

  1. vm.panic_on_oom=0
  2. vm.overcommit_memory=1
  3. kernel.panic=10
  4. kernel.panic_on_oops=1

Kubernetes Runtime Requirements

The runtime requirements to comply with the CIS Benchmark are centered around pod security (via PSP or PSA), network policies and API Server auditing logs. These are outlined in this section.

By default, K3s does not include any pod security or network policies. However, K3s ships with a controller that will enforce network policies, if any are created. K3s doesn’t enable auditing by default, so audit log configuration and audit policy must be created manually. By default, K3s runs with the both the PodSecurity and NodeRestriction admission controllers enabled, among others.

Pod Security

  • v1.25 and Newer
  • v1.24 and Older

K3s v1.25 and newer support Pod Security Admissions (PSAs) for controlling pod security. PSAs are enabled by passing the following flag to the K3s server:

  1. --kube-apiserver-arg="admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml"

The policy should be written to a file named psa.yaml in /var/lib/rancher/k3s/server directory.

Here is an example of a compliant PSA:

  1. apiVersion: apiserver.config.k8s.io/v1
  2. kind: AdmissionConfiguration
  3. plugins:
  4. - name: PodSecurity
  5. configuration:
  6. apiVersion: pod-security.admission.config.k8s.io/v1beta1
  7. kind: PodSecurityConfiguration
  8. defaults:
  9. enforce: "restricted"
  10. enforce-version: "latest"
  11. audit: "restricted"
  12. audit-version: "latest"
  13. warn: "restricted"
  14. warn-version: "latest"
  15. exemptions:
  16. usernames: []
  17. runtimeClasses: []
  18. namespaces: [kube-system, cis-operator-system]

K3s v1.24 and older support Pod Security Policies (PSPs) for controlling pod security. PSPs are enabled by passing the following flag to the K3s server:

  1. --kube-apiserver-arg="enable-admission-plugins=NodeRestriction,PodSecurityPolicy"

This will have the effect of maintaining the NodeRestriction plugin as well as enabling the PodSecurityPolicy.

When PSPs are enabled, a policy can be applied to satisfy the necessary controls described in section 5.2 of the CIS Benchmark.

Here is an example of a compliant PSP:

  1. apiVersion: policy/v1beta1
  2. kind: PodSecurityPolicy
  3. metadata:
  4. name: restricted-psp
  5. spec:
  6. privileged: false # CIS - 5.2.1
  7. allowPrivilegeEscalation: false # CIS - 5.2.5
  8. requiredDropCapabilities: # CIS - 5.2.7/8/9
  9. - ALL
  10. volumes:
  11. - 'configMap'
  12. - 'emptyDir'
  13. - 'projected'
  14. - 'secret'
  15. - 'downwardAPI'
  16. - 'csi'
  17. - 'persistentVolumeClaim'
  18. - 'ephemeral'
  19. hostNetwork: false # CIS - 5.2.4
  20. hostIPC: false # CIS - 5.2.3
  21. hostPID: false # CIS - 5.2.2
  22. runAsUser:
  23. rule: 'MustRunAsNonRoot' # CIS - 5.2.6
  24. seLinux:
  25. rule: 'RunAsAny'
  26. supplementalGroups:
  27. rule: 'MustRunAs'
  28. ranges:
  29. - min: 1
  30. max: 65535
  31. fsGroup:
  32. rule: 'MustRunAs'
  33. ranges:
  34. - min: 1
  35. max: 65535
  36. readOnlyRootFilesystem: false

For the above PSP to be effective, we need to create a ClusterRole and a ClusterRoleBinding. We also need to include a “system unrestricted policy” which is needed for system-level pods that require additional privileges, and an additional policy that allows sysctls necessary for servicelb to function properly.

Combining the configuration above with the Network Policy described in the next section, a single file can be placed in the /var/lib/rancher/k3s/server/manifests directory. Here is an example of a policy.yaml file:

  1. apiVersion: policy/v1beta1
  2. kind: PodSecurityPolicy
  3. metadata:
  4. name: restricted-psp
  5. spec:
  6. privileged: false
  7. allowPrivilegeEscalation: false
  8. requiredDropCapabilities:
  9. - ALL
  10. volumes:
  11. - 'configMap'
  12. - 'emptyDir'
  13. - 'projected'
  14. - 'secret'
  15. - 'downwardAPI'
  16. - 'csi'
  17. - 'persistentVolumeClaim'
  18. - 'ephemeral'
  19. hostNetwork: false
  20. hostIPC: false
  21. hostPID: false
  22. runAsUser:
  23. rule: 'MustRunAsNonRoot'
  24. seLinux:
  25. rule: 'RunAsAny'
  26. supplementalGroups:
  27. rule: 'MustRunAs'
  28. ranges:
  29. - min: 1
  30. max: 65535
  31. fsGroup:
  32. rule: 'MustRunAs'
  33. ranges:
  34. - min: 1
  35. max: 65535
  36. readOnlyRootFilesystem: false
  37. ---
  38. apiVersion: policy/v1beta1
  39. kind: PodSecurityPolicy
  40. metadata:
  41. name: system-unrestricted-psp
  42. annotations:
  43. seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  44. spec:
  45. allowPrivilegeEscalation: true
  46. allowedCapabilities:
  47. - '*'
  48. fsGroup:
  49. rule: RunAsAny
  50. hostIPC: true
  51. hostNetwork: true
  52. hostPID: true
  53. hostPorts:
  54. - max: 65535
  55. min: 0
  56. privileged: true
  57. runAsUser:
  58. rule: RunAsAny
  59. seLinux:
  60. rule: RunAsAny
  61. supplementalGroups:
  62. rule: RunAsAny
  63. volumes:
  64. - '*'
  65. ---
  66. apiVersion: policy/v1beta1
  67. kind: PodSecurityPolicy
  68. metadata:
  69. name: svclb-psp
  70. annotations:
  71. seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  72. spec:
  73. allowPrivilegeEscalation: false
  74. allowedCapabilities:
  75. - NET_ADMIN
  76. allowedUnsafeSysctls:
  77. - net.ipv4.ip_forward
  78. - net.ipv6.conf.all.forwarding
  79. fsGroup:
  80. rule: RunAsAny
  81. hostPorts:
  82. - max: 65535
  83. min: 0
  84. runAsUser:
  85. rule: RunAsAny
  86. seLinux:
  87. rule: RunAsAny
  88. supplementalGroups:
  89. rule: RunAsAny
  90. ---
  91. apiVersion: rbac.authorization.k8s.io/v1
  92. kind: ClusterRole
  93. metadata:
  94. name: psp:restricted-psp
  95. rules:
  96. - apiGroups:
  97. - policy
  98. resources:
  99. - podsecuritypolicies
  100. verbs:
  101. - use
  102. resourceNames:
  103. - restricted-psp
  104. ---
  105. apiVersion: rbac.authorization.k8s.io/v1
  106. kind: ClusterRole
  107. metadata:
  108. name: psp:system-unrestricted-psp
  109. rules:
  110. - apiGroups:
  111. - policy
  112. resources:
  113. - podsecuritypolicies
  114. resourceNames:
  115. - system-unrestricted-psp
  116. verbs:
  117. - use
  118. ---
  119. apiVersion: rbac.authorization.k8s.io/v1
  120. kind: ClusterRole
  121. metadata:
  122. name: psp:svclb-psp
  123. rules:
  124. - apiGroups:
  125. - policy
  126. resources:
  127. - podsecuritypolicies
  128. resourceNames:
  129. - svclb-psp
  130. verbs:
  131. - use
  132. ---
  133. apiVersion: rbac.authorization.k8s.io/v1
  134. kind: ClusterRoleBinding
  135. metadata:
  136. name: default:restricted-psp
  137. roleRef:
  138. apiGroup: rbac.authorization.k8s.io
  139. kind: ClusterRole
  140. name: psp:restricted-psp
  141. subjects:
  142. - kind: Group
  143. name: system:authenticated
  144. apiGroup: rbac.authorization.k8s.io
  145. ---
  146. apiVersion: rbac.authorization.k8s.io/v1
  147. kind: ClusterRoleBinding
  148. metadata:
  149. name: system-unrestricted-node-psp-rolebinding
  150. roleRef:
  151. apiGroup: rbac.authorization.k8s.io
  152. kind: ClusterRole
  153. name: psp:system-unrestricted-psp
  154. subjects:
  155. - apiGroup: rbac.authorization.k8s.io
  156. kind: Group
  157. name: system:nodes
  158. ---
  159. apiVersion: rbac.authorization.k8s.io/v1
  160. kind: RoleBinding
  161. metadata:
  162. name: system-unrestricted-svc-acct-psp-rolebinding
  163. namespace: kube-system
  164. roleRef:
  165. apiGroup: rbac.authorization.k8s.io
  166. kind: ClusterRole
  167. name: psp:system-unrestricted-psp
  168. subjects:
  169. - apiGroup: rbac.authorization.k8s.io
  170. kind: Group
  171. name: system:serviceaccounts
  172. ---
  173. apiVersion: rbac.authorization.k8s.io/v1
  174. kind: RoleBinding
  175. metadata:
  176. name: svclb-psp-rolebinding
  177. namespace: kube-system
  178. roleRef:
  179. apiGroup: rbac.authorization.k8s.io
  180. kind: ClusterRole
  181. name: psp:svclb-psp
  182. subjects:
  183. - kind: ServiceAccount
  184. name: svclb
  185. ---
  186. kind: NetworkPolicy
  187. apiVersion: networking.k8s.io/v1
  188. metadata:
  189. name: intra-namespace
  190. namespace: kube-system
  191. spec:
  192. podSelector: {}
  193. ingress:
  194. - from:
  195. - namespaceSelector:
  196. matchLabels:
  197. name: kube-system
  198. ---
  199. kind: NetworkPolicy
  200. apiVersion: networking.k8s.io/v1
  201. metadata:
  202. name: intra-namespace
  203. namespace: default
  204. spec:
  205. podSelector: {}
  206. ingress:
  207. - from:
  208. - namespaceSelector:
  209. matchLabels:
  210. name: default
  211. ---
  212. kind: NetworkPolicy
  213. apiVersion: networking.k8s.io/v1
  214. metadata:
  215. name: intra-namespace
  216. namespace: kube-public
  217. spec:
  218. podSelector: {}
  219. ingress:
  220. - from:
  221. - namespaceSelector:
  222. matchLabels:
  223. name: kube-public

Note: The Kubernetes critical additions such as CNI, DNS, and Ingress are run as pods in the kube-system namespace. Therefore, this namespace will have a policy that is less restrictive so that these components can run properly.

NetworkPolicies

CIS requires that all namespaces have a network policy applied that reasonably limits traffic into namespaces and pods.

Network policies should be placed the /var/lib/rancher/k3s/server/manifests directory, where they will automatically be deployed on startup.

Here is an example of a compliant network policy.

  1. kind: NetworkPolicy
  2. apiVersion: networking.k8s.io/v1
  3. metadata:
  4. name: intra-namespace
  5. namespace: kube-system
  6. spec:
  7. podSelector: {}
  8. ingress:
  9. - from:
  10. - namespaceSelector:
  11. matchLabels:
  12. kubernetes.io/metadata.name: kube-system

With the applied restrictions, DNS will be blocked unless purposely allowed. Below is a network policy that will allow for traffic to exist for DNS.

  1. apiVersion: networking.k8s.io/v1
  2. kind: NetworkPolicy
  3. metadata:
  4. name: default-network-dns-policy
  5. namespace: <NAMESPACE>
  6. spec:
  7. ingress:
  8. - ports:
  9. - port: 53
  10. protocol: TCP
  11. - port: 53
  12. protocol: UDP
  13. podSelector:
  14. matchLabels:
  15. k8s-app: kube-dns
  16. policyTypes:
  17. - Ingress

The metrics-server and Traefik ingress controller will be blocked by default if network policies are not created to allow access. Traefik v1 as packaged in K3s version 1.20 and below uses different labels than Traefik v2. Ensure that you only use the sample yaml below that is associated with the version of Traefik present on your cluster.

  • v1.21 and Newer
  • v1.20 and Older
  1. apiVersion: networking.k8s.io/v1
  2. kind: NetworkPolicy
  3. metadata:
  4. name: allow-all-metrics-server
  5. namespace: kube-system
  6. spec:
  7. podSelector:
  8. matchLabels:
  9. k8s-app: metrics-server
  10. ingress:
  11. - {}
  12. policyTypes:
  13. - Ingress
  14. ---
  15. apiVersion: networking.k8s.io/v1
  16. kind: NetworkPolicy
  17. metadata:
  18. name: allow-all-svclbtraefik-ingress
  19. namespace: kube-system
  20. spec:
  21. podSelector:
  22. matchLabels:
  23. svccontroller.k3s.cattle.io/svcname: traefik
  24. ingress:
  25. - {}
  26. policyTypes:
  27. - Ingress
  28. ---
  29. apiVersion: networking.k8s.io/v1
  30. kind: NetworkPolicy
  31. metadata:
  32. name: allow-all-traefik-v121-ingress
  33. namespace: kube-system
  34. spec:
  35. podSelector:
  36. matchLabels:
  37. app.kubernetes.io/name: traefik
  38. ingress:
  39. - {}
  40. policyTypes:
  41. - Ingress
  42. ---
  1. apiVersion: networking.k8s.io/v1
  2. kind: NetworkPolicy
  3. metadata:
  4. name: allow-all-metrics-server
  5. namespace: kube-system
  6. spec:
  7. podSelector:
  8. matchLabels:
  9. k8s-app: metrics-server
  10. ingress:
  11. - {}
  12. policyTypes:
  13. - Ingress
  14. ---
  15. apiVersion: networking.k8s.io/v1
  16. kind: NetworkPolicy
  17. metadata:
  18. name: allow-all-svclbtraefik-ingress
  19. namespace: kube-system
  20. spec:
  21. podSelector:
  22. matchLabels:
  23. svccontroller.k3s.cattle.io/svcname: traefik
  24. ingress:
  25. - {}
  26. policyTypes:
  27. - Ingress
  28. ---
  29. apiVersion: networking.k8s.io/v1
  30. kind: NetworkPolicy
  31. metadata:
  32. name: allow-all-traefik-v120-ingress
  33. namespace: kube-system
  34. spec:
  35. podSelector:
  36. matchLabels:
  37. app: traefik
  38. ingress:
  39. - {}
  40. policyTypes:
  41. - Ingress
  42. ---

CIS Hardening Guide - 图1info

Operators must manage network policies as normal for additional namespaces that are created.

API Server audit configuration

CIS requirements 1.2.22 to 1.2.25 are related to configuring audit logs for the API Server. K3s doesn’t create by default the log directory and audit policy, as auditing requirements are specific to each user’s policies and environment.

The log directory, ideally, must be created before starting K3s. A restrictive access permission is recommended to avoid leaking potential sensitive information.

  1. sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs

A starter audit policy to log request metadata is provided below. The policy should be written to a file named audit.yaml in /var/lib/rancher/k3s/server directory. Detailed information about policy configuration for the API server can be found in the Kubernetes documentation.

  1. apiVersion: audit.k8s.io/v1
  2. kind: Policy
  3. rules:
  4. - level: Metadata

Both configurations must be passed as arguments to the API Server as:

  • config
  • cmdline
  1. kube-apiserver-arg:
  2. - 'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml'
  3. - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  4. - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'
  5. - 'audit-log-maxage=30'
  6. - 'audit-log-maxbackup=10'
  7. - 'audit-log-maxsize=100'
  1. --kube-apiserver-arg='audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  2. --kube-apiserver-arg='audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'

K3s must be restarted to load the new configuration.

  1. sudo systemctl daemon-reload
  2. sudo systemctl restart k3s.service

Configuration for Kubernetes Components

The configuration below should be placed in the configuration file, and contains all the necessary remediations to harden the Kubernetes components.

  • v1.25 and Newer
  • v1.24 and Older
  1. protect-kernel-defaults: true
  2. secrets-encryption: true
  3. kube-apiserver-arg:
  4. - "enable-admission-plugins=NodeRestriction,EventRateLimit"
  5. - 'admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml'
  6. - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  7. - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'
  8. - 'audit-log-maxage=30'
  9. - 'audit-log-maxbackup=10'
  10. - 'audit-log-maxsize=100'
  11. kube-controller-manager-arg:
  12. - 'terminated-pod-gc-threshold=10'
  13. kubelet-arg:
  14. - 'streaming-connection-idle-timeout=5m'
  15. - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"
  1. protect-kernel-defaults: true
  2. secrets-encryption: true
  3. kube-apiserver-arg:
  4. - 'enable-admission-plugins=NodeRestriction,PodSecurityPolicy,NamespaceLifecycle,ServiceAccount'
  5. - 'audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log'
  6. - 'audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml'
  7. - 'audit-log-maxage=30'
  8. - 'audit-log-maxbackup=10'
  9. - 'audit-log-maxsize=100'
  10. kube-controller-manager-arg:
  11. - 'terminated-pod-gc-threshold=10'
  12. kubelet-arg:
  13. - 'streaming-connection-idle-timeout=5m'
  14. - 'make-iptables-util-chains=true'
  15. - "tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"

Manual Operations

The following are controls that K3s currently does not pass by with the above configuration applied. These controls require manual intervention to fully comply with the CIS Benchmark.

Control 1.1.20

Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual)

Details

Remediation K3s PKI certificate files are stored in /var/lib/rancher/k3s/server/tls/ with permission 644. To remediate, run the following command:

  1. chmod -R 600 /var/lib/rancher/k3s/server/tls/*.crt

Control 1.2.9

Ensure that the admission control plugin EventRateLimit is set

Details

Remediation Follow the Kubernetes documentation and set the desired limits in a configuration file. For this and other psa configuration, this documentation uses /var/lib/rancher/k3s/server/psa.yaml. Then, edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameters.

  1. kube-apiserver-arg:
  2. - "enable-admission-plugins=NodeRestriction,EventRateLimit"
  3. - "admission-control-config-file=/var/lib/rancher/k3s/server/psa.yaml"

Control 1.2.11

Ensure that the admission control plugin AlwaysPullImages is set

Details

Remediation Permissive, per CIS guidelines, “This setting could impact offline or isolated clusters, which have images pre-loaded and do not have access to a registry to pull in-use images. This setting is not appropriate for clusters which use this configuration.” Edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameter.

  1. kube-apiserver-arg:
  2. - "enable-admission-plugins=...,AlwaysPullImages,..."

Control 1.2.21

Ensure that the —request-timeout argument is set as appropriate

Details

Remediation Permissive, per CIS guidelines, “it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed”. Edit the K3s config file /etc/rancher/k3s/config.yaml and set the below parameter if needed. For example,

  1. kube-apiserver-arg:
  2. - "request-timeout=300s"

Control 4.2.13

Ensure that a limit is set on pod PIDs

Details

Remediation Decide on an appropriate level for this parameter and set it, If using a K3s config file /etc/rancher/k3s/config.yaml, edit the file to set podPidsLimit to

  1. kubelet-arg:
  2. - "pod-max-pids=<value>"

Control 5.X

All the 5.X Controls are related to Kubernetes policy configuration. These controls are not enforced by K3s by default.

Refer to CIS 1.8 Section 5 for more information on how to create and apply these policies.

Conclusion

If you have followed this guide, your K3s cluster will be configured to comply with the CIS Kubernetes Benchmark. You can review the CIS 1.8 Self-Assessment Guide to understand the expectations of each of the benchmark’s checks and how you can do the same on your cluster.