RKE 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24

本文档是 RKE 加固指南的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。

本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes：

本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 Not Applicable 的结果。

本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。

有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在互联网安全中心 (CIS)创建免费账户后下载 benchmark。

测试方法

Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。

在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 kubectl（带有有效的配置文件）和 jq 工具，在测试和评估测试结果时这些工具是必需的。

备注

本指南仅涵盖 automated（之前称为 scored）测试。

Controls

1.1 Control Plane Node Configuration Files

1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time.

1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time.

1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time.

1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time.

1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time.

1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time.

1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time.

1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time.

1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual)

Result: warn

Remediation: Run the below command (based on the file location on your system) on the control plane node. For example, chmod 600 <path/to/cni/files>

Audit:

ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a

Expected Result:

permissions has permissions 644, expected 600 or more restrictive

Returned Value:

permissions=600 permissions=644

1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated)

Result: pass

Remediation: Run the below command (based on the file location on your system) on the control plane node. For example, chown root:root <path/to/cni/files>

Audit:

ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G

Expected Result:

'root:root' is present

Returned Value:

root:root root:root

1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated)

Result: pass

Remediation: On the etcd server node, get the etcd data directory, passed as an argument —data-dir, from the command ‘ps -ef | grep etcd’. Run the below command (based on the etcd data directory found above). For example, chmod 700 /var/lib/etcd

Audit:

stat -c %a /node/var/lib/etcd

Expected Result:

'700' is equal to '700'

Returned Value:

700

1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)

Result: Not Applicable

Remediation: On the etcd server node, get the etcd data directory, passed as an argument —data-dir, from the command ‘ps -ef | grep etcd’. Run the below command (based on the etcd data directory found above). For example, chown etcd:etcd /var/lib/etcd

1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.

1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.

1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time.

1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time.

1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time.

1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated)

Result: Not Applicable

Remediation: Cluster provisioned by RKE doesn’t require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time.

1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated)

Result: pass

Remediation: Run the below command (based on the file location on your system) on the master node. For example, chown -R root:root /etc/kubernetes/pki/

Audit Script: check_files_owner_in_dir.sh

#!/usr/bin/env bash
# This script is used to ensure the owner is set to root:root for
# the given directory and all the files in it
#
# inputs:
#   $1 = /full/path/to/directory
#
# outputs:
#   true/false
INPUT_DIR=$1
if [[ "${INPUT_DIR}" == "" ]]; then
    echo "false"
    exit
fi
if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then
    echo "false"
    exit
fi
statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*)
while read -r statInfoLine; do
  f=$(echo ${statInfoLine} | cut -d' ' -f1)
  p=$(echo ${statInfoLine} | cut -d' ' -f2)
  if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then
    if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then
      echo "false"
      exit
    fi
  else
    if [[ "$p" != "root:root" ]]; then
      echo "false"
      exit
    fi
  fi
done <<< "${statInfoLines}"
echo "true"
exit

Audit Execution:

./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl

Expected Result:

'true' is equal to 'true'

Returned Value:

true

1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual)

Result: warn

Remediation: Run the below command (based on the file location on your system) on the control plane node. For example, find /node/etc/kubernetes/ssl/ -name ‘_.pem’ ! -name ‘_key.pem’ -exec chmod -R 600 {} +

Audit:

find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a

Expected Result:

permissions has permissions 644, expected 600 or more restrictive

Returned Value:

permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600

1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated)

Result: pass

Remediation: Run the below command (based on the file location on your system) on the control plane node. For example, chmod -R 600 /etc/kubernetes/ssl/*key.pem

Audit Script: check_files_permissions.sh

#!/usr/bin/env bash
# This script is used to ensure the file permissions are set to 644 or
# more restrictive for all files in a given directory or a wildcard
# selection of files
#
# inputs:
#   $1 = /full/path/to/directory or /path/to/fileswithpattern
#                                   ex: !(*key).pem
#
#   $2 (optional) = permission (ex: 600)
#
# outputs:
#   true/false
# Turn on "extended glob" for use of '!' in wildcard
shopt -s extglob
# Turn off history to avoid surprises when using '!'
set -H
USER_INPUT=$1
if [[ "${USER_INPUT}" == "" ]]; then
  echo "false"
  exit
fi
if [[ -d ${USER_INPUT} ]]; then
  PATTERN="${USER_INPUT}/*"
else
  PATTERN="${USER_INPUT}"
fi
PERMISSION=""
if [[ "$2" != "" ]]; then
  PERMISSION=$2
fi
FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN})
while read -r fileInfo; do
  p=$(echo ${fileInfo} | cut -d' ' -f2)
  if [[ "${PERMISSION}" != "" ]]; then
    if [[ "$p" != "${PERMISSION}" ]]; then
      echo "false"
      exit
    fi
  else
    if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then
      echo "false"
      exit
    fi
  fi
done <<< "${FILES_PERMISSIONS}"
echo "true"
exit

Audit Execution:

./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem'

Expected Result:

'true' is equal to 'true'

Returned Value:

true

1.2 API Server

1.2.1 Ensure that the —anonymous-auth argument is set to false (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the below parameter. —anonymous-auth=false

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--anonymous-auth' is equal to 'false'

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.2 Ensure that the —token-auth-file parameter is not set (Automated)

Result: pass

Remediation: Follow the documentation and configure alternate mechanisms for authentication. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and remove the —token-auth-file=<filename> parameter.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--token-auth-file' is not present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.3 Ensure that the —DenyServiceExternalIPs is not set (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and remove the DenyServiceExternalIPs from enabled admission plugins.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.4 Ensure that the —kubelet-https argument is set to true (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and remove the —kubelet-https parameter.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--kubelet-https' is present OR '--kubelet-https' is not present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.5 Ensure that the —kubelet-client-certificate and —kubelet-client-key arguments are set as appropriate (Automated)

Result: pass

Remediation: Follow the Kubernetes documentation and set up the TLS connection between the apiserver and kubelets. Then, edit API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the kubelet client certificate and key parameters as below. —kubelet-client-certificate=<path/to/client-certificate-file> —kubelet-client-key=<path/to/client-key-file>

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.6 Ensure that the —kubelet-certificate-authority argument is set as appropriate (Automated)

Result: pass

Remediation: Follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelets. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. —kubelet-certificate-authority=<ca-string>

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--kubelet-certificate-authority' is present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.7 Ensure that the —authorization-mode argument is not set to AlwaysAllow (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —authorization-mode parameter to values other than AlwaysAllow. One such example could be as below. —authorization-mode=RBAC

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--authorization-mode' does not have 'AlwaysAllow'

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.8 Ensure that the —authorization-mode argument includes Node (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —authorization-mode parameter to a value that includes Node. —authorization-mode=Node,RBAC

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--authorization-mode' has 'Node'

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.9 Ensure that the —authorization-mode argument includes RBAC (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —authorization-mode parameter to a value that includes RBAC, for example --authorization-mode=Node,RBAC.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--authorization-mode' has 'RBAC'

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.10 Ensure that the admission control plugin EventRateLimit is set (Automated)

Result: pass

Remediation: Follow the Kubernetes documentation and set the desired limits in a configuration file. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml and set the below parameters. —enable-admission-plugins=…,EventRateLimit,… —admission-control-config-file=<path/to/configuration/file>

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--enable-admission-plugins' has 'EventRateLimit'

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and either remove the —enable-admission-plugins parameter, or set it to a value that does not include AlwaysAdmit.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual)

Result: warn

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —enable-admission-plugins parameter to include AlwaysPullImages. —enable-admission-plugins=…,AlwaysPullImages,…

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--enable-admission-plugins' has 'AlwaysPullImages'

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual)

Result: warn

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —enable-admission-plugins parameter to include SecurityContextDeny, unless PodSecurityPolicy is already in place. —enable-admission-plugins=…,SecurityContextDeny,…

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated)

Result: pass

Remediation: Follow the documentation and create ServiceAccount objects as per your environment. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and ensure that the —disable-admission-plugins parameter is set to a value that does not include ServiceAccount.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s

1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated)

Result: pass

Remediation: Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the —disable-admission-plugins parameter to ensure it does not include NamespaceLifecycle.

Audit:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Expected Result:

'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present

Returned Value:

root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s