Using sysctls in containers

Sysctl settings are exposed through Kubernetes, allowing users to modify certain kernel parameters at runtime. Only sysctls that are namespaced can be set independently on pods. If a sysctl is not namespaced, called node-level, you must use another method of setting the sysctl, such as by using the Node Tuning Operator.

Network sysctls are a special category of sysctl. Network sysctls include:

• System-wide sysctls, for example net.ipv4.ip_local_port_range, that are valid for all networking. You can set these independently for each pod on a node.

• Interface-specific sysctls, for example net.ipv4.conf.IFNAME.accept_local, that only apply to a specific additional network interface for a given pod. You can set these independently for each additional network configuration. You set these by using a configuration in the tuning-cni after the network interfaces are created.

Moreover, only those sysctls considered safe are whitelisted by default; you can manually enable other unsafe sysctls on the node to be available to the user.

Additional resources

• Node Tuning Operator

About sysctls

In Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available from the _/proc/sys/_ virtual process file system. The parameters cover various subsystems, such as:

• kernel (common prefix: _kernel._)

• networking (common prefix: _net._)

• virtual memory (common prefix: _vm._)

• MDADM (common prefix: _dev._)

More subsystems are described in Kernel documentation. To get a list of all parameters, run:

$ sudo sysctl -a

Namespaced and node-level sysctls

A number of sysctls are namespaced in the Linux kernels. This means that you can set them independently for each pod on a node. Being namespaced is a requirement for sysctls to be accessible in a pod context within Kubernetes.

The following sysctls are known to be namespaced:

• _kernel.shm*_

• _kernel.msg*_

• _kernel.sem_

• _fs.mqueue.*_

Additionally, most of the sysctls in the net.* group are known to be namespaced. Their namespace adoption differs based on the kernel version and distributor.

Sysctls that are not namespaced are called node-level and must be set manually by the cluster administrator, either by means of the underlying Linux distribution of the nodes, such as by modifying the _/etc/sysctls.conf_ file, or by using a daemon set with privileged containers. You can use the Node Tuning Operator to set node-level sysctls.

Safe and unsafe sysctls

Sysctls are grouped into safe and unsafe sysctls.

For system-wide sysctls to be considered safe, they must be namespaced. A namespaced sysctl ensures there is isolation between namespaces and therefore pods. If you set a sysctl for one pod it must not add any of the following:

• Influence any other pod on the node

• Harm the node health

• Gain CPU or memory resources outside of the resource limits of a pod

Any sysctl that is not added to the allowed list on OKD is considered unsafe for OKD.

Unsafe sysctls are not allowed by default. For system-wide sysctls the cluster administrator must manually enable them on a per-node basis. Pods with disabled unsafe sysctls are scheduled but do not launch.

OKD adds the following system-wide and interface-specific safe sysctls to an allowed safe list:

Updating the interface-specific safe sysctls list

OKD includes a predefined list of safe interface-specific sysctls. You can modify this list by updating the cni-sysctl-allowlist in the openshift-multus namespace.

Follow this procedure to modify the predefined list of safe sysctls. This procedure describes how to extend the default allow list.

Procedure

1. View the existing predefined list by running the following command:

   $ oc get cm -n openshift-multus cni-sysctl-allowlist -oyaml

   Expected output

   apiVersion: v1
   data:
     allowlist.conf: |-
       ^net.ipv4.conf.IFNAME.accept_redirects$
       ^net.ipv4.conf.IFNAME.accept_source_route$
       ^net.ipv4.conf.IFNAME.arp_accept$
       ^net.ipv4.conf.IFNAME.arp_notify$
       ^net.ipv4.conf.IFNAME.disable_policy$
       ^net.ipv4.conf.IFNAME.secure_redirects$
       ^net.ipv4.conf.IFNAME.send_redirects$
       ^net.ipv6.conf.IFNAME.accept_ra$
       ^net.ipv6.conf.IFNAME.accept_redirects$
       ^net.ipv6.conf.IFNAME.accept_source_route$
       ^net.ipv6.conf.IFNAME.arp_accept$
       ^net.ipv6.conf.IFNAME.arp_notify$
       ^net.ipv6.neigh.IFNAME.base_reachable_time_ms$
       ^net.ipv6.neigh.IFNAME.retrans_time_ms$
   kind: ConfigMap
   metadata:
     annotations:
       kubernetes.io/description: |
         Sysctl allowlist for nodes.
       release.openshift.io/version: 4.14.0-0.nightly-2022-11-16-003434
     creationTimestamp: "2022-11-17T14:09:27Z"
     name: cni-sysctl-allowlist
     namespace: openshift-multus
     resourceVersion: "2422"
     uid: 96d138a3-160e-4943-90ff-6108fa7c50c3

2. Edit the list by using the following command:

   $ oc edit cm -n openshift-multus cni-sysctl-allowlist -oyaml

   For example, to allow you to be able to implement stricter reverse path forwarding you need to add ^net.ipv4.conf.IFNAME.rp_filter$ and ^net.ipv6.conf.IFNAME.rp_filter$ to the list as shown here:

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: v1
   data:
     allowlist.conf: |-
       ^net.ipv4.conf.IFNAME.accept_redirects$
       ^net.ipv4.conf.IFNAME.accept_source_route$
       ^net.ipv4.conf.IFNAME.arp_accept$
       ^net.ipv4.conf.IFNAME.arp_notify$
       ^net.ipv4.conf.IFNAME.disable_policy$
       ^net.ipv4.conf.IFNAME.secure_redirects$
       ^net.ipv4.conf.IFNAME.send_redirects$
       ^net.ipv4.conf.IFNAME.rp_filter$
       ^net.ipv6.conf.IFNAME.accept_ra$
       ^net.ipv6.conf.IFNAME.accept_redirects$
       ^net.ipv6.conf.IFNAME.accept_source_route$
       ^net.ipv6.conf.IFNAME.arp_accept$
       ^net.ipv6.conf.IFNAME.arp_notify$
       ^net.ipv6.neigh.IFNAME.base_reachable_time_ms$
       ^net.ipv6.neigh.IFNAME.retrans_time_ms$
       ^net.ipv6.conf.IFNAME.rp_filter$

3. Save the changes to the file and exit.

   The removal of sysctls is also supported. Edit the file, remove the sysctl or sysctls then save the changes and exit.

Verification

Follow this procedure to enforce stricter reverse path forwarding for IPv4. For more information on reverse path forwarding see Reverse Path Forwarding.

1. Create a network attachment definition, such as reverse-path-fwd-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: tuningnad
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "tuningnad",
       "plugins": [{
         "type": "bridge"
         },
         {
         "type": "tuning",
         "sysctl": {
            "net.ipv4.conf.IFNAME.rp_filter": "1"
           }
       }
     ]
   }'

2. Apply the yaml by running the following command:

   $ oc apply -f reverse-path-fwd-example.yaml

   Example output

   networkattachmentdefinition.k8.cni.cncf.io/tuningnad created

3. Create a pod such as examplepod.yaml using the following YAML:

   apiVersion: v1
   kind: Pod
   metadata:
     name: example
     labels:
       app: httpd
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: tuningnad  (1)
   spec:
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
     containers:
       - name: httpd
         image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest'
         ports:
           - containerPort: 8080
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
             drop:
               - ALL

   1	Specify the name of the configured NetworkAttachmentDefinition.

4. Apply the yaml by running the following command:

   $ oc apply -f examplepod.yaml

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME      READY   STATUS    RESTARTS   AGE
   example   1/1     Running   0          47s

6. Log in to the pod by running the following command:

   $ oc rsh example

7. Verify the value of the configured sysctl flag. For example, find the value net.ipv4.conf.net1.rp_filter by running the following command:

   sh-4.4# sysctl net.ipv4.conf.net1.rp_filter

   Expected output

   net.ipv4.conf.net1.rp_filter = 1

Additional resources

• Configuring the tuning CNI

• Linux networking documentation

Starting a pod with safe sysctls

You can set sysctls on pods using the pod’s securityContext. The securityContext applies to all containers in the same pod.

Safe sysctls are allowed by default.

This example uses the pod securityContext to set the following safe sysctls:

• kernel.shm_rmid_forced

• net.ipv4.ip_local_port_range

• net.ipv4.tcp_syncookies

• net.ipv4.ping_group_range

Use this procedure to start a pod with the configured sysctl settings.

Procedure

1. Create a YAML file sysctl_pod.yaml that defines an example pod and add the securityContext spec, as shown in the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sysctl-example
     namespace: default
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 (1)
         runAsGroup: 3000 (2)
         allowPrivilegeEscalation: false (3)
         capabilities: (4)
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true (5)
       seccompProfile: (6)
         type: RuntimeDefault
       sysctls:
       - name: kernel.shm_rmid_forced
         value: "1"
       - name: net.ipv4.ip_local_port_range
         value: "32770       60666"
       - name: net.ipv4.tcp_syncookies
         value: "0"
       - name: net.ipv4.ping_group_range
         value: "0           200000000"

   1	runAsUser controls which user ID the container is run with.
   2	runAsGroup controls which primary group ID the containers is run with.
   3	allowPrivilegeEscalation determines if a pod can request to allow privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.
   4	capabilities permit privileged actions without giving full root access. This policy ensures all capabilities are dropped from the pod.
   5	runAsNonRoot: true requires that the container will run with a user with any UID other than 0.
   6	RuntimeDefault enables the default seccomp profile for a pod or container workload.

2. Create the pod by running the following command:

   $ oc apply -f sysctl_pod.yaml

3. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME              READY   STATUS            RESTARTS   AGE
   sysctl-example    1/1     Running           0          14s

4. Log in to the pod by running the following command:

   $ oc rsh sysctl-example

5. Verify the values of the configured sysctl flags. For example, find the value kernel.shm_rmid_forced by running the following command:

   sh-4.4# sysctl kernel.shm_rmid_forced

   Expected output

   kernel.shm_rmid_forced = 1

Starting a pod with unsafe sysctls

A pod with unsafe sysctls fails to launch on any node unless the cluster administrator explicitly enables unsafe sysctls for that node. As with node-level sysctls, use the taints and toleration feature or labels on nodes to schedule those pods onto the right nodes.

The following example uses the pod securityContext to set a safe sysctl kernel.shm_rmid_forced and two unsafe sysctls, net.core.somaxconn and kernel.msgmax. There is no distinction between safe and unsafe sysctls in the specification.

The following example illustrates what happens when you add safe and unsafe sysctls to a pod specification:

Procedure

1. Create a YAML file sysctl-example-unsafe.yaml that defines an example pod and add the securityContext specification, as shown in the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sysctl-example-unsafe
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000
         runAsGroup: 3000
         allowPrivilegeEscalation: false
         capabilities:
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
       sysctls:
       - name: kernel.shm_rmid_forced
         value: "0"
       - name: net.core.somaxconn
         value: "1024"
       - name: kernel.msgmax
         value: "65536"

2. Create the pod using the following command:

   $ oc apply -f sysctl-example-unsafe.yaml

3. Verify that the pod is scheduled but does not deploy because unsafe sysctls are not allowed for the node using the following command:

   $ oc get pod

   Example output

   NAME                       READY             STATUS            RESTARTS   AGE
   sysctl-example-unsafe      0/1               SysctlForbidden   0          14s

Enabling unsafe sysctls

A cluster administrator can allow certain unsafe sysctls for very special situations such as high performance or real-time application tuning.

If you want to use unsafe sysctls, a cluster administrator must enable them individually for a specific type of node. The sysctls must be namespaced.

You can further control which sysctls are set in pods by specifying lists of sysctls or sysctl patterns in the allowedUnsafeSysctls field of the Security Context Constraints.

• The allowedUnsafeSysctls option controls specific needs such as high performance or real-time application tuning.

Procedure

1. List existing MachineConfig objects for your OKD cluster to decide how to label your machine config by running the following command:

   $ oc get machineconfigpool

   Example output

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-bfb92f0cd1684e54d8e234ab7423cc96   True      False      False      3              3                   3                     0                      42m
   worker   rendered-worker-21b6cb9a0f8919c88caf39db80ac1fce   True      False      False      3              3                   3                     0                      42m

2. Add a label to the machine config pool where the containers with the unsafe sysctls will run by running the following command:

   $ oc label machineconfigpool worker custom-kubelet=sysctl

3. Create a YAML file set-sysctl-worker.yaml that defines a KubeletConfig custom resource (CR):

   apiVersion: machineconfiguration.openshift.io/v1
   kind: KubeletConfig
   metadata:
     name: custom-kubelet
   spec:
     machineConfigPoolSelector:
       matchLabels:
         custom-kubelet: sysctl (1)
     kubeletConfig:
       allowedUnsafeSysctls: (2)
         - "kernel.msg*"
         - "net.core.somaxconn"

   1	Specify the label from the machine config pool.
   2	List the unsafe sysctls you want to allow.

4. Create the object by running the following command:

   $ oc apply -f set-sysctl-worker.yaml

5. Wait for the Machine Config Operator to generate the new rendered configuration and apply it to the machines by running the following command:

   $ oc get machineconfigpool worker -w

   After some minutes the UPDATING status changes from True to False:

   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   worker   rendered-worker-f1704a00fc6f30d3a7de9a15fd68a800   False     True       False      3              2                   2                     0                      71m
   worker   rendered-worker-f1704a00fc6f30d3a7de9a15fd68a800   False     True       False      3              2                   3                     0                      72m
   worker   rendered-worker-0188658afe1f3a183ec8c4f14186f4d5   True      False      False      3              3                   3                     0                      72m

6. Create a YAML file sysctl-example-safe-unsafe.yaml that defines an example pod and add the securityContext spec, as shown in the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: sysctl-example-safe-unsafe
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000
         runAsGroup: 3000
         allowPrivilegeEscalation: false
         capabilities:
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true
       seccompProfile:
         type: RuntimeDefault
       sysctls:
       - name: kernel.shm_rmid_forced
         value: "0"
       - name: net.core.somaxconn
         value: "1024"
       - name: kernel.msgmax
         value: "65536"

7. Create the pod by running the following command:

   $ oc apply -f sysctl-example-safe-unsafe.yaml

   Expected output

   Warning: would violate PodSecurity "restricted:latest": forbidden sysctls (net.core.somaxconn, kernel.msgmax)
   pod/sysctl-example-safe-unsafe created

8. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME                         READY   STATUS    RESTARTS   AGE
   sysctl-example-safe-unsafe   1/1     Running   0          19s

9. Log in to the pod by running the following command:

   $ oc rsh sysctl-example-safe-unsafe

10. Verify the values of the configured sysctl flags. For example, find the value net.core.somaxconn by running the following command:

    sh-4.4# sysctl net.core.somaxconn

    Expected output

    net.core.somaxconn = 1024

The unsafe sysctl is now allowed and the value is set as defined in the securityContext spec of the updated pod specification.

Additional resources

• Setting interface-level network sysctls