Advanced Security Profiles Operator tasks

Use advanced tasks to enable metrics, configure webhooks, or restrict syscalls.

Restrict the allowed syscalls in seccomp profiles

The Security Profiles Operator does not restrict syscalls in seccomp profiles by default. You can define the list of allowed syscalls in the spod configuration.

Procedure

  • To define the list of allowedSyscalls, adjust the spec parameter by running the following command:

    1. $ oc -n openshift-security-profiles patch spod spod --type merge \
    2.     -p '{"spec":{"allowedSyscalls": ["exit", "exit_group", "futex", "nanosleep"]}}'

The Operator will install only the seccomp profiles, which have a subset of syscalls defined into the allowed list. All profiles not complying with this ruleset are rejected.

When the list of allowed syscalls is modified in the spod configuration, the Operator will identify the already installed profiles which are non-compliant and remove them automatically.

Base syscalls for a container runtime

You can use the baseProfileName attribute to establish the minimum required syscalls for a given runtime to start a container.

Procedure

  • Edit the SeccompProfile kind object and add baseProfileName: runc-v1.0.0 to the spec field:

    1. apiVersion: security-profiles-operator.x-k8s.io/v1beta1
    2. kind: SeccompProfile
    3. metadata:
    4.   namespace: my-namespace
    5.   name: example-name
    6. spec:
    7.   defaultAction: SCMP_ACT_ERRNO
    8.   baseProfileName: runc-v1.0.0
    9.   syscalls:
    10.     - action: SCMP_ACT_ALLOW
    11.       names:
    12.         - exit_group

Enabling memory optimization in the spod daemon

The controller running inside of spod daemon process watches all pods available in the cluster when profile recording is enabled. This can lead to very high memory usage in large clusters, resulting in the spod daemon running out of memory or crashing.

To prevent crashes, the spod daemon can be configured to only load the pods labeled for profile recording into the cache memory.

+

SPO memory optimization is not enabled by default.

Procedure

  1. Enable memory optimization by running the following command:

    1. $ oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"enableMemoryOptimization":true}}'

  2. To record a security profile for a pod, the pod must be labeled with spo.x-k8s.io/enable-recording: "true":

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: my-recording-pod
    5.   labels:
    6.     spo.x-k8s.io/enable-recording: "true"

Customizing daemon resource requirements

The default resource requirements of the daemon container can be adjusted by using the field daemonResourceRequirements from the spod configuration.

Procedure

  • To specify the memory and cpu requests and limits of the daemon container, run the following command:

    1. $ oc -n openshift-security-profiles patch spod spod --type merge -p \
    2.     '{"spec":{"daemonResourceRequirements": { \
    3.     "requests": {"memory": "256Mi", "cpu": "250m"}, \
    4.     "limits": {"memory": "512Mi", "cpu": "500m"}}}}'

Setting a custom priority class name for the spod daemon pod

The default priority class name of the spod daemon pod is set to system-node-critical. A custom priority class name can be configured in the spod configuration by setting a value in the priorityClassName field.

Procedure

  • Configure the priority class name by running the following command:

    1. $ oc -n openshift-security-profiles patch spod spod --type=merge -p '{"spec":{"priorityClassName":"my-priority-class"}}'

    Example output

    1. securityprofilesoperatordaemon.openshift-security-profiles.x-k8s.io/spod patched

Using metrics

The openshift-security-profiles namespace provides metrics endpoints, which are secured by the kube-rbac-proxy container. All metrics are exposed by the metrics service within the openshift-security-profiles namespace.

The Security Profiles Operator includes a cluster role and corresponding binding spo-metrics-client to retrieve the metrics from within the cluster. There are two metrics paths available:

  • metrics.openshift-security-profiles/metrics: for controller runtime metrics

  • metrics.openshift-security-profiles/metrics-spod: for the Operator daemon metrics

Procedure

  1. To view the status of the metrics service, run the following command:

    1. $ oc get svc/metrics -n openshift-security-profiles

    Example output

    1. NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    2. metrics   ClusterIP   10.0.0.228   <none>        443/TCP   43s

  2. To retrieve the metrics, query the service endpoint using the default ServiceAccount token in the openshift-security-profiles namespace by running the following command:

    1. $ oc run --rm -i --restart=Never --image=registry.fedoraproject.org/fedora-minimal:latest \
    2.     -n openshift-security-profiles metrics-test -- bash -c \
    3.     'curl -ks -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-security-profiles/metrics-spod'

    Example output

    1. # HELP security_profiles_operator_seccomp_profile_total Counter about seccomp profile operations.
    2. # TYPE security_profiles_operator_seccomp_profile_total counter
    3. security_profiles_operator_seccomp_profile_total{operation="delete"} 1
    4. security_profiles_operator_seccomp_profile_total{operation="update"} 2

  3. To retrieve metrics from a different namespace, link the ServiceAccount to the spo-metrics-client ClusterRoleBinding by running the following command:

    1. $ oc get clusterrolebinding spo-metrics-client -o wide

    Example output

    1. NAME                 ROLE                             AGE   USERS   GROUPS   SERVICEACCOUNTS
    2. spo-metrics-client   ClusterRole/spo-metrics-client   35m                    openshift-security-profiles/default

controller-runtime metrics

The controller-runtime metrics and the DaemonSet endpoint metrics-spod provide a set of default metrics. Additional metrics are provided by the daemon, which are always prefixed with security_profiles_operator_.

Table 1. Available controller-runtime metrics
Metric keyPossible labelsTypePurpose

seccomp_profile_total

operation={delete,update}

Counter

Amount of seccomp profile operations.

seccomp_profile_audit_total

node, namespace, pod, container, executable, syscall

Counter

Amount of seccomp profile audit operations. Requires the log enricher to be enabled.

seccomp_profile_bpf_total

node, mount_namespace, profile

Counter

Amount of seccomp profile bpf operations. Requires the bpf recorder to be enabled.

seccomp_profile_error_total

reason={
SeccompNotSupportedOnNode,
InvalidSeccompProfile,
CannotSaveSeccompProfile,
CannotRemoveSeccompProfile,
CannotUpdateSeccompProfile,
CannotUpdateNodeStatus
}

Counter

Amount of seccomp profile errors.

selinux_profile_total

operation={delete,update}

Counter

Amount of SELinux profile operations.

selinux_profile_audit_total

node, namespace, pod, container, executable, scontext,tcontext

Counter

Amount of SELinux profile audit operations. Requires the log enricher to be enabled.

selinux_profile_error_total

reason={
CannotSaveSelinuxPolicy,
CannotUpdatePolicyStatus,
CannotRemoveSelinuxPolicy,
CannotContactSelinuxd,
CannotWritePolicyFile,
CannotGetPolicyStatus
}

Counter

Amount of SELinux profile errors.

Using the log enricher

The Security Profiles Operator contains a log enrichment feature, which is disabled by default. The log enricher container runs with privileged permissions to read the audit logs from the local node. The log enricher runs within the host PID namespace, hostPID.

The log enricher must have permissions to read the host processes.

Procedure

  1. Patch the spod configuration to enable the log enricher by running the following command:

    1. $ oc -n openshift-security-profiles patch spod spod \
    2.     --type=merge -p '{"spec":{"enableLogEnricher":true}}'

    Example output

    1. securityprofilesoperatordaemon.security-profiles-operator.x-k8s.io/spod patched

    The Security Profiles Operator will re-deploy the spod daemon set automatically.

  2. View the audit logs by running the following command:

    1. $ oc -n openshift-security-profiles logs -f ds/spod log-enricher

    Example output

    1. I0623 12:51:04.257814 1854764 deleg.go:130] setup "msg"="starting component: log-enricher"  "buildDate"="1980-01-01T00:00:00Z" "compiler"="gc" "gitCommit"="unknown" "gitTreeState"="clean" "goVersion"="go1.16.2" "platform"="linux/amd64" "version"="0.4.0-dev"
    2. I0623 12:51:04.257890 1854764 enricher.go:44] log-enricher "msg"="Starting log-enricher on node: 127.0.0.1"
    3. I0623 12:51:04.257898 1854764 enricher.go:46] log-enricher "msg"="Connecting to local GRPC server"
    4. I0623 12:51:04.258061 1854764 enricher.go:69] log-enricher "msg"="Reading from file /var/log/audit/audit.log"
    5. 2021/06/23 12:51:04 Seeked /var/log/audit/audit.log - &{Offset:0 Whence:2}

Using the log enricher to trace an application

You can use the Security Profiles Operator log enricher to trace an application.

Procedure

  1. To trace an application, create a SeccompProfile logging profile:

    1. apiVersion: security-profiles-operator.x-k8s.io/v1beta1
    2. kind: SeccompProfile
    3. metadata:
    4.   name: log
    5.   namespace: default
    6. spec:
    7.   defaultAction: SCMP_ACT_LOG

  2. Create a pod object to use the profile:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: log-pod
    5. spec:
    6.   securityContext:
    7.     seccompProfile:
    8.       type: Localhost
    9.       localhostProfile: operator/default/log.json
    10.   containers:
    11.     - name: log-container
    12.       image: quay.io/security-profiles-operator/test-nginx-unprivileged:1.21

  3. Examine the log enricher output by running the following command:

    1. $ oc -n openshift-security-profiles logs -f ds/spod log-enricher

    Example output

    1. …
    2. I0623 12:59:11.479869 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=3 "syscallName"="close" "timestamp"="1624453150.205:1061" "type"="seccomp"
    3. I0623 12:59:11.487323 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=157 "syscallName"="prctl" "timestamp"="1624453150.205:1062" "type"="seccomp"
    4. I0623 12:59:11.492157 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=157 "syscallName"="prctl" "timestamp"="1624453150.205:1063" "type"="seccomp"
    5. …
    6. I0623 12:59:20.258523 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=12 "syscallName"="brk" "timestamp"="1624453150.235:2873" "type"="seccomp"
    7. I0623 12:59:20.263349 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=21 "syscallName"="access" "timestamp"="1624453150.235:2874" "type"="seccomp"
    8. I0623 12:59:20.354091 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=257 "syscallName"="openat" "timestamp"="1624453150.235:2875" "type"="seccomp"
    9. I0623 12:59:20.358844 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=5 "syscallName"="fstat" "timestamp"="1624453150.235:2876" "type"="seccomp"
    10. I0623 12:59:20.363510 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=9 "syscallName"="mmap" "timestamp"="1624453150.235:2877" "type"="seccomp"
    11. I0623 12:59:20.454127 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=3 "syscallName"="close" "timestamp"="1624453150.235:2878" "type"="seccomp"
    12. I0623 12:59:20.458654 1854764 enricher.go:111] log-enricher "msg"="audit"  "container"="log-container" "executable"="/usr/sbin/nginx" "namespace"="default" "node"="127.0.0.1" "pid"=1905792 "pod"="log-pod" "syscallID"=257 "syscallName"="openat" "timestamp"="1624453150.235:2879" "type"="seccomp"
    13. …

Configuring webhooks

Profile binding and profile recording objects can use webhooks. Profile binding and recording object configurations are MutatingWebhookConfiguration CRs, managed by the Security Profiles Operator.

To change the webhook configuration, the spod CR exposes a webhookOptions field that allows modification of the failurePolicy, namespaceSelector, and objectSelector variables. This allows you to set the webhooks to “soft-fail” or restrict them to a subset of a namespaces so that even if the webhooks failed, other namespaces or resources are not affected.

Procedure

  1. Set the recording.spo.io webhook configuration to record only pods labeled with spo-record=true by creating the following patch file:

    1. spec:
    2.   webhookOptions:
    3.     - name: recording.spo.io
    4.       objectSelector:
    5.         matchExpressions:
    6.           - key: spo-record
    7.             operator: In
    8.             values:
    9.               - "true"

  2. Patch the spod/spod instance by running the following command:

    1. $ oc -n openshift-security-profiles patch spod \
    2.     spod -p $(cat /tmp/spod-wh.patch) --type=merge

  3. To view the resulting MutatingWebhookConfiguration object, run the following command:

    1. $ oc get MutatingWebhookConfiguration \
    2.     spo-mutating-webhook-configuration -oyaml