Understanding the Custom Resource Definitions

The Compliance Operator in the OKD provides you with several Custom Resource Definitions (CRDs) to accomplish the compliance scans. To run a compliance scan, it leverages the predefined security policies, which are derived from the ComplianceAsCode community project. The Compliance Operator converts these security policies into CRDs, which you can use to run compliance scans and get remediations for the issues found.

CRDs workflow

The CRD provides you the following workflow to complete the compliance scans:

  1. Define your compliance scan requirements

  2. Configure the compliance scan settings

  3. Process compliance requirements with compliance scans settings

  4. Monitor the compliance scans

  5. Check the compliance scan results

Defining the compliance scan requirements

By default, the Compliance Operator CRDs include ProfileBundle and Profile objects, in which you can define and set the rules for your compliance scan requirements. You can also customize the default profiles by using a TailoredProfile object.

ProfileBundle object

When you install the Compliance Operator, it includes ready-to-run ProfileBundle objects. The Compliance Operator parses the ProfileBundle object and creates a Profile object for each profile in the bundle. It also parses Rule and Variable objects, which are used by the Profile object.

Example ProfileBundle object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: ProfileBundle
  3.   name: <profile bundle name>
  4.   namespace: openshift-compliance
  5. status:
  6.   dataStreamStatus: VALID (1)
1Indicates whether the Compliance Operator was able to parse the content files.

When the contentFile fails, an errorMessage attribute appears, which provides details of the error that occurred.

Troubleshooting

When you roll back to a known content image from an invalid image, the ProfileBundle object stops responding and displays PENDING state. As a workaround, you can move to a different image than the previous one. Alternatively, you can delete and re-create the ProfileBundle object to return to the working state.

Profile object

The Profile object defines the rules and variables that can be evaluated for a certain compliance standard. It contains parsed out details about an OpenSCAP profile, such as its XCCDF identifier and profile checks for a Node or Platform type. You can either directly use the Profile object or further customize it using a TailorProfile object.

You cannot create or modify the Profile object manually because it is derived from a single ProfileBundle object. Typically, a single ProfileBundle object can include several Profile objects.

Example Profile object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. description: <description of the profile>
  3. id: xccdf_org.ssgproject.content_profile_moderate (1)
  4. kind: Profile
  5. metadata:
  6.   annotations:
  7.     compliance.openshift.io/product: <product name>
  8.     compliance.openshift.io/product-type: Node (2)
  9.   creationTimestamp: "YYYY-MM-DDTMM:HH:SSZ"
  10.   generation: 1
  11.   labels:
  12.     compliance.openshift.io/profile-bundle: <profile bundle name>
  13.   name: rhcos4-moderate
  14.   namespace: openshift-compliance
  15.   ownerReferences:
  16.   - apiVersion: compliance.openshift.io/v1alpha1
  17.     blockOwnerDeletion: true
  18.     controller: true
  19.     kind: ProfileBundle
  20.     name: <profile bundle name>
  21.     uid: <uid string>
  22.   resourceVersion: "<version number>"
  23.   selfLink: /apis/compliance.openshift.io/v1alpha1/namespaces/openshift-compliance/profiles/rhcos4-moderate
  24.   uid: <uid string>
  25. rules: (3)
  26. - rhcos4-account-disable-post-pw-expiration
  27. - rhcos4-accounts-no-uid-except-zero
  28. - rhcos4-audit-rules-dac-modification-chmod
  29. - rhcos4-audit-rules-dac-modification-chown
  30. title: <title of the profile>
1Specify the XCCDF name of the profile. Use this identifier when you define a ComplianceScan object as the value of the profile attribute of the scan.
2Specify either a Node or Platform. Node profiles scan the cluster nodes and platform profiles scan the Kubernetes platform.
3Specify the list of rules for the profile. Each rule corresponds to a single check.

Rule object

The Rule object, which forms the profiles, are also exposed as objects. Use the Rule object to define your compliance check requirements and specify how it could be fixed.

Example Rule object

  1.     apiVersion: compliance.openshift.io/v1alpha1
  2.     checkType: Platform (1)
  3.     description: <description of the rule>
  4.     id: xccdf_org.ssgproject.content_rule_configure_network_policies_namespaces (2)
  5.     instructions: <manual instructions for the scan>
  6.     kind: Rule
  7.     metadata:
  8.       annotations:
  9.         compliance.openshift.io/rule: configure-network-policies-namespaces
  10.         control.compliance.openshift.io/CIS-OCP: 5.3.2
  11.         control.compliance.openshift.io/NERC-CIP: CIP-003-3 R4;CIP-003-3 R4.2;CIP-003-3
  12.           R5;CIP-003-3 R6;CIP-004-3 R2.2.4;CIP-004-3 R3;CIP-007-3 R2;CIP-007-3 R2.1;CIP-007-3
  13.           R2.2;CIP-007-3 R2.3;CIP-007-3 R5.1;CIP-007-3 R6.1
  14.         control.compliance.openshift.io/NIST-800-53: AC-4;AC-4(21);CA-3(5);CM-6;CM-6(1);CM-7;CM-7(1);SC-7;SC-7(3);SC-7(5);SC-7(8);SC-7(12);SC-7(13);SC-7(18)
  15.       labels:
  16.         compliance.openshift.io/profile-bundle: ocp4
  17.       name: ocp4-configure-network-policies-namespaces
  18.       namespace: openshift-compliance
  19.     rationale: <description of why this rule is checked>
  20.     severity: high (3)
  21.     title: <summary of the rule>
1Specify the type of check this rule executes. Node profiles scan the cluster nodes and Platform profiles scan the Kubernetes platform. An empty value indicates there is no automated check.
2Specify the XCCDF name of the rule, which is parsed directly from the datastream.
3Specify the severity of the rule when it fails.

The Rule object gets an appropriate label for an easy identification of the associated ProfileBundle object. The ProfileBundle also gets specified in the OwnerReferences of this object.

TailoredProfile object

Use the TailoredProfile object to modify the default Profile object based on your organization requirements. You can enable or disable rules, set variable values, and provide justification for the customization. After validation, the TailoredProfile object creates a ConfigMap, which can be referenced by a ComplianceScan object.

You can use the TailoredProfile object by referencing it in a ScanSettingBinding object. For more information about ScanSettingBinding, see ScanSettingBinding object.

Example TailoredProfile object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: TailoredProfile
  3. metadata:
  4.   name: rhcos4-with-usb
  5. spec:
  6.   extends: rhcos4-moderate (1)
  7.   title: <title of the tailored profile>
  8.   disableRules:
  9.     - name: <name of a rule object to be disabled>
  10.       rationale: <description of why this rule is checked>
  11. status:
  12.   id: xccdf_compliance.openshift.io_profile_rhcos4-with-usb (2)
  13.   outputRef:
  14.     name: rhcos4-with-usb-tp (3)
  15.     namespace: openshift-compliance
  16.   state: READY (4)
1This is optional. Name of the Profile object upon which the TailoredProfile is built. If no value is set, a new profile is created from the enableRules list.
2Specifies the XCCDF name of the tailored profile.
3Specifies the ConfigMap name, which can be used as the value of the tailoringConfigMap.name attribute of a ComplianceScan.
4Shows the state of the object such as READY, PENDING, and FAILURE. If the state of the object is ERROR, then the attribute status.errorMessage provides the reason for the failure.

With the TailoredProfile object, it is possible to create a new Profile object using the TailoredProfile construct. To create a new Profile, set the following configuration parameters :

  • an appropriate title

  • extends value must be empty

  • scan type annotation on the TailoredProfile object:

    1. compliance.openshift.io/product-type: Platform/Node

    If you have not set the product-type annotation, the Compliance Operator defaults to Platform scan type. Adding the -node suffix to the name of the TailoredProfile object results in node scan type.

Configuring the compliance scan settings

After you have defined the requirements of the compliance scan, you can configure it by specifying the type of the scan, occurrence of the scan, and location of the scan. To do so, Compliance Operator provides you with a ScanSetting object.

ScanSetting object

Use the ScanSetting object to define and reuse the operational policies to run your scans. By default, the Compliance Operator creates the following ScanSetting objects:

  • default - it runs a scan every day at 1 AM on both master and worker nodes using a 1Gi Persistent Volume (PV) and keeps the last three results. Remediation is neither applied nor updated automatically.

  • default-auto-apply - it runs a scan every day at 1AM on both control plane and worker nodes using a 1Gi Persistent Volume (PV) and keeps the last three results. Both autoApplyRemediations and autoUpdateRemediations are set to true.

Example ScanSetting object

  1. Name:                      default-auto-apply
  2. Namespace:                 openshift-compliance
  3. Labels:                    <none>
  4. Annotations:               <none>
  5. API Version:               compliance.openshift.io/v1alpha1
  6. Auto Apply Remediations:   true
  7. Auto Update Remediations:  true
  8. Kind:                      ScanSetting
  9. Metadata:
  10.   Creation Timestamp:  2022-10-18T20:21:00Z
  11.   Generation:          1
  12.   Managed Fields:
  13.     API Version:  compliance.openshift.io/v1alpha1
  14.     Fields Type:  FieldsV1
  15.     fieldsV1:
  16.       f:autoApplyRemediations: (1)
  17.       f:autoUpdateRemediations: (2)
  18.       f:rawResultStorage:
  19.         .:
  20.         f:nodeSelector:
  21.           .:
  22.           f:node-role.kubernetes.io/master:
  23.         f:pvAccessModes:
  24.         f:rotation:
  25.         f:size:
  26.         f:tolerations:
  27.       f:roles:
  28.       f:scanTolerations:
  29.       f:schedule:
  30.       f:showNotApplicable:
  31.       f:strictNodeScan:
  32.     Manager:         compliance-operator
  33.     Operation:       Update
  34.     Time:            2022-10-18T20:21:00Z
  35.   Resource Version:  38840
  36.   UID:               8cb0967d-05e0-4d7a-ac1c-08a7f7e89e84
  37. Raw Result Storage:
  38.   Node Selector:
  39.     node-role.kubernetes.io/master:
  40.   Pv Access Modes:
  41.     ReadWriteOnce
  42.   Rotation:  3 (3)
  43.   Size:      1Gi (4)
  44.   Tolerations:
  45.     Effect:              NoSchedule
  46.     Key:                 node-role.kubernetes.io/master
  47.     Operator:            Exists
  48.     Effect:              NoExecute
  49.     Key:                 node.kubernetes.io/not-ready
  50.     Operator:            Exists
  51.     Toleration Seconds:  300
  52.     Effect:              NoExecute
  53.     Key:                 node.kubernetes.io/unreachable
  54.     Operator:            Exists
  55.     Toleration Seconds:  300
  56.     Effect:              NoSchedule
  57.     Key:                 node.kubernetes.io/memory-pressure
  58.     Operator:            Exists
  59. Roles: (6)
  60.   master
  61.   worker
  62. Scan Tolerations:
  63.   Operator:           Exists
  64. Schedule:             "0 1 * * *" (5)
  65. Show Not Applicable:  false
  66. Strict Node Scan:     true
  67. Events:               <none>
1Set to true to enable auto remediations. Set to false to disable auto remediations.
2Set to true to enable auto remediations for content updates. Set to false to disable auto remediations for content updates.
3Specify the number of stored scans in the raw result format. The default value is 3. As the older results get rotated, the administrator must store the results elsewhere before the rotation happens.
4Specify the storage size that should be created for the scan to store the raw results. The default value is 1Gi
5Specify how often the scan should be run in cron format.

To disable the rotation policy, set the value to 0.

6Specify the node-role.kubernetes.io label value to schedule the scan for Node type. This value has to match the name of a MachineConfigPool.

Processing the compliance scan requirements with compliance scans settings

When you have defined the compliance scan requirements and configured the settings to run the scans, then the Compliance Operator processes it using the ScanSettingBinding object.

ScanSettingBinding object

Use the ScanSettingBinding object to specify your compliance requirements with reference to the Profile or TailoredProfile object. It is then linked to a ScanSetting object, which provides the operational constraints for the scan. Then the Compliance Operator generates the ComplianceSuite object based on the ScanSetting and ScanSettingBinding objects.

Example ScanSettingBinding object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: ScanSettingBinding
  3. metadata:
  4.   name: <name of the scan>
  5. profiles: (1)
  6.   # Node checks
  7.   - name: rhcos4-with-usb
  8.     kind: TailoredProfile
  9.     apiGroup: compliance.openshift.io/v1alpha1
  10.   # Cluster checks
  11.   - name: ocp4-moderate
  12.     kind: Profile
  13.     apiGroup: compliance.openshift.io/v1alpha1
  14. settingsRef: (2)
  15.   name: my-companys-constraints
  16.   kind: ScanSetting
  17.   apiGroup: compliance.openshift.io/v1alpha1
1Specify the details of Profile or TailoredProfile object to scan your environment.
2Specify the operational constraints, such as schedule and storage size.

The creation of ScanSetting and ScanSettingBinding objects results in the compliance suite. To get the list of compliance suite, run the following command:

  1. $ oc get compliancesuites

If you delete ScanSettingBinding, then compliance suite also is deleted.

Tracking the compliance scans

After the creation of compliance suite, you can monitor the status of the deployed scans using the ComplianceSuite object.

ComplianceSuite object

The ComplianceSuite object helps you keep track of the state of the scans. It contains the raw settings to create scans and the overall result.

For Node type scans, you should map the scan to the MachineConfigPool, since it contains the remediations for any issues. If you specify a label, ensure it directly applies to a pool.

Example ComplianceSuite object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: ComplianceSuite
  3. metadata:
  4.   name: <name of the scan>
  5. spec:
  6.   autoApplyRemediations: false (1)
  7.   schedule: "0 1 * * *" (2)
  8.   scans: (3)
  9.     - name: workers-scan
  10.       scanType: Node
  11.       profile: xccdf_org.ssgproject.content_profile_moderate
  12.       content: ssg-rhcos4-ds.xml
  13.       contentImage: quay.io/complianceascode/ocp4:latest
  14.       rule: "xccdf_org.ssgproject.content_rule_no_netrc_files"
  15.       nodeSelector:
  16.         node-role.kubernetes.io/worker: ""
  17. status:
  18.   Phase: DONE (4)
  19.   Result: NON-COMPLIANT (5)
  20.   scanStatuses:
  21.   - name: workers-scan
  22.     phase: DONE
  23.     result: NON-COMPLIANT
1Set to true to enable auto remediations. Set to false to disable auto remediations.
2Specify how often the scan should be run in cron format.
3Specify a list of scan specifications to run in the cluster.
4Indicates the progress of the scans.
5Indicates the overall verdict of the suite.

The suite in the background creates the ComplianceScan object based on the scans parameter. You can programmatically fetch the ComplianceSuites events. To get the events for the suite, run the following command:

  1. $ oc get events --field-selector involvedObject.kind=ComplianceSuite,involvedObject.name=<name of the suite>

You might create errors when you manually define the ComplianceSuite, since it contains the XCCDF attributes.

Advanced ComplianceScan Object

The Compliance Operator includes options for advanced users for debugging or integrating with existing tooling. While it is recommended that you not create a ComplianceScan object directly, you can instead manage it using a ComplianceSuite object.

Example Advanced ComplianceScan object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: ComplianceScan
  3. metadata:
  4.   name: <name of the scan>
  5. spec:
  6.   scanType: Node (1)
  7.   profile: xccdf_org.ssgproject.content_profile_moderate (2)
  8.   content: ssg-ocp4-ds.xml
  9.   contentImage: quay.io/complianceascode/ocp4:latest (3)
  10.   rule: "xccdf_org.ssgproject.content_rule_no_netrc_files" (4)
  11.   nodeSelector: (5)
  12.     node-role.kubernetes.io/worker: ""
  13. status:
  14.   phase: DONE (6)
  15.   result: NON-COMPLIANT (7)
1Specify either Node or Platform. Node profiles scan the cluster nodes and platform profiles scan the Kubernetes platform.
2Specify the XCCDF identifier of the profile that you want to run.
3Specify the container image that encapsulates the profile files.
4It is optional. Specify the scan to run a single rule. This rule has to be identified with the XCCDF ID, and has to belong to the specified profile.

If you skip the rule parameter, then scan runs for all the available rules of the specified profile.

5If you are on the OKD and wants to generate a remediation, then nodeSelector label has to match the MachineConfigPool label.

If you do not specify nodeSelector parameter or match the MachineConfig label, scan will still run, but it will not create remediation.

6Indicates the current phase of the scan.
7Indicates the verdict of the scan.

If you delete a ComplianceSuite object, then all the associated scans get deleted.

When the scan is complete, it generates the result as Custom Resources of the ComplianceCheckResult object. However, the raw results are available in ARF format. These results are stored in a Persistent Volume (PV), which has a Persistent Volume Claim (PVC) associated with the name of the scan. You can programmatically fetch the ComplianceScans events. To generate events for the suite, run the following command:

  1. oc get events --field-selector involvedObject.kind=ComplianceScan,involvedObject.name=<name of the suite>

Viewing the compliance results

When the compliance suite reaches the DONE phase, you can view the scan results and possible remediations.

ComplianceCheckResult object

When you run a scan with a specific profile, several rules in the profiles are verified. For each of these rules, a ComplianceCheckResult object is created, which provides the state of the cluster for a specific rule.

Example ComplianceCheckResult object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: ComplianceCheckResult
  3. metadata:
  4.   labels:
  5.     compliance.openshift.io/check-severity: medium
  6.     compliance.openshift.io/check-status: FAIL
  7.     compliance.openshift.io/suite: example-compliancesuite
  8.     compliance.openshift.io/scan-name: workers-scan
  9.   name: workers-scan-no-direct-root-logins
  10.   namespace: openshift-compliance
  11.   ownerReferences:
  12.   - apiVersion: compliance.openshift.io/v1alpha1
  13.     blockOwnerDeletion: true
  14.     controller: true
  15.     kind: ComplianceScan
  16.     name: workers-scan
  17. description: <description of scan check>
  18. instructions: <manual instructions for the scan>
  19. id: xccdf_org.ssgproject.content_rule_no_direct_root_logins
  20. severity: medium (1)
  21. status: FAIL (2)
1Describes the severity of the scan check.
2Describes the result of the check. The possible values are:

  • PASS: check was successful.

  • FAIL: check was unsuccessful.

  • INFO: check was successful and found something not severe enough to be considered an error.

  • MANUAL: check cannot automatically assess the status and manual check is required.

  • INCONSISTENT: different nodes report different results.

  • ERROR: check run successfully, but could not complete.

  • NOTAPPLICABLE: check did not run as it is not applicable.

To get all the check results from a suite, run the following command:

  1. oc get compliancecheckresults \
  2. -l compliance.openshift.io/suite=workers-compliancesuite

ComplianceRemediation object

For a specific check you can have a datastream specified fix. However, if a Kubernetes fix is available, then the Compliance Operator creates a ComplianceRemediation object.

Example ComplianceRemediation object

  1. apiVersion: compliance.openshift.io/v1alpha1
  2. kind: ComplianceRemediation
  3. metadata:
  4.   labels:
  5.     compliance.openshift.io/suite: example-compliancesuite
  6.     compliance.openshift.io/scan-name: workers-scan
  7.     machineconfiguration.openshift.io/role: worker
  8.   name: workers-scan-disable-users-coredumps
  9.   namespace: openshift-compliance
  10.   ownerReferences:
  11.   - apiVersion: compliance.openshift.io/v1alpha1
  12.     blockOwnerDeletion: true
  13.     controller: true
  14.     kind: ComplianceCheckResult
  15.     name: workers-scan-disable-users-coredumps
  16.     uid: <UID>
  17. spec:
  18.   apply: false (1)
  19.   object:
  20.     current: (2)
  21.        apiVersion: machineconfiguration.openshift.io/v1
  22.        kind: MachineConfig
  23.        spec:
  24.          config:
  25.            ignition:
  26.              version: 2.2.0
  27.            storage:
  28.              files:
  29.              - contents:
  30.                  source: data:,%2A%20%20%20%20%20hard%20%20%20core%20%20%20%200
  31.                filesystem: root
  32.                mode: 420
  33.                path: /etc/security/limits.d/75-disable_users_coredumps.conf
  34.     outdated: {} (3)
1true indicates the remediation was applied. false indicates the remediation was not applied.
2Includes the definition of the remediation.
3Indicates remediation that was previously parsed from an earlier version of the content. The Compliance Operator still retains the outdated objects to give the administrator a chance to review the new remediations before applying them.

To get all the remediations from a suite, run the following command:

  1. oc get complianceremediations \
  2. -l compliance.openshift.io/suite=workers-compliancesuite

To list all failing checks that can be remediated automatically, run the following command:

  1. oc get compliancecheckresults \
  2. -l 'compliance.openshift.io/check-status in (FAIL),compliance.openshift.io/automated-remediation'

To list all failing checks that can be remediated manually, run the following command:

  1. oc get compliancecheckresults \
  2. -l 'compliance.openshift.io/check-status in (FAIL),!compliance.openshift.io/automated-remediation'