Writing Kuttl Scorecard Tests

This guide outlines the steps which can be followed to implement scorecard tests using the kuttl project and specifically the scorecard kuttl test image.

Defining kuttl Tests in Scorecard

Scorecard users can include kuttl tests within their operator bundles as follows:

  1. .
  2. ├── manifests
  3. │   ├── cache.example.com_memcacheds_crd.yaml
  4. │   └── memcached-operator.clusterserviceversion.yaml
  5. ├── metadata
  6. │   └── annotations.yaml
  7. └── tests
  8.     └── scorecard
  9.         ├── config.yaml
  10.         └── kuttl
  11.             ├── kuttl-test.yaml
  12.             └── list-pods
  13.                 ├── 00-assert.yaml
  14.                 └── 00-pod.yaml
  15.             └── list-other
  16.                 ├── 00-assert.yaml
  17.                 └── 00-pod.yaml
  • bundle/ - Contains bundle manifests and metadata under test.
  • bundle/tests/scorecard/config.yaml - Configuration yaml to define and run scorecard tests.
  • tests/kuttl - Contains tests written for kuttl to execute
  • tests/kuttl/kuttl-test.yaml - Contains the kuttl configuration, it is here that you would add any kuttl specific configuration settings that you might require.
  • tests/kuttl/list-pods - Contains a kuttl test case
  • tests/kuttl/list-pods/00-assert.yaml - Contains a kuttl test case assert
  • tests/kuttl/list-pods/00-pod.yaml - Contains a kuttl test case step
  • tests/kuttl/list-other - Contains another kuttl test case

When the scorecard kuttl binary is executed, it will process all the test cases under the scorecard/kuttl directory within the bundle contents.

Configuring kuttl Tests in the Scorecard Configuration

In the scorecard configuration file, you might have the following definition of what the selector suite=kuttlsuite will translate to:

  1. tests:
  2. - name: "kuttltest1"
  3.   image: quay.io/operator-framework/scorecard-test-kuttl:dev
  4.   labels:
  5.     suite: kuttlsuite
  6.     test: kuttltest1
  7.   description: an ISV custom test that does...

This test configuration will execute the scorecard-test-kuttl image which executes kuttl. The kuttl output is translated into scorecard compliant output which is displayed back to the end user along with any other test results.

With the above kuttl test configuration, you can execute that kuttl test using scorecard as follows:

  1. operator-sdk alpha scorecard deploy/olm-catalog/memcached-operator --selector=suite=kuttlsuite

Defining kuttl Specific Configuration Options

The kuttl configuration file is documented within the kuttl project.

An example of the kuttl configuration file is as follows:

  1. apiVersion: kudo.dev/v1beta1
  2. kind: TestSuite
  3. parallel: 4
  4. timeout: 120
  5. startControlPlane: false

The important fields to note here are:

  • startControlPlane - Set to false since scorecard assumes it is running within a control plane already.

Other kuttl configurations settings are available for more advanced kuttl use cases. See kuttl configuration for more details on kuttl configuration.

kuttl Tests Explained

The kuttl test tool looks for tests to execute within the bundle following a naming convention as follows:

  1.         └── kuttl
  2.             ├── kuttl-test.yaml
  3.             └── list-pods
  4.                 ├── 00-assert.yaml
  5.                 └── 00-pod.yaml
  6.             └── list-other
  7.                 ├── 00-assert.yaml
  8.                 └── 00-pod.yaml

The important fields to note here are:

  • kuttl-test.yaml - The name required for your kuttl configuration file.
  • list-pods, list-other - The names given by you for these test cases.
  • 00-assert.yaml - The assert file is executed to test whether or not the test was successful, this assertion determines whether or not the test passed or failed.
  • 00-pod.yaml - The pod file is used to define what the test will create, in this case a pod will be created based on the manifest within 00-pod.yaml.

The number in front of the assert and pod manifests is used to determine the order in which kuttl will execute the files.

See kuttl tests for a detailed description of how kuttl tests are named and executed.

kuttl Test Privileges

The kuttl tests a user might write can vary widely in functionality and in particular require special Kubernetes RBAC priviledges outside of what your default service account might have. It is therefore very likely you will be required to run scorecard with a custom service account that holds the required RBAC permissions.

You can specify a custom service account in scorecard as follows:

  1. operator-sdk alpha scorecard --service-account=mycustomsa

Also, you can set up a non-default namespace that your tests will be executed within using the following namespace flag:

  1. operator-sdk alpha scorecard --namespace=mycustomns

If you do not specify either of these flags, the default namespace and service account will be used by the scorecard to run test pods.

It is worth noting that scorecard-test-kuttl specifies a namespace to the kubectl-kuttl command which causes kuttl to not create a namespace for each test. This might impact your kuttl tests in that you might need to perform resource cleanup in your tests instead of depending upon namespace deletion to perform that cleanup.

Last modified July 9, 2020: bump kuttl version, handle a kuttl report file name change with the new version, and update kuttl user docs (#3369) (e0dd4caf)