Writing Custom Scorecard Tests

Custom Tests using Operator SDK Scorecard

This guide outlines the steps which can be followed to extend the existing scorecard tests and implement operator specific custom tests.

Run scorecard with custom tests:

Building test image:

The following steps explain creating of a custom test image which can be used with Scorecard to run operator specific tests. As an example, let us start by creating a sample go repository containing the test bundle data, custom scorecard tests and a Makefile to help us build a test image.

The sample test image repository present here has the following project structure:

  1. ├── Makefile
  2. ├── bundle
  3. │   ├── manifests
  4. │   │   ├── cache.example.com_memcached_crd.yaml
  5. │   │   └── memcached-operator.clusterserviceversion.yaml
  6. │   ├── metadata
  7. │   │   └── annotations.yaml
  8. │   └── tests
  9. │       └── scorecard
  10. │           └── config.yaml
  11. ├── go.mod
  12. ├── go.sum
  13. ├── images
  14. │   └── custom-scorecard-tests
  15. │       ├── Dockerfile
  16. │       ├── bin
  17. │       │   ├── entrypoint
  18. │       │   └── user_setup
  19. │       ├── cmd
  20. │       │   └── test
  21. │       │       └── main.go
  22. │       └── custom-scorecard-tests
  23. └── internal
  24.     └── tests
  25.         └── tests.go
  1. bundle/ - Contains bundle manifests and metadata under test.
  2. bundle/tests/scorecard/config.yaml - Configuration yaml to define and run scorecard tests.
  3. images/custom-scorecard-tests/cmd/test/main.go - Scorecard test binary.
  4. internal/tests/tests.go - Contains the implementation of custom tests specific to the operator.

Writing custom test logic:

Scorecard currently implements a few basic and olm tests for the image bundle, custom resources and custom resource definitions. Additional tests specific to the operator can also be included in the test suite of scorecard.

The tests.go file is where the custom tests are implemented in the sample test image project. These tests use scapiv1alpha2.ScorecardTestResult struct to populate the result, which is then converted to json format for the output. For example, the format of a simple custom sample test can be as follows:

  1. package tests
  3. import (
  4.     "github.com/operator-framework/operator-registry/pkg/registry"
  5.     scapiv1alpha2 "github.com/operator-framework/operator-sdk/pkg/apis/scorecard/v1alpha2"
  6. )
  8. const (
  9.     CustomTest1Name = "customtest1"
  10. )
  12. // CustomTest1 
  13. func CustomTest1(bundle registry.Bundle) scapiv1alpha2.ScorecardTestResult {
  14.     r := scapiv1alpha2.ScorecardTestResult{}
  15.     r.Name = CustomTest1Name
  16.     r.Description = "Custom Test 1"
  17.     r.State = scapiv1alpha2.PassState
  18.     r.Errors = make([]string, 0)
  19.     r.Suggestions = make([]string, 0)
  21.   // Implement relevant custom test logic here
  23.     return r
  24. }

Scorecard Configuration file:

The configuration file includes test definitions and metadata to run the test. For the example CustomTest1 function, the following fields should be specified in config.yaml.

  1. tests:
  2. - name: "customtest1"
  3.   image: quay.io/username/custom-scorecard-tests:dev
  4.   entrypoint: 
  5.   - custom-scorecard-tests
  6.   - customtest1
  7.   labels:
  8.     suite: custom
  9.     test: customtest1
  10.   description: an ISV custom test

The important fields to note here are:

  1. image - name and tag of the test image which was specified in the Makefile.
  2. labels - the name of the test and suite the test function belongs to. This can be specified in the operator-sdk alpha scorecard command to run the desired test.

Note: The default location of config.yaml inside the bundle is <bundle directory>/tests/scorecard/config.yaml. It can be overridden using the --config flag. For more details regarding the configuration file refer to user docs.

Scorecard binary:

The scorecard test image implementation requires the bundle under test to be present in the test image. The GetBundle() function reads the pod’s bundle to fetch the manifests and scorecard configuration from desired path.

  1.     cfg, err := GetBundle("/bundle")
  2.     if err != nil {
  3.         log.Fatal(err.Error())
  4.     }

The scorecard binary uses config.yaml file to locate tests and execute the them as Pods which scorecard creates. Custom test images are included into Pods that scorecard creates, passing in the bundle contents on a shared mount point to the test image container. The specific custom test that is executed is driven by the config.yaml’s entry-point command and arguments.

An example scorecard binary is present here.

The names with which the tests are identified in config.yaml and would be passed in the scorecard command, are to be specified here.

  1. ...     
  2. switch entrypoint[0] {
  3. case tests.CustomTest1Name:
  4.     result = tests.CustomTest1(*cfg)
  5.     ...
  6. }
  7. ...

The result of the custom tests which is in scapiv1alpha2.ScorecardTestResult format, is converted to json for output.

  1. prettyJSON, err := json.MarshalIndent(result, "", "    ")
  2.     if err != nil {
  3.         log.Fatal("Failed to generate json", err)
  4.     }
  5. fmt.Printf("%s\n", string(prettyJSON))

The names of the custom tests are also included in printValidTests() function:

  1. func printValidTests() (result v1alpha2.ScorecardTestResult) {
  2. ...
  3.     str := fmt.Sprintf("Valid tests for this image include: %s,
  4.             tests.CustomTest1Name
  5.     result.Errors = append(result.Errors, str")
  6. ...

Building the project

The project makefile is to help us build the go project and test image using docker. An example of the makefile script can be found in the sample test image.

To build the project, use the docker build command and specify the desired name of the image in the format: <repository_name>/<username>/<image_name>:tag.

Push the image to a remote repository by running the docker command:

  1. docker push <repository_name>/<username>/<image_name>:tag

Running scorecard command

The operator-sdk alpha scorecard command is used to execute the scorecard tests by specifying the location of test bundle in the command. The name or suite of the tests which are to be executed can be specified with the --selector flag. The command will create scorecard pods with the image specified in config.yaml for the respective test. For example, the CustomTest1Name test provides the following json output.

  1. operator-sdk alpha scorecard bundle/ --selector=suite=custom -o json --wait-time=32s --skip-cleanup=false
  2. {
  3.   "metadata": {
  4.     "creationTimestamp": null
  5.   },
  6.   "log": "",
  7.   "results": [
  8.     {
  9.       "name": "customtest1",
  10.       "description": "",
  11.       "state": "pass",
  12.       "log": "an ISV custom test"
  13.     }
  14.   ]
  15. }

Note: More details on the usage of operator-sdk alpha scorecard command and its flags can be found in the scorecard user documentation

Last modified May 26, 2020: [doc] Update links in custom scorecard doc (#3098) (3299e19f)