Image Signatures

Overview

Container image signing on Red Hat Enterprise Linux (RHEL) systems provides a means of:

  • Validating where a container image came from,

  • Checking that the image has not been tampered with, and

  • Setting policies to determine which validated images can be pulled to a host.

For a more complete understanding of the architecture of container image signing on RHEL systems, see the Container Image Signing Integration Guide.

The OpenShift Container Registry allows the ability to store signatures via REST API. The oc CLI can be used to verify image signatures, with their validated displayed in the web console or CLI.

Signing Images Using Atomic CLI

OKD does not automate image signing. Signing requires a developer’s private GPG key, typically stored securely on a workstation. This document describes that workflow.

The atomic command line interface (CLI), version 1.12.5 or greater, provides commands for signing container images, which can be pushed to an OpenShift Container Registry. The atomic CLI is available on Red Hat-based distributions: RHEL, Centos, and Fedora.

The atomic CLI uses the authenticated credentials from oc login. Be sure to use the same user on the same host for both atomic and oc commands. For example, if you execute atomic CLI as sudo, be sure to log in to OKD using sudo oc login.

In order to attach the signature to the image, the user must have the image-signer cluster role. Cluster administrators can add this using:

  1. $ oc adm policy add-cluster-role-to-user system:image-signer <user_name>

Images may be signed at push time:

  1. $ atomic push [--sign-by <gpg_key_id>] --type atomic <image>

Signatures are stored in OKD when the atomic transport type argument is specified. See Signature Transports for more information.

For full details on how to set up and perform image signing using the atomic CLI, see the RHEL Atomic Host Managing Containers: Signing Container Images documentation or the atomic push --help output for argument details.

A specific example workflow of working with the atomic CLI and an OpenShift Container Registry is documented in the Container Image Signing Integration Guide.

Verifying Image Signatures Using OpenShift CLI

You can verify the signatures of an image imported to an OpenShift Container Registry using the oc adm verify-image-signature command. This command verifies if the image identity contained in the image signature can be trusted by using the public GPG key to verify the signature itself then match the provided expected identity with the identity (the pull spec) of the given image.

By default, this command uses the public GPG keyring located in $GNUPGHOME/pubring.gpg, typically in path ~/.gnupg. By default, this command does not save the result of the verification back to the image object. To do so, you must specify the --save flag, as shown below.

In order to verify the signature of an image, the user must have the image-auditor cluster role. Cluster administrators can add this using:

  1. $ oc adm policy add-cluster-role-to-user system:image-auditor <user_name>

Using the —save flag on already verified image together with invalid GPG key or invalid expected identity causes the saved verification status and all signatures to be removed, and the image will become unverified.

In order to avoid deleting all signatures by mistake, you can run the command without the —save flag first and check the logs for potential issues.

To verify an image signature use the following format:

  1. $ oc adm verify-image-signature <image> --expected-identity=<pull_spec> [--save] [options]

The <pull_spec> can be found by describing the image stream. The <image> may be found by describing the image stream tag. See the following example command output.

Example Image Signature Verification

  1. $ oc describe is nodejs -n openshift
  2. Name:             nodejs
  3. Namespace:        openshift
  4. Created:          2 weeks ago
  5. Labels:           <none>
  6. Annotations:      openshift.io/display-name=Node.js
  7.                   openshift.io/image.dockerRepositoryCheck=2017-07-05T18:24:01Z
  8. Docker Pull Spec: 172.30.1.1:5000/openshift/nodejs
  9. ...
  11. $ oc describe istag nodejs:latest -n openshift
  12. Image Name:    sha256:2bba968aedb7dd2aafe5fa8c7453f5ac36a0b9639f1bf5b03f95de325238b288
  13. ...
  15. $ oc adm verify-image-signature \
  16.     sha256:2bba968aedb7dd2aafe5fa8c7453f5ac36a0b9639f1bf5b03f95de325238b288 \
  17.     --expected-identity 172.30.1.1:5000/openshift/nodejs:latest \
  18.     --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release \
  19.     --save

If the oc adm verify-image-signature command returns an x509: certificate signed by unknown authority error, you might need to add the registry’s certificate authority (CA) to the list of CAs trusted on the system. You can do this by performing the following steps:

  1. Transfer the CA certificate from the cluster to the client machine.

    For example, to add the CA for docker-registry.default.svc, transfer the file located at /etc/docker/certs.d/docker-registry.default.svc\:5000/node-client-ca.crt.

  2. Copy the CA certificate to the /etc/pki/ca-trust/source/anchors/ directory. For example:

    1. # cp </path_to_file>/node-client-ca.crt \
    2.     /etc/pki/ca-trust/source/anchors/

  3. Execute update-ca-trust to update the list of trusted CAs:

    1. # update-ca-trust

Accessing Image Signatures Using Registry API

The OpenShift Container Registry provides an extensions endpoint that allows you to write and read image signatures. The image signatures are stored in the OKD key-value store via the container image registry API.

This endpoint is experimental and not supported by the upstream container image registry project. See the upstream API documentation for general information about the container image registry API.

Writing Image Signatures via API

In order to add a new signature to the image, you can use the HTTP PUT method to send a JSON payload to the extensions endpoint:

  1. PUT /extensions/v2/<namespace>/<name>/signatures/<digest>
  1. $ curl -X PUT --data @signature.json http://<user>:<token>@<registry_endpoint>:5000/extensions/v2/<namespace>/<name>/signatures/sha256:<digest>

The JSON payload with the signature content should have the following structure:

  1. {
  2.   "version": 2,
  3.   "type":    "atomic",
  4.   "name":    "sha256:4028782c08eae4a8c9a28bf661c0a8d1c2fc8e19dbaae2b018b21011197e1484@cddeb7006d914716e2728000746a0b23",
  5.   "content": "<cryptographic_signature>"
  6. }

The name field contains the name of the image signature, which must be unique and in the format <digest>@<name>. The <digest> represents an image name and the <name> is the name of the signature. The signature name must be 32 characters long. The <cryptographic_signature> must follow the specification documented in the containers/image library.

Reading Image Signatures via API

Assuming a signed image has already been pushed into the OpenShift Container Registry, you can read the signatures using the following command:

  1. GET /extensions/v2/<namespace>/<name>/signatures/<digest>
  1. $ curl http://<user>:<token>@<registry_endpoint>:5000/extensions/v2/<namespace>/<name>/signatures/sha256:<digest>

The <namespace> represents the OKD project name or registry repository name and the <name> refers to the name of the image repository. The digest represents the SHA-256 checksum of the image.

If the given image contains the signature data, the output of the command above should produce following JSON response:

  1. {
  2.   "signatures": [
  3.   {
  4.     "version": 2,
  5.     "type":    "atomic",
  6.     "name":    "sha256:4028782c08eae4a8c9a28bf661c0a8d1c2fc8e19dbaae2b018b21011197e1484@cddeb7006d914716e2728000746a0b23",
  7.     "content": "<cryptographic_signature>"
  8.   }
  9.   ]
  10. }

The name field contains the name of the image signature, which must be unique and in the format <digest>@<name>. The <digest> represents an image name and the <name> is the name of the signature. The signature name must be 32 characters long. The <cryptographic_signature> must follow the specification documented in the containers/image library.

Importing Image Signatures Automatically from Signature Stores

OKD can automatically import image signatures if a signature store is configured on all OKD master nodes through the registries configuration directory.

The registries configuration directory contains the configuration for various registries (servers storing remote container images) and for the content stored in them. The single directory ensures that the configuration does not have to be provided in command-line options for each command, so that it can be shared by all the users of the containers/image.

The default registries configuration directory is located in the /etc/containers/registries.d/default.yaml file.

A sample configuration that will cause image signatures to be imported automatically for all Red Hat images:

  1. docker:
  2.   registry.redhat.io:
  3.     sigstore: https://registry.redhat.io/containers/sigstore (1)
1Defines the URL of a signature store. This URL is used for reading existing signatures.

Signatures imported automatically by OKD will be unverified by default and will have to be verified by image administrators.

For more details about the registries configuration directory, see Registries Configuration Directory.