Container image signatures

Red Hat delivers signatures for the images in the Red Hat Container Registries. Those signatures can be automatically verified when being pulled to OKD 4 clusters by using the Machine Config Operator (MCO).

Quay.io serves most of the images that make up OKD, and only the release image is signed. Release images refer to the approved OKD images, offering a degree of protection against supply chain attacks. However, some extensions to OKD, such as logging, monitoring, and service mesh, are shipped as Operators from the Operator Lifecycle Manager (OLM). Those images ship from the Red Hat Ecosystem Catalog Container images registry.

To verify the integrity of those images between Red Hat registries and your infrastructure, enable signature verification.

Enabling signature verification for Red Hat Container Registries

Enabling container signature validation for Red Hat Container Registries requires writing a signature verification policy file specifying the keys to verify images from these registries. The registries are already defined in /etc/containers/registries.d by default.

Procedure

  1. Create a Butane config file, 51-worker-rh-registry-trust.bu, containing the necessary configuration for the worker nodes.

    See “Creating machine configs with Butane” for information about Butane.

    1. variant: openshift
    2. version: 4.10.0
    3. metadata:
    4.   name: 51-worker-rh-registry-trust
    5.   labels:
    6.     machineconfiguration.openshift.io/role: worker
    7. storage:
    8.   files:
    9.   - path: /etc/containers/policy.json
    10.     mode: 0644
    11.     overwrite: true
    12.     contents:
    13.       inline: |
    14.         {
    15.           "default": [
    16.             {
    17.               "type": "insecureAcceptAnything"
    18.             }
    19.           ],
    20.           "transports": {
    21.             "docker": {
    22.               "registry.access.redhat.com": [
    23.                 {
    24.                   "type": "signedBy",
    25.                   "keyType": "GPGKeys",
    26.                   "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
    27.                 }
    28.               ],
    29.               "registry.redhat.io": [
    30.                 {
    31.                   "type": "signedBy",
    32.                   "keyType": "GPGKeys",
    33.                   "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
    34.                 }
    35.               ]
    36.             },
    37.             "docker-daemon": {
    38.               "": [
    39.                 {
    40.                   "type": "insecureAcceptAnything"
    41.                 }
    42.               ]
    43.             }
    44.           }
    45.         }

  2. Use Butane to generate a machine config YAML file, 51-worker-rh-registry-trust.yaml, containing the file to be written to disk on the worker nodes:

    1. $ butane 51-worker-rh-registry-trust.bu -o 51-worker-rh-registry-trust.yaml

  3. Apply the created machine config:

    1. $ oc apply -f 51-worker-rh-registry-trust.yaml

  4. Create a Butane config and corresponding machine config which writes the configuration to disk on the control plane nodes:

    1. $ sed -e 's,\(machineconfiguration.openshift.io/role: \)worker,\1master,' \
    2.     -e 's,\(name: 51-\)worker,\1master,' 51-worker-rh-registry-trust.bu \
    3.     > 51-master-rh-registry-trust.bu
    4. $ butane 51-master-rh-registry-trust.bu -o 51-master-rh-registry-trust.yaml

  5. Apply the control plane machine config to the cluster:

    1. $ oc apply -f 51-master-rh-registry-trust.yaml

Verifying the signature verification configuration

After you apply the machine configs to the cluster, the Machine Config Controller detects the new MachineConfig object and generates a new rendered-worker-<hash> version.

Prerequisites

  • You enabled signature verification by using a machine config file.

Procedure

  1. On the command line, run the following command to display information about a desired worker:

    1. $ oc describe machineconfigpool/worker

    Example output of initial worker monitoring

    1. Name:         worker
    2. Namespace:
    3. Labels:       machineconfiguration.openshift.io/mco-built-in=
    4. Annotations:  <none>
    5. API Version:  machineconfiguration.openshift.io/v1
    6. Kind:         MachineConfigPool
    7. Metadata:
    8.   Creation Timestamp:  2019-12-19T02:02:12Z
    9.   Generation:          3
    10.   Resource Version:    16229
    11.   Self Link:           /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
    12.   UID:                 92697796-2203-11ea-b48c-fa163e3940e5
    13. Spec:
    14.   Configuration:
    15.     Name:  rendered-worker-f6819366eb455a401c42f8d96ab25c02
    16.     Source:
    17.       API Version:  machineconfiguration.openshift.io/v1
    18.       Kind:         MachineConfig
    19.       Name:         00-worker
    20.       API Version:  machineconfiguration.openshift.io/v1
    21.       Kind:         MachineConfig
    22.       Name:         01-worker-container-runtime
    23.       API Version:  machineconfiguration.openshift.io/v1
    24.       Kind:         MachineConfig
    25.       Name:         01-worker-kubelet
    26.       API Version:  machineconfiguration.openshift.io/v1
    27.       Kind:         MachineConfig
    28.       Name:         51-worker-rh-registry-trust
    29.       API Version:  machineconfiguration.openshift.io/v1
    30.       Kind:         MachineConfig
    31.       Name:         99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
    32.       API Version:  machineconfiguration.openshift.io/v1
    33.       Kind:         MachineConfig
    34.       Name:         99-worker-ssh
    35.   Machine Config Selector:
    36.     Match Labels:
    37.       machineconfiguration.openshift.io/role:  worker
    38.   Node Selector:
    39.     Match Labels:
    40.       node-role.kubernetes.io/worker:
    41.   Paused:                              false
    42. Status:
    43.   Conditions:
    44.     Last Transition Time:  2019-12-19T02:03:27Z
    45.     Message:
    46.     Reason:
    47.     Status:                False
    48.     Type:                  RenderDegraded
    49.     Last Transition Time:  2019-12-19T02:03:43Z
    50.     Message:
    51.     Reason:
    52.     Status:                False
    53.     Type:                  NodeDegraded
    54.     Last Transition Time:  2019-12-19T02:03:43Z
    55.     Message:
    56.     Reason:
    57.     Status:                False
    58.     Type:                  Degraded
    59.     Last Transition Time:  2019-12-19T02:28:23Z
    60.     Message:
    61.     Reason:
    62.     Status:                False
    63.     Type:                  Updated
    64.     Last Transition Time:  2019-12-19T02:28:23Z
    65.     Message:               All nodes are updating to rendered-worker-f6819366eb455a401c42f8d96ab25c02
    66.     Reason:
    67.     Status:                True
    68.     Type:                  Updating
    69.   Configuration:
    70.     Name:  rendered-worker-d9b3f4ffcfd65c30dcf591a0e8cf9b2e
    71.     Source:
    72.       API Version:            machineconfiguration.openshift.io/v1
    73.       Kind:                   MachineConfig
    74.       Name:                   00-worker
    75.       API Version:            machineconfiguration.openshift.io/v1
    76.       Kind:                   MachineConfig
    77.       Name:                   01-worker-container-runtime
    78.       API Version:            machineconfiguration.openshift.io/v1
    79.       Kind:                   MachineConfig
    80.       Name:                   01-worker-kubelet
    81.       API Version:            machineconfiguration.openshift.io/v1
    82.       Kind:                   MachineConfig
    83.       Name:                   99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
    84.       API Version:            machineconfiguration.openshift.io/v1
    85.       Kind:                   MachineConfig
    86.       Name:                   99-worker-ssh
    87.   Degraded Machine Count:     0
    88.   Machine Count:              1
    89.   Observed Generation:        3
    90.   Ready Machine Count:        0
    91.   Unavailable Machine Count:  1
    92.   Updated Machine Count:      0
    93. Events:                       <none>

  2. Run the oc describe command again:

    1. $ oc describe machineconfigpool/worker

    Example output after the worker is updated

    1. ...
    2.     Last Transition Time:  2019-12-19T04:53:09Z
    3.     Message:               All nodes are updated with rendered-worker-f6819366eb455a401c42f8d96ab25c02
    4.     Reason:
    5.     Status:                True
    6.     Type:                  Updated
    7.     Last Transition Time:  2019-12-19T04:53:09Z
    8.     Message:
    9.     Reason:
    10.     Status:                False
    11.     Type:                  Updating
    12.   Configuration:
    13.     Name:  rendered-worker-f6819366eb455a401c42f8d96ab25c02
    14.     Source:
    15.       API Version:            machineconfiguration.openshift.io/v1
    16.       Kind:                   MachineConfig
    17.       Name:                   00-worker
    18.       API Version:            machineconfiguration.openshift.io/v1
    19.       Kind:                   MachineConfig
    20.       Name:                   01-worker-container-runtime
    21.       API Version:            machineconfiguration.openshift.io/v1
    22.       Kind:                   MachineConfig
    23.       Name:                   01-worker-kubelet
    24.       API Version:            machineconfiguration.openshift.io/v1
    25.       Kind:                   MachineConfig
    26.       Name:                   51-worker-rh-registry-trust
    27.       API Version:            machineconfiguration.openshift.io/v1
    28.       Kind:                   MachineConfig
    29.       Name:                   99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
    30.       API Version:            machineconfiguration.openshift.io/v1
    31.       Kind:                   MachineConfig
    32.       Name:                   99-worker-ssh
    33.   Degraded Machine Count:     0
    34.   Machine Count:              3
    35.   Observed Generation:        4
    36.   Ready Machine Count:        3
    37.   Unavailable Machine Count:  0
    38.   Updated Machine Count:      3
    39. ...

    The Observed Generation parameter shows an increased count based on the generation of the controller-produced configuration. This controller updates this value even if it fails to process the specification and generate a revision. The Configuration Source value points to the 51-worker-rh-registry-trust configuration.

  3. Confirm that the policy.json file exists with the following command:

    1. $ oc debug node/<node> -- chroot /host cat /etc/containers/policy.json

    Example output

    1. Starting pod/<node>-debug ...
    2. To use host binaries, run `chroot /host`
    3. {
    4.   "default": [
    5.     {
    6.       "type": "insecureAcceptAnything"
    7.     }
    8.   ],
    9.   "transports": {
    10.     "docker": {
    11.       "registry.access.redhat.com": [
    12.         {
    13.           "type": "signedBy",
    14.           "keyType": "GPGKeys",
    15.           "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
    16.         }
    17.       ],
    18.       "registry.redhat.io": [
    19.         {
    20.           "type": "signedBy",
    21.           "keyType": "GPGKeys",
    22.           "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
    23.         }
    24.       ]
    25.     },
    26.     "docker-daemon": {
    27.       "": [
    28.         {
    29.           "type": "insecureAcceptAnything"
    30.         }
    31.       ]
    32.     }
    33.   }
    34. }

  4. Confirm that the registry.redhat.io.yaml file exists with the following command:

    1. $ oc debug node/<node> -- chroot /host cat /etc/containers/registries.d/registry.redhat.io.yaml

    Example output

    1. Starting pod/<node>-debug ...
    2. To use host binaries, run `chroot /host`
    3. docker:
    4.      registry.redhat.io:
    5.          sigstore: https://registry.redhat.io/containers/sigstore

  5. Confirm that the registry.access.redhat.com.yaml file exists with the following command:

    1. $ oc debug node/<node> -- chroot /host cat /etc/containers/registries.d/registry.access.redhat.com.yaml

    Example output

    1. Starting pod/<node>-debug ...
    2. To use host binaries, run `chroot /host`
    3. docker:
    4.      registry.access.redhat.com:
    5.          sigstore: https://access.redhat.com/webassets/docker/content/sigstore

Additional resources