Deploying a Registry on Existing Clusters

You are viewing documentation for a release that is no longer supported. The latest supported version of version 3 is [3.11]. For the most recent version 4, see [4]

You are viewing documentation for a release that is no longer supported. The latest supported version of version 3 is [3.11]. For the most recent version 4, see [4]

Overview

If the integrated registry was not previously deployed automatically during the initial installation of your OKD cluster, or if it is no longer running successfully and you need to redeploy it on your existing cluster, see the following sections for options on deploying a new registry.

This topic is not required if you installed a stand-alone registry.

Deploying the Registry

To deploy the integrated Docker registry, use the oc adm registry command from the admin.kubeconfig file’s location, as a user with cluster administrator privileges:

  1. $ oc adm registry --config=admin.kubeconfig \(1)
  2.     --service-account=registry (2)
1—config is the path to the CLI configuration file for the cluster administrator.
2—service-account is the service account used to run the registry’s pod.

This creates a service and a deployment configuration, both called docker-registry. Once deployed successfully, a pod is created with a name similar to docker-registry-1-cpty9.

To see a full list of options that you can specify when creating the registry:

  1. $ oc adm registry --help

The value for --fs-group must be permitted by the SCC used by the registry (typically, the restricted SCC).

Deploying the Registry as a DaemonSet

Use the oc adm registry command to deploy the registry as a DaemonSet with the --daemonset option.

Daemonsets ensure that when nodes are created, they contain copies of a specified pod. When the nodes are removed, the pods are garbage collected.

For more information on DaemonSets, see Using Daemonsets.

Registry Compute Resources

By default, the registry is created with no settings for compute resource requests or limits. For production, it is highly recommended that the deployment configuration for the registry be updated to set resource requests and limits for the registry pod. Otherwise, the registry pod will be considered a BestEffort pod.

See Compute Resources for more information on configuring requests and limits.

Storage for the Registry

The registry stores container images and metadata. If you simply deploy a pod with the registry, it uses an ephemeral volume that is destroyed if the pod exits. Any images anyone has built or pushed into the registry would disappear.

This section lists the supported registry storage drivers. See the Docker registry documentation for more information.

The following list includes storage drivers that need to be configured in the registry’s configuration file:

General registry storage configuration options are supported. See the Docker registry documentation for more information.

The following storage options need to be configured through the filesystem driver:

For more information on supported persistent storage drivers, see Configuring Persistent Storage and Persistent Storage Examples.

Production Use

For production use, attach a remote volume or define and use the persistent storage method of your choice.

For example, to use an existing persistent volume claim:

  1. $ oc volume deploymentconfigs/docker-registry --add --name=registry-storage -t pvc \
  2.      --claim-name=<pvc_name> --overwrite

Testing shows issues with using the RHEL NFS server as a storage backend for the container image registry. This includes the OpenShift Container Registry and Quay. Therefore, using the RHEL NFS server to back PVs used by core services is not recommended.

Other NFS implementations on the marketplace might not have these issues. Contact the individual NFS implementation vendor for more information on any testing that was possibly completed against these OpenShift core components.

Use Amazon S3 as a Storage Back-end

There is also an option to use Amazon Simple Storage Service storage with the internal Docker registry. It is a secure cloud storage manageable through AWS Management Console. To use it, the registry’s configuration file must be manually edited and mounted to the registry pod. However, before you start with the configuration, look at upstream’s recommended steps.

Take a default YAML configuration file as a base and replace the filesystem entry in the storage section with s3 entry such as below. The resulting storage section may look like this:

  1. storage:
  2.   cache:
  3.     layerinfo: inmemory
  4.   delete:
  5.     enabled: true
  6.   s3:
  7.     accesskey: awsaccesskey (1)
  8.     secretkey: awssecretkey (2)
  9.     region: us-west-1
  10.     regionendpoint: http://myobjects.local
  11.     bucket: bucketname
  12.     encrypt: true
  13.     keyid: mykeyid
  14.     secure: true
  15.     v4auth: false
  16.     chunksize: 5242880
  17.     rootdirectory: /s3/object/name/prefix
1Replace with your Amazon access key.
2Replace with your Amazon secret key.

All of the s3 configuration options are documented in upstream’s driver reference documentation.

Overriding the registry configuration will take you through the additional steps on mounting the configuration file into pod.

When the registry runs on the S3 storage back-end, there are reported issues.

If you want to use a S3 region that is not supported by the integrated registry you are using, see S3 Driver Configuration.

Non-Production Use

For non-production use, you can use the --mount-host=<path> option to specify a directory for the registry to use for persistent storage. The registry volume is then created as a host-mount at the specified <path>.

The —mount-host option mounts a directory from the node on which the registry container lives. If you scale up the docker-registry deployment configuration, it is possible that your registry pods and containers will run on different nodes, which can result in two or more registry containers, each with its own local storage. This will lead to unpredictable behavior, as subsequent requests to pull the same image repeatedly may not always succeed, depending on which container the request ultimately goes to.

The --mount-host option requires that the registry container run in privileged mode. This is automatically enabled when you specify --mount-host. However, not all pods are allowed to run privileged containers by default. If you still want to use this option:

  1. Create a new service account in the default project for the registry to run as. The following example creates a service account named registry:

    1. $ oc create serviceaccount registry -n default

  2. To add the new registry service account in the default namespace to the list of users allowed to run privileged containers:

    1. $ oc adm policy add-scc-to-user privileged system:serviceaccount:default:registry

  3. Create the registry and specify that it use the new registry service account:

    1. $ oc adm registry --service-account=registry \
    2.     --config=admin.kubeconfig \
    3.     --mount-host=<path>

The Docker registry pod runs as user 1001. This user must be able to write to the host directory. You may need to change directory ownership to user ID 1001 with this command:

  1. $ sudo chown 1001:root <path>

Enabling the Registry Console

OKD provides a web-based interface to the integrated registry. This registry console is an optional component for browsing and managing images. It is deployed as a stateless service running as a pod.

If you installed OKD as a stand-alone registry, the registry console is already deployed and secured automatically during installation.

If Cockpit is already running, you’ll need to shut it down before proceeding in order to avoid a port conflict (9090 by default) with the registry console.

Deploying the Registry Console

You must first have exposed the registry.

  1. Install template in the default namespace

    1. oc create -n default -f https://raw.githubusercontent.com/openshift/openshift-ansible/master/roles/openshift_hosted_templates/files/v1.5/origin/registry-console.yaml

  2. Create a passthrough route in the default project. You will need this when creating the registry console application in the next step.

    1. $ oc create route passthrough --service registry-console \
    2.     --port registry-console \
    3.     -n default

  3. Deploy the registry console application. Replace <openshift_oauth_url> with the URL of the OKD OAuth provider, which is typically the master.

    1. $ oc new-app -n default --template=registry-console \
    2.     -p OPENSHIFT_OAUTH_PROVIDER_URL="https://<openshift_oauth_url>:8443" \
    3.     -p REGISTRY_HOST=$(oc get route docker-registry -n default --template='{{ .spec.host }}') \
    4.     -p COCKPIT_KUBE_URL=$(oc get route registry-console -n default --template='https://{{ .spec.host }}')

    If the redirection URL is wrong when you are trying to log in to the registry console, check your OAuth client with oc get oauthclients.

  4. Finally, use a web browser to view the console using the route URI.

Securing the Registry Console

By default, the registry console generates self-signed TLS certificates if deployed manually per the steps in Deploying the Registry Console. See Troubleshooting the Registry Console for more information.

Use the following steps to add your organization’s signed certificates as a secret volume. This assumes your certificates are available on the oc client host.

  1. Create a .cert file containing the certificate and key. Format the file with:

    • One or more BEGIN CERTIFICATE blocks for the server certificate and the intermediate certificate authorities

    • A block containing a BEGIN PRIVATE KEY or similar for the key. The key must not be encrypted

      For example:

      1. -----BEGIN CERTIFICATE-----
      2. MIIDUzCCAjugAwIBAgIJAPXW+CuNYS6QMA0GCSqGSIb3DQEBCwUAMD8xKTAnBgNV
      3. BAoMIGI0OGE2NGNkNmMwNTQ1YThhZTgxOTEzZDE5YmJjMmRjMRIwEAYDVQQDDAls
      4. ...
      5. -----END CERTIFICATE-----
      6. -----BEGIN CERTIFICATE-----
      7. MIIDUzCCAjugAwIBAgIJAPXW+CuNYS6QMA0GCSqGSIb3DQEBCwUAMD8xKTAnBgNV
      8. BAoMIGI0OGE2NGNkNmMwNTQ1YThhZTgxOTEzZDE5YmJjMmRjMRIwEAYDVQQDDAls
      9. ...
      10. -----END CERTIFICATE-----
      11. -----BEGIN PRIVATE KEY-----
      12. MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCyOJ5garOYw0sm
      13. 8TBCDSqQ/H1awGMzDYdB11xuHHsxYS2VepPMzMzryHR137I4dGFLhvdTvJUH8lUS
      14. ...
      15. -----END PRIVATE KEY-----

    • The secured registry should contain the following Subject Alternative Names (SAN) list:

      • Two service hostnames.

        For example:

        1. docker-registry.default.svc.cluster.local
        2. docker-registry.default.svc

      • Service IP address.

        For example:

        1. 172.30.124.220

        Use the following command to get the Docker registry service IP address:

        1. oc get service docker-registry --template='{{.spec.clusterIP}}'

      • Public hostname.

        For example:

        1. docker-registry-default.apps.example.com

        Use the following command to get the Docker registry public hostname:

        1. oc get route docker-registry --template '{{.spec.host}}'

        For example, the server certificate should contain SAN details similar to the following:

        1. X509v3 Subject Alternative Name:
        2.                DNS:docker-registry-public.openshift.com, DNS:docker-registry.default.svc, DNS:docker-registry.default.svc.cluster.local, DNS:172.30.2.98, IP Address:172.30.2.98

        The registry console loads a certificate from the /etc/cockpit/ws-certs.d directory. It uses the last file with a .cert extension in alphabetical order. Therefore, the .cert file should contain at least two PEM blocks formatted in the OpenSSL style.

        If no certificate is found, a self-signed certificate is created using the openssl command and stored in the 0-self-signed.cert file.

  1. Create the secret:

    1. $ oc create secret generic console-secret \
    2.     --from-file=/path/to/console.cert

  2. Add the secrets to the registry-console deployment configuration:

    1. $ oc volume dc/registry-console --add --type=secret \
    2.     --secret-name=console-secret -m /etc/cockpit/ws-certs.d

    This triggers a new deployment of the registry console to include your signed certificates.

Troubleshooting the Registry Console

Debug Mode

The registry console debug mode is enabled using an environment variable. The following command redeploys the registry console in debug mode:

  1. $ oc set env dc registry-console G_MESSAGES_DEBUG=cockpit-ws,cockpit-wrapper

Enabling debug mode allows more verbose logging to appear in the registry console’s pod logs.

Display SSL Certificate Path

To check which certificate the registry console is using, a command can be run from inside the console pod.

  1. List the pods in the default project and find the registry console’s pod name:

    1. $ oc get pods -n default
    2. NAME                       READY     STATUS    RESTARTS   AGE
    3. registry-console-1-rssrw   1/1       Running   0          1d

  2. Using the pod name from the previous command, get the certificate path that the cockpit-ws process is using. This example shows the console using the auto-generated certificate:

    1. $ oc exec registry-console-1-rssrw remotectl certificate
    2. certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert