Extended Registry Configuration

Maintaining the Registry IP Address

OKD refers to the integrated registry by its service IP address, so if you decide to delete and recreate the docker-registry service, you can ensure a completely transparent transition by arranging to re-use the old IP address in the new service. If a new IP address cannot be avoided, you can minimize cluster disruption by rebooting only the masters.

Re-using the Address

To re-use the IP address, you must save the IP address of the old docker-registry service prior to deleting it, and arrange to replace the newly assigned IP address with the saved one in the new docker-registry service.

1. Make a note of the clusterIP for the service:

   $ oc get svc/docker-registry -o yaml | grep clusterIP:

2. Delete the service:

   $ oc delete svc/docker-registry dc/docker-registry

3. Create the registry definition in registry.yaml, replacing <options> with, for example, those used in step 3 of the instructions in the Non-Production Use section:

   $ oc adm registry <options> -o yaml > registry.yaml

4. Edit registry.yaml, find the Service there, and change its clusterIP to the address noted in step 1.

5. Create the registry using the modified registry.yaml:

   $ oc create -f registry.yaml

Rebooting the Masters

If you are unable to re-use the IP address, any operation that uses a pull specification that includes the old IP address will fail. To minimize cluster disruption, you must reboot the masters:

# master-restart api
# master-restart controllers

This ensures that the old registry URL, which includes the old IP address, is cleared from the cache.

Configuring an External Registry Search List

You can use the /etc/containers/registries.conf file to create a list of Docker registries to search for container images.

The /etc/containers/registries.conf file is a list of registry servers that OKD should search against when a user pulls an image using the image short name, such as: myimage:latest. You can customize the order of the search, specify secure and insecure registries, and define a blocked registry list. OKD does not search or allow pulls from registries on the blocked list.

For example, if a user wants to pull the myimage:latest image, OKD searches the registries in the order they appear in the list until it finds the myimage:latest.

The registry search list allows you to curate a set of images and templates that are available for download by OKD users. You can place these images in one or more Docker registries, add the registry to the list, and pull those images into your cluster.

To configure a registry search list:

1. Edit the /etc/containers/registries.conf file to add or edit the following parameters as needed:

   [registries.search] (1)
   registries = ["reg1.example.com", "reg2.example.com"]
   [registries.insecure] (2)
   registries = ["reg3.example.com"]
   [registries.block] (3)
   registries = ['docker.io']

   1	Specify the secure registries from which users can download images using SSL/TLS.
   2	Specify the insecure registries from which users can download images without TLS.
   3	Specify the registries from which users cannot download images.

Setting the Registry Host Name

You can configure the host name and port the registry is known by for both internal and external references. By doing this, image streams will provide hostname based push and pull specifications for images, allowing consumers of the images to be isolated from changes to the registry service IP and potentially allowing image streams and their references to be portable between clusters.

To set the hostname used to reference the registry from within the cluster, set the internalRegistryHostname in the imagePolicyConfig section of the master configuration file. The external host name is controlled by setting the externalRegistryHostname value in the same location.

Image Policy Configuration

imagePolicyConfig:
  internalRegistryHostname: docker-registry.default.svc.cluster.local:5000
  externalRegistryHostname: docker-registry.mycompany.com

The registry itself must be configured with the same internal hostname value. This can be accomplished by setting the **REGISTRY_OPENSHIFT_SERVER_ADDR** environment variable on the registry deployment configuration, or by setting the value in the OpenShift section of the registry configuration.

Overriding the Registry Configuration

You can override the integrated registry’s default configuration, found by default at /config.yml in a running registry’s container, with your own custom configuration.

To enable management of the registry configuration file directly and deploy an updated configuration using a **ConfigMap**:

1. Deploy the registry.

2. Edit the registry configuration file locally as needed. The initial YAML file deployed on the registry is provided below. Review supported options.

   Registry Configuration File

   version: 0.1
   log:
     level: debug
   http:
     addr: :5000
   storage:
     cache:
       blobdescriptor: inmemory
     filesystem:
       rootdirectory: /registry
     delete:
       enabled: true
   auth:
     openshift:
       realm: openshift
   middleware:
     registry:
       - name: openshift
     repository:
       - name: openshift
         options:
           acceptschema2: true
           pullthrough: true
           enforcequota: false
           projectcachettl: 1m
           blobrepositorycachettl: 10m
     storage:
       - name: openshift
   openshift:
     version: 1.0
     metrics:
       enabled: false
       secret: <secret>

3. Create a **ConfigMap** holding the content of each file in this directory:

   $ oc create configmap registry-config \
       --from-file=</path/to/custom/registry/config.yml>/

4. Add the registry-config ConfigMap as a volume to the registry’s deployment configuration to mount the custom configuration file at /etc/docker/registry/:

   $ oc set volume dc/docker-registry --add --type=configmap \
       --configmap-name=registry-config -m /etc/docker/registry/

5. Update the registry to reference the configuration path from the previous step by adding the following environment variable to the registry’s deployment configuration:

   $ oc set env dc/docker-registry \
       REGISTRY_CONFIGURATION_PATH=/etc/docker/registry/config.yml

This may be performed as an iterative process to achieve the desired configuration. For example, during troubleshooting, the configuration may be temporarily updated to put it in debug mode.

To update an existing configuration:

1. Edit the local registry configuration file, config.yml.

2. Delete the registry-config configmap:

   $ oc delete configmap registry-config

3. Recreate the configmap to reference the updated configuration file:

   $ oc create configmap registry-config\
       --from-file=</path/to/custom/registry/config.yml>/

4. Redeploy the registry to read the updated configuration:

   $ oc rollout latest docker-registry

Registry Configuration Reference

There are many configuration options available in the upstream docker distribution library. Not all configuration options are supported or enabled. Use this section as a reference when overriding the registry configuration.

Log

Upstream options are supported.

Example:

log:
  level: debug
  formatter: text
  fields:
    service: registry
    environment: staging

Hooks

Mail hooks are not supported.

Storage

This section lists the supported registry storage drivers. See the container image registry documentation for more information.

The following list includes storage drivers that need to be configured in the registry’s configuration file:

• Filesystem. Filesystem is the default and does not need to be configured.

• S3. See the CloudFront configuration documentation for more information.

• OpenStack Swift

• Google Cloud Storage (GCS)

• Microsoft Azure

• Aliyun OSS

General registry storage configuration options are supported. See the container image registry documentation for more information.

The following storage options need to be configured through the filesystem driver:

• GlusterFS Storage

• Ceph Rados Block Device

General Storage Configuration Options

storage:
  delete:
    enabled: true (1)
  redirect:
    disable: false
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false

Auth

Auth options should not be altered. The openshift extension is the only supported option.

auth:
  openshift:
    realm: openshift

Middleware

The repository middleware extension allows to configure OKD middleware responsible for interaction with OKD and image proxying.

middleware:
  registry:
    - name: openshift (1)
  repository:
    - name: openshift (1)
      options:
        acceptschema2: true (2)
        pullthrough: true (3)
        mirrorpullthrough: true (4)
        enforcequota: false (5)
        projectcachettl: 1m (6)
        blobrepositorycachettl: 10m (7)
  storage:
    - name: openshift (1)

S3 Driver Configuration

If you want to use a S3 region that is not supported by the integrated registry you are using, then you can specify a regionendpoint to avoid the region validation error.

For more information about using Amazon Simple Storage Service storage, see Amazon S3 as a Storage Back-end.

For example:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: inmemory
  delete:
    enabled: true
  s3:
    accesskey: BJKMSZBRESWJQXRWMAEQ
    secretkey: 5ah5I91SNXbeoUXXDasFtadRqOdy62JzlnOW1goS
    bucket: docker.myregistry.com
    region: eu-west-3
    regionendpoint: https://s3.eu-west-3.amazonaws.com
 auth:
  openshift:
    realm: openshift
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
  storage:
    - name: openshift

The regionendpoint can also be useful if you use a S3 storage different from the Amazon S3.

CloudFront Middleware

The CloudFront middleware extension can be added to support AWS, CloudFront CDN storage provider. CloudFront middleware speeds up distribution of image content internationally. The blobs are distributed to several edge locations around the world. The client is always directed to the edge with the lowest latency.

The following is an example of minimal configuration of S3 storage driver with a CloudFront middleware:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: inmemory
  delete:
    enabled: true
  s3: (1)
    accesskey: BJKMSZBRESWJQXRWMAEQ
    secretkey: 5ah5I91SNXbeoUXXDasFtadRqOdy62JzlnOW1goS
    region: us-east-1
    bucket: docker.myregistry.com
auth:
  openshift:
    realm: openshift
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
  storage:
    - name: cloudfront (2)
      options:
        baseurl: https://jrpbyn0k5k88bi.cloudfront.net/ (3)
        privatekey: /etc/docker/cloudfront-ABCEDFGHIJKLMNOPQRST.pem (4)
        keypairid: ABCEDFGHIJKLMNOPQRST (5)
    - name: openshift

Overriding Middleware Configuration Options

The middleware section cannot be overridden using environment variables. There are a few exceptions, however. For example:

middleware:
  repository:
    - name: openshift
      options:
        acceptschema2: true (1)
        pullthrough: true (2)
        mirrorpullthrough: true (3)
        enforcequota: false (4)
        projectcachettl: 1m (5)
        blobrepositorycachettl: 10m (6)

Image Pullthrough

If enabled, the registry will attempt to fetch requested blob from a remote registry unless the blob exists locally. The remote candidates are calculated from DockerImage entries stored in status of the image stream, a client pulls from. All the unique remote registry references in such entries will be tried in turn until the blob is found.

Pullthrough will only occur if an image stream tag exists for the image being pulled. For example, if the image being pulled is **docker-registry.default.svc:5000/yourproject/yourimage:prod** then the registry will look for an image stream tag named **yourimage:prod** in the project **yourproject**. If it finds one, it will attempt to pull the image using the **dockerImageReference** associated with that image stream tag.

When performing pullthrough, the registry will use pull credentials found in the project associated with the image stream tag that is being referenced. This capability also makes it possible for you to pull images that reside on a registry they do not have credentials to access, as long as you have access to the image stream tag that references the image.

You must ensure that your registry has appropriate certificates to trust any external registries you do a pullthrough against. The certificates need to be placed in the /etc/pki/tls/certs directory on the pod. You can mount the certificates using a configuration map or secret. Note that the entire /etc/pki/tls/certs directory must be replaced. You must include the new certificates and replace the system certificates in your secret or configuration map that you mount.

Note that by default image stream tags use a reference policy type of **Source** which means that when the image stream reference is resolved to an image pull specification, the specification used will point to the source of the image. For images hosted on external registries, this will be the external registry and as a result the resource will reference and pull the image by the external registry. For example, **registry.redhat.io/openshift3/jenkins-2-rhel7** and pullthrough will not apply. To ensure that resources referencing image streams use a pull specification that points to the internal registry, the image stream tag should use a reference policy type of **Local**. More information is available on Reference Policy.

This feature is on by default. However, it can be disabled using a configuration option.

By default, all the remote blobs served this way are stored locally for subsequent faster access unless mirrorpullthrough is disabled. The downside of this mirroring feature is an increased storage usage.

$ oc get imagestreamtag/${IS}:${TAG} -o jsonpath=’{ .image.dockerImageLayers[*].name }’ | \
  xargs -n1 -I {} curl -H “Range: bytes=0-1” -u user:${TOKEN} \
  http://${REGISTRY_IP}:${PORT}/v2/default/mysql/blobs/{}

Manifest Schema v2 Support

Each image has a manifest describing its blobs, instructions for running it and additional metadata. The manifest is versioned, with each version having different structure and fields as it evolves over time. The same image can be represented by multiple manifest versions. Each version will have different digest though.

The registry currently supports manifest v2 schema 1 (schema1) and manifest v2 schema 2 (schema2). The former is being obsoleted but will be supported for an extended amount of time.

The default configuration is to store schema2.

You should be wary of compatibility issues with various Docker clients:

• Docker clients of version 1.9 or older support only schema1. Any manifest this client pulls or pushes will be of this legacy schema.

• Docker clients of version 1.10 support both schema1 and schema2. And by default, it will push the latter to the registry if it supports newer schema.

The registry, storing an image with schema1 will always return it unchanged to the client. Schema2 will be transferred unchanged only to newer Docker client. For the older one, it will be converted on-the-fly to schema1.

This has significant consequences. For example an image pushed to the registry by a newer Docker client cannot be pulled by the older Docker by its digest. That’s because the stored image’s manifest is of schema2 and its digest can be used to pull only this version of manifest.

Once you’re confident that all the registry clients support schema2, you’ll be safe to enable its support in the registry. See the middleware configuration reference above for particular option.

OpenShift

This section reviews the configuration of global settings for features specific to OKD. In a future release, openshift-related settings in the Middleware section will be obsoleted.

Currently, this section allows you to configure registry metrics collection:

openshift:
  version: 1.0 (1)
  server:
    addr: docker-registry.default.svc (2)
  metrics:
    enabled: false (3)
    secret: <secret> (4)
  requests:
    read:
      maxrunning: 10 (5)
      maxinqueue: 10 (6)
      maxwaitinqueue 2m (7)
    write:
      maxrunning: 10 (8)
      maxinqueue: 10 (9)
      maxwaitinqueue 2m (10)

See Accessing Registry Metrics for usage information.

Reporting

Reporting is unsupported.

HTTP

Upstream options are supported. Learn how to alter these settings via environment variables. Only the tls section should be altered. For example:

http:
  addr: :5000
  tls:
    certificate: /etc/secrets/registry.crt
    key: /etc/secrets/registry.key

Notifications

Upstream options are supported. The REST API Reference provides more comprehensive integration options.

Example:

notifications:
  endpoints:
    - name: registry
      disabled: false
      url: https://url:port/path
      headers:
        Accept:
          - text/plain
      timeout: 500
      threshold: 5
      backoff: 1000

Redis

Redis is not supported.

Health

Upstream options are supported. The registry deployment configuration provides an integrated health check at /healthz.

Proxy

Proxy configuration should not be enabled. This functionality is provided by the OKD repository middleware extension, pullthrough: true.

Cache

The integrated registry actively caches data to reduce the number of calls to slow external resources. There are two caches:

1. The storage cache that is used to cache blobs metadata. This cache does not have an expiration time and the data is there until it is explicitly deleted.

2. The application cache contains association between blobs and repositories. The data in this cache has an expiration time.

In order to completely turn off the cache, you need to change the configuration:

version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    blobdescriptor: "" (1)
openshift:
  version: 1.0
  cache:
    disabled: true (2)
    blobrepositoryttl: 10m