Using a Private Docker Registry

Providing access to a private Docker registry

To supply authentication credentials which allows agents to pull from a private Docker registry, there are several methods to choose from:

  1. Use the DC/OS configuration parameter cluster_docker_credentials to set Docker credentials on each agent. This will enable any task to pull from the registry without any other configuration needed to services.

  2. Create an archive (.tar.gz file) of the Docker credentials. Then add it as a URI in each service or pod definition that will pull from the registry. DC/OS packages and other Mesos frameworks may not support the ability to add Docker credentials to their tasks using this method.

  3. Upload the Docker credentials into DC/OS Secrets. Then add that secret in each service or pod definition that will pull from the registry. DC/OS packages and other Mesos frameworks may not support the ability to add Docker credentials to their tasks using this method. Enterprise

DC/OS does not support external credential stores or credential helpers.

In addition to supplying credentials, the CA certificate of the private Docker registry may also be needed to be provided to DC/OS.

Prerequisites

Create a Docker credentials configuration file

  1. Log in to the private registry manually. Login creates a .docker folder and a .docker/config.json file in your home directory.

    1. docker login some.docker.host.com
    2. Username: foo
    3. Password:
    4. Email: foo@bar.com

DC/OS will not use this file directly but it will be used in several of the procedures listed below.

Using cluster_docker_credentials to set cluster-wide registry credentials

DC/OS has several parameters which control Docker credentials for all tasks on the cluster regardless of whether the task uses Docker or UCR as its containerizer. See the DC/OS Configuration Reference for details on each of these parameters. With the following configuration, DC/OS will create a credentials file with a blank configuration:

  1. cluster_docker_credentials = "{}"
  2. cluster_docker_credentials_enabled = "true"
  3. cluster_docker_credentials_write_to_etc = "true"
  4. cluster_docker_credentials_dcos_owned = "false"

Rather than using a blank configuration, operators can choose to include their full Docker credentials configuration. However, this is not recommended as it leaves sensitive information exposed in DC/OS configuration. Instead, the file /etc/mesosphere/docker_credentials can be created prior to DC/OS installation or modified after installation to include the correct configuration and real credentials. The DC/OS agent service must be restarted after making a change to the file: sudo systemctl restart dcos-mesos-slave or sudo systemctl restart dcos-mesos-slave-public. An empty file or invalid configuration will prevent the agent service from starting.

The Docker credentials file location can be changed by setting cluster_docker_credentials_path.

Referencing private Docker registry credentials as a URI

Create an archive of your Docker credentials, then add it as a URI in your service or pod definition.

Step 1: Create an archive of the Docker credentials

  1. Compress your .docker folder and its contents.

    1. cd ~
    2. tar -czf docker.tar.gz .docker

  2. Verify that both files are in the archive.

    1.   tar -tvf ~/docker.tar.gz
    3.   drwx------ root/root         0 2015-07-28 02:54 .docker/
    4.   -rw------- root/root       114 2015-07-28 01:31 .docker/config.json

  3. Put the archive file in a location that is accessible to your application definition.

    1. cp docker.tar.gz /etc/

IMPORTANT: The URI must be accessible by all nodes that will start your application. You can distribute the file to the local filesystem of all nodes, for example via RSYNC/SCP, or store it on a shared network drive like Amazon S3. Consider the security implications of your chosen approach carefully.

Step 2: Add the path to each service definition

  1. Add the path to the archive file login credentials to your service definition.

    1. "fetch": [
    2.   {
    3.     "uri": "file:///etc/docker.tar.gz"
    4.   }
    5. ]

    For example:

    1. {
    2.   "id": "/some/name/or/id",
    3.   "cpus": 1,
    4.   "mem": 1024,
    5.   "instances": 1,
    6.   "container": {
    7.     "type": "DOCKER",
    8.     "docker": {
    9.       "image": "some.docker.host.com/namespace/repo"
    10.     }
    11.   },
    12.   "fetch": [
    13.     {
    14.       "uri": "file:///etc/docker.tar.gz"
    15.     }
    16.   ]
    17. }

    The Docker image will now pull using the provided security credentials.

Reference private Docker registry credentials in DC/OS Secrets Enterprise

Follow these steps to add your Docker registry credentials to the DC/OS Enterprise secrets store, and then reference that secret in your service definition.

IMPORTANT: This functionality is available only with the Universal Container Runtime. If you need to use the Docker Engine, follow the URI instructions above.

Step 1: Create the secret

  1. Check that you have the ~/.docker/config.json file.

    1. ls ~/.docker
    2. config.json

    Your config.json file should look like this, where value of auth is a base64-encoded username:password string.

    1. {
    2.   "auths": {
    3.       "https://index.docker.io/v1/": {
    4.           "auth": "XXXXX",
    5.           "email": "<your-email>"
    6.       }
    7.   }
    8. }

    If you are using Mac OS, you will need to manually encode your username:password string and modify your config.json to match the snippet above. Be sure to omit a trailing new-line when base64 encoding the pair:

    1. echo -n myuser@domain.com:hard-to-guess-password | base64

  2. Add the config.json file to the DC/OS secret store. Learn more about creating secrets.

    1. dcos security secrets create --file=config.json <path/to/secret>

    If you plan to follow the example below, enter the following command to add the secret:

    1. dcos security secrets create --file=config.json mesos-docker/pullConfig

Step 2: Add the secret to your service or pod definition

For a service

  1. Add a location for the secret in the secrets parameter and a reference to the secret in the docker.pullConfig parameter.

    IMPORTANT: This functionality is only supported with the Universal Container Runtime: container.type must be MESOS.

    1. {
    2.    "id": "/mesos-docker",
    3.    "container": {
    4.      "docker": {
    5.        "image": "<your/private/image>",
    6.        "pullConfig": {
    7.          "secret": "pullConfigSecret"
    8.        }
    9.      },
    10.      "type": "MESOS"
    11.    },
    12.    "secrets": {
    13.      "pullConfigSecret": {
    14.        "source": "/mesos-docker/pullConfig"
    15.      }
    16.    },
    17.    "args": ["hello"],
    18.    "cpus": 0.2,
    19.    "mem": 16.0,
    20.    "instances": 1
    21. }

  2. Add the service to DC/OS. If you are using the example above, <svc-name> is mesos-docker.

    1. dcos marathon app add <svc-name>.json

    The Docker image will now pull using the provided security credentials.

For a pod

  1. Add a location for the secret in the secrets parameter and a reference to the secret in the containers.image.pullConfig parameter.

    IMPORTANT: This functionality is only supported if image.kind is set to DOCKER.

    1. {
    2.      "id":"/simple-pod",
    3.      "containers":[
    4.         {
    5.           "name":"simpletask1",
    6.           "exec":{
    7.               "command":{
    8.                 "shell":"env && sleep 1000"
    9.               }
    10.             },
    11.           "resources":{
    12.             "cpus":0.1,
    13.             "mem":32
    14.         },
    15.         "image":{
    16.             "kind":"DOCKER",
    17.             "id":"<your/private/image>",
    18.             "pullConfig":{
    19.               "secret":"pullConfigSecret"
    20.            }
    21.         }
    22.      }
    23.   ],
    24.   "networks":[
    25.       {
    26.         "mode":"host"
    27.       }
    28.   ],
    29.   "secrets":{
    30.       "pullConfigSecret":{
    31.         "source":"/pod/pullConfig"
    32.      }
    33.   }
    34. }

  2. Add the pod to DC/OS. If you are using the example above, <pod-name> is simple-pod.

    1. dcos marathon pod add <pod-name>.json

    The Docker image will now pull using the provided security credentials.

Configuring agents to use a custom certificate for the Docker registry

Some organizations require both user credentials and valid TLS certificates to authorize access to the Docker registry. For example, some registry configurations require a certificate to encrypt the communications between the client and the registry, while user credentials determine who gets to access to the registry after the connection to the registry is successful.

If your private registry uses a certificate to secure communications, you must configure the agent nodes to trust the certificate you use to access the private Docker registry.

To configure a custom certificate for accessing the private Docker registry and DC/OS UCR, complete the following steps:

Step 1: Create the certificate and obtain its CA certificate.

  1. Create or identify a custom certificate that you want to use as a trusted certificate for accessing the Docker registry. You can use OpenSSL, DC/OS Enterprise CLI, or another program for generating public and private keys, certificate requests, and encrypted client and server certificates.

  2. After you create or identify a certificate, you can configure the registry to use this certificate by following the instructions provided by the registry provider.

  3. Finally, obtain the certificate of the certificate authority (CA) which signed the registry certificate and use that in the instructions below.

Step 2: Add custom certificate to Docker

  1. Download or copy the certificate to the following location on each agent:

    1. /etc/docker/certs.d/<registry_name>:<registry_port>/ca.crt

    For the path to the trusted CA certificate on each agent, replace the <registry_name> and <registry_port> with the specific registry name and port number appropriate for your installation. For example, if you are configuring the DC/OS ca.crt certificate as a trusted certificate and the local Docker registry is referenced as registry.mycompany.com:5000, you can download a copy of the ca.crt file and set it as trusted using a command similar to the following:

    1. sudo mkdir -p /etc/docker/certs.d/registry.mycompany.com:5000
    2. sudo cp /path/to/ca.crt etc/docker/certs.d/registry.mycompany.com:5000/ca.crt

  2. Restart the Docker daemon:

    1. sudo systemctl restart docker

    WARNING: Restarting Docker will cause all Docker containers to stop. Drain the agent of any tasks which are using Docker prior to restarting it.

Step 3: Add custom certificate to Mesos

  1. Download or copy the certificate to the following location on each agent:

    1. /var/lib/dcos/pki/tls/certs/docker-registry-ca.crt

  2. Create a symbolic link from the trusted certificate to the /var/lib/dcos/pki/tls/certs directory on each agent:

    1. sudo ln -s /var/lib/dcos/pki/tls/certs/docker-registry-ca.crt /var/lib/dcos/pki/tls/certs/$(openssl x509 -hash -noout -in /var/lib/dcos/pki/tls/certs/docker-registry-ca.crt).0

Pushing a custom image to a private registry from a tarball

If you asked your sales representative for an enterprise version of Marathon, you may have been given a Docker image in a .tar archive. Follow these steps to deploy it to your registry:

Step 1: Import in the local machine

  1. Load the tarball into your local Docker client, passing the path to your custom tarball. For example, marathon-dcos-ee.<version>.tar:

    1. docker load -i marathon-dcos-ee.<version>.tar

    Tip: You can view the Marathon image with this command.

    1. docker images

    You should see output similar to this:

    1. REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
    2. mesosphere/marathon-dcos-ee   1.4.0-RC4_1.9.4     d1ffa68a50c0        3 months ago        926.4 MB

Step 2: Push the image to the repository

  1. Re-tag the file to match the repository that you are using in your private Docker registry:

    1. docker tag \
    2.  mesosphere/marathon-dcos-ee:<mesosphere-tag> \
    3.  <your-repo>/marathon-dcos-ee:<your-tag>

    Where:

    • <mesosphere-tag> is the tag of the image from Mesosphere. Typically, this will match the version number in the filename.
    • <your-repo> is the name of the private repository that you want to store the image in.
    • <your-tag> is the tag for the image. It is recommended that you use the same tag as the Mesosphere image.

  2. Push the new image to your private Docker registry:

    1. docker push <your-repo>/marathon-dcos-ee:<your-tag>