10.部署私有 docker registry

注意：本文档介绍使用 docker 官方的 registry v2 镜像部署私有仓库的步骤，你也可以部署 Harbor 私有仓库（部署 Harbor 私有仓库）。

本文档讲解部署一个 TLS 加密、HTTP Basic 认证、用 ceph rgw 做后端存储的私有 docker registry 步骤，如果使用其它类型的后端存储，则可以从 “创建 docker registry” 节开始；

示例两台机器 IP 如下：

  • ceph rgw: 10.64.3.9
  • docker registry: 10.64.3.7

部署 ceph RGW 节点

  1. $ ceph-deploy rgw create 10.64.3.9 # rgw 默认监听7480端口
  2. $

创建测试账号 demo

  1. $ radosgw-admin user create --uid=demo --display-name="ceph rgw demo user"
  2. $

创建 demo 账号的子账号 swift

当前 registry 只支持使用 swift 协议访问 ceph rgw 存储，暂时不支持 s3 协议；

  1. $ radosgw-admin subuser create --uid demo --subuser=demo:swift --access=full --secret=secretkey --key-type=swift
  2. $

创建 demo:swift 子账号的 sercret key

  1. $ radosgw-admin key create --subuser=demo:swift --key-type=swift --gen-secret
  2. {
  3.     "user_id": "demo",
  4.     "display_name": "ceph rgw demo user",
  5.     "email": "",
  6.     "suspended": 0,
  7.     "max_buckets": 1000,
  8.     "auid": 0,
  9.     "subusers": [
  10.         {
  11.             "id": "demo:swift",
  12.             "permissions": "full-control"
  13.         }
  14.     ],
  15.     "keys": [
  16.         {
  17.             "user": "demo",
  18.             "access_key": "5Y1B1SIJ2YHKEHO5U36B",
  19.             "secret_key": "nrIvtPqUj7pUlccLYPuR3ntVzIa50DToIpe7xFjT"
  20.         }
  21.     ],
  22.     "swift_keys": [
  23.         {
  24.             "user": "demo:swift",
  25.             "secret_key": "aCgVTx3Gfz1dBiFS4NfjIRmvT0sgpHDP6aa0Yfrh"
  26.         }
  27.     ],
  28.     "caps": [],
  29.     "op_mask": "read, write, delete",
  30.     "default_placement": "",
  31.     "placement_tags": [],
  32.     "bucket_quota": {
  33.         "enabled": false,
  34.         "max_size_kb": -1,
  35.         "max_objects": -1
  36.     },
  37.     "user_quota": {
  38.         "enabled": false,
  39.         "max_size_kb": -1,
  40.         "max_objects": -1
  41.     },
  42.         "temp_url_keys": []
  43. }
  • aCgVTx3Gfz1dBiFS4NfjIRmvT0sgpHDP6aa0Yfrh 为子账号 demo:swift 的 secret key；

创建 docker registry

创建 registry 使用的 x509 证书

  1. $ mdir -p registry/{auth,certs}
  2. $ cat registry-csr.json
  3. {
  4.   "CN": "registry",
  5.   "hosts": [
  6.       "127.0.0.1",
  7.       "10.64.3.7"
  8.   ],
  9.   "key": {
  10.     "algo": "rsa",
  11.     "size": 2048
  12.   },
  13.   "names": [
  14.     {
  15.       "C": "CN",
  16.       "ST": "BeiJing",
  17.       "L": "BeiJing",
  18.       "O": "k8s",
  19.       "OU": "4Paradigm"
  20.     }
  21.   ]
  22. }
  23. $ cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  24.     -ca-key=/etc/kubernetes/cert/ca-key.pem \
  25.     -config=/etc/kubernetes/cert/ca-config.json \
  26.     -profile=kubernetes registry-csr.json | cfssljson -bare registry
  27. $ cp registry.pem registry-key.pem registry/certs
  28. $
  • 这里复用以前创建的 CA 证书和秘钥文件；
  • hosts 字段指定 registry 的 NodeIP；

创建 HTTP Baisc 认证文件

  1. $ docker run --entrypoint htpasswd registry:2 -Bbn foo foo123  > auth/htpasswd
  2. $ cat auth/htpasswd
  3. foo:$2y$05$I60z69MdluAQ8i1Ka3x3Neb332yz1ioow2C4oroZSOE0fqPogAmZm

配置 registry 参数

  1. $ export RGW_AUTH_URL="http://10.64.3.9:7480/auth/v1"
  2. $ export RGW_USER="demo:swift"
  3. $ export RGW_SECRET_KEY="aCgVTx3Gfz1dBiFS4NfjIRmvT0sgpHDP6aa0Yfrh"
  4. $ cat > config.yml << EOF
  5. # https://docs.docker.com/registry/configuration/#list-of-configuration-options
  6. version: 0.1
  7. log:
  8.   level: info
  9.   fromatter: text
  10.   fields:
  11.     service: registry
  13. storage:
  14.   cache:
  15.     blobdescriptor: inmemory
  16.   delete:
  17.     enabled: true
  18.   swift:
  19.     authurl: ${RGW_AUTH_URL}
  20.     username: ${RGW_USER}
  21.     password: ${RGW_SECRET_KEY}
  22.     container: registry
  24. auth:
  25.   htpasswd:
  26.     realm: basic-realm
  27.     path: /auth/htpasswd
  29. http:
  30.   addr: 0.0.0.0:8000
  31.   headers:
  32.     X-Content-Type-Options: [nosniff]
  33.   tls:
  34.     certificate: /certs/registry.pem
  35.     key: /certs/registry-key.pem
  37. health:
  38.   storagedriver:
  39.     enabled: true
  40.     interval: 10s
  41.     threshold: 3
  42. EOF
  • storage.swift 指定后端使用 swfit 接口协议的存储，这里配置的是 ceph rgw 存储参数；
  • auth.htpasswd 指定了 HTTP Basic 认证的 token 文件路径；
  • http.tls 指定了 registry http 服务器的证书和秘钥文件路径；

创建 docker registry

  1. $ docker run -d -p 8000:8000 \
  2.     -v $(pwd)/registry/auth/:/auth \
  3.     -v $(pwd)/registry/certs:/certs \
  4.     -v $(pwd)/config.yml:/etc/docker/registry/config.yml \
  5.     --name registry registry:2
  • 执行该 docker run 命令的机器 IP 为 10.64.3.7；

向 registry push image

将签署 registry 证书的 CA 证书拷贝到 /etc/docker/certs.d/10.64.3.7:8000 目录下

  1. $ sudo mkdir -p /etc/docker/certs.d/10.64.3.7:8000
  2. $ sudo cp /etc/kubernetes/cert/ca.pem /etc/docker/certs.d/10.64.3.7:8000/ca.crt
  3. $

登陆私有 registry

  1. $ docker login 10.64.3.7:8000
  2. Username: foo
  3. Password:
  4. Login Succeeded

登陆信息被写入 ~/.docker/config.json 文件

  1. $ cat ~/.docker/config.json
  2. {
  3.         "auths": {
  4.                 "10.64.3.7:8000": {
  5.                         "auth": "Zm9vOmZvbzEyMw=="
  6.                 }
  7.         }
  8. }

将本地的 image 打上私有 registry 的 tag

  1. $ docker tag docker.io/kubernetes/pause 10.64.3.7:8000/zhangjun3/pause
  2. $ docker images |grep pause
  3. docker.io/kubernetes/pause                            latest              f9d5de079539        2 years ago         239.8 kB
  4. 10.64.3.7:8000/zhangjun3/pause                        latest              f9d5de079539        2 years ago         239.8 kB

将 image push 到私有 registry

  1. $ docker push 10.64.3.7:8000/zhangjun3/pause
  2. The push refers to a repository [10.64.3.7:8000/zhangjun3/pause]
  3. 5f70bf18a086: Pushed
  4. e16a89738269: Pushed
  5. latest: digest: sha256:9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359 size: 916

查看 ceph 上是否已经有 push 的 pause 容器文件

  1. $ rados lspools
  2. rbd
  3. .rgw.root
  4. default.rgw.control
  5. default.rgw.data.root
  6. default.rgw.gc
  7. default.rgw.log
  8. default.rgw.users.uid
  9. default.rgw.users.keys
  10. default.rgw.users.swift
  11. default.rgw.buckets.index
  12. default.rgw.buckets.data
  14. $ rados --pool default.rgw.buckets.data ls|grep pause
  15. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_layers/sha256/f9d5de0795395db6c50cb1ac82ebed1bd8eb3eefcebb1aa724e01239594e937b/link
  16. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_layers/sha256/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845/link
  17. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link
  18. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_manifests/tags/latest/current/link
  19. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_manifests/tags/latest/index/sha256/9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359/link
  20. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_manifests/revisions/sha256/9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359/link

私有 registry 的运维操作

查询私有镜像中的 images

  1. $ curl  --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/_catalog
  2. {"repositories":["library/redis","zhangjun3/busybox","zhangjun3/pause","zhangjun3/pause2"]}

查询某个镜像的 tags 列表

  1. $ curl  --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/tags/list
  2. {"name":"zhangjun3/busybox","tags":["latest"]}

获取 image 或 layer 的 digest

向 v2/<repoName>/manifests/<tagName> 发 GET 请求，从响应的头部 Docker-Content-Digest 获取 image digest，从响应的 body 的 fsLayers.blobSum 中获取 layDigests;

注意，必须包含请求头：Accept: application/vnd.docker.distribution.manifest.v2+json：

  1. $ curl -v -H "Accept: application/vnd.docker.distribution.manifest.v2+json" --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/manifests/latest
  3. > GET /v2/zhangjun3/busybox/manifests/latest HTTP/1.1
  4. > User-Agent: curl/7.29.0
  5. > Host: 10.64.3.7:8000
  6. > Accept: application/vnd.docker.distribution.manifest.v2+json
  7. >
  8. < HTTP/1.1 200 OK
  9. < Content-Length: 527
  10. < Content-Type: application/vnd.docker.distribution.manifest.v2+json
  11. < Docker-Content-Digest: sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5
  12. < Docker-Distribution-Api-Version: registry/2.0
  13. < Etag: "sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5"
  14. < X-Content-Type-Options: nosniff
  15. < Date: Tue, 21 Mar 2017 15:19:42 GMT
  16. <
  17. {
  18.    "schemaVersion": 2,
  19.    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  20.    "config": {
  21.       "mediaType": "application/vnd.docker.container.image.v1+json",
  22.       "size": 1465,
  23.       "digest": "sha256:00f017a8c2a6e1fe2ffd05c281f27d069d2a99323a8cd514dd35f228ba26d2ff"
  24.    },
  25.    "layers": [
  26.       {
  27.          "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
  28.          "size": 701102,
  29.          "digest": "sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515"
  30.       }
  31.    ]
  32. }

删除 image

向 /v2/<name>/manifests/<reference> 发送 DELETE 请求，reference 为上一步返回的 Docker-Content-Digest 字段内容：

  1. $ curl -X DELETE  --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/manifests/sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5
  2. $

删除 layer

向 /v2/<name>/blobs/<digest>发送 DELETE 请求，其中 digest 是上一步返回的 fsLayers.blobSum 字段内容：

  1. $ curl -X DELETE  --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
  2. $ curl -X DELETE  --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515
  3. $