部署私有 docker registry

注意:本文档介绍使用 docker 官方的 registry v2 镜像部署私有仓库的步骤,你也可以部署 Harbor 私有仓库(部署 Harbor 私有仓库)。

本文档讲解部署一个 TLS 加密、HTTP Basic 认证、用 ceph rgw 做后端存储的私有 docker registry 步骤,如果使用其它类型的后端存储,则可以从 “创建 docker registry” 节开始;

示例两台机器 IP 如下:

  • ceph rgw: 10.64.3.9
  • docker registry: 10.64.3.7

部署 ceph RGW 节点

  1. $ ceph-deploy rgw create 10.64.3.9 # rgw 默认监听7480端口
  2. $

创建测试账号 demo

  1. $ radosgw-admin user create --uid=demo --display-name="ceph rgw demo user"
  2. $

创建 demo 账号的子账号 swift

当前 registry 只支持使用 swift 协议访问 ceph rgw 存储,暂时不支持 s3 协议;

  1. $ radosgw-admin subuser create --uid demo --subuser=demo:swift --access=full --secret=secretkey --key-type=swift
  2. $

创建 demo:swift 子账号的 sercret key

  1. $ radosgw-admin key create --subuser=demo:swift --key-type=swift --gen-secret
  2. {
  3. "user_id": "demo",
  4. "display_name": "ceph rgw demo user",
  5. "email": "",
  6. "suspended": 0,
  7. "max_buckets": 1000,
  8. "auid": 0,
  9. "subusers": [
  10. {
  11. "id": "demo:swift",
  12. "permissions": "full-control"
  13. }
  14. ],
  15. "keys": [
  16. {
  17. "user": "demo",
  18. "access_key": "5Y1B1SIJ2YHKEHO5U36B",
  19. "secret_key": "nrIvtPqUj7pUlccLYPuR3ntVzIa50DToIpe7xFjT"
  20. }
  21. ],
  22. "swift_keys": [
  23. {
  24. "user": "demo:swift",
  25. "secret_key": "aCgVTx3Gfz1dBiFS4NfjIRmvT0sgpHDP6aa0Yfrh"
  26. }
  27. ],
  28. "caps": [],
  29. "op_mask": "read, write, delete",
  30. "default_placement": "",
  31. "placement_tags": [],
  32. "bucket_quota": {
  33. "enabled": false,
  34. "max_size_kb": -1,
  35. "max_objects": -1
  36. },
  37. "user_quota": {
  38. "enabled": false,
  39. "max_size_kb": -1,
  40. "max_objects": -1
  41. },
  42. "temp_url_keys": []
  43. }
  • aCgVTx3Gfz1dBiFS4NfjIRmvT0sgpHDP6aa0Yfrh 为子账号 demo:swift 的 secret key;

创建 docker registry

创建 registry 使用的 TLS 证书

  1. $ mdir -p registry/{auth,certs}
  2. $ cat registry-csr.json
  3. {
  4. "CN": "registry",
  5. "hosts": [
  6. "127.0.0.1",
  7. "10.64.3.7"
  8. ],
  9. "key": {
  10. "algo": "rsa",
  11. "size": 2048
  12. },
  13. "names": [
  14. {
  15. "C": "CN",
  16. "ST": "BeiJing",
  17. "L": "BeiJing",
  18. "O": "k8s",
  19. "OU": "System"
  20. }
  21. ]
  22. }
  23. $ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  24. -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  25. -config=/etc/kubernetes/ssl/ca-config.json \
  26. -profile=kubernetes registry-csr.json | cfssljson -bare registry
  27. $ cp registry.pem registry-key.pem registry/certs
  28. $
  • 这里复用以前创建的 CA 证书和秘钥文件;
  • hosts 字段指定 registry 的 NodeIP;

创建 HTTP Baisc 认证文件

  1. $ docker run --entrypoint htpasswd registry:2 -Bbn foo foo123 > auth/htpasswd
  2. $ cat auth/htpasswd
  3. foo:$2y$05$I60z69MdluAQ8i1Ka3x3Neb332yz1ioow2C4oroZSOE0fqPogAmZm

配置 registry 参数

  1. $ export RGW_AUTH_URL="http://10.64.3.9:7480/auth/v1"
  2. $ export RGW_USER="demo:swift"
  3. $ export RGW_SECRET_KEY="aCgVTx3Gfz1dBiFS4NfjIRmvT0sgpHDP6aa0Yfrh"
  4. $ cat > config.yml << EOF
  5. # https://docs.docker.com/registry/configuration/#list-of-configuration-options
  6. version: 0.1
  7. log:
  8. level: info
  9. fromatter: text
  10. fields:
  11. service: registry
  12. storage:
  13. cache:
  14. blobdescriptor: inmemory
  15. delete:
  16. enabled: true
  17. swift:
  18. authurl: ${RGW_AUTH_URL}
  19. username: ${RGW_USER}
  20. password: ${RGW_SECRET_KEY}
  21. container: registry
  22. auth:
  23. htpasswd:
  24. realm: basic-realm
  25. path: /auth/htpasswd
  26. http:
  27. addr: 0.0.0.0:8000
  28. headers:
  29. X-Content-Type-Options: [nosniff]
  30. tls:
  31. certificate: /certs/registry.pem
  32. key: /certs/registry-key.pem
  33. health:
  34. storagedriver:
  35. enabled: true
  36. interval: 10s
  37. threshold: 3
  38. EOF
  • storage.swift 指定后端使用 swfit 接口协议的存储,这里配置的是 ceph rgw 存储参数;
  • auth.htpasswd 指定了 HTTP Basic 认证的 token 文件路径;
  • http.tls 指定了 registry http 服务器的证书和秘钥文件路径;

创建 docker registry

  1. $ docker run -d -p 8000:8000 \
  2. -v $(pwd)/registry/auth/:/auth \
  3. -v $(pwd)/registry/certs:/certs \
  4. -v $(pwd)/config.yml:/etc/docker/registry/config.yml \
  5. --name registry registry:2
  • 执行该 docker run 命令的机器 IP 为 10.64.3.7;

向 registry push image

将签署 registry 证书的 CA 证书拷贝到 /etc/docker/certs.d/10.64.3.7:8000 目录下

  1. $ sudo mkdir -p /etc/docker/certs.d/10.64.3.7:8000
  2. $ sudo cp /etc/kubernetes/ssl/ca.pem /etc/docker/certs.d/10.64.3.7:8000/ca.crt
  3. $

登陆私有 registry

  1. $ docker login 10.64.3.7:8000
  2. Username: foo
  3. Password:
  4. Login Succeeded

登陆信息被写入 ~/.docker/config.json 文件

  1. $ cat ~/.docker/config.json
  2. {
  3. "auths": {
  4. "10.64.3.7:8000": {
  5. "auth": "Zm9vOmZvbzEyMw=="
  6. }
  7. }
  8. }

将本地的 image 打上私有 registry 的 tag

  1. $ docker tag docker.io/kubernetes/pause 10.64.3.7:8000/zhangjun3/pause
  2. $ docker images |grep pause
  3. docker.io/kubernetes/pause latest f9d5de079539 2 years ago 239.8 kB
  4. 10.64.3.7:8000/zhangjun3/pause latest f9d5de079539 2 years ago 239.8 kB

将 image push 到私有 registry

  1. $ docker push 10.64.3.7:8000/zhangjun3/pause
  2. The push refers to a repository [10.64.3.7:8000/zhangjun3/pause]
  3. 5f70bf18a086: Pushed
  4. e16a89738269: Pushed
  5. latest: digest: sha256:9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359 size: 916

查看 ceph 上是否已经有 push 的 pause 容器文件

  1. $ rados lspools
  2. rbd
  3. .rgw.root
  4. default.rgw.control
  5. default.rgw.data.root
  6. default.rgw.gc
  7. default.rgw.log
  8. default.rgw.users.uid
  9. default.rgw.users.keys
  10. default.rgw.users.swift
  11. default.rgw.buckets.index
  12. default.rgw.buckets.data
  13. $ rados --pool default.rgw.buckets.data ls|grep pause
  14. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_layers/sha256/f9d5de0795395db6c50cb1ac82ebed1bd8eb3eefcebb1aa724e01239594e937b/link
  15. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_layers/sha256/f72a00a23f01987b42cb26f259582bb33502bdb0fcf5011e03c60577c4284845/link
  16. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link
  17. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_manifests/tags/latest/current/link
  18. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_manifests/tags/latest/index/sha256/9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359/link
  19. 9c2d5a9d-19e6-4003-90b5-b1cbf15e890d.4310.1_files/docker/registry/v2/repositories/zhangjun3/pause/_manifests/revisions/sha256/9a6b437e896acad3f5a2a8084625fdd4177b2e7124ee943af642259f2f283359/link

私有 registry 的运维操作

查询私有镜像中的 images

  1. $ curl --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/_catalog
  2. {"repositories":["library/redis","zhangjun3/busybox","zhangjun3/pause","zhangjun3/pause2"]}

查询某个镜像的 tags 列表

  1. $ curl --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/tags/list
  2. {"name":"zhangjun3/busybox","tags":["latest"]}

获取 image 或 layer 的 digest

v2/<repoName>/manifests/<tagName> 发 GET 请求,从响应的头部 Docker-Content-Digest 获取 image digest,从响应的 body 的 fsLayers.blobSum 中获取 layDigests;

注意,必须包含请求头:Accept: application/vnd.docker.distribution.manifest.v2+json

  1. $ curl -v -H "Accept: application/vnd.docker.distribution.manifest.v2+json" --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/manifests/latest
  2. > GET /v2/zhangjun3/busybox/manifests/latest HTTP/1.1
  3. > User-Agent: curl/7.29.0
  4. > Host: 10.64.3.7:8000
  5. > Accept: application/vnd.docker.distribution.manifest.v2+json
  6. >
  7. < HTTP/1.1 200 OK
  8. < Content-Length: 527
  9. < Content-Type: application/vnd.docker.distribution.manifest.v2+json
  10. < Docker-Content-Digest: sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5
  11. < Docker-Distribution-Api-Version: registry/2.0
  12. < Etag: "sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5"
  13. < X-Content-Type-Options: nosniff
  14. < Date: Tue, 21 Mar 2017 15:19:42 GMT
  15. <
  16. {
  17. "schemaVersion": 2,
  18. "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  19. "config": {
  20. "mediaType": "application/vnd.docker.container.image.v1+json",
  21. "size": 1465,
  22. "digest": "sha256:00f017a8c2a6e1fe2ffd05c281f27d069d2a99323a8cd514dd35f228ba26d2ff"
  23. },
  24. "layers": [
  25. {
  26. "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
  27. "size": 701102,
  28. "digest": "sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515"
  29. }
  30. ]
  31. }

删除 image

/v2/<name>/manifests/<reference> 发送 DELETE 请求,reference 为上一步返回的 Docker-Content-Digest 字段内容:

  1. $ curl -X DELETE --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/manifests/sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5
  2. $

删除 layer

/v2/<name>/blobs/<digest>发送 DELETE 请求,其中 digest 是上一步返回的 fsLayers.blobSum 字段内容:

  1. $ curl -X DELETE --user zhangjun3:xxx --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
  2. $ curl -X DELETE --cacert /etc/docker/certs.d/10.64.3.7\:8000/ca.crt https://10.64.3.7:8000/v2/zhangjun3/busybox/blobs/sha256:04176c8b224aa0eb9942af765f66dae866f436e75acef028fe44b8a98e045515
  3. $