11.部署 harbor 私有仓库

本文档介绍使用 docker-compose 部署 harbor 私有仓库的步骤，你也可以使用 docker 官方的 registry 镜像部署私有仓库(部署 Docker Registry)。

使用的变量

本文档用到的变量定义如下：

  1. $ export NODE_IP=172.27.136.1 # 当前部署 harbor 的节点 IP
  2. $

下载文件

从 docker compose 发布页面下载最新的 docker-compose 二进制文件

  1. wget https://github.com/docker/compose/releases/download/1.22.0/docker-compose-Linux-x86_64
  2. mv docker-compose-Linux-x86_64 /opt/k8s/bin/docker-compose
  3. chmod a+x  /opt/k8s/bin/docker-compose
  4. export PATH=/opt/k8s/bin:$PATH

从 harbor 发布页面下载最新的 harbor 离线安装包

  1. cd /opt/k8s
  2. wget  --continue https://storage.googleapis.com/harbor-releases/release-1.5.0/harbor-offline-installer-v1.5.2.tgz
  3. tar -xzvf harbor-offline-installer-v1.5.2.tgz

导入 docker images

导入离线安装包中 harbor 相关的 docker images：

  1. cd harbor
  2. docker load -i harbor.v1.5.2.tar.gz

创建 harbor nginx 服务器使用的 x509 证书

创建 harbor 证书签名请求：

  1. cd /opt/k8s/work
  2. cat > harbor-csr.json <<EOF
  3. {
  4.   "CN": "harbor",
  5.   "hosts": [
  6.     "127.0.0.1",
  7.     "172.27.136.1",
  8.     "172.27.136.3",
  9.     "172.27.136.2"
  10.   ],
  11.   "key": {
  12.     "algo": "rsa",
  13.     "size": 2048
  14.   },
  15.   "names": [
  16.     {
  17.       "C": "CN",
  18.       "ST": "BeiJing",
  19.       "L": "BeiJing",
  20.       "O": "k8s",
  21.       "OU": "4Paradigm"
  22.     }
  23.   ]
  24. }
  25. EOF
  • hosts 字段指定授权使用该证书的当前部署节点 IP，如果后续使用域名访问 harbor 则还需要添加域名；

生成 harbor 证书和私钥：

  1. $ cfssl gencert -ca=/opt/k8s/work/ca.pem \
  2.   -ca-key=/opt/k8s/work/ca-key.pem \
  3.   -config=/opt/k8s/work/ca-config.json \
  4.   -profile=kubernetes harbor-csr.json | cfssljson -bare harbor
  6. $ ls harbor*
  7. harbor.csr  harbor-csr.json  harbor-key.pem harbor.pem
  9. $ sudo mkdir -p /etc/harbor/ssl
  10. $ sudo cp harbor*.pem /etc/harbor/ssl

创建一个 harbor 使用的对象存储的账号的 pool

[ceph@m7-common-dfs04 ceph-cluster]$ sudo netstat -lnpt|grep sgw tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 260853/radosgw

[ceph@m7-common-dfs04 ceph-cluster]$ curl http://172.27.128.100 <?xml version=”1.0” encoding=”UTF-8”?>anonymous

$ radosgw-admin user create —uid=zhangjun —display-name=”ceph rgw by zhangjun” { “user_id”: “zhangjun”, “display_name”: “ceph rgw by zhangjun”, “email”: “”, “suspended”: 0, “max_buckets”: 1000, “auid”: 0, “subusers”: [], “keys”: [ { “user”: “zhangjun”, “access_key”: “LX3BDCVRQONMEYDGICQZ”, “secret_key”: “JBXxxcTn32zAZ8djFcQsFMp5afOrQtKd6g68kkTj” } ], “swift_keys”: [], “caps”: [], “op_mask”: “read, write, delete”, “default_placement”: “”, “placement_tags”: [], “bucket_quota”: { “enabled”: false, “check_on_raw”: false, “max_size”: -1, “max_size_kb”: 0, “max_objects”: -1 }, “user_quota”: { “enabled”: false, “check_on_raw”: false, “max_size”: -1, “max_size_kb”: 0, “max_objects”: -1 }, “temp_url_keys”: [], “type”: “rgw” }

$ radosgw-admin subuser create —uid zhangjun —subuser=zhangjun:swift —access=full —secret=secretkey —key-type=swift { “user_id”: “zhangjun”, “display_name”: “ceph rgw by zhangjun”, “email”: “”, “suspended”: 0, “max_buckets”: 1000, “auid”: 0, “subusers”: [ { “id”: “zhangjun:swift”, “permissions”: “full-control” } ], “keys”: [ { “user”: “zhangjun”, “access_key”: “LX3BDCVRQONMEYDGICQZ”, “secret_key”: “JBXxxcTn32zAZ8djFcQsFMp5afOrQtKd6g68kkTj” } ], “swift_keys”: [ { “user”: “zhangjun:swift”, “secret_key”: “secretkey” } ], “caps”: [], “op_mask”: “read, write, delete”, “default_placement”: “”, “placement_tags”: [], “bucket_quota”: { “enabled”: false, “check_on_raw”: false, “max_size”: -1, “max_size_kb”: 0, “max_objects”: -1 }, “user_quota”: { “enabled”: false, “check_on_raw”: false, “max_size”: -1, “max_size_kb”: 0, “max_objects”: -1 }, “temp_url_keys”: [], “type”: “rgw” }

$ radosgw-admin key create —subuser=zhangjun:swift —key-type=swift —gen-secret { “user_id”: “zhangjun”, “display_name”: “ceph rgw by zhangjun”, “email”: “”, “suspended”: 0, “max_buckets”: 1000, “auid”: 0, “subusers”: [ { “id”: “zhangjun:swift”, “permissions”: “full-control” } ], “keys”: [ { “user”: “zhangjun”, “access_key”: “LX3BDCVRQONMEYDGICQZ”, “secret_key”: “JBXxxcTn32zAZ8djFcQsFMp5afOrQtKd6g68kkTj” } ], “swift_keys”: [ { “user”: “zhangjun:swift”, “secret_key”: “OcOkMZFwSxLAPzBKTkesoM4M3CW7DxMRVSWxNtUb” } ], “caps”: [], “op_mask”: “read, write, delete”, “default_placement”: “”, “placement_tags”: [], “bucket_quota”: { “enabled”: false, “check_on_raw”: false, “max_size”: -1, “max_size_kb”: 0, “max_objects”: -1 }, “user_quota”: { “enabled”: false, “check_on_raw”: false, “max_size”: -1, “max_size_kb”: 0, “max_objects”: -1 }, “temp_url_keys”: [], “type”: “rgw” }

修改 harbor.cfg 文件

  1. $ cp harbor.cfg{,.bak}
  2. $ vim harbor.cfg
  3. $ diff harbor.cfg{,.bak}
  4. 7c7
  5. < hostname = 172.27.129.81
  6. ---
  7. > hostname = reg.mydomain.com
  8. 11c11
  9. < ui_url_protocol = https
  10. ---
  11. > ui_url_protocol = http
  12. 23,24c23,24
  13. < ssl_cert =  /etc/harbor/ssl/harbor.pem
  14. < ssl_cert_key = /etc/harbor/ssl/harbor-key.pem
  15. ---
  16. > ssl_cert = /data/cert/server.crt
  17. > ssl_cert_key = /data/cert/server.key
  19. $ cp prepare{,.bak}
  20. $ vim prepare
  21. $ diff prepare{,.bak}
  22. 453a454
  23. >         print("%s %w", args, kw)
  24. 490c491
  25. <     empty_subj = "/"
  26. ---
  27. >     empty_subj = "/C=/ST=/L=/O=/CN=/"

  • 需要修改 prepare 脚本的 empyt_subj 参数，否则后续 install 时出错退出：

    Fail to generate key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt

参考：https://github.com/vmware/harbor/issues/2920

加载和启动 harbor 镜像

  1. $ sudo mkdir /data
  2. $ sudo chmod 777 /var/run/docker.sock /data
  3. $ sudo apt-get install python
  4. $ ./install.sh --with-clair 
  6. [Step 0]: checking installation environment ...
  8. Note: docker version: 18.03.0
  10. Note: docker-compose version: 1.22.0
  12. [Step 1]: loading Harbor images ...
  13. Loaded image: vmware/clair-photon:v2.0.1-v1.5.2
  14. Loaded image: vmware/postgresql-photon:v1.5.2
  15. Loaded image: vmware/harbor-adminserver:v1.5.2
  16. Loaded image: vmware/registry-photon:v2.6.2-v1.5.2
  17. Loaded image: vmware/photon:1.0
  18. Loaded image: vmware/harbor-migrator:v1.5.2
  19. Loaded image: vmware/harbor-ui:v1.5.2
  20. Loaded image: vmware/redis-photon:v1.5.2
  21. Loaded image: vmware/nginx-photon:v1.5.2
  22. Loaded image: vmware/mariadb-photon:v1.5.2
  23. Loaded image: vmware/notary-signer-photon:v0.5.1-v1.5.2
  24. Loaded image: vmware/harbor-log:v1.5.2
  25. Loaded image: vmware/harbor-db:v1.5.2
  26. Loaded image: vmware/harbor-jobservice:v1.5.2
  27. Loaded image: vmware/notary-server-photon:v0.5.1-v1.5.2
  30. [Step 2]: preparing environment ...
  31. loaded secret from file: /data/secretkey
  32. Generated configuration file: ./common/config/nginx/nginx.conf
  33. Generated configuration file: ./common/config/adminserver/env
  34. Generated configuration file: ./common/config/ui/env
  35. Generated configuration file: ./common/config/registry/config.yml
  36. Generated configuration file: ./common/config/db/env
  37. Generated configuration file: ./common/config/jobservice/env
  38. Generated configuration file: ./common/config/jobservice/config.yml
  39. Generated configuration file: ./common/config/log/logrotate.conf
  40. Generated configuration file: ./common/config/jobservice/config.yml
  41. Generated configuration file: ./common/config/ui/app.conf
  42. Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
  43. The configuration files are ready, please use docker-compose to start the service.
  46. [Step 3]: checking existing instance of Harbor ...
  49. [Step 4]: starting Harbor ...
  50. Creating network "harbor_harbor" with the default driver
  51. Creating harbor-log ... done
  52. Creating redis              ... done
  53. Creating harbor-adminserver ... done
  54. Creating harbor-db          ... done
  55. Creating registry           ... done
  56. Creating harbor-ui          ... done
  57. Creating harbor-jobservice  ... done
  58. Creating nginx              ... done
  60. ✔ ----Harbor has been installed and started successfully.----
  62. Now you should be able to visit the admin portal at https://172.27.129.81.
  63. For more details, please visit https://github.com/vmware/harbor .

访问管理界面

确认所有组件都工作正常：

  1. $ docker-compose  ps
  2.        Name                     Command                  State                                    Ports
  3. -------------------------------------------------------------------------------------------------------------------------------------
  4. harbor-adminserver   /harbor/start.sh                 Up (healthy)
  5. harbor-db            /usr/local/bin/docker-entr ...   Up (healthy)   3306/tcp
  6. harbor-jobservice    /harbor/start.sh                 Up
  7. harbor-log           /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
  8. harbor-ui            /harbor/start.sh                 Up (healthy)
  9. nginx                nginx -g daemon off;             Up (healthy)   0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->80/tcp
  10. redis                docker-entrypoint.sh redis ...   Up             6379/tcp
  11. registry             /entrypoint.sh serve /etc/ ...   Up (healthy)   5000/tcp

浏览器访问 https://${NODE_IP}，示例的是 https://172.27.129.81；

由于是在 virtualbox 虚机 m7-demo-136002 中运行，所以需要做下端口转发，Vagrant 文件中已经指定 host 端口为 4443，也可以在 virtualbox 的 GUI 中直接添加端口转发：

virtualbox-harbor

浏览器访问 https://127.0.0.1:443，用账号 admin 和 harbor.cfg 配置文件中的默认密码 Harbor12345 登陆系统。

harbor

harbor 运行时产生的文件、目录

harbor 将日志打印到 /var/log/harbor 的相关目录下，使用 docker logs XXX 或 docker-compose logs XXX 将看不到容器的日志。

  1. $ # 日志目录
  2. $ ls /var/log/harbor
  3. adminserver.log  jobservice.log  mysql.log  proxy.log  registry.log  ui.log
  4. $ # 数据目录，包括数据库、镜像仓库
  5. $ ls /data/
  6. ca_download  config  database  job_logs registry  secretkey

docker 客户端登陆

将签署 harbor 证书的 CA 证书拷贝到 /etc/docker/certs.d/172.27.129.81 目录下

  1. $ sudo mkdir -p /etc/docker/certs.d/172.27.129.81
  2. $ sudo cp /opt/k8s/work/ca.pem /etc/docker/certs.d/172.27.129.81/ca.crt
  3. $

登陆 harbor

  1. $ docker login 172.27.129.81
  2. Username: admin
  3. Password:

认证信息自动保存到 ~/.docker/config.json 文件。

其它操作

下列操作的工作目录均为 解压离线安装文件后 生成的 harbor 目录。

  1. $ # 停止 harbor
  2. $ docker-compose down -v
  3. $ # 修改配置
  4. $ vim harbor.cfg
  5. $ # 更修改的配置更新到 docker-compose.yml 文件
  6. $ ./prepare
  7. Clearing the configuration file: ./common/config/ui/app.conf
  8. Clearing the configuration file: ./common/config/ui/env
  9. Clearing the configuration file: ./common/config/ui/private_key.pem
  10. Clearing the configuration file: ./common/config/db/env
  11. Clearing the configuration file: ./common/config/registry/root.crt
  12. Clearing the configuration file: ./common/config/registry/config.yml
  13. Clearing the configuration file: ./common/config/jobservice/app.conf
  14. Clearing the configuration file: ./common/config/jobservice/env
  15. Clearing the configuration file: ./common/config/nginx/cert/admin.pem
  16. Clearing the configuration file: ./common/config/nginx/cert/admin-key.pem
  17. Clearing the configuration file: ./common/config/nginx/nginx.conf
  18. Clearing the configuration file: ./common/config/adminserver/env
  19. loaded secret from file: /data/secretkey
  20. Generated configuration file: ./common/config/nginx/nginx.conf
  21. Generated configuration file: ./common/config/adminserver/env
  22. Generated configuration file: ./common/config/ui/env
  23. Generated configuration file: ./common/config/registry/config.yml
  24. Generated configuration file: ./common/config/db/env
  25. Generated configuration file: ./common/config/jobservice/env
  26. Generated configuration file: ./common/config/jobservice/app.conf
  27. Generated configuration file: ./common/config/ui/app.conf
  28. Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
  29. The configuration files are ready, please use docker-compose to start the service.
  30. $ sudo chmod -R 666 common ## 防止容器进程没有权限读取生成的配置
  31. $ # 启动 harbor
  32. $ docker-compose up -d