harbor (docker warehouse) warehouse deployment

1. Harbor overview

Although Docker officially provides a public image warehouse, from the aspects of security and efficiency, it is also very necessary to deploy the Registry in our private environment.

Harbor is an enterprise-level Docker Registry management project open sourced by VMware. Compared with official docker, it has richer permissions and complete architectural design, and is suitable for large-scale docker cluster deployment to provide warehouse services.

It mainly provides the Dcoker Registry management interface UI, which can provide role-based access control, image replication, AD/LDAP integration, log audit and other functions, and fully supports Chinese.

2. Main functions of harbor

• Role-based access control
  Users and Docker image repositories are organized and managed through "projects". A user can have different permissions on multiple image repositories in the same namespace (project).

• Mirror-based replication strategy
  Mirrors can be replicated in multiple Registry instances (the mirrors in the warehouse can be synchronized to the remote Harbor, similar to the MySQL master-slave synchronization function), especially suitable for load balancing, high availability, hybrid cloud and multi-cloud scene.

• Graphical user interface
  Users can browse through the browser, retrieve the current Docker image warehouse, and manage projects and namespaces.

• Supports AD/LDAP
  Harbor can integrate existing AD/LDAP within the enterprise for authentication management.

• Image deletion and garbage collection
  Harbor supports deleting images on the Web, recycling useless images, and freeing up disk space. The image can be deleted and the space occupied by the image reclaimed.

• Audit Management
  ​ All operations on the image warehouse can be recorded and traced for audit management.

• RESTful API
  RESTful API provides administrators with more control over Harbor, making it easier to integrate with other management software.

• Simple deployment.
  Online and offline installation tools are provided. It can also be installed on the vSphere platform (OVA method) virtual device.

3. Introduction to harbor architecture

Architecture component diagram:

As described in the above figure, Harbor consists of 6 large modules:

• 1. Proxy: Harbor's registry, UI, token services and other components are all behind a reverse proxy. The proxy forwards requests from browsers and docker clients to the backend service.

• 2. Registry: Responsible for storing Docker images and processing Docker push/pull requests. Because Harbor enforces permission control on access to the image, the Registry will force the client to obtain a valid token from the token service during each push/pull request.

• 3. Core services: Harbor’s core functions mainly include the following three services:

  • UI: graphical interface
  • WebHook: Get the status changes of the image on the registry in a timely manner, configure webhook on the registry, and pass the status changes to the UI module.
  • Token service: Responsible for issuing tokens for each docker push/pull command based on user permissions. If the request initiated by the Docker client to the Registry service does not contain a token, it will be redirected here. After obtaining the token, the request will be made to the Registry again.

• 4. Database: Provides database services for core services and is responsible for storing user permissions, audit logs, Docker image grouping information and other data.

• 5. Job services: Mainly used for image replication. Local images can be synchronized to remote Harbor instances.

• 6. Log collector: Responsible for collecting logs from other components to one place

It is very difficult to deploy Harbor on a physical machine. In order to simplify the application of Harbor, Harbor officials directly made Harbor an application that runs in a container, and this container relies on many storage systems such as redis, mysql, pgsql, etc. in Harbor. Therefore, it needs to orchestrate many containers to work together. Therefore, when deploying and using VMWare Harbor, it needs to be Docker composeimplemented with the help of Docker's stand-alone orchestration tool ( ).

4. Docker Compose

Docker Compose official document
CHAPTER 12 Compose (1)

4.1 Introduction to compose

The Docker-Compose project is Docker's official open source project, responsible for realizing the rapid orchestration of Docker container clusters.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you can use YML files to configure all the services your application requires. Then, using a single command, you can create and start all services from the YML file configuration.

4.2 docker-compose tool command

docker-compose up	//启动所有服务，默认读取名为docker-compose.yml文件
docker-compose up -d	//在后台启动所有服务

docker-compose build 	//在运行项目前先构建项目所需镜像

docker-compose ps	//列出本地docker-compose.yml文件里定义的正在运行的所有服务

docker-compose stop	//停止所有服务，如果服务没有停止，可以使用docker-compose kill强制杀死服务

docker-compose start	//开启所有服务

docker-compose rm	//删除所有服务

5. harbor deployment

Harbor official document
Harbor download site

Things to note when using Harbor:

• When uploading an image on the client, you must remember to execute docker login for user authentication, otherwise you cannot push directly.
• If you are not using https when using the client, you must configure the insecure-registries parameter in the client's /etc/docker/daemon.json configuration file.
• The data storage path should be configured in the configuration file to a shared storage with sufficient capacity.
• Harbor is managed using the docker-compose command. If you need to stop Harbor, you can also use docker-compose stop to stop it. Please use –help for other parameters.

5.1 Deploy Docker Compose

This time, binary deployment docker compose is used
to download the address: docker compose

[root@node-253 opt]# docker -v
Docker version 19.03.9, build 9d988398e7
[root@node-253 opt]# mv docker-compose-linux-x86_64 docker-compose
[root@node-253 opt]# cp docker-compose /usr/local/bin/
cp: overwrite ‘/usr/local/bin/docker-compose’? y
[root@node-253 opt]# chmod a+x /usr/local/bin/docker-compose
[root@node-253 opt]# docker-compose -v
Docker Compose version v2.17.2

5.2 Deploy harbor

//下载harbor压缩包
[root@harbor_node01 ~]# wget https://github.com/goharbor/harbor/releases/download/v2.4.3/harbor-offline-installer-v2.4.3.tgz
//把harbor解压至/usr/local目录下
[root@harbor_node01 ~]# tar xf harbor-offline-installer-v2.4.3.tgz -C /usr/local/
//进入harbor目录
[root@harbor_node01 ~]# cd /usr/local/harbor/
[root@harbor_node01 harbor]# ls
common.sh  harbor.v2.4.3.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare
//把配置文件模板拷贝一份出来
root@harbor_node01 harbor]# cp harbor.yml.tmpl harbor.yml
//查看当前主机的主机名全称
[root@harbor_node01 harbor]# hostname
harbor_node01.example.com
[root@harbor_node01 harbor]# vim harbor.yml
..........
hostname: harbor_node01.example.com		//这里的主机名换成本机的主机名或IP地址
.........
#https:									//把https这行及子行都注释掉，因为没有弄ssl证书加密
  # https port for harbor, default is 443
  #port: 443
  # The path of cert and key files for nginx
  #certificate: /your/certificate/path
  #private_key: /your/private/key/path
.........
//启动harbor
[root@harbor_node01 harbor]# ./install.sh
..........启动过程略.............
✔ ----Harbor has been installed and started successfully.----	 //显示此行说明启动成功！
//查看harbor运行起来的容器
[root@harbor_node01 harbor]# docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED              STATUS                    PORTS                                   NAMES
1602065b6558   goharbor/nginx-photon:v2.4.3         "nginx -g 'daemon of…"   About a minute ago   Up 51 seconds (healthy)   0.0.0.0:80->8080/tcp, :::80->8080/tcp   nginx
807abe81e1e8   goharbor/harbor-jobservice:v2.4.3    "/harbor/entrypoint.…"   About a minute ago   Up 52 seconds (healthy)                                           harbor-jobservice
525886fff9b0   goharbor/harbor-core:v2.4.3          "/harbor/entrypoint.…"   About a minute ago   Up 53 seconds (healthy)                                           harbor-core
9d40e1cf4131   goharbor/harbor-db:v2.4.3            "/docker-entrypoint.…"   About a minute ago   Up 54 seconds (healthy)                                           harbor-db
e7405e5ba6b1   goharbor/registry-photon:v2.4.3      "/home/harbor/entryp…"   About a minute ago   Up 54 seconds (healthy)                                           registry
a86ae3d1848e   goharbor/redis-photon:v2.4.3         "redis-server /etc/r…"   About a minute ago   Up 54 seconds (healthy)                                           redis
814e2bfc178e   goharbor/harbor-registryctl:v2.4.3   "/home/harbor/start.…"   About a minute ago   Up 54 seconds (healthy)                                           registryctl
340de1da037f   goharbor/harbor-portal:v2.4.3        "nginx -g 'daemon of…"   About a minute ago   Up 54 seconds (healthy)                                           harbor-portal
c3eceadcb92b   goharbor/harbor-log:v2.4.3           "/bin/sh -c /usr/loc…"   About a minute ago   Up 59 seconds (healthy)   127.0.0.1:1514->10514/tcp               harbor-log
//查看启用的端口
[root@harbor_node01 harbor]# ss -anlt
State       Recv-Q       Send-Q             Local Address:Port             Peer Address:Port      Process
LISTEN      0            128                    127.0.0.1:1514                  0.0.0.0:*
LISTEN      0            128                      0.0.0.0:80                    0.0.0.0:*
LISTEN      0            128                      0.0.0.0:22                    0.0.0.0:*
LISTEN      0            128                         [::]:80                       [::]:*
LISTEN      0            128                         [::]:22                       [::]:*

Browser access: The username and password can harbor.ymlbe found in the configuration file. If you change the password, change it here
admin/Harbor123456

5.3 Deploy harbor to start automatically at boot

[root@harbor_node01 ~]# vim /etc/rc.local
#!/bin/bash
cd /usr/local/harbor		//添加这行与下面一行
docker-compose start
[root@harbor_node01 ~]# ll /etc/rc.local
lrwxrwxrwx. 1 root root 13 Oct  5  2021 /etc/rc.local -> rc.d/rc.local
[root@harbor_node01 ~]# ll /etc/rc.d/rc.local
-rw-r--r--. 1 root root 516 Aug 11 21:17 /etc/rc.d/rc.local		//默认该文件无执行权限，需添加执行权限
//添加执行权限，否则该文件无法生效
[root@harbor_node01 ~]# chmod +x /etc/rc.d/rc.local
//重启主机验证配置的开机自启harbor的效果
[root@harbor_node01 ~]# reboot

//重启之后的效果，可以看到harbor开启成功
[root@harbor_node01 ~]# docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED       STATUS                   PORTS                                   NAMES
1602065b6558   goharbor/nginx-photon:v2.4.3         "nginx -g 'daemon of…"   2 hours ago   Up 2 minutes (healthy)   0.0.0.0:80->8080/tcp, :::80->8080/tcp   nginx
807abe81e1e8   goharbor/harbor-jobservice:v2.4.3    "/harbor/entrypoint.…"   2 hours ago   Up 2 minutes (healthy)                                           harbor-jobservice
525886fff9b0   goharbor/harbor-core:v2.4.3          "/harbor/entrypoint.…"   2 hours ago   Up 2 minutes (healthy)                                           harbor-core
9d40e1cf4131   goharbor/harbor-db:v2.4.3            "/docker-entrypoint.…"   2 hours ago   Up 2 minutes (healthy)                                           harbor-db
e7405e5ba6b1   goharbor/registry-photon:v2.4.3      "/home/harbor/entryp…"   2 hours ago   Up 2 minutes (healthy)                                           registry
a86ae3d1848e   goharbor/redis-photon:v2.4.3         "redis-server /etc/r…"   2 hours ago   Up 2 minutes (healthy)                                           redis
814e2bfc178e   goharbor/harbor-registryctl:v2.4.3   "/home/harbor/start.…"   2 hours ago   Up 2 minutes (healthy)                                           registryctl
340de1da037f   goharbor/harbor-portal:v2.4.3        "nginx -g 'daemon of…"   2 hours ago   Up 2 minutes (healthy)                                           harbor-portal
c3eceadcb92b   goharbor/harbor-log:v2.4.3           "/bin/sh -c /usr/loc…"   2 hours ago   Up 2 minutes (healthy)   127.0.0.1:1514->10514/tcp               harbor-log

5.4 harbor installation script

ip=10.48.14.50
 
curl -L "https://get.daocloud.io/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version
 
mkdir /usr/local/harbor && cd /usr/local/harbor
wget --continue https://storage.googleapis.com/harbor-releases/release-1.8.0/harbor-online-installer-v1.8.1.tgz
tar -zxf harbor-online-installer-v1.8.1.tgz
cd harbor
sed -i 's/hostname: reg.mydomain.com/hostname: '$ip'/g' harbor.yml
./prepare
./install.sh
docker-compose ps
 
#配置免https验证
cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://uvbz951q.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "insecure-registries": ["10.48.14.50:8888"]
}
EOF
 
systemctl daemon-reload
systemctl restart docker
docker-compose stop
docker-compose up -d

5.5 Harbor core services are unavailable

报错：pgsql.conf permission denied，failed to connect to tcp://postgresql:5432

#harbor报错时访问下面路径，根据日志信息分析排错。
[root@node4 database]# cd /var/log/harbor/
[root@node4 harbor]# ls
core.log  jobservice.log  portal.log  postgresql.log  proxy.log  redis.log  registryctl.log  registry.log

solve

[root@node4 ~]# cd /data/database/
[root@node4 database]# ls
pg13

chown -R polkitd:input pg13
chmod -R 700 pg13

#如果还不行则重启harbor所有服务
cd /harbor-haproxy-sslFile
docker-compose down
./prepare
docker-compose up -d

6. Harbor application

6.1 Configure client to access harbor

Enable another docker host as a client to access harbor

//将harbor服务端的主机域名进行映射
[root@localhost ~]# vim /etc/hosts
192.168.71.253	node-253 //添加harbor服务端的IP地址与主机名
//测试是否能与harbor端ping通
[root@localhost ~]# ping -c 2 node-253
PING harbor_node01.example.com (192.168.92.130) 56(84) bytes of data.
64 bytes from harbor_node01.example.com (192.168.92.130): icmp_seq=1 ttl=64 time=0.322 ms
64 bytes from harbor_node01.example.com (192.168.92.130): icmp_seq=2 ttl=64 time=0.696 ms

--- harbor_node01.example.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1028ms
rtt min/avg/max/mdev = 0.322/0.509/0.696/0.187 ms
//使用insecure-registries参数添加http支持
[root@node-252 ~]# cat /etc/docker/daemon.json
{
    
    
  "registry-mirrors": ["https://3s9106.mirror.alncs.com"],
  "insecure-registries": ["192.168.71.253"]
}
[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart docker

//登录到harbor
[root@localhost ~]# docker login 192.168.71.253
Username: admin		//输入harbor的用户名
Password:			//输入harbor的密码
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
//修改镜像名与标签，把该镜像上传至harbor的library项目里
[root@localhost ~]# docker images
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
busybox      latest    beae173ccac6   7 months ago   1.24MB
[root@localhost ~]# docker tag busybox:latest 192.168.92.130/library/busybox:v0.1
[root@localhost ~]# docker images
REPOSITORY                                  TAG       IMAGE ID       CREATED        SIZE
busybox                                     latest    beae173ccac6   7 months ago   1.24MB
192.168.92.130/library/busybox   v0.1      beae173ccac6   7 months ago   1.24MB
[root@localhost ~]# docker push 192.168.92.130/library/busybox:v0.1
The push refers to repository [192.168.92.130/library/busybox]
01fd6df81c8e: Pushed
v0.1: digest: sha256:62ffc2ed7554e4c6d360bce40bbcf196573dd27c4ce080641a2c59867e732dee size: 527

6.2 Harbor’s web management interface

On Harbor's web management page, enter the library project and check whether the upload is successful. Here we can create projects and manage the images in the project.

Click [Push Command] to see the command syntax of various container services for official push images.

Click [User Management], and then [Create User] and modify The user authorizes or resets the password or deletes the user.

After adding the user, add it to the project

. Fill in the user name and grant permissions (role), and set the permissions to developer.

As shown in the picture below, you can see that the user has joined this project. admin is the administrator authority, and yurq is the developer.

Currently, I am an admin user and have administrator rights for this project. I can delete this project and add or remove users from the project.

Click [Warehouse Management] on the left, and then you can [New Target] to remotely manage other warehouse services. The optional warehouse service types are as follows. Next,

add a warehouse service. Select the warehouse type as [DOcker Hub] and specify the warehouse name [Docker Hub]. Since it is the official warehouse of docker, the URL does not need to be filled in manually. The access id and password can be filled in with the user name and password registered in Docker Hub. . Since this is an official repository, you can check [Verify remote certificate]. If the remote instance uses a 'self-signed' or 'untrusted certificate', please uncheck this option.

7. K8s containerd docking with harbor

In the newer version of K8s, the container engine containerd is replaced. Next, we configure containerd to connect to harbor.

Harbor recommends configuring https. If https is not configured, it is not easy to connect at present.

7.1 Harbor self-signed certificate configuration https

[root@node-134 harbor]# openssl genrsa -out ca.key 4096
[root@node-134 harbor]# openssl req -x509 -new -nodes -sha512 -days 3650  -subj "/CN=harbor.yh.com"  -key ca.key  -out ca.crt
[root@node-134 harbor]# openssl genrsa -out server.key 4096
[root@node-134 harbor]# openssl req  -new -sha512  -subj "/CN=harbor.yh.com"  -key server.key  -out  server.csr
[root@node-134 harbor]# cat certs/v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment,dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=harbor.yh.com
[root@node-134 harbor]# pwd
/usr/local/harbor
[root@node-134 harbor]# openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in server.csr -out server.crt
Signature ok
subject=/CN=harbor.yh.com
Getting CA Private Key
[root@node-134 harbor]# ll certs/
total 28
-rw-r--r-- 1 root root 1797 Jul 25 23:54 ca.crt
-rw-r--r-- 1 root root 3243 Jul 25 23:54 ca.key
-rw-r--r-- 1 root root   17 Jul 25 23:55 ca.srl
-rw-r--r-- 1 root root 1834 Jul 25 23:55 server.crt
-rw-r--r-- 1 root root 1590 Jul 25 23:54 server.csr
-rw-r--r-- 1 root root 3243 Jul 25 23:54 server.key
-rw-r--r-- 1 root root  237 Jul 25 23:16 v3.ext

$vim harbor.yml 
hostname: harbor.yh.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 8080

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/server/harbor/certs/server.crt
  private_key: /data/server/harbor/certs/server.key
  
./prepare 
docker-compose down
docker-compose up -d
ss -lntup|egrep '80|443'
tcp    LISTEN     0      128    [::]:80                 [::]:*                   users:(("docker-proxy",pid=130988,fd=4))
tcp    LISTEN     0      128    [::]:443                [::]:*                   users:(("docker-proxy",pid=130962,fd=4))

harbor.yml

[root@node-134 harbor]# egrep -v "^ *#|^$" harbor.yml
hostname: harbor.yh.com
http:
  port: 80
https:
  port: 443
  certificate: /usr/local/harbor/certs/server.crt
  private_key: /usr/local/harbor/certs/server.key
harbor_admin_password: Harbor12345
database:
  password: root123
  max_idle_conns: 100
  max_open_conns: 900
data_volume: /data
trivy:
  ignore_unfixed: false
  skip_update: false
  offline_scan: false
  insecure: false
jobservice:
  max_job_workers: 10
notification:
  webhook_job_max_retry: 10
chart:
  absolute_url: disabled
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
_version: 2.4.0
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

#查看证书是否过期
[root@node-128 home]# openssl x509 -in /etc/ssl/certs/server.crt  -noout -dates
notBefore=Jul 26 05:55:47 2023 GMT
notAfter=Jul 23 05:55:47 2033 GMT

7.2 containerd docking with harbor (operating on all nodes)

• nerdctl installation

[21:29:54 root@c7-3~]#containerd --version
containerd containerd.io 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
wget https://github.com/containerd/nerdctl/releases/download/v0.22.2/nerdctl-0.22.2-linux-amd64.tar.gz
tar -zxvf nerdctl-0.22.2-linux-amd64.tar.gz
mv nerdctl /usr/local/bin

• Install CNI plug-in

wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
mkdir -p /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin/

• docking harbor

The server.crt here is copied from the certificate production machine

scp node-134:/usr/local/harbor/certs/server.crt .
yum install -y ca-certificates
cp -a server.crt /etc/pki/ca-trust/source/anchors/ 
ln -s /etc/pki/ca-trust/source/anchors/server.crt /etc/ssl/certs/
update-ca-trust    
[root@node-128 home]# nerdctl login -u admin harbor.yh.com
Enter Password:
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded