Harbor Mirror private warehouse (under)
7. Harbor HA: Environment and ready
- Install two new harbor, but do not worry execute ./install (if you want to clean up under the old data)
- Harbor availability to do, we need to prepare an NFS server to share two Harbor data storage directory, in a production environment, we can choose Ceph or Glusterfs
- We need a Postgres database vulnerability scanning component Harbor to clair of use
- We need a MySQL database to use Harbor
- We need a redis database to use session
| CPU name | IP | use | VIP |
|---|---|---|---|
| Harbor-master | 192.168.200.16 | Harbor Mirror warehouse - Main | 192.168.200.20 |
| Harbor-slave | 192.168.200.18 | Mirror Harbor warehouse - Preparation of | |
| Docker-client | 192.168.200.17 | NFS server (Harbor HA file storage) | |
| LDNS | 192.168.200.19 | DNS server |
7.1 nfs server set up on Docker-client
[root@Docker-client ~]# mkdir -p /data/nfs
[root@Docker-client ~]# yum -y install nfs-utils
[root@Docker-client ~]# rpm -qa nfs-utils
nfs-utils-1.3.0-0.61.el7.x86_64
[root@Docker-client ~]# cat /etc/exports
/data/nfs 192.168.200.0/24(rw,no_root_squash)
[root@Docker-client ~]# systemctl start nfs7.2 Creating a data directory mounted on the harbor in the primary and standby, and install nfs node support package
#harbor主备都进行如下操作
[root@Harbor-master ~]# yum -y install nfs-utils
[root@Harbor-master ~]# rpm -qa nfs-utils
nfs-utils-1.3.0-0.61.el7.x86_64
[root@Harbor-master ~]# mkdir -p /data/storage
[root@Harbor-master ~]# mount 192.168.200.17:/data/nfs /data/storage
[root@Harbor-master ~]# df -hT | grep /data/nfs
192.168.200.17:/data/nfs nfs4 17G 2.8G 15G 17% /data/storage
[root@Harbor-slave ~]# df -hT | grep /data/nfs
192.168.200.17:/data/nfs nfs4 17G 2.8G 15G 17% /data/storage7.3 redis start a container docker-client (NFS server)
#下载一个redis镜像（alpine系统目前docker领域很火，因为它容量很小，比centos小很多）
[root@Docker-client ~]# docker pull redis:alpine
[root@Docker-client ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mongo latest 9c02a5a12c52 41 hours ago 413MB
www.yunjisuan.com/library/mongo latest 9c02a5a12c52 41 hours ago 413MB
redis alpine d975eaec5f68 13 days ago 51.1MB
#启动redis镜像，映射端口
[root@Docker-client ~]# docker run -dit --name redis_test -p 6379:6379 redis:alpine
448de2a11cf1677c20e7280301ce869d878c2a0a6627019082e44cc337a6d71f
[root@Docker-client ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
448de2a11cf1 redis:alpine "docker-entrypoint.s…" 5 seconds ago Up 3 seconds 0.0.0.0:6379->6379/tcp redis_test
7.4 postgreSQL start a database container docker-client (NFS server)
#下载postgres
[root@Docker-client ~]# docker pull postgres
[root@Docker-client ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mongo latest 9c02a5a12c52 43 hours ago 413MB
www.yunjisuan.com/library/mongo latest 9c02a5a12c52 43 hours ago 413MB
postgres latest 53912975086f 7 days ago 312MB
redis alpine d975eaec5f68 13 days ago 51.1MB[root@Docker-client ~]# docker run -dit --name postgres_test -p 5432:5432 -e POSTGRES_PASSWORD=123123 postgres
04c883f32fdc8fffb6c9f90539a0093ffb302cbb9d2ec4c4bcb73b90133d3952
[root@Docker-client ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
04c883f32fdc postgres "docker-entrypoint.s…" 12 seconds ago Up 11 seconds 0.0.0.0:5432->5432/tcp postgres_test
448de2a11cf1 redis:alpine "docker-entrypoint.s…" 2 hours ago Exited (0) 52 minutes ago redis_test7.5 start a MySQL database container docker-client (NFS) server
#下载MySQL5.6版镜像
[root@Docker-client ~]# docker pull mysql:5.6
[root@Docker-client ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mongo latest 9c02a5a12c52 43 hours ago 413MB
www.yunjisuan.com/library/mongo latest 9c02a5a12c52 43 hours ago 413MB
mysql 5.6 7b01f1418bd7 2 days ago 256MB
postgres latest 53912975086f 7 days ago 312MB
redis alpine d975eaec5f68 13 days ago 51.1MB#启动MySQL容器，并映射端口
[root@Docker-client ~]# docker run -dit --name mysql_test -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123123 mysql:5.6 --character-set-server=utf8
bfe4d57f424e27e48553a735aee8e2e1f0d65dc51691069db43bc92986ca4b70
[root@Docker-client ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bfe4d57f424e mysql:5.6 "docker-entrypoint.s…" 11 seconds ago Up 10 seconds 0.0.0.0:3306->3306/tcp mysql_test
04c883f32fdc postgres "docker-entrypoint.s…" 5 minutes ago Up 5 minutes 0.0.0.0:5432->5432/tcp postgres_test
448de2a11cf1 redis:alpine "docker-entrypoint.s…" 2 hours ago Up 1 seconds 0.0.0.0:6379->6379/tcp session7.6 Finally, in accordance with the purpose, we are about to change the name of the database
[root@Docker-client ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bfe4d57f424e mysql:5.6 "docker-entrypoint.s…" 50 seconds ago Up 49 seconds 0.0.0.0:3306->3306/tcp mysql_test
04c883f32fdc postgres "docker-entrypoint.s…" 6 minutes ago Up 6 minutes 0.0.0.0:5432->5432/tcp postgres_test
448de2a11cf1 redis:alpine "docker-entrypoint.s…" 2 hours ago Exited (0) 58 minutes ago redis_test
[root@Docker-client ~]# docker rename postgres_test clair_db
[root@Docker-client ~]# docker rename mysql_test harbor_db
[root@Docker-client ~]# docker rename redis_test session
[root@Docker-client ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bfe4d57f424e mysql:5.6 "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:3306->3306/tcp harbor_db
04c883f32fdc postgres "docker-entrypoint.s…" 6 minutes ago Up 6 minutes 0.0.0.0:5432->5432/tcp clair_db
448de2a11cf1 redis:alpine "docker-entrypoint.s…" 2 hours ago Up 10 seconds 0.0.0.0:6379->6379/tcp session8. Harbor HA: modify the configuration
8.1 mysql_db introduced into container data table (192.168.200.17)
#在解压后的harbor目录里的ha目录下的registry.sql表导入到我们之前在NFS服务端上创建的MySQL容器里
[root@Harbor-master ~]# cd /data/install/harbor
[root@Harbor-master harbor]# ls
common docker-compose.notary.yml ha harbor.cfg.bak install.sh NOTICE
docker-compose.clair.yml docker-compose.yml harbor.cfg harbor.v1.5.0.tar.gz LICENSE prepare
[root@Harbor-master harbor]# tree ha
ha
├── docker-compose.clair.tpl
├── docker-compose.clair.yml
├── docker-compose.tpl
├── docker-compose.yml #需要修改的配置文件
├── registry.sql #需要导入的mysql表格
└── sample
├── active_active
│ ├── check.sh
│ └── keepalived_active_active.conf
└── active_standby
├── check_harbor.sh
└── keepalived_active_standby.conf
3 directories, 9 files#Harbor-master本地安装mysql客户端连接程序
[root@Harbor-master harbor]# yum -y install mysql
[root@Harbor-master harbor]# which mysql
/usr/bin/mysql
#远程连接到192.168.200.17（NFS服务器端）的3306端口，导入表registry.sql
[root@Harbor-master harbor]# mysql -uroot -p123123 -h192.168.200.17 -P3306
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.45 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.01 sec)
MySQL [(none)]> source ha/registry.sql #导入表格
#以下省略若干。。。
MySQL [registry]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| registry |
+--------------------+
4 rows in set (0.00 sec)
MySQL [registry]> Special Note: If you import a table there are mistakes
specified key was too long max key length is 767bytes
because the index table to import established more than mysql default on the line 767bytes> = 254, so we need to modify the import table
[root@harbor harbor]# cat -n ha/registry.sql | sed -n '220p;291p'
220 repository varchar(256) NOT NULL, #将256修改成254
291 resource_name varchar(256), #将256修改成254
#改完之后再进行表格导入就不会报错了8.2 modify the configuration file
#修改/data/install/ha/docker-compose.yml配置文件
[root@Harbor-master harbor]# cat -n ha/docker-compose.yml | sed -n '19p'
19 - /data/registry:/storage:z
#修改成我们的nfs共享目录
[root@Harbor-master harbor]# cat -n ha/docker-compose.yml | sed -n '19p'
19 - /data/storage:/storage:z#修改/data/install/harbor.cfg文件
[root@Harbor-master harbor]# cat -n harbor.cfg.bak | sed -n '7p;11p;23p;24p;68p;130p;133p;136p;139p;145p;150p;154p;157p;160p;163p'
7 hostname = reg.mydomain.com #harbor的访问域名（不能用IP地址）
11 ui_url_protocol = http #web访问连接方式
23 ssl_cert = /data/cert/server.crt #ca证书路径
24 ssl_cert_key = /data/cert/server.key #ca密钥路径
68 harbor_admin_password = Harbor12345 #harbor默认初始密码
130 db_host = mysql #harbor连接的mysql_db的IP
133 db_password = root123 #MySQL连接密码
136 db_port = 3306 #Mysql连接端口
139 db_user = root #MySQL连接用户
145 redis_url = redis:6379 #session连接的redis数据库路径
150 clair_db_host = postgres #clair漏洞检测组件连接的数据库IP
154 clair_db_password = password #postgres数据库连接密码
157 clair_db_port = 5432 #postgres数据库连接端口
160 clair_db_username = postgres #默认的连接用户名
163 clair_db = postgres #默认的库名
#修改成如下所示
[root@Harbor-master harbor]# cat -n harbor.cfg | sed -n '7p;11p;23p;24p;68p;130p;133p;136p;139p;145p;150p;154p;157p;160p;163p'
7 hostname = www.yunjisuan.com
11 ui_url_protocol = https
23 ssl_cert = /etc/ssl/harbor/www.yunjisuan.com.crt
24 ssl_cert_key = /etc/ssl/harbor/www.yunjisuan.com.key
68 harbor_admin_password = Harbor12345
130 db_host = 192.168.200.17
133 db_password = 123123
136 db_port = 3306
139 db_user = root
145 redis_url = 192.168.200.17:6379
150 clair_db_host = 192.168.200.17
154 clair_db_password = 123123
157 clair_db_port = 5432
160 clair_db_username = postgres
163 clair_db = postgres9. Harbor HA: Start Harbor
[root@Harbor-master harbor]# pwd
/data/install/harbor
[root@Harbor-master harbor]# ./install.sh --with-clair --ha
#因此使用了自定义存储路径，安装中途需要yes确认Browser to access the test: http: //192.168.200.16
#在docker-client（NFS共享存储服务器端）进行镜像上传测试：
[root@Docker-client ~]# docker login www.yunjisuan.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@Docker-client ~]# docker tag redis:alpine www.yunjisuan.com/library/redis
[root@Docker-client ~]# docker push www.yunjisuan.com/library/redis
The push refers to repository [www.yunjisuan.com/library/redis]
8fdd7da74c31: Pushed
2166e8ad934d: Pushed
c921f5478449: Pushed
9b8719029b64: Pushed
bd23b36e1125: Pushed
1bfeebd65323: Pushed
latest: digest: sha256:6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4 size: 1571
#查看NFS服务器共享目录
[root@Docker-client ~]# tree /data/nfs/
/data/nfs/
└── docker
└── registry
└── v2
├── blobs
│ └── sha256
│ ├── 05
│ │ └── 0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa
│ │ └── data
│ ├── 33
│ │ └── 3348f84e43d019f5288bf0f3143725683ec3e95d771af1dc60b2ec08ab33e919
│ │ └── data
│ ├── 6e
│ │ └── 6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4
│ │ └── data
│ ├── 7a
│ │ └── 7a3fdc0143e12cb63356b93af0fae6daacaf9fda239e776a8ab5c121ff184dc7
│ │ └── data
│ ├── ab
│ │ └── ab70e0f222721845b57e1a38fa16eee604153e6430df3e209ffc47b2874f3d5d
│ │ └── data
│ ├── d4
│ │ └── d48f315c369d800f68a3c7b2ed1b713df08231f439f3dcdcb7110fa87609fe4e
│ │ └── data
│ ├── d9
│ │ └── d975eaec5f68eddceab6bbc3f8c96fa3418978acd431c2a8cab1d7860372b1d1
│ │ └── data
│ └── ec
│ └── ecf40235d2c75d0220ad5f7c654d05cff5b527ca9f231f4a0203f0c19e5fc519
│ └── data
└── repositories
└── library
└── redis
├── _layers
│ └── sha256
│ ├── 0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa
│ │ └── link
│ ├── 3348f84e43d019f5288bf0f3143725683ec3e95d771af1dc60b2ec08ab33e919
│ │ └── link
│ ├── 7a3fdc0143e12cb63356b93af0fae6daacaf9fda239e776a8ab5c121ff184dc7
│ │ └── link
│ ├── ab70e0f222721845b57e1a38fa16eee604153e6430df3e209ffc47b2874f3d5d
│ │ └── link
│ ├── d48f315c369d800f68a3c7b2ed1b713df08231f439f3dcdcb7110fa87609fe4e
│ │ └── link
│ ├── d975eaec5f68eddceab6bbc3f8c96fa3418978acd431c2a8cab1d7860372b1d1
│ │ └── link
│ └── ecf40235d2c75d0220ad5f7c654d05cff5b527ca9f231f4a0203f0c19e5fc519
│ └── link
├── _manifests
│ ├── revisions
│ │ └── sha256
│ │ └── 6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4
│ │ └── link
│ └── tags
│ └── latest
│ ├── current
│ │ └── link
│ └── index
│ └── sha256
│ └── 6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4
│ └── link
└── _uploads
44 directories, 18 files10. Harbor HA: Keepalived installation configuration and testing
10.1 First we install a harbor-slave library equipment (192.168.200.18)
Repeat the previous operation, the process is slightly (harbor HA, require the same domain)
10.2 upload test
[root@Docker-client ~]# cd /data/nfs/
[root@Docker-client nfs]# docker tag mysql:5.6 www.yunjisuan.com/library/mysql
[root@Docker-client nfs]# docker push www.yunjisuan.com/library/mysql
The push refers to repository [www.yunjisuan.com/library/mysql]
a1e3e0513114: Pushed
6c621d0720e2: Pushed
d86d34816513: Pushed
b314ec235321: Pushed
812e5f94ac49: Pushed
d355dacb791d: Pushed
2f1b41b24201: Pushed
007a7f930352: Pushed
c6926fcee191: Pushed
b78ec9586b34: Pushed
d56055da3352: Pushed
latest: digest: sha256:ce58204b5f01bac11838b2ce2f379492841a11206a71a379bb47a68f63d057bf size: 2621Browser to access the test:
https://192.168.200.16
https://192.168.200.18
10.3 Harbor-master和Harbor-slave安装keepalived
#在Harbor-master进行如下操作
[root@Harbor-master harbor]# yum -y install keepalived
[root@Harbor-master harbor]# which keepalived
/usr/sbin/keepalived
[root@Harbor-master harbor]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id harbor01
}
vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 55
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.200.20 dev ens32
}
}
[root@Harbor-master harbor]# systemctl start keepalived
[root@Harbor-master harbor]# systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
inet 192.168.200.20/32 scope global ens32#在Harbor-slave进行如下操作
[root@Harbor-slave harbor]# yum -y install keepalived
[root@Harbor-slave harbor]# which keepalived
/usr/sbin/keepalived
[root@Harbor-slave harbor]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id harbor01
}
vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 55
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.200.20 dev ens32
}
}
[root@Harbor-slave harbor]# systemctl start keepalived
[root@Harbor-slave harbor]# systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@Harbor-slave harbor]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since 五 2019-07-26 11:56:38 CST; 1min 31s ago
Main PID: 69765 (keepalived)
CGroup: /system.slice/keepalived.service
├─69765 /usr/sbin/keepalived -D
├─69766 /usr/sbin/keepalived -D
└─69767 /usr/sbin/keepalived -D
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: Registering gratuitous ARP shared channel
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: Opening file '/etc/keepalived/keepalived.conf'.
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) removing protocol VIPs.
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: Using LinkWatch kernel netlink reflector...
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP sockpool: [ifindex(2), proto(112), unicast(0...1)]
7月 26 11:56:38 Harbor-slave Keepalived_healthcheckers[69766]: Opening file '/etc/keepalived/keepalived...'.
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) Transition to MASTER STATE
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) Received advert with higher p...100
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) Entering BACKUP STATE
7月 26 11:56:38 Harbor-slave systemd[1]: Started LVS and VRRP High Availability Monitor.
Hint: Some lines were ellipsized, use -l to show in full.10.4 VIP Switch Testing
在Harbor-master上操作
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
inet 192.168.200.20/32 scope global ens32
[root@Harbor-master harbor]# systemctl stop keepalived
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
在Harbor-slave上验证
[root@Harbor-slave harbor]# ip addr | grep 192.168.200.20
inet 192.168.200.20/32 scope global ens32
在Harbor-master上操作
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
[root@Harbor-master harbor]# systemctl start keepalived
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
inet 192.168.200.20/32 scope global ens32
在Harbor-slave上验证
[root@Harbor-slave harbor]# ip addr | grep 192.168.200.20