k8s--04 部署harbor作为k8s镜像仓库

k8s实战

部署harbor作为k8s镜像仓库

1.实验目标

部署k8s私有镜像仓库harbor
把demo小项目需要的镜像上传到harbor上
修改demo项目的资源配置清单，镜像地址修改为harbord的地址

2.在node1上安装harbor

[root@node1 ~]# cd /opt/
#上传harbor软件包
[root@node1 /opt]# rz -E
rz waiting to receive.
#解压
[root@node1 /opt]# tar zxf harbor-offline-installer-v1.9.0-rc1.tgz
#进入解压后的文件目录
[root@node1 /opt]# cd harbor/

3.编辑harbor配置文件

#备份
[root@node1 /opt/harbor]# cp harbor.yml harbor.yml.bak

#编辑配置文件
[root@node1 /opt/harbor]# vim harbor.yml 
#需要更改的地方
hostname: 10.0.0.11
port: 8888
harbor_admin_password: 123456
data_volume: /data/harbor

4.执行安装

#在安装harbor是许诺先安装docker-compose，否则报错
[root@node1 /opt/harbor]# yum install docker-compose -y

#安装harbor（注意命令执行的所在目录）
[root@node1 /opt/harbor]# ./install.sh

5.浏览器访问

http://10.0.0.11:8888
用户：admin
密码：123456

6.建立镜像仓库

这里有2种访问级别：
公开：任何人都可以直接访问并下载镜像
私有：登陆授权后才允许下载镜像

#注意
如果创建私有仓库，k8s是不能直接下载的，需要配置安全文件

7. 所有节点都配置docker信任harbor仓库并重启docker 注意：所有节点

#配置信任仓库
cat >/etc/docker/daemon.json <<EOF
    {
      "registry-mirrors": ["https://ig2l319y.mirror.aliyuncs.com"],
      "exec-opts": ["native.cgroupdriver=systemd"],
      "insecure-registries" : ["http://10.0.0.11:8888"]
    }
EOF
#重启docker
systemctl restart docker

###############注意###############
在node1上重启docker后，如果harbor不正常了，重启harbor即可
[root@node1 ~]# cd /opt/harbor
[root@node1 /opt/harbor]# docker-compose restart 
Restarting harbor-jobservice ... done
Restarting nginx             ... done
Restarting harbor-core       ... done
Restarting registryctl       ... done
Restarting registry          ... done
Restarting harbor-portal     ... done
Restarting harbor-db         ... done
Restarting redis             ... done
Restarting harbor-log        ... done

8.docker登陆harbor ( 所有节点 都执行 )

[root@node1 /opt/harbor]# docker login 10.0.0.11:8888
Username: admin
Password: #密码 123456
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

9.下载镜像修改tag并push到harbor上 ( 注意：从节点执行 )

1.在主节点查询镜像存放的节点位置
[root@node1 ~]# kubectl get pod -o wide 
NAME                     READY   STATUS    RESTARTS   AGE   IP         NODE    NOMINATED NODE   READINESS GATES
mysql-8fcd9f64-vqkm9     1/1     Running   1          18m   10.2.1.4   node2   <none>           <none>
myweb-6f974fdbdc-gsncp   1/1     Running   1          18m   10.2.1.5   node2   <none>           <none>
myweb-6f974fdbdc-ngngv   1/1     Running   1          18m   10.2.2.3   node3   <none>           <none>

2.根据主节点获取的信息在从节点执行打标签
[root@node2 ~]# docker tag kubeguide/tomcat-app:v1 10.0.0.11:8888/k8s/tomcat-app:v1
[root@node2 ~]# docker tag mysql:5.7 10.0.0.11:8888/k8s/mysql:5.7

3.将打好的标签的镜像上传到harbor仓库
[root@node2 ~]# docker push 10.0.0.11:8888/k8s/tomcat-app:v1 
[root@node2 ~]# docker push 10.0.0.11:8888/k8s/mysql:5.7

10.节点上删除镜像

#注意需要先删除标签镜像在删除源镜像
docker rmi 10.0.0.11:8888/k8s/mysql:5.7 
docker rmi 10.0.0.11:8888/k8s/tomcat-app:v1
docker rmi mysql:5.7 
docker rmi kubeguide/tomcat-app:v1

11.删除以前的demo项目 注意：主节点执行

[root@node1 ~]# kubectl delete -f tomcat-demo.yaml
deployment.apps "mysql" deleted
service "mysql" deleted
deployment.apps "myweb" deleted
service "myweb" deleted

12.修改demo项目的资源配置清单里的镜像地址

[root@node1 ~]# vim tomcat-demo.yaml   #注意更改的位置
原来image: mysql:5.7  变更为： image: 10.0.0.11:8888/k8s/mysql:5.7
原来image: k8s/tomcat-app:v1   变更为： image: 10.0.0.11:8888/k8s/tomcat-app:v1

13.应用资源配置清单

[root@node1 ~]# kubectl create -f tomcat-demo.yaml 
deployment.apps/mysql created
service/mysql created
deployment.apps/myweb created
service/myweb created

14.报错

#此时查看pod状态会发现镜像拉取失败了
[root@node1 ~]# kubectl get pod
NAME                     READY   STATUS             RESTARTS   AGE
mysql-7d746b5577-wtxtm   0/1     ErrImagePull       0          15s
myweb-764df5ffdd-jvvmf   0/1     ImagePullBackOff   0          15s
myweb-764df5ffdd-rc9pc   0/1     ImagePullBackOff   0          15s


#查看pod创建的详细信息
[root@node1 ~]# kubectl describe pod mysql-7d746b5577-可以tab自己的数据


#关键报错信息：
Failed to pull image "10.0.0.11:8888/k8s/mysql:5.7": rpc error: code = Unknown desc = Error response from daemon: pull access denied for 10.0.0.11:8888/k8s/mysql, repository does not exist or may require 'docker login'

翻译：项目不出在或者需要登录

15.查看docker登陆的密码文件

[root@node1 ~]# docker login 10.0.0.11:8888
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

#查看加密密码文件
[root@node1 ~]# cat /root/.docker/config.json
{
    "auths": {
        "10.0.0.11:8888": {
            "auth": "YWRtaW46MTIzNDU2"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/18.09.9 (linux)"
    }

16.将docker密码文件解码成base64编码 解码：base64

[root@node1 ~]# cat /root/.docker/config.json|base64
ewoJImF1dGhzIjogewoJCSIxMC4wLjAuMTE6ODg4OCI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNDZNVEl6TkRVMiIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTguMDkuOSAobGludXgpIgoJfQp9

#每一个人的都不一样

17.创建并应用docker登陆的Secret资源

#注意！！！
1.dockerconfigjson: xxx直接写base64的编码，不需要换行
2.base64编码是一整行，不是好几行
3.最后的type字段不能少

[root@node1 ~]# cat >harbor-secret.yaml<<EOF 
apiVersion: v1
kind: Secret
metadata:
  name: harbor-secret
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxMC4wLjAuMTE6ODg4OCI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNDZNVEl6TkRVMiIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTguMDkuOSAobGludXgpIgoJfQp9
  
type: kubernetes.io/dockerconfigjson
EOF

#注意密码是一行


[root@node1 ~]# kubectl create -f harbor-secret.yaml
secret/harbor-secret created


[root@node1 ~]# kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-tslz6   kubernetes.io/service-account-token   3      23h
harbor-secret         kubernetes.io/dockerconfigjson        1      46s

18.修改demo资源配置清单，添加拉取镜像的参数

查看命令帮助
kubectl explain deployment.spec.template.spec.imagePullSecrets

修改资源配置清单
修改文件
----------------------------
      imagePullSecrets: 
      - name: harbor-secret
----------------------------  
#注意：mysql和tomcat都需要增加

[root@node1 ~/demo]# cat tomcat-demo.yaml 
apiVersion: apps/v1
kind: Deployment 
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: 10.0.0.11:8888/k8s/mysql:5.7
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
      imagePullSecrets:
      - name: harbor-secret
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
---
apiVersion: apps/v1
kind: Deployment 
metadata:
  name: myweb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myweb
  template:
    metadata:
      labels:
        app: myweb
    spec:
      containers:
      - name: myweb
        image: 10.0.0.11:8888/k8s/tomcat-app:v1
        ports:
        - containerPort: 8080
        env:
        - name: MYSQL_SERVICE_HOST
          value: 'mysql'
        - name: MYSQL_SERVICE_PORT
          value: '3306'
      imagePullSecrets:
      - name: harbor-secret
---
apiVersion: v1
kind: Service
metadata:
  name: myweb
spec:
  type: NodePort
  ports:
    - port: 8080
      nodePort: 30001
  selector:
    app: myweb

19.应用资源配置清单并查看

1.删除资源配置清单
[root@node1 ~]# kubectl delete -f tomcat-demo.yaml

2.创建新的资源
[root@node1 ~]# kubectl create -f tomcat-demo.yaml 
deployment.apps/mysql created
service/mysql created
deployment.apps/myweb created
service/myweb created

3.查询下载的资源

kubectl get pod -o wide

20.浏览器查看

http://10.0.0.11:30001/demo

报错总结：

#报错总结：
1.如果要删除的镜像正在被容器使用，那么你是删不了的
2.harbor卸载不干净，/data/harbor/目录下的数据也要删除
3.secret配置只写了一个dp，实际上有几个deployment就需要写几个



重做k8s使用harbor作为私有仓库
1.停止harbor正在运行的容器
2.删除harbor的容器
docker ps -a|grep "goharbor"|awk '{print "docker rm "$1}'
3.删除harbor的镜像
dockerimages|grep "goharbor"|awk '{print "docker rmi "$1":"$2}'

4.解压并修改harbor配置文件
hostname: 10.0.0.11
port: 8888
harbor_admin_password: 123456
data_volume: /data/harbor

5.执行安装并访问
./install.sh
http://10.0.0.11:8888

6.创建一个私有仓库k8s

7.配置docker信任仓库并重启--三台服务器都操作!!!
    {
      "registry-mirrors": ["https://ig2l319y.mirror.aliyuncs.com"],
      "exec-opts": ["native.cgroupdriver=systemd"],
      "insecure-registries" : ["http://10.0.0.11:8888"]
    }

systemctl restart docker

注意！！！node1重启后harbor会失效，需要重启harbor
cd /opt/harbor
docker-compose stop
docker-compose start

8.docker登陆harbor
docker login 10.0.0.11:8888

9.将docker登陆凭证转化为k8s能识别的base64编码
[root@node1 ~]# cat /root/.docker/config.json|base64
ewoJImF1dGhzIjogewoJCSIxMC4wLjAuMTE6ODg4OCI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNDZN
VEl6TkRVMiIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tl
ci1DbGllbnQvMTguMDkuOSAobGludXgpIgoJfQp9

10.编写Secert资源配置清单
[root@node1 ~/demo]# cat harbor-secret.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: harbor-secret
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxMC4wLjAuMTE6ODg4OCI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNDZNVEl6TkRVMiIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTguMDkuOSAobGludXgpIgoJfQp9
type: kubernetes.io/dockerconfigjson

11.应用Secret资源
kubectl delete -f harbor-secret.yaml 
kubectl create -f harbor-secret.yaml
kubectl get secrets

12.修改镜像tag并上传到harbor
docker tag kubeguide/tomcat-app:v1 10.0.0.11:8888/k8s/tomcat-app:v1
docker tag mysql:5.7 10.0.0.11:8888/k8s/mysql:5.7

docker push 10.0.0.11:8888/k8s/tomcat-app:v1
docker push 10.0.0.11:8888/k8s/mysql:5.7 


13.修改demo资源配置清单
####mysql
imagePullSecrets:
      - name: harbor-secret

###tomcat
imagePullSecrets:
      - name: harbor-secret


14.应用资源清单并查看
kubectl apply -f .
kubectl get pod