Gitlab CI/CD 生产环境docker化+Docker-Registry私有云实验

实验环境

一共用了4台主机,预安装的Ubuntu 18.04.1 LTS系统和Docker-ce 18.09.2版本.

Registry端IP:192.168.99.100
Gitlab端IP:192.168.99.101
Runner端IP:192.168.99.102
Nginx端IP:192.168.99.103

安装使用存储库

#卸载docker，确保系统内不docker应用
$ sudo apt-get remove docker docker-engine docker.io containerd runc

#更新 apt 
$ apt-get	update

#安装包以允许apt通过HTTPS使用存储库：
$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common

#添加Docker的官方GPG密钥：
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-key fingerprint 0EBFCD88

#设置稳定存储库
$ sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

安装docker-ce

$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io

查看docker版本

$ docker version

Docker私有云

本地获取registry镜像

$ docker search registry
$ docker pull registry

创建registry容器

$ docker run -d \
--name registry	\
--restart=always \
--publish 5000:5000 \
-v /opt/data/registry:/var/lib/registry \
registry

测试私有库

$ curl -X GET http://192.168.99.100:5000/v2/_catalog
{"repositories":[]}
#因为我们还没有像私有容器提交镜像，所以这里返回空。

创建好私有仓库之后，就可以使用 docker tag 来标记一个镜像，然后推送它到仓库。

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               latest              f09fe80eb0e7        11 days ago         109MB

$ docker tag nginx:latest 192.168.99.100:5000/nginx:nginx
$ docker push 192.168.99.100:5000/nginx:nginx 

注意事项
如果你想让本网段的其他主机也能把镜像推送到私有仓库。
你就得把例如 192.168.199.100:5000 这样的内网地址作为私有仓库地址，这时你会发现无法成功推送镜像。
这是因为 Docker 默认不允许非 HTTPS 方式推送镜像。我们可以通过 Docker 的配置选项来取消这个限制。

第一次执行"docker push 192.168.99.100:5000/nginx:nginx "可能报如下异常：
$ docker push 192.168.99.100:5000/nginx:nginx
The push refers to repository [192.168.99.100:5000/nginx]
Get https://192.168.99.100:5000/v2/: http: server gave HTTP response to HTTPS client

解决方案

在/etc/docker目录下新建 daemon.json, 文件中写入：

{
  "insecure-registries": ["192.168.99.100:5000"] 
}

注意书写格式,然后重启,重新运行registry,再次运行上传镜像到私有仓库.Gitlab-Runner与发布主机都要使用私有库,所以都要修改.

$ systemctl daemon-reload 
$ systemctl rstart docker
$ docker push 192.168.99.100:5000/nginx:nginx 
The push refers to repository [192.168.99.100:5000/nginx]
6b5e2ed60418: Pushed 
92c15149e23b: Pushed 
0a07e81f5da3: Pushed 
latest: digest: sha256:5b49c8e2c890fbb0a35f6050ed3c5109c5bb47b9e774264f4f3aa85bb69e2033 size: 948

再次测试私有云

$ curl -X GET http://192.168.99.100:5000/v2/_catalog
{"repositories":["nginx"]}

部署Gitlab

本地获取Gitlab中文社区版镜像

$ docker search gitlab
$ docker pull twang2218/gitlab-ce-zh 

运行Gitlba容器,添加端口映射和数据卷挂载等设置

$ docker run --detach \
--hostname 192.168.99.101  \
--publish 443:443 \
--publish 80:80 \
--publish 2222:22  \
--name gitlab  \
--restart always  \
--volume /srv/gitlab/config:/etc/gitlab  \
--volume /srv/gitlab/logs:/var/log/gitlab  \
--volume /srv/gitlab/data:/var/opt/gitlab \
twang2218/gitlab-ce-zh:latest 

b60a63fc8110b6c9f2fa14c77173f794652b0f994b7a11534c727755ad4b7748

查看本地容器

$ docker ps

CONTAINER ID        IMAGE                           COMMAND             CREATED             STATUS                 PORTS                                                            NAMES
e3c6902469f8        twang2218/gitlab-ce-zh:latest   "/assets/wrapper"   4 hours ago         Up 4 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp   gitlab

进入Gitlab容器并初始化Gtlab

$ docker exec -it gitlab gitlab-ctl reconfigure

Starting Chef Client, version 13.6.4
lving cookbooks for run list: ["gitlab"]
hronizing Cookbooks:
.
.
.
Running handlers:
ing handlers complete
Client finished, 2/537 resources updated in 05 minutes 42 seconds
gitlab Reconfigured!

初始化完成后就可以通过web页面访问Gitlab进行使用了.首次初始化比较慢,大概15分钟左右.

部署Gitlab-Runner

本地获取Gitlab-Runner镜像

$ docker search gitlab-runner
$ docker pull gitlab/gitlab-runner

运行Gitlba-Runner容器,添加数据卷挂在等设置

$ docker run -d \
--name gitlab-runner \
--restart always \
-v /srv/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:latest    

e707d7834d69a6d2a9852b89ce5cbed170d9a791b9b4e94f672e614708abae27

创建Runner调用的docker基础镜像并上传到私有云

$

进入Gitlab-Runner容器,设置向Gitlab注册,gitlab-I的url和token需查看Gitlab获取.

$ docker exec -it gitlab-runner /bin/bash
root@7411c4aacbdb:/# gitlab-runner register

Runtime platform                                    arch=amd64 os=linux pid=43 revision=8bb608ff version=11.7.0
Running in system-mode.                            
                                               
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://192.168.99.101/
Please enter the gitlab-ci token for this runner:
MqMwm1JSsRmezGCxj3Bq
Please enter the gitlab-ci description for this runner:
[7411c4aacbdb]: test-runner
Please enter the gitlab-ci tags for this runner (comma separated):
my-tag,test-tag
Registering runner... succeeded                     runner=MqMwm1JS
Please enter the executor: parallels, docker+machine, docker-ssh+machine, kubernetes, docker, docker-ssh, shell, ssh, virtualbox:
docker
Please enter the default Docker image (e.g. ruby:2.1):
docker:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 

注册完成后,在Girlab的web页面中管理区域下的Runners分页可以查看到注册信息.

Runner免密登录设置

创建公私密钥并发送给发布主机

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/docker/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/docker/.ssh/id_rsa.
Your public key has been saved in /home/docker/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:GjxLmRIfkW7USqK2zaLthZNNw8Dvg0ChBF2tpNuwam0 docker@test1
The key's randomart image is:
+---[RSA 2048]----+
|o+ ....o         |
|o.o o =..        |
|..o+.*..         |
|. =+.+++         |
|.. O*.O S        |
| .=B++ =         |
| +*.= o          |
|o..E .           |
|..o              |
+----[SHA256]-----+

$ ssh-copy-id [email protected]
/usr/local/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/docker/.ssh/id_rsa.pub"
The authenticity of host '192.168.99.103 (192.168.99.103)' can't be established.
ECDSA key fingerprint is SHA256:FsYHsFkz3+y9BmBj1yydl6fHKLAcnJbOd6lkprC0ywc.
Are you sure you want to continue connecting (yes/no)? yes
/usr/local/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
expr: warning: '^ERROR: ': using '^' as the first character
of a basic regular expression is not portable; it is ignored
/usr/local/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:   tcuser

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

$ ssh [email protected]

#   **tcuser  是虚拟机中docker用户默认密码**

实现CI/CD持续继承,持集交付,持续部署功能

Gitlab的项目中添加.gitlab-ci.yml文件

stages:
  - build
  - deploy

variables:    
    CONTAINER_NAME: nginx
    REGISTRY_NAME: 192.168.99.100
    PORT_NAME: 5000

build_push_image_prod:
    stage: build
    before_script:
      - docker login --username=docker --password="tcuser" $REGISTRY_NAME:$PORT_NAME

    script:
      - docker build -t $REGISTRY_NAME:$PORT_NAME/$CONTAINER_NAME:$CONTAINER_NAME .
      - docker push $REGISTRY_NAME:$PORT_NAME/$CONTAINER_NAME:$CONTAINER_NAME
    tags:
      - my-tag
    only:
      - master

remote_run_image_prod:
    stage: deploy

    script:
      - bash /.runner/runner.sh $CONTAINER_NAME  80  80  $REGISTRY_NAME:$PORT_NAME/$CONTAINER_NAME:$CONTAINER_NAME   docker 192.168.99.103
    tags:
      - test-tag
    only:
      - master
    when: on_success

Gitlab中的Dockerfile文件

FROM 192.168.99.100:5000/nginx:nginx
ENV RUN_USER nginx 
ENV RUN_GROUP nginx 
ENV DATA_DIR /data/web
ENV LOG_DIR /data/log/nginx
RUN mkdir /data/log/nginx -p 
RUN chown nginx.nginx -R /data/log/nginx
ADD web /data/web
ADD nginx.conf /etc/nginx/nginx.conf
ADD default.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Dockerfile做了这么几件事：

1、拉取一个nginx镜像。
2、设置了几个变量。
3、创建了几个需要的目录。
4、把当前目录下的web程序复制到镜像的/data/web目录。
5、把nginx.conf配置文件和default.conf配置文件复制到镜像中。
6、设置一个默认端口。
7、最后设置了容器启动时执行的命令，我用来启动nginx程序，注意这个命令不能错，不然容器启动不了。这样设置后，当你docker run运行此镜像时不需要在后面再次执行需要执行的命令了。

gitlab-runner中的~/.runner/runner.sh脚本

#!/bin/bash
ssh $5@$6  <<EOF
cd;bash /.runner/fdeploy.sh $1 $2  $3  $4
EOF

gitlab-runner中的/.runner.sh/fdeploy.sh脚本

#!/bin/bash
conName=$1
eonPort=$2
conPort=$3
images=$4
count=`docker ps -a  |grep "$conName"|wc -l `
if [ $count -eq 0 ];then
       echo "$conName container  is not exit"
   else 
       id=$(docker ps -a |grep "$conName" |awk '{print $1}')

       for i in $id
       do
           docker stop $i
       docker rm $i
       done
       #cName=`docker ps --format "{{.Names}}" |grep "$conName"
       #eport=`docker ps --format "{{.Ports}}:{{.Names}}" |grep "web_activity_assistant" |awk -F ":" '{print $2}' |awk -F "->" '{print $1}'`
       #cport=`docker ps --format "{{.Ports}}:{{.Names}}" |grep "web_activity_assistant" |awk -F ":" '{print $2}' |awk -F "->" '{print $1}'|awk -F "/" '{print $1}'`
fi
     docker rmi $images
     docker run -dit --restart=always     --name $conName -p $eonPort:$conPort $images