[CI/CD] Simple CI process practice based on Jenkins+Docker+Git (Part 1)

Simple CI process practice based on Jenkins+Docker+Git (Part 1)

In today's Internet age, with the continuous increase in the complexity of software development, software development and release management are becoming more and more important. At present, a set of standard processes has been formed, the most important components of which are continuous integration , continuous delivery, and deployment . Here, we use a simple case to practice the CI process on JD Cloud.

1. Introduction to related concepts and tools

1.1 CI/CD

Continuous Integration (CI), which is an automated process for developers. Successful CI means that new changes to the application code are regularly built, tested, and merged into the shared repository. This solution can solve the problem of having too many application branches in one development, resulting in conflicts.

Continuous Delivery (CD), typically when changes to the application by developers are automatically error-tested and uploaded to a repository (such as GitHub or a container registry), which is then deployed by the operations team to a live production environment . This is intended to address poor visibility and communication between development and operations teams. Therefore, the purpose of continuous delivery is to ensure that the effort required to deploy new code is as small as possible.

Continuous Deployment (CD), another type of "CD", refers to the automatic release of developer changes from the repository to production for consumption by customers. It is primarily intended to address the problem of overloading operations teams due to manual processes that slow down application delivery.

1.2 Jenkins

Jenkins is an open source software project. It is a continuous integration tooldeveloped based on Java

1.3 Docker

Docker is an open source application container engine that allows developers to package their applications and dependencies into a portable container, and then publish it to any popular Linux machine, and can also implement virtualization. Containers use a sandbox mechanism completely, and there will be no interfaces between them.

1.4 Git

Git is an open source distributed version control system that provides a code repository that can effectively and quickly handle project version management from small to very large. Git is$LinusTorvaldsis$ an open source version control software developed to help manage Linux$kernel$$development$$.$

2. Preliminary preparation

2.1 CI process design diagram

2.2 Workflow

• Developers submit code to the Git repository.
• Jenkins manually/timing triggers project builds.
• Jenkins pulls the code, compiles the code, packages the image, and pushes it to the mirror warehouse.
• Jenkins creates a container on the Docker host and publishes it.

2.3 Host environment planning

host name	introduce	public net	Intranet
docker-jenkins	Project construction; pull code, code compile, package mirror, push mirror to mirror warehouse	116.196.85.174	10.0.0.20
docker-git	code repository	116.196.86.207	10.0.0.22
docker-harbor	Private Mirror Warehouse	116.196.88.91	10.0.0.21
buildimage	build docker image	116.196.89.139	10.0.0.4

3. Host creation

Create 4 cloud hosts in the JD Cloud console, address: https://console.jdcloud.com/

The configuration is as follows, the quantity is directly selected when purchasing $4.$ After the purchase is completed, modify the name, respectively:docker-jenkins,docker-git,docker-harbor,buildimage.

After creating and modifying the name as follows:

4. Environment configuration

4.1 cloud host docker-git

(1) Modify the hostname to:docker-git

[root@112 ~]# hostnamectl set-hostname docker-git
[root@112 ~]# hostname  docker-git
[root@112 ~]# logout
[root@docker-git ~]#

Ctrl+DRe-login after logout to take effect.

(2) Deploy the Git code version warehouse

Install:

[root@docker-git ~]# yum install git -y

configure gituser:

[root@docker-git ~]# useradd git
[root@docker-git ~]# passwd git

Create library:

[root@docker-git ~]# su git
[git@docker-git root]$ cd
[git@docker-git ~]$ mkdir tomcat-java-demo.git
[git@docker-git ~]$ cd tomcat-java-demo.git/
[git@docker-git tomcat-java-demo.git]$ git --bare init
Initialized empty Git repository in /home/git/tomcat-java-demo.git/
[git@docker-git tomcat-java-demo.git]$ ls
branches  config  description  HEAD  hooks  info  objects  refs
[git@docker-git tomcat-java-demo.git]$ 

4.2 Cloud host docker-jenkins

(1) Modify the hostname to:docker-jenkins

[root@113 ~]# hostnamectl set-hostname docker-jenkins
[root@113 ~]# hostname  docker-jenkins
[root@113 ~]# logout
[root@docker-jenkins ~]#

Ctrl+DRe-login after logout to take effect.

(2) Jenkins environment deployment

Deployment jdkenvironment and maven:

[root@docker-jenkins tomcat-java-demo]# cd
[root@docker-jenkins ~]# mkdir tools
[root@docker-jenkins ~]# cd tools
[root@docker-jenkins tools]# wget https://pocenv-hcc.oss.cn-north-1.jcloudcs.com/jdk-8u191-linux-x64.tar.gz;tar zxf jdk-8u191-linux-x64.tar.gz;mv jdk1.8.0_191/ /usr/local/;ln -s /usr/local/jdk1.8.0_191/ /usr/local/jdk;
[root@docker-jenkins tools]# vim /etc/profile
######## JDK #######
JAVA_HOME=/usr/local/jdk1.8.0_191
JAVA_BIN=/usr/local/jdk1.8.0_191/bin
PATH=$PATH:$JAVA_BIN
CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
[root@docker-jenkins tools]# source /etc/profile
[root@docker-jenkins tools]# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) 
[root@docker-jenkins tools]# wget https://pocenv-hcc.oss.cn-north-1.jcloudcs.com/apache-maven-3.5.0-bin.tar.gz;tar zxf apache-maven-3.5.0-bin.tar.gz;mv apache-maven-3.5.0 /usr/local/maven
[root@docker-jenkins tools]# 

Install Jenkins, download Tomcatthe binary package, and warmove the package to webapps.

[root@docker-jenkins tools]# wget https://pocenv-hcc.oss.cn-north-1.jcloudcs.com/jenkins.war
[root@docker-jenkins tools]# wget https://pocenv-hcc.oss.cn-north-1.jcloudcs.com/apache-tomcat-8.5.38.tar.gz
[root@docker-jenkins tools]# tar zxf apache-tomcat-8.5.38.tar.gz
[root@docker-jenkins tools]# ls
apache-maven-3.5.0-bin.tar.gz  apache-tomcat-8.5.38  apache-tomcat-8.5.38.tar.gz  jdk-8u191-linux-x64.tar.gz  jenkins.war
[root@docker-jenkins tools]# mv apache-tomcat-8.5.38 /usr/local/tomcat-jenkins
[root@docker-jenkins tools]# ls /usr/local/tomcat-jenkins/webapps/
docs  examples  host-manager  manager  ROOT
[root@docker-jenkins tools]# rm -rf /usr/local/tomcat-jenkins/webapps/*
[root@docker-jenkins tools]# mv jenkins.war /usr/local/tomcat-jenkins/webapps/ROOT.war
[root@docker-jenkins tools]# ll /usr/local/tomcat-jenkins/webapps/
total 75520
-rw-r--r--. 1 root root 77330344 Mar 15 00:55 ROOT.war
[root@docker-jenkins tools]# cd /usr/local/tomcat-jenkins/bin/
[root@docker-jenkins bin]# ./startup.sh
Using CATALINA_BASE:   /usr/local/tomcat-jenkins
Using CATALINA_HOME:   /usr/local/tomcat-jenkins
Using CATALINA_TMPDIR: /usr/local/tomcat-jenkins/temp
Using JRE_HOME:        /usr/local/jdk1.8
Using CLASSPATH:       /usr/local/tomcat-jenkins/bin/bootstrap.jar:/usr/local/tomcat-jenkins/bin/tomcat-juli.jar
Tomcat started.
[root@docker-jenkins bin]#

After startup, browser access ( docker-jenkins): http://Jenkins主机公网IP:8080/, enter the password as prompted, and log in.

In /root/.jenkins/secrets/initialAdminPasswordthe file, check the password and fill it in.

Install plugins according to your own needs.

Set up administrators.

Get started Jenkins.

(3) Install DOCKER CE (Docker's Community Edition)

Install required packages:

yum install -y yum-utils device-mapper-persistent-data lvm2 -y

Set up a stable repository:

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo -y

Installation DOCKER CE(this step is slow, wait patiently!):

yum install docker-ce docker-ce-cli containerd.io -y

start Docker:

systemctl start docker

4.3 Cloud host docker-harbor

(1) Modify the hostname to:docker-harbor

[root@c-dfjgjesgqe ~]# hostnamectl set-hostname docker-harbor
[root@c-dfjgjesgqe ~]# hostname  docker-harbor

Ctrl+DRe-login after logout to take effect.

(2) Enterprise-level Harbormirror warehouse deployment

Habor is an open source container mirror warehouse by VMWare. In fact, Habor has made corresponding enterprise-level extensions on Docker Registry, thus gaining wider application. These new enterprise-level features include: management user interface , role-based access control , AD/LDAP integration and audit logs etc., enough to meet basic business needs.

Introduction to each component of Harbor:

components	Function
harbor-adminserver	Configuration Management Center
harbor-db	MySQL database
harbor-jobservice	responsible for mirroring
harbor-log	record operation log
harbor-ui	Web management pages and API
nginx	Front-end proxy, responsible for front-end pages and mirror upload/download forwarding
redis	conversation
registry	mirror storage

Harbor installed with $3$ ways

• Online installation: Download Harbor related images from Docker Hub, so the installation package is very small.
• Offline installation: The installation package contains the relevant images for deployment, so the installation package is relatively large.
• OVA Installer: When user has a vCenter environment, use this installer to start Harb after deploying OVA.

Here we use the second offline installation method to build an httpsaccess-based harbormirror warehouse.

1) Download and unzip the offline installation package

harborOffline package download address: https://github.com/goharbor/harbor/releases

For the convenience of downloading, I also saved a copy on JD Cloud Object Storage, which can be directly wget:https://pocenv-hcc.oss.cn-north-1.jcloudcs.com/harbor-offline-installer-v1.7.4.tgz

[root@docker-harbor ~]# yum install vim wget openssl -y
[root@docker-harbor ~]# wget https://pocenv-hcc.oss.cn-north-1.jcloudcs.com/harbor-offline-installer-v1.7.4.tgz
[root@docker-harbor ~]# tar zxf harbor-offline-installer-v1.7.4.tgz
[root@docker-harbor ~]# cd harbor
[root@docker-harbor harbor]# ll
total 570744
drwxr-xr-x 3 root root        23 Apr  1 15:05 common
-rw-r--r-- 1 root root       939 Mar  4 15:33 docker-compose.chartmuseum.yml
-rw-r--r-- 1 root root       975 Mar  4 15:33 docker-compose.clair.yml
-rw-r--r-- 1 root root      1434 Mar  4 15:33 docker-compose.notary.yml
-rw-r--r-- 1 root root      5608 Mar  4 15:33 docker-compose.yml
-rw-r--r-- 1 root root      8033 Mar  4 15:33 harbor.cfg
-rw-r--r-- 1 root root 583086399 Mar  4 15:33 harbor.v1.7.4.tar.gz
-rwxr-xr-x 1 root root      5739 Mar  4 15:33 install.sh
-rw-r--r-- 1 root root     11347 Mar  4 15:33 LICENSE
-rw-r--r-- 1 root root   1263409 Mar  4 15:33 open_source_license
-rwxr-xr-x 1 root root     36337 Mar  4 15:33 prepare

2) Self-signed http certificate

Obtain an authoritative certification certificate.

[root@docker-harbor harbor]# mkdir ssl
[root@docker-harbor harbor]# cd ssl
[root@docker-harbor ssl]# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
...................................++
.....................................................................................................................................++
e is 65537 (0x10001)
[root@docker-harbor ssl]# openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=ZH/ST=ShangHai/L=ShangHai/O=example/OU=Personal/CN=reg.marin.com" -key ca.key -out ca.crt
[root@docker-harbor ssl]# ll
total 8
-rw-r--r-- 1 root root 2037 Apr  4 18:41 ca.crt
-rw-r--r-- 1 root root 3243 Apr  4 18:41 ca.key

Get the server certificate.

1.Create your own Private Key:
[root@docker-harbor ssl]# openssl genrsa -out reg.marin.com.key 4096
Generating RSA private key, 4096 bit long modulus
.............................................++
............................................................................................................................................................................................................................++
e is 65537 (0x10001)
[root@docker-harbor ssl]# openssl req -sha512 -new -subj "/C=ZH/ST=ShangHai/L=ShangHai/O=example/OU=Personal/CN=reg.marin.com" -key reg.marin.com.key -out reg.marin.com.csr
[root@docker-harbor ssl]# ll
total 16
-rw-r--r-- 1 root root 2037 Apr  4 18:41 ca.crt
-rw-r--r-- 1 root root 3243 Apr  4 18:41 ca.key
-rw-r--r-- 1 root root 1708 Apr  4 18:42 reg.marin.com.csr
-rw-r--r-- 1 root root 3243 Apr  4 18:42 reg.marin.com.key
[root@docker-harbor ssl]# cat > v3.ext <<-EOF
> authorityKeyIdentifier=keyid,issuer
> basicConstraints=CA:FALSE
> keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = @alt_names
> 
> [alt_names]
> DNS.1=reg.marin.com
> DNS.2=reg.marin
> DNS.3=marin
> EOF
[root@docker-harbor ssl]# openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in reg.marin.com.csr -out reg.marin.com.crt
Signature ok
subject=/C=ZH/ST=ShangHai/L=ShangHai/O=example/OU=Personal/CN=reg.marin.com
Getting CA Private Key
[root@docker-harbor ssl]# ll
total 28
-rw-r--r-- 1 root root 2037 Apr  4 18:41 ca.crt
-rw-r--r-- 1 root root 3243 Apr  4 18:41 ca.key
-rw-r--r-- 1 root root   17 Apr  4 18:44 ca.srl
-rw-r--r-- 1 root root 2098 Apr  4 18:44 reg.marin.com.crt
-rw-r--r-- 1 root root 1708 Apr  4 18:42 reg.marin.com.csr
-rw-r--r-- 1 root root 3243 Apr  4 18:42 reg.marin.com.key
-rw-r--r-- 1 root root  260 Apr  4 18:43 v3.ext

Modify harborthe configuration, and Dockerconfigure the server certificate for, keyand CA.

[root@docker-harbor ssl]# cd ..
[root@docker-harbor harbor]# vim harbor.cfg
......
hostname = reg.marin.com
ui_url_protocol = https
ssl_cert = ./ssl/reg.marin.com.crt
ssl_cert_key = ./ssl/reg.marin.com.key
harbor_admin_password = 123456
......

The password can also be left unchanged, the default login user: admin, password: Harbor12345.

The Docker daemon will .crtinterpret the file as a CA certificate and .certthe file as a client certificate, first .crtconverting the file to a .certfile.

[root@docker-harbor harbor]# cd ssl/
[root@docker-harbor ssl]# mkdir -p /etc/docker/certs.d/reg.marin.com
[root@docker-harbor ssl]# openssl x509 -inform PEM -in reg.marin.com.crt -out reg.marin.com.cert
[root@docker-harbor ssl]# cp reg.marin.com.cert reg.marin.com.key ca.crt /etc/docker/certs.d/reg.marin.com/

This self-signed success!

3) Install Docker CE

Install required packages:

yum install -y yum-utils device-mapper-persistent-data lvm2 -y

Set up a stable repository:

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo -y

Installation DOCKER CE(this step is slow, wait patiently!):

yum install docker-ce docker-ce-cli containerd.io -y

start Docker:

systemctl start docker

hello-worldVerify that it was installed correctly by running the mirror Docker CE:

docker run hello-world

4) Initialization and installation verification

Initial installation:

[root@docker-harbor ssl]# 
[root@docker-harbor ssl]# cd ..
[root@docker-harbor harbor]# ./prepare
Generated and saved secret to file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/core/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/config.yml
Generated configuration file: ./common/config/log/logrotate.conf
Generated configuration file: ./common/config/registryctl/env
Generated configuration file: ./common/config/core/app.conf
Generated certificate, key file: ./common/config/core/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

Execute install.shthe script to install harborthe repository.

NOTE: install.shBefore executing the script, check two issues:

• docker-composeWhether to install it, otherwise install.shit will fail when running, and report "✖ Need to install docker-compose(1.7.1+) by yourself first and run this script again."
• dockerinstall.shWhether the service is running normally, otherwise it will fail when running , and report an error "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

install Compose.

Run this command to download Docker Composethe current stable release of :

curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Apply executable permissions to the binary:

chmod +x /usr/local/bin/docker-compose

Execute install.shthe script to install harborthe repository:

[root@docker-harbor harbor]# ./install.sh

[Step 0]: checking installation environment ...

Note: docker version: 18.09.4

Note: docker-compose version: 1.24.0

[Step 1]: loading Harbor images ...
bffe2a0fec66: Loading layer [==================================================>]  33.22MB/33.22MB
38e174bed467: Loading layer [==================================================>]  8.964MB/8.964MB
427e4936ae66: Loading layer [==================================================>]  35.77MB/35.77MB
3bfd5214250a: Loading layer [==================================================>]  2.048kB/2.048kB
f30df776629d: Loading layer [==================================================>]  3.072kB/3.072kB
f87afad43f43: Loading layer [==================================================>]   22.8MB/22.8MB
......
953717aa0afc: Loading layer [==================================================>]   22.8MB/22.8MB
Loaded image: goharbor/registry-photon:v2.6.2-v1.7.4


[Step 2]: preparing environment ...
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/core/env
Clearing the configuration file: ./common/config/core/app.conf
Clearing the configuration file: ./common/config/core/private_key.pem
Clearing the configuration file: ./common/config/db/env
......
Generated certificate, key file: ./common/config/core/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.


[Step 3]: checking existing instance of Harbor ...


[Step 4]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating redis              ... done
Creating registryctl        ... done
Creating harbor-db          ... done
Creating harbor-adminserver ... done
Creating registry           ... done
Creating harbor-core        ... done
Creating harbor-jobservice  ... done
Creating harbor-portal      ... done
Creating nginx              ... done

✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at https://reg.marin.com. 
For more details, please visit https://github.com/goharbor/harbor .

Browser access verification.

Domain name resolution is required for browser access, so add: to the local hosts( ) file .C:\Windows\System32\drivers\etc\hosts116.196.88.91 reg.marin.com

Visit: https://reg.marin.com, and log in.

Basic operation of the interface after login:

• Create a new project test.

• Create a new user marin.

• Set the user marinas testa project administrator.