Project Deployment_Continuous Integration

1 Today’s content introduction

1.1 What is continuous integration

Continuous integration (CI for short) refers to integrating code into the trunk frequently (multiple times a day)

Components of continuous integration

An automatic build process, from checking out code, compiling and building, running tests, result recording, test statistics, etc., is completed automatically without manual intervention.

A code repository requires version control software to ensure the maintainability of the code. It also serves as a material library for the construction process. SVN or Git is generally used.

A continuous integration server, Jenkins is a continuous integration server that is simple to configure and easy to use .

1.2 Benefits of continuous integration

1. Reduce risks. Since continuous integration continuously builds, compiles and tests, problems can be discovered very early , so the cost of repair is low;

2. Continuously check the system health to reduce problems caused by release risks ;

3. Reduce repetitive work;

4. Continuous deployment, providing deployable unit packages;

5. Continuously deliver available versions;

6. Enhance team confidence;

1.3 Today’s content

2 Software development model

2.1 Software development life cycle

Software development life cycle is also called SDLC (Software Development Life Cycle), which is a collection of planning, development, testing and deployment processes . As shown below:

demand analysis

This is the first phase of the life cycle where the team performs an analysis of a feasibility plan based on project requirements . Project requirements may come from within the company or from customers. This stage is mainly about the collection of information, and it may also be about improving existing projects or starting a new project. It is also necessary to analyze the budget of the project , what aspects it can benefit from and its layout. This is also the goal of project creation.

design

The second stage is the design stage, the system architecture and satisfactory status (that is, what it will look like and what functions it will have), and the creation of a project plan. Plans can be presented using diagrams, layouts, or text .

accomplish

The third phase is the implementation phase, where the project manager creates and assigns work to developers , who develop code based on the tasks and goals defined in the design phase. Depending on the size and complexity of the project, it can take several months or longer to complete.

test

Testers conduct code testing , including functional testing, code testing, stress testing, etc.

evolution

The final stage is the continuous evolution, improvement and maintenance stage of the product . Depending on the user's usage, certain functions may need to be modified, bug fixed, functions added, etc.

2.2 Waterfall model of software development

The waterfall model is the most famous and commonly used software development model. The waterfall model is a series of software development processes. It was bred from manufacturing. A highly structured process flows in one direction, somewhat like a production line . At the beginning of the creation of the waterfall model, there was no other developed model, and many things were left to developers to guess and develop. Such a model is only suitable for simple software development, but it is no longer suitable for current development.

The figure below illustrates the software development model.

Advantage	Disadvantages
Simple to use and understand	The division of each stage is completely fixed, and a large number of documents are generated between stages, which greatly increases the workload.
Once the previous phase is complete, you only need to focus on subsequent phases.	Since the development model is linear, users cannot see the development results until the end of the entire process, thus increasing development risks.
Provides inspection nodes divided by stages for the project	The outstanding disadvantage of the waterfall model is that it cannot adapt to changes in user needs.

2.3 Agile development of software

What is agile development?

The core of Agile Development is Iterative Development and Incremental Development.

What is iterative development?

For large software projects, the traditional development method is to develop in a large cycle (such as one year), and the entire process is a "big development"; the iterative development method is different, it , that is, one "big development" becomes multiple "small developments". Each small development has the same process, so it looks like the same steps are being repeated.

For example, SpaceX wants to build a high-thrust rocket to send humans to Mars. However, instead of building a big rocket from the beginning, it first built the most rudimentary small rocket, Falcon 1. As a result, it exploded on the first launch, and it was not until the fourth launch that it successfully entered orbit. Then, the medium rocket Falcon 9 was developed, which was launched 70 times in nine years. Finally, the Falcon Heavy rocket was developed. If SpaceX didn't use iterative development, it might not be able to fly until now.

What is incremental development?

Each version of the software will add a complete function that users can perceive . In other words, iterations are divided according to new features.

For example, a real estate company develops a 10-building community. If the incremental development model is adopted, the company's first iteration is to deliver Building 1, and the second iteration is to deliver Building 2... Each iteration is to complete a complete building. Instead of digging the foundations of 10 buildings in the first iteration, building the skeleton of each building in the second iteration, and erecting the roof in the third iteration...

How to iterate in agile development?

Although agile development divides software development into multiple iterations , it also requires that each iteration is a complete software development cycle, and formal process management must be carried out in accordance with software engineering methodology.

What are the benefits of agile development?
- early delivery

The first benefit of agile development is early delivery, which greatly reduces costs .

Let's take the real estate company in the previous section as an example. If we follow the traditional "waterfall development model", we first dig the foundations of 10 buildings, then build the skeletons, and then erect the roofs. Each stage waits until the previous stage is completed. It may take It will take two years to deliver 10 buildings at once. In other words, if pre-sales are not taken into account, the project will have to wait until two years to get the money back. Agile development requires building No. 1 to be delivered after six months, and one building to be delivered every two months thereafter . Therefore, 10% of the money can be returned in half a year, and there will be cash flow every month, which greatly reduces the financial pressure.

reduce risk

The second benefit of agile development is to understand market needs in a timely manner and reduce the risk of product incompatibility . Please think about which situation will cause the smaller loss: building all 10 buildings and then discovering that they cannot be sold, or building the first building and then discovering that it cannot be sold and thus improving or stopping the construction of the next 9 buildings. ?

3 Jenkins installation configuration

3.1 Introduction to Jenkins

Jenkins is a popular open source continuous integration (Continuous Integration) tool that is widely used in project development and has functions such as automated build, testing and deployment. Official website: Jenkins .

Jenkins features:

The open source Java language develops continuous integration tools that support continuous integration and continuous deployment.
Easy to install , deploy and configure: It can be installed through yum, or downloaded war package and quickly installed and deployed through docker container, which can facilitate web interface configuration management.
Message notification and test report : Integrate RSS/E-mail to publish the build results through RSS or notify via e-mail when the build is completed to generate a JUnit/TestNG test report.
Distributed build : Supports Jenkins to allow multiple computers to build/test together.
File identification: Jenkins can track which build generates which jars, which build uses which version of jar, etc.
Rich plug-in support:
- Supporting extension plug-ins, you can develop tools suitable for your own team, such as git, svn, maven, docker, etc.

Jenkins installation and continuous integration environment configuration

1) First, developers submit code every day to the Git repository

2) Then, as a continuous integration tool, Jenkins uses the Git tool to pull the code from the Git warehouse to the integration server , and then cooperates with JDK, Maven and other software to complete code compilation, code testing and review, testing, packaging and other tasks. In this process, every If one step goes wrong, the entire process must be re-executed.

3) Finally, Jenkins distributes the generated jar or war package to the test server or production server, and testers or users can access the application.

3.2 Jenkins environment setup

3.2.1 Jenkins installation and configuration

Images from the data can be imported:

服务器用户名：root   密码：itcast
jenkins  用户名：itcast   密码：itcast

Install using YUM method

Add jenkins installation source:

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo --no-check-certificate

sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

Execute the yum command to install:

yum -y install jenkins

Use RPM installation package method

Jenkins installation package download address

wget https://pkg.jenkins.io/redhat-stable/jenkins-2.190.1-1.1.noarch.rpm

To perform the installation:

rpm -ivh jenkins-2.190.1-1.1.noarch.rpm

Configuration:

Modify configuration file:

vi /etc/sysconfig/jenkins

Modification content:

# 修改为对应的目标用户， 这里使用的是root
$JENKINS_USER="root"
# 服务监听端口
JENKINS_PORT="16060"

Directory permissions:

chown -R root:root /var/lib/jenkins
chown -R root:root /var/cache/jenkins
chown -R root:root /var/log/jenkins

Restart:

systemctl restart jenkins

If startup fails, an error message appears:

Starting Jenkins bash: /usr/bin/java: No such file or directory

Create a soft link to the JAVA environment:

ln -s /usr/local/jdk/bin/java /usr/bin/java

Management background initialization settings

http://192.168.200.100:16060/

An administrative password is required, which can be found at:

cat /var/lib/jenkins/secrets/initialAdminPassword

According to the default settings, install all recommended plug-ins.

This step takes a long time to wait. After the installation is completed, create an administrator user:

Configure access address:

After the configuration is completed, it will be restarted, and then you can see the management background:

3.2.2 Jenkins plug-in installation

Before implementing continuous integration, you need to ensure that the following plug-ins are installed successfully .

Maven Integration plugin: Maven integration management plugin.
Docker plugin: Docker integration plugin.
GitLab Plugin: GitLab integration plug-in.
Publish Over SSH: Remote file publishing plug-in.
SSH: Remote script execution plugin.

installation method:

Enter [System Management]-[Plug-in Management]
Click [Optional Plugins] on the tab page

Search for the plugin name in the filter box

Check the plug-in and click to install it directly.

Note that if there is no installation button, you need to change the configuration.
In the advanced configuration of installing the plug-in, modify the connection of the upgrade site to: http://updates.jenkins.io/update-center.json Save

3.2.3 Git installation configuration

yum installation method

yum -y install git

Install using source code package
- Install dependency packages

yum -y install curl-devel expat-devel gettext-devel openssl-devel zlib-devel
yum -y install gcc perl-ExtUtils-MakeMaker

If an old version has been installed before, uninstall it first. If it is not installed, ignore it.

yum remove git

Download source code package

cd /usr/local
wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-1.8.3.1.tar.gz
tar -xvf git-1.8.3.1.tar.gz

You can also install other versions, address: Index of /pub/software/scm/git/

Compile and install

cd git-1.8.3.1
make prefix=/usr/local/git all
make prefix=/usr/local/git install
echo "export PATH=$PATH:/usr/local/git/bin" >> /etc/bashrc
source /etc/bashrc

Check git version

[root@localhost jenkins]# git version
git version 1.8.3.1

3.2.4 Maven installation configuration

Download the installation package

Download address: Maven – Download Apache Maven

Unzip the installation package

cd /usr/local
unzip -o apache-maven-3.6.1.zip

Upload the local warehouse and decompress it

Configuration

Environment variable configuration

vi /etc/profile

Increase:

export MAVEN_HOME=/usr/local/maven/apache-maven-3.6.1
export PATH=$PATH:$MAVEN_HOME/bin

If the permissions are not enough, you need to increase the permissions of the current directory

chmod 777 /usr/local/maven/apache-maven-3.6.1/bin/mvn

Modify the image warehouse configuration:

vi /usr/local/maven/apache-maven-3.6.1/conf/settings.xml

You need to package and upload the local warehouse to the server (it will be automatically downloaded if you do not upload it)

Then specify the warehouse configuration after uploading

3.2.5 Docker installation configuration

Update package version

yum -y update

Uninstall old version

yum -y remove docker  docker-common docker-selinux docker-engine

Install software dependency packages

yum install -y yum-utils device-mapper-persistent-data lvm2

Set the yum source to Alibaba Cloud

sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

Check the docker version after installation

docker -v

start up

Set up startup:

systemctl enable docker

Start docker

systemctl start docker

3.2.6 Docker Registry private warehouse installation and configuration

3.3 Jenkins tool configuration

Enter [System Management] --> [Global Tool Configuration]

MAVEN configure global settings

Specify JDK configuration

Specify MAVEN directory

Specify DOCKER directory

If you don’t know the directory where docker is installed, you can use the command to view the directory where whereis dockerdocker is installed.

4 Backend project deployment

4.1 Multiple environment switching

In the process of project development and deployment, there are generally three sets of project environments

Development: development environment
Production: production environment
Test: test environment

For example: the development environment's mysql connection is local, and the production environment needs to connect to the online mysql environment.

4.2 Multi-environment switching-multi-environment configuration in microservices

1. Add new configuration in bootstrap.yml in microservice

server:
  port: 51801
spring:
  application:
    name: leadnews-user
  cloud:
    nacos:
      discovery:
        server-addr: 192.168.200.130:8848
      config:
        server-addr: 192.168.200.130:8848
        file-extension: yml
  profiles:
    active: dev

2. Add configuration files for each environment in the configuration center of nacos, for example, add them to the user microservice

Modify bootstrap.yml to add content

spring:
  profiles:
    active: dev

Create the corresponding nacos multi-environment configuration:

Precautions:

The DataID attribute naming is standardized:

prefix，
- By default, ${spring.application.name} is used , which can also be configured through spring.cloud.nacos.config.prefix.
spring.profile.active is the profile corresponding to the current environment.
- For details, please refer to the Spring Boot documentation. Note: When spring.profile.active is empty, the corresponding connector - will not exist, and the splicing format of dataId becomes ${prefix}.${file-extension}
file-exetension, which is the data format of configuration content, can be configured through the configuration item spring.cloud.nacos.config.file-extension . Currently only properties and yaml types are supported.

4.3 Overall idea

Goal: Deploy the app-related microservices of Heima Toutiao to the server 192.168.200.100

4.4 Service integration Docker configuration

Goal: Each microservice deployed is to first create a docker image and then create the corresponding container to start.

Method 1: The local microservice is packaged and uploaded to the server , and the Dockerfile is written.

Method 2: Using the dockerfile-maven-plugin plug-in, you can directly create the microservice as a mirror (more trouble-free)

Service integration Docker configuration

Each microservice introduces this dependency, taking the heima-leadnews-user microservice as an example

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>heima-leadnews-service</artifactId>
        <groupId>com.heima</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>heima-leadnews-user</artifactId>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <docker.image>docker_storage</docker.image>
    </properties>

    <build>
        <finalName>heima-leadnews-user</finalName>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.spotify</groupId>
                <artifactId>dockerfile-maven-plugin</artifactId>
                <version>1.3.6</version>
                <configuration>
                    <repository>${docker.image}/${project.artifactId}</repository>
                    <buildArgs>
                        <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
                    </buildArgs>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Service integration Dockerfile file

# 设置JAVA版本
FROM java:8
# 指定存储卷, 任何向/tmp写入的信息都不会记录到容器存储层
VOLUME /tmp
# 拷贝运行JAR包
ARG JAR_FILE
COPY ${JAR_FILE} app.jar
# 设置JVM运行参数， 这里限定下内存大小，减少开销
ENV JAVA_OPTS="\
-server \
-Xms256m \
-Xmx512m \
-XX:MetaspaceSize=256m \
-XX:MaxMetaspaceSize=512m"
#空参数，方便创建容器时传参
ENV PARAMS=""
# 入口点， 执行JAVA运行命令
ENTRYPOINT ["sh","-c","java -jar $JAVA_OPTS /app.jar $PARAMS"]

4.5 Jenkins basic dependency packaging configuration

Before running the microservice, you need to install the dependent jar packages in the local warehouse, so the first step should be to pull the code from git and install the basic dependencies into the warehouse.

1. Parent project heima-leadnews

2. Find the git repository you specified and set the username and password.

3. Install the basic dependency information to the local warehouse on the server

4. Execution

Execution log, partial screenshots, the following is the code pulled from git

Execution logs, partial screenshots, compilation and packaging

Execution log, partial screenshots, successful execution

4.6 Jenkins microservice packaging configuration

All microservices are packaged in a similar way, taking the heima-leadnews-user microservice as an example

1. Create a new task

2. Find the git repository you specified and set the username and password.

3. Execute maven command

clean install -Dmaven.test.skip=true  dockerfile:build -f heima-leadnews/heima-leadnews-service/heima-leadnews-user/pom.xml

Note: Configure according to your actual code path

-Dmaven.test.skip=true skips testing

dockerfile:build starts the dockerfile plug-in to build the container

-f heima-leadnews-user/pom.xml specifies the file to be built (must be pom)

4. And execute the shell script

if [ -n  "$(docker ps -a -f  name=$JOB_NAME  --format '{
     
     {.ID}}' )" ]
 then
 #删除之前的容器
 docker rm -f $(docker ps -a -f  name=$JOB_NAME  --format '{
     
     {.ID}}' )
fi
 # 清理镜像
docker image prune -f 
 # 启动docker服务
docker run -d --net=host -e PARAMS="--spring.profiles.active=prod"  --name $JOB_NAME docker_storage/$JOB_NAME

5. Execution log

Pull code

Compile and package

Build image

Clean the container and create a new container

4.7 Deploy services to remote servers

Goal: Use jenkins (192.168.200.100) to package and deploy microservices to the 192.168.200.130 server

4.7.1 Install and configure private warehouse

For the configuration of the continuous integration environment, Jenkins will publish a large number of microservices. To interact with multiple machines, you can use the save and export functions of docker images combined with SSH . However, this interaction is cumbersome, unstable, and inconvenient to manage. Here We achieve this by building a Docker private warehouse, which is somewhat similar to a GIT warehouse. It centrally manages resources and can be pulled or updated by the client.

Download the latest Registry image

docker pull registry:latest

Start the Registry mirror service

docker run -d -p 5000:5000 --name registry -v /usr/local/docker/registry:/var/lib/registry registry:latest

Mapping port 5000; -v associates the mirror data volume in the Registry with local files to facilitate the management and maintenance of data in the Registry.

View warehouse resources

Access address: http://192.168.200.100:5000/v2/_catalog

The startup is normal and you can see the return:

{"repositories":[]}

There is currently no image uploaded and empty data is displayed.

If the upload is successful, you can see the data:

Configure Docker client

For use in a normal production environment, the HTTPS service must be configured to ensure security. For internal development or testing of integrated LAN environments, a simple method can be used without security control.

First ensure that the Docker client has been installed on the machine in the continuous integration environment, and then make the following modifications:

vi /lib/systemd/system/docker.service

Modification content:

ExecStart=/usr/bin/dockerd --insecure-registry 192.168.200.100:5000

Point to the service IP and port where the Registry is installed.

Restart takes effect:

systemctl daemon-reolad
systemctl restart docker.service

4.7.2 Install plug-ins in jenkins

4.7.3 Jenkins system configuration remote server link

Location: Manage Jenkins-->Configure System

Need to add credentials

Location: Manage Jenkins-->Manage CreDentials

Add username and password to link to 130 server

4.7.4 Jenkins project creation is the same as other microservices

Create a project with reference to the previously created user microservice

4.7.5 Setting parameters

4.7.6 Build and execute Execute shell

maven command

clean install -Dmaven.test.skip=true dockerfile:build -f heima-leadnews/heima-leadnews-service/heima-leadnews-article/pom.xml

shell script

image_tag=$docker_registry/docker_storage/$JOB_NAME
echo '================docker镜像清理================'
if [ -n  "$(docker ps -a -f  name=$JOB_NAME  --format '{
     
     {.ID}}' )" ]
 then
 #删除之前的容器
 docker rm -f $(docker ps -a -f  name=$JOB_NAME  --format '{
     
     {.ID}}' )
fi
 # 清理镜像
docker image prune -f 

# 创建TAG
docker tag docker_storage/$JOB_NAME $image_tag
echo '================docker镜像推送================'
# 推送镜像
docker push $image_tag
# 删除TAG
docker rmi $image_tag
echo '================docker tag 清理 ================'

4.7.7 Executing scripts on remote servers

Shell script executed by remote server

echo '================拉取最新镜像================'
docker pull $docker_registry/docker_storage/$JOB_NAME

echo '================删除清理容器镜像================'
if [ -n  "$(docker ps -a -f  name=$JOB_NAME  --format '{
     
     {.ID}}' )" ]
 then
 #删除之前的容器
 docker rm -f $(docker ps -a -f  name=$JOB_NAME  --format '{
     
     {.ID}}' )
fi
 # 清理镜像
docker image prune -f 

echo '===============启动容器================'
docker run -d   --net=host -e PARAMS="--spring.profiles.active=prod" --name $JOB_NAME $docker_registry/docker_storage/$JOB_NAME

4.7.8 After the construction is completed, you can log in to the 130 server to check whether there are related images and containers.

mirror

container

4.8 Joint debugging test

1. Deploy the app-side gateway by referring to the heima-leadnews-user microservice in Jenkins.

2. Modify the configuration reverse proxy address in local nginx to 100 this server: heima-leadnews-app.conf

upstream  heima-app-gateway{
	server 192.168.200.100:51601;
}

3. Start nginx and open the page for testing

5 jenkins trigger configuration

5.1 URL triggers remote build

Trigger remote build and modify jenkins configuration as follows

Trigger build url: http://192.168.200.100:16060/job/leadnews-admin/build?token=88888888

5.2 Triggered after other projects are built

Configure the projects that need to be triggered

5.3 Scheduled build

Build periodically

The timing strings from left to right are: time, day, month, week

Timing construction-timing expression