An efficient and reliable CI/CD pipeline is the basis for an IT organization to achieve rapid delivery of software services. Nowadays, a large number of enterprises use Jenkins clusters to build their delivery pipelines. However, how to manage the differentiation of a large number of Jenkins slaves? How to easily and quickly implement the horizontal expansion of Jenkins capabilities? How to achieve high availability of pipelines? How to effectively utilize idle Jenkins Slave resources? The above problems have always plagued cluster administrators. With the rapid development of virtualization technology in the past two years, modern tools such as Docker and Kubernetes have completely subverted the delivery process of the delivery team. Management provides a new way of thinking.
There are many popular CI tools in Setup
. Since this article discusses the application of CI/CD pipelines in enterprises, and considering that enterprises will not be willing to open source code access rights to third parties, SaaS-based CI tools such as Travis CI are naturally Passed, because of Jenkins' dominance in the CI field, the CI tools in this article only involve Jenkins. In addition, this article assumes that you have some basic understanding of Jenkins, Docker, Kubernetes and other tools.
Introduction to Continuous Delivery Pipelines
A brief introduction to the continuous delivery pipeline, as shown in the figure below, in simple terms, the pipeline workflow is like this. When the code base has code changes, the continuous integration server (Jenkins) will listen to the code changes and automatically trigger the first stage - continuous Integration phase, what this phase does is automatic build, unit testing, static code analysis, and report generation. If all the steps are successfully passed as expected, it will automatically enter the next stage - the automated testing stage. Before running the automated test suite, you must first pass the artifact of the first stage. If you choose java as the development language If it is the generated war package, if it is ruby, it is the gem file, which is deployed to the test server. After the deployment is completed, the automated test suite is triggered, including the acceptance test, capacity test, performance test, etc., which will be automatically executed. If all test cases pass successfully (some If the company also needs to do some manual testing (exploratory testing, etc.), then this version will be marked as a release candidate. Once the business needs it, the corresponding candidate version will be released to the production environment through one-click deployment. If you want to learn more about continuous delivery, please read the famous book "Continues Delivery" by David Farley and Jez Humble.
Practice and Pain Points
The above process needs to be done in this way. First, the work of each stage in the above figure needs to be scripted. Specifically, a build script needs to be written to complete the steps of compilation, unit testing, static code analysis, and report generation. In order to complete the work of the continuous integration stage, then the automated test script, which can trigger the automated test suite and generate relevant reports, and finally need to write a deployment script to deploy the output of the continuous integration stage to the test environment (of course This script will also be reused in the final release stage), what needs to be paid attention to here is to ensure that the deployment is repeatable, the effect of deploying the same output N times is the same as the effect of deploying only once, which is often said Idempotent. Next, it's Jenkins' turn to appear. First, we need to configure Jenkins to monitor changes in the code base, which means that as long as there is code moved in, the corresponding pipeline will be triggered, and then we use Jenkins job or pipeline to transfer these The scripts of the stages are connected in series (the figure below is an example of a job), and such a simple CI/CD pipeline is completed.
As shown in the figure above, each Job is responsible for running a script in a certain stage. It can be simply analogized to run the script in the continuous integration stage for Job1. The source code is migrated from the code base, compiled, unit tested, static code analysis and report generation. Job2 First deploy the output of the continuous integration phase to the test environment and run the automated test suite, generate reports and mark the output, and Job3 is used for on-demand release. The language or running environment of the script run by each Job will be different, and it can be selected to run on the corresponding Slave through the label method.
However, in the large-scale application of CI/CD pipelines at the enterprise level, due to the existence of multiple teams and multiple products in the enterprise, different teams will choose different technical implementation methods according to the characteristics of the products themselves, which means that the product implementation language will have There are many kinds, such as java, C#, ruby, python, nodejs ... Then the corresponding CI/CD pipeline needs to provide the compilation environment of all languages and install the relevant dependency packages, of course, in order to reduce dependencies, avoid conflicts and better Smart administrators who manage these compilation environments will choose to allow each slave to only run Jobs in a specific programming language, as shown in the following figure:
The Jenkins cluster Maser in the picture above is only used to schedule and collect logs. All jobs will be diverted by the Master to the corresponding Slaves to run according to different labels. The implementation of this cluster is our last resort in the VM era. Although it can solve most problems, it also brings a lot of troubles.
Let's take a look at the good news brought by virtualization technology. The following picture is a Jenkins cluster built based on Kubernetes and Docker. To avoid confusion, I omitted the Master node in the Kubernetes cluster. We see that Jenkins Mater runs on a Kubernetes Node in the form of a Docker container and stores all Jenkins related data in a volume. Jenkins Slave also runs in each Node in the form of a Docker container. The reason why the dashed line is used to represent the Slave is Because the slave does not always exist, it will be dynamically created on demand and automatically deleted. Briefly introduce this method of dynamically creating and registering a slave. Its workflow is that when the Jenkins Master receives a build request, it will dynamically create a Jenkins Slave running in a Docker container according to the requirements of the label and register it with On the Master, and then run the corresponding Job, when the Job is completed, the Slave will be logged out, and the Docker container where it is located will also be automatically deleted.
This CI/CD pipeline based on Docker and Kubernetes brings many benefits to the Jenkins cluster:
Let's complete a simple implementation:
1. First, you need to install a Kubernetes cluster, please refer to https://kubernetes.io/docs/setup/ After the Kubernetes installation is completed, you first need to use the following two commands and files to deploy A Jenkins to Kubernetes cluster:
tip: The part marked in red is very important. The open port 8080 is used to access the Jenkins web portal; and the dynamically created Jenkins slave will establish a connection with the master through the 50000 (modifiable) port by default.
Check the jenkins installation is running:
2. View the Jenkins log, log in to Jenkins with the administrator password, and install the Kubernetes plugin.
3. Configure Kubernetes cloud
Manage Jenkins/Configure System/ Add a new cloud/ Kubernetes:
Add pod template / Kubernetes Pod Template ：
Click Save and you're done. Let's test whether the Jenkins cluster with dynamically registered Slave is working properly. First, log in to Jenkins to create a simple free style job, and specify that this Job can only be run on the agent whose Label is "jnlp"
.
Click build now, and you will find that a miracle has happened. Jenkins, which has not registered any Slave, dynamically creates a Slave and registers it on the Master, and then runs the corresponding Job. When the Job is finished, the Slave is automatically cleared.
PS: This is just a simple implementation. In the practice of the enterprise, we need different build environments. We need to build our own Jenkins slave Image based on the image of jenkinsci/jnlp-slave and save it to the private Registry. The corresponding Kubernetes needs to pull the Image from the private Registry.
references:
There are many popular CI tools in Setup
. Since this article discusses the application of CI/CD pipelines in enterprises, and considering that enterprises will not be willing to open source code access rights to third parties, SaaS-based CI tools such as Travis CI are naturally Passed, because of Jenkins' dominance in the CI field, the CI tools in this article only involve Jenkins. In addition, this article assumes that you have some basic understanding of Jenkins, Docker, Kubernetes and other tools.
Introduction to Continuous Delivery Pipelines
A brief introduction to the continuous delivery pipeline, as shown in the figure below, in simple terms, the pipeline workflow is like this. When the code base has code changes, the continuous integration server (Jenkins) will listen to the code changes and automatically trigger the first stage - continuous Integration phase, what this phase does is automatic build, unit testing, static code analysis, and report generation. If all the steps are successfully passed as expected, it will automatically enter the next stage - the automated testing stage. Before running the automated test suite, you must first pass the artifact of the first stage. If you choose java as the development language If it is the generated war package, if it is ruby, it is the gem file, which is deployed to the test server. After the deployment is completed, the automated test suite is triggered, including the acceptance test, capacity test, performance test, etc., which will be automatically executed. If all test cases pass successfully (some If the company also needs to do some manual testing (exploratory testing, etc.), then this version will be marked as a release candidate. Once the business needs it, the corresponding candidate version will be released to the production environment through one-click deployment. If you want to learn more about continuous delivery, please read the famous book "Continues Delivery" by David Farley and Jez Humble.
Practice and Pain Points
The above process needs to be done in this way. First, the work of each stage in the above figure needs to be scripted. Specifically, a build script needs to be written to complete the steps of compilation, unit testing, static code analysis, and report generation. In order to complete the work of the continuous integration stage, then the automated test script, which can trigger the automated test suite and generate relevant reports, and finally need to write a deployment script to deploy the output of the continuous integration stage to the test environment (of course This script will also be reused in the final release stage), what needs to be paid attention to here is to ensure that the deployment is repeatable, the effect of deploying the same output N times is the same as the effect of deploying only once, which is often said Idempotent. Next, it's Jenkins' turn to appear. First, we need to configure Jenkins to monitor changes in the code base, which means that as long as there is code moved in, the corresponding pipeline will be triggered, and then we use Jenkins job or pipeline to transfer these The scripts of the stages are connected in series (the figure below is an example of a job), and such a simple CI/CD pipeline is completed.
As shown in the figure above, each Job is responsible for running a script in a certain stage. It can be simply analogized to run the script in the continuous integration stage for Job1. The source code is migrated from the code base, compiled, unit tested, static code analysis and report generation. Job2 First deploy the output of the continuous integration phase to the test environment and run the automated test suite, generate reports and mark the output, and Job3 is used for on-demand release. The language or running environment of the script run by each Job will be different, and it can be selected to run on the corresponding Slave through the label method.
However, in the large-scale application of CI/CD pipelines at the enterprise level, due to the existence of multiple teams and multiple products in the enterprise, different teams will choose different technical implementation methods according to the characteristics of the products themselves, which means that the product implementation language will have There are many kinds, such as java, C#, ruby, python, nodejs ... Then the corresponding CI/CD pipeline needs to provide the compilation environment of all languages and install the relevant dependency packages, of course, in order to reduce dependencies, avoid conflicts and better Smart administrators who manage these compilation environments will choose to allow each slave to only run Jobs in a specific programming language, as shown in the following figure:
The Jenkins cluster Maser in the picture above is only used to schedule and collect logs. All jobs will be diverted by the Master to the corresponding Slaves to run according to different labels. The implementation of this cluster is our last resort in the VM era. Although it can solve most problems, it also brings a lot of troubles.
- A single point of dependence on Jenkins Master becomes a single point. Once the Jenkins Master goes down, it will be catastrophic, and the entire CI/CD pipeline will be in an unavailable state.
- It is difficult to maintain a large number of differentiated Jenkins Slaves, which are very difficult to manage. Due to the existence of differences, maintenance and upgrades are almost always done manually, and labor costs are high.
- It is not easy to expand. For example, we found that the number of requests sent by the pipeline to the Java7 Slave is relatively large, and queuing often occurs. In order to alleviate this situation, the administrator needs to add a Slave that can compile Java7 applications, so what should the administrator do? First, you need to prepare a VM, then install java7 and all dependent software packages, and finally configure the Slave-related information label, etc. and register the newly installed Slave to the Master, which basically requires human intervention, which is very inconvenient to expand.
- Waste of resources Every Jenkins Slave Server is a real running VM. When the Slave Sever is idle, the resources it occupies cannot be released, because the Slave may be required to complete the related Job at any time.
Let's take a look at the good news brought by virtualization technology. The following picture is a Jenkins cluster built based on Kubernetes and Docker. To avoid confusion, I omitted the Master node in the Kubernetes cluster. We see that Jenkins Mater runs on a Kubernetes Node in the form of a Docker container and stores all Jenkins related data in a volume. Jenkins Slave also runs in each Node in the form of a Docker container. The reason why the dashed line is used to represent the Slave is Because the slave does not always exist, it will be dynamically created on demand and automatically deleted. Briefly introduce this method of dynamically creating and registering a slave. Its workflow is that when the Jenkins Master receives a build request, it will dynamically create a Jenkins Slave running in a Docker container according to the requirements of the label and register it with On the Master, and then run the corresponding Job, when the Job is completed, the Slave will be logged out, and the Docker container where it is located will also be automatically deleted.
This CI/CD pipeline based on Docker and Kubernetes brings many benefits to the Jenkins cluster:
- The highly available Jenkins Master is deployed on the Kubernetes cluster. Once the container runs abnormally and unexpectedly exits, Kubernetes will automatically start a new Jenkins with the same Docker image and attach the volume to the newly created Docker container to ensure no Any data will be lost, achieving high availability of the Jenkins cluster.
- Automatic scaling Because every time a job is run, the Jenkins master will dynamically create a Jenkins slave. After the job is completed, the Docker container where the slave will be logged out will also be automatically deleted, and the occupied resources will be automatically released. That is to say, the more jobs are requested at the same time, the more slave containers will be generated, and the more resources will be occupied, and vice versa, and this dynamic scaling does not require human intervention at all.
- Complete isolation Since each job is run in a brand new Jenkins slave, the possibility of conflict between jobs running at the same time is avoided.
- Ease of maintenance Compared to the previous practice that each Jenkins Slave was a fixed VM, the cluster built in this way no longer maintains a fixed VM but the Docker image required to create a dynamic Slave. We can easily use Docker File to Build works with our own Docker images and store them in a private Docker registry, which is very easy to maintain.
- Easy to expand When we find that there are a large number of jobs waiting to be executed in the Jenkins queue because the resources of the kubernetes cluster are insufficient, it is easy to initialize a kubernetes node and add it to the cluster, which is very convenient to achieve horizontal expansion.
Let's complete a simple implementation:
1. First, you need to install a Kubernetes cluster, please refer to https://kubernetes.io/docs/setup/ After the Kubernetes installation is completed, you first need to use the following two commands and files to deploy A Jenkins to Kubernetes cluster:
- kubectl create -f jenkins-deployment.yaml
- kybectl create -f jenkins-service.yaml
tip: The part marked in red is very important. The open port 8080 is used to access the Jenkins web portal; and the dynamically created Jenkins slave will establish a connection with the master through the 50000 (modifiable) port by default.
Check the jenkins installation is running:
2. View the Jenkins log, log in to Jenkins with the administrator password, and install the Kubernetes plugin.
3. Configure Kubernetes cloud
Manage Jenkins/Configure System/ Add a new cloud/ Kubernetes:
Add pod template / Kubernetes Pod Template ：
Click Save and you're done. Let's test whether the Jenkins cluster with dynamically registered Slave is working properly. First, log in to Jenkins to create a simple free style job, and specify that this Job can only be run on the agent whose Label is "jnlp"
.
Click build now, and you will find that a miracle has happened. Jenkins, which has not registered any Slave, dynamically creates a Slave and registers it on the Master, and then runs the corresponding Job. When the Job is finished, the Slave is automatically cleared.
PS: This is just a simple implementation. In the practice of the enterprise, we need different build environments. We need to build our own Jenkins slave Image based on the image of jenkinsci/jnlp-slave and save it to the private Registry. The corresponding Kubernetes needs to pull the Image from the private Registry.
references: