Learn about continuous integration testing practices based on Jenkins and Kubernetes!

Author | Liu Chunming, Editor-in-Chief | Carol

Produced | CSDN Cloud Computing (ID: CSDNcloud)

Cover Image | CSDN Download to Visual China

At present, in order to reduce the cost of using the machine, the company has conducted an inventory of all AWS virtual machines and found that some of the machines with low utilization are used by the test team as Jenkins Slave machines. This is not what we expected. Using a virtual machine as a Jenkins Slave, there will be a lot of waste, because after the test job is completed, when the slave is idle, the virtual machine resources are not released.

In addition to low resource utilization, the virtual machine as a Jenkins Slave has other disadvantages, such as uneven resource allocation , some Slave jobs to be queued, and some Slaves may be idle. In addition, capacity expansion is inconvenient. Using a virtual machine as a slave, if you want to add Jenkins Slave, you need to manually mount the virtual machine to the Jenkins Master and configure the environment for the slave, which makes management very inconvenient and maintenance-consuming.

In 2019, the operation and maintenance team built a Kubernetes container cloud platform. In order to achieve the company's goal of reducing the cost of machine use, my connected car testing team considered fully moving Jenkins Slave to the Kubernetes container cloud platform. The main purpose is to improve the resource utilization of Jenkins Slave, and provide more flexible and flexible capacity expansion to meet the needs of more and more test jobs for Slave.

This article is a summary of our practice.

Overall structure

We know that Jenkins uses the Master-Slave architecture. The Master is responsible for managing the Job, and the Slave is responsible for running the Job. In our company, the Master is built on a virtual machine, and the Slave comes from the Kubernetes platform. Each Slave is a Pod in the Kubernetes platform. The Pod is the atomic scheduling unit of Kubernetes. More basic knowledge of Kubernetes is not introduced too much. In this article, all you need to remember is that Pod is Jenkins Slave.

The schematic diagram of the Jenkins Slave cluster built on Kubernetes is as follows.

In this architecture, Jenkins Master is responsible for managing and testing the job. In order to be able to use the resources on the Kubernetes platform, you need to install Kubernetes-plugin on the Master.

The Kubernetes platform is responsible for generating Pods, which are used as Jenkins Slave to execute Job tasks. When Job is scheduled on Jenkins Master, Jenkins Master initiated by Kubernetes-plugin to Kubernetes internet request, Kubernetes generate the corresponding Pod objects according Pod template, Pod objects initiates JNLP request to Jenkins Master, so as to connect the Jenkins Master, once After the connection is successful, you can execute the job on the Pod.

The container images used in Pods come from Harbor. Here, three images are used in a Pod, which are Java images , Python images , and JNLP images . The Java mirror provides a Java environment that can be used to compile and execute test code written in Java. The Python mirror provides a Python environment that is used to execute test code written in Python. The JNLP mirror is an official Slave mirror provided by Jenkins.

Using Kubernetes as Jenkins Slave, how to solve the aforementioned problems of low resource utilization and uneven resource allocation when using virtual machines, and realize the dynamic and flexible expansion of Slave?

First of all, only when the Jenkins Master has a job scheduled, it will apply to the Kubernetes Pod to create a Jenkins Slave. After the test job is executed, the slave used will be recycled by Kubernetes. It is not like when a virtual machine is used as a slave, there are situations where the slave is idle, thereby improving the utilization rate of computing resources.

Second, the main problem of uneven resource allocation is between different test groups, because of different test environments and dependencies, Jenkins Slave cannot be shared. The Kubernetes platform breaks the barrier of sharing. As long as there are computing resources in the Kubernetes cluster, you can apply for Jenkins Slave suitable for your project from it, so that the phenomenon of Job queuing will no longer occur.

With the help of Kubernetes, it is even simpler to implement Slave's dynamic elastic expansion. Because Kubernetes inherently supports elastic expansion. When it is monitored that Kubernetes resources are insufficient, you only need to add Node nodes to it through the operation and maintenance platform. For testing, this step is completely transparent.

Configure Jenkins Master

To use Kubernetes as a Jenkins Slave, the first step is to install the Kubernetes plugin on the Jenkins Master. The installation method is very simple. Log in to Jenkins with a Jenkisn administrator account, search for Kubernetes on the Manage Plugin page, check and install.

The next step is to configure Kubernetes connection information on Jenkins Master. Jenkins Master connection Kubernetes cloud need to configure the three key messages: the name , the address and certificate . All configuration information is shown in the figure below.

The name will be used in Jenkins Pipeline. When configuring multiple Kubernetes clouds, you need to specify a different name for each cloud.

The Kubernetes address refers to the address of the Kubernetes API server. The Jenkins Master initiates a request to schedule Pods to this address through the Kubernetes plugin.

The Kubernetes service certificate key is used to establish a connection with the Kubernetes API server. The method is to obtain the content of certificate-authority-data in /root/.kube/config from the /root/.kube/config file of the Kubernetes API server , And converted to base64 encoded files.

# echo certificate-authority-data的内容 | base64 -D > ~/ca.crt

The content of ca.crt is the Kubernetes service certificate key.

The credential in the picture above is a pxf file generated using the client's certificate and key. First, convert the contents of client-certificate-data and client-key-data in /root/.kube/config into base64 encoded files.

# echo client-certificate-data的内容 | base64 -D > ~/client.crt
# echo client-key-data的内容 | base64 -D > ~/client.crt

Make a pxf file based on these two files:

# openssl pkcs12 -export -out ~/cert.pfx -inkey ~/client.key -in ~/client.crt -certfile ~/ca.crt
# Enter Export Password:
# Verifying - Enter Export Password:

Customize a password and keep in mind.

Click Add, select Certificate, click Upload certificate, select the cert.pfx file generated earlier, enter the password when generating the cert.pfx file, and the certificate is added.

Then configure the Jenkins URL and the number of pods that can be scheduled at the same time.

After the configuration is complete, you can click the "Test Connection" button to test whether you can connect to Kubernetes. If Connection test successful is displayed, the connection is successful and there is no problem with the configuration.

After configuring the Kubernetes plug-in, configure some public tools on the Jenkins Master as needed. For example, I configured allure to generate reports. In this way, when these tools are used in Jenkins Slave, they will be automatically installed in Jenkins Slave.

Customize Jenkins Pipeline

After configuring the Kubernetes connection information, you can use kubernetes as an agent in the test Job's Pipeline. The difference from using a virtual machine as Jenkins Slave is mainly in the pipeline.agent part. The following code is the complete Jenkinsfile content.

pipeline {
    agent {
      kubernetes{
          cloud 'kubernetes-bj' //Jenkins Master上配置的Kubernetes名称
          label 'SEQ-AUTOTEST-PYTHON36' //Jenkins slave的前缀
          defaultContainer 'python36' // stages和post步骤中默认用到的container。如需指定其他container，可用语法 container("jnlp"){...}
          idleMinutes 10 //所创建的pod在job结束后的空闲生存时间
          yamlFile "jenkins/jenkins_pod_template.yaml" // pod的yaml文件
      }
    }
    environment {
        git_url = '[email protected]:liuchunming033/seq_jenkins_template.git'
        git_key = 'c8615bc3-c995-40ed-92ba-d5b66'
        git_branch = 'master'
        email_list = '[email protected]'
    }
    options {
        buildDiscarder(logRotator(numToKeepStr: '30'))  //保存的job构建记录总数
        timeout(time: 30, unit: 'MINUTES')  //job超时时间
        disableConcurrentBuilds() //不允许同时执行流水线
    }
    stages {
        stage('拉取测试代码') {
            steps {
                git branch: "${git_branch}", credentialsId: "${git_key}", url: "${git_url}"
            }
        }
        stage('安装测试依赖') {
            steps {
                sh "pipenv install"
            }
        }
        stage('执行测试用例') {
            steps {
                sh "pipenv run py.test"
            }
        }
    }
    post {
        always{
            container("jnlp"){ //在jnlp container中生成测试报告
                allure includeProperties: false, jdk: '', report: 'jenkins-allure-report', results: [[path: 'allure-results']]
            }   
        }
    }
}

In the above Pipeline, the core part related to this article is the agent.kubernetes section. This section describes how to generate Jenkins Slave on the kubernetes platform.

cloud is the Kubernetes name configured on the Jenkins Master, which is used to identify which Kubernetes cloud is used by the current Pipeline.

The label is the prefix of the Jenkins Slave name. It is used to distinguish different Jenkins Slaves. When an exception occurs, you can use this name to debug in the Kubernetes cloud.

defaultContainer, in Jenkins Slave, I defined three containers, which were introduced earlier. The defaultContainer represents the default container in which the code runs in stages and post stages in the pipeline. In other words, if you do not specify a container in the stages and post stages, the code is run in the defaultContainer by default. If you want to use other containers to run the code, you need to specify it in a way similar to container ("jnlp") {...}.

idleMinutes specifies the length of time that Jenkins Slave can be retained after the test job running on Jenkins Slave ends. During this time, Jenkins Slave will not be recycled by Kubernetes. If there is a test job with the same label scheduled during this time, you can continue to use this idle Jenkins Slave. The purpose of this is to improve the utilization of Jenkins Slave and avoid frequent scheduling by Kubernetes, because it is time-consuming to successfully generate a Jenkins Slave.

yamlFile, this file is a standard Kubernetes Pod template file. Kubernetes generates Pod objects based on this file, which are used as Jenkins Slave. This file defines three containers (Container) and scheduling rules and external storage. This file uses Kubernetes as the core file of the Jenkins Slave cluster. The content of this file will be described in detail below.

At this point, the Pipeline of the test job is established.

Custom Jenkins Slave template

When using a virtual machine as a Jenkins Slave, if a new virtual machine is added, we need to initialize the virtual machine, mainly to install tool software, dependent packages, and connect to the Jenkins Master. The same is true when using Kubernetes cloud as a Jenkins Slave cluster. You need to define the operating system, dependent software, and external disks used by Jenkins Slave. It's just that the information is written in a Yaml file, which is a standard template file for Kubernetes' Pod objects. Kubernetes will generate a Pod based on this Yaml file and connect it to the Jenkins Master.

The contents of this Yaml file are as follows:

apiVersion: v1
kind: Pod
metadata:
  # ① 指定 Pod 将产生在Kubernetes的哪个namespace下，需要有这个namespace的权限
  namespace: sqe-test  
spec:
  containers:
    # ② 必选，负责连接Jenkins Master，注意name一定要是jnlp
    - name: jnlp
      image: swc-harbor.nioint.com/sqe/jnlp-slave:root_user
      imagePullPolicy: Always
      # 将Jenkins的WORKSPACE（/home/jenkins/agent）挂载到jenkins-slave
      volumeMounts:
        - mountPath: /home/jenkins/agent
          name: jenkins-slave

    # ③ 可选，python36环境，已安装pipenv，负责执行python编写的测试代码
    - name: python36
      image: swc-harbor.nioint.com/sqe/automation_python36:v1
      imagePullPolicy: Always
      # 通过cat命令，让这个container保持持续运行
      command:
        - cat
      tty: true
      env:
        # 设置pipenv的虚拟环境路径变量 WORKON_HOME
        - name: WORKON_HOME 
          value: /home/jenkins/agent/.local/share/virtualenvs/
      # 创建/home/jenkins/agent目录并挂载到jenkins-slave Volume上
      volumeMounts: 
        - mountPath: /home/jenkins/agent
          name: jenkins-slave
      # 可以对Pod使用的资源进行限定，可调。尽量不要用太多，够用即可。
      resources: 
        limits:
          cpu: 300m
          memory: 500Mi

    # ④ 可选，Java8环境，已安装maven，负责执行Java编写的测试代码
    - name: java8
      image: swc-harbor.nioint.com/sqe/automation_java8:v2
      imagePullPolicy: Always
      command:
        - cat
      tty: true
      volumeMounts:
        - mountPath: /home/jenkins/agent
          name: jenkins-slave

  # ⑤ 声明一个名称为 jenkins-slave 的 NFS Volume，多个container共享
  volumes:
    - name: jenkins-slave
      nfs:
        path: /data/jenkins-slave-nfs/
        server: 10.125.234.64
  # ⑥ 指定在Kubernetes的哪些Node节点上产生Pod
  nodeSelector:
    node-app: normal
    node-dept: sqe

Through the above Yaml file, you can see that three containers are defined in the Pod through spec.containers, namely jnlp responsible for connecting to Jenkins Master, python36 responsible for running Python code, and java8 responsible for running Java code. We can compare Jenkins Slave to a pod, and the container inside is like a bean in the pod. Each bean has a different function.

At the same time, a volume called jenkins-slave is also declared. The jnlp container mounts the Jenkins WORKSPACE directory (/ home / jenkins / agent) to jenkins-slave. At the same time, the two containers python36 and java8 also mount the directory / home / jenkins / agent to the jenkins-slave. Therefore, the modification of the / home / jenkins / agent directory in any container can read the modified content in the other two containers. The main benefit of mounting external storage is that the test results and the virtual environment can be persisted. Especially after the virtual environment is persisted, instead of performing a test each time to create a new virtual environment, but reuse the existing virtual environment, Speed up the entire test execution process.

In addition, it also specifies which Namespace namespace of kubernetes to use and on which Node nodes Jenkins Slave is generated. Regarding other details of this Yaml file, I have written it in the comments of the file, you can refer to it for understanding.

Custom container image

The three containers used in Jenkins Slave were introduced earlier. Let's take a look at the images of these three containers separately.

First of all, DockerHub (https://hub.docker.com/r/jenkinsci/jnlp-slave) provides the official image of Jenkins Slave. Here we switch the default user in the official image to the root user, otherwise when executing test cases , Permission problems may occur. The Dockerfile of the JNLP container image is as follows:

FROM jenkinsci/jnlp-slave:latest
LABEL maintainer="[email protected]"
USER root

The Python image is pipenv installed in the official Python 3.6.4 image. Because the current Python projects of our team are all dependent on pipenv to manage the projects. Let me talk about it here, pipenv is an upgraded version of pip, which can not only create an independent virtual environment for your project, but also automatically maintain and manage the dependent packages of the project. Unlike pip which uses requirements.txt to manage dependencies, pipenv uses Pipefile to manage dependencies. The benefits here will not be introduced. Interested friends can check the official documentation of pipenv https://github.com/pypa/pipenv. The Dockerfile of the Python image is as follows:

FROM python:3.6.4
LABEL maintainer="[email protected]"
USER root
RUN pip install --upgrade pip
RUN pip3 install pipenv

The Java image is based on the maven image extension on DockerHub. The main change is to put the maven configuration file settings.xml used inside the company into the mirror. The complete Dockerfile is as follows:

FROM maven:3.6.3-jdk-8
LABEL maintainer="[email protected]"
USER root

# 设置系统时区为北京时间
RUN mv /etc/localtime /etc/localtime.bak && \
    ln -s /usr/share/zoneinfo/Asia/Shanghai  /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone # 解决JVM与linux系统时间不一致问题
# 支持中文
RUN apt-get update && \
    apt-get install locales -y && \
    echo "zh_CN.UTF-8 UTF-8" > /etc/locale.gen && \
    locale-gen
# 更新资源地址
ADD settings.xml /root/.m2/

# 安装jacococli
COPY jacoco-plugin/jacococli.jar  /usr/bin
RUN  chmod +x /usr/bin/jacococli.jar

After making the container image, we will push it to the company's internal harbor so that kubernetes can quickly pull the image. You can make your own container image according to your actual situation and project needs.

Perform automated testing

Through the previous steps, we have completed the preparation of using Kubernetes as Jenkins Slave. The next step is to execute the test job. Compared with using a virtual machine to perform a test job, this step is actually the same.

Create a Pipeline-style Job and configure it as follows:

After the configuration is complete, click Build to start the test.

Performance optimization

Unlike the virtual machine as Jenkins Salve, Kubernetes generating Jenkins Slave is a dynamic creation process, because it is dynamically created, it involves efficiency issues. There are two ways to solve the efficiency problem. On the one hand, try to use the existing Jenkins Slave to run the test job. On the other hand, it is to speed up the efficiency of generating Jenkins Slave. Below we will look at the specific optimization measures from these two aspects.

7.1 Make full use of the existing Jenkins Slave

Make full use of the existing Jenkins Slave, you can start from two aspects.

On the one hand, setting idleMinutes allows Jenkins Slave to not be destroyed immediately after executing the test job, but can be idle for a period of time. If a test job is started during this time, it can be assigned to be executed on it, which improves The existing utilization rate of Jenkins Slave also avoids the time-consuming creation of Jenkins Slave.

On the other hand, in more test job pipelines, the same label is used, so that after the current test job ends, the Jenkins Slave used can also be used by the test job that will be started and uses the same lable. For example, the Jenkins Slave used by job1 is

DD-SEQ-AUTOTEST-PYTHON, then when test job1 ends, use the same test job2 to start, you can directly use the Jenkins Slave used by test job1.

7.2 Speed up the scheduling efficiency of Jenkins Slave

The complete process of generating Jenkins Slave on Kubernetes and adding it to Jenkins Master is:

Jenkins Master calculates the current load situation;
According to the load, Jenkins Master initiates requests to the Kubernetes API server through the Kubernetes Plugin as needed;
The Kubernetes API server schedules Pods to the Kubernetes cluster;
After the Pod is generated, it is automatically connected to the Jenkins Master through the JNLP protocol.

The last three steps are very fast, mainly affected by the network. In the first step, after a series of algorithm calculations, Jenkins Master found that no Jenkins Slave was available before deciding to initiate a request to the Kubernetes API server. This process is not efficient under the default startup configuration of Jenkins Master. It often causes a new test job to wait for a while before it starts to generate Pods on Kubernetes.

Therefore, it is necessary to modify the startup items of Jenkins Master, which mainly involves the following parameters:

-Dhudson.model.LoadStatistics.clock=2000 
-Dhudson.slaves.NodeProvisioner.recurrencePeriod=5000 
-Dhudson.slaves.NodeProvisioner.initialDelay=0 
-Dhudson.model.LoadStatistics.decay=0.5 
-Dhudson.slaves.NodeProvisioner.MARGIN=50 
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

Jenkins Master will calculate the cluster load at intervals. The time interval is determined by hudson.model.LoadStatistics.clock. The default is 10 seconds. We will adjust it to 2 seconds to speed up the frequency of the Master to calculate the cluster load, so as to know more Changes in load. For example, it used to take up to 10 seconds to know how many jobs need to be scheduled for execution. Now it only takes 2 seconds.

When Jenkins Master calculated the cluster load, it was found that there was no Jenkins Slave available. The Jenkins master will notify the NodeProvisioner of the Kubernetes Plugin to produce Pods at recurrencePeriod intervals. Therefore, the recurrencePeriod value cannot be smaller than hudson.model.LoadStatistics.clock, otherwise multiple Jenkins slaves will be generated.

initialDelay is a delay time, originally used to ensure that the static Jenkins Slave and Master are connected, because we are using the Kubernetes plugin to dynamically generate the Jenkins slave, there is no static Jenkins Slave, so we set the parameter to 0.

The original meaning of the parameter hudson.model.LoadStatistics.decay is to suppress the jitter of the master load, which has a great influence on the load value obtained by the evaluation. The default decay is 0.9. We set decay to 0.5 to allow relatively large fluctuations in the load. The load evaluated by Jenkins Master is the actual load as much as possible, and the number of Jenkins Slaves required for evaluation.

hudson.slaves.NodeProvisioner.MARGIN and hudson.slaves.NodeProvisioner.MARGIN0, these two parameters align the calculated load up to an integer, which may generate an additional slave to improve efficiency.

Add the above parameters to the Jenkins Mater startup process, and restart Jenkins Master to take effect.

java -Dhudson.model.LoadStatistics.clock=2000 -Dxxx -jar jenkins.war

to sum up

This article introduces the advantages of using Kubernetes as a continuous integration test environment, and details the use method and optimizes its performance. In this way, the shortcomings of virtual machine as Jenkins Slave are perfectly solved.

In addition to the benefits that automated testing can benefit from Kubernetes, in the process of setting up a performance testing environment, with the help of Kubernetes' dynamic elastic expansion mechanism, the creation of large-scale pressure test clusters has obvious advantages in terms of efficiency and convenience.

Author introduction: Liu Chunming, software testing technology evangelist, ten-year test veteran, CSDN blog expert, MSTC conference lecturer, ArchSummit lecturer, running the "Ming Shuo software testing" public account. Good at test framework development, test platform development, continuous integration, test environment governance, etc., familiar with server-side testing, APP testing, Web testing and performance testing.

【END】