Jihu GitLab enterprise-level CI/CD large-scale implementation practice guide (1)

Table of contents

template reference, reducing code redundancy and enhancing CI/CD build scalability

Problem 1: Redundant code and inefficient practices

Problem 2: Difficult to maintain and heavy workload

➤ local

➤ file

➤ remote

➤ template

Benefit 1: One modification takes effect in many places

Benefit 2: Efficient construction, simple and convenient

Component, creating a single source of trust for CI/CD Pipeline and simplifying CI/CD Pipeline construction

Construction of Component warehouse

Release of Component

Reference to Component

 Runner, a tool for efficient CI/CD construction

Exclusive + Shared, more flexible options

Dynamically expand and shrink capacity to improve resource utilization

Compliance assembly line, assisting the safe and compliant use of assembly lines

---

JiHu GitLab CI is built into the JiHu GitLab integrated platform, providing out-of-the-box CI/CD capabilities , and is also one of the CI tools loved by many users. Jihu GitLab CI's unique design mechanism and enterprise-level functional features can help enterprises improve CI/CD construction efficiency and reduce pipeline maintenance costs when implementing CI/CD practices on a large scale , while maintaining sufficient security compliance .

This article starts from the construction of CI/CD Pipeline and describes the use of Jihu GitLab CI in three major aspects:

1. Use  template , component to shorten pipeline writing time and improve maintainability;

2. Use Runner’s “fancy” gameplay to meet CI/CD Pipeline operation needs in different scenarios while reducing usage costs;

3. Use a compliance framework to ensure the compliant use of CI/CD Pipeline.

template reference, reducing code redundancy and enhancing CI/CD build scalability

Within an enterprise, a very common scenario is that different teams or different product lines have their own unique projects, and each project has a corresponding CI/CD pipeline. As the number of projects increases, the number of pipelines will continue to increase. Increasingly, there may be hundreds or even thousands of assembly lines within an enterprise.

Because the CI/CD pipeline is an automated form of software delivery (from coding to online), most pipelines will have a relatively high degree of similarity , and some stages or jobs are even exactly the same, such as in cloud native delivery scenarios. The application needs to be packaged into a mirror. The code to build using Jihu GitLab CI is as follows:

build:
  image: docker:latest
  stage: build
  services:
    - docker:20.10.7-dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:1.0.0 .
    - docker push $CI_REGISTRY_IMAGE:1.0.0

In addition, if they are all java or golang projects, the compilation or testing commands may be similar. This "repetition" will increase as the pipeline increases, and the following problems will also arise:

Problem 1: Redundant code and inefficient practices

If each pipeline has a similar stage or job with about 10 lines of code, then the number of duplicate codes in hundreds or thousands of pipelines is tens of thousands. This kind of code redundancy is itself an inefficient practice in the field of software development. If it is not refactored in time, it will become technical debt as the project evolves.

Problem 2: Difficult to maintain and heavy workload

In the process of optimizing the CI/CD Pipeline, some parts of the pipeline need to be modified, such as upgrading the dind version, or changing the build method from dind to kaniko in order to build the image safely, then the corresponding code must be changed. become:

  services:
    - docker:24.0.3-dind

and:

build:
  stage: build
  image:
    name: registry.jihulab.com/jh-xiaomage-devops/go-demo/kaniko:debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:1.0.0"

At this time, all pipelines must be transformed. The transformation of hundreds or even thousands of pipelines will be a huge workload, and the "copy and paste" process of a large amount of code is difficult to avoid without errors.

In the field of software research and development, an important way to solve redundant code is through abstraction + reuse: that is, abstracting the same (or similar) content into templates, "store" the templates in a certain place, and simply refer to the templates in other places. That’s it .

The same is true for CI/CD Pipeline. GiFox GitLab template is the built-in template engine function of GiFox GitLab CI. It can store abstracted templates in the project warehouse, and other projects can  include reference the template through syntax.

The usage of Jihu GitLab template is relatively flexible. First, you need to "make" the template, that is, extract the "duplicate" code and save it in a YAML file. For example, the above image build content can be written to a docker-image-build.gitlab-ci.yml file. Next use  include citation. Depending on where the template is stored, include there are four ways to reference it:

➤ local

Templates are located in the current project and  local are referenced using keywords . The usage syntax is as follows:

include:  
  - local: '/templates/docker-image-build.gitlab-ci.yml'

➤ file

The template and project are located in the same instance, but in different repositories, and are  file referenced using keywords . The usage syntax is as follows:

include  
  - project: xiaomage/templates  
  - ref: main    
  file: /templates/docker-image-build.gitlab-ci.yml

➤ remote

References to pipelines in remote warehouses, usually between different instances . The usage syntax is as follows:

include:   
  - remote: 'https://jihulab.com/xiaomage/teamplates/raw/main/docker-image-build.gitlab-ci.yml'

➤ template

A reference to the built-in template of JiHu GitLab . Based on its many years of experience, GitLab has accumulated many templates that can be directly reused, which can be used using template syntax. The most typical one is the reference of Jihu GitLab DevSecOps template. GitLab DevSecOps has key scanning, dependency scanning, SAST, DAST, container image scanning, fuzz testing and license compliance detection functions. All functions can be turned on with two lines of code.

Therefore, using template can bring the following benefits:

Benefit 1: One modification takes effect in many places

If you need to optimize the content of the pipeline, such as  dind upgrading the version of , you only need to make modifications in the template, and other referenced places will take effect, truly realizing "modify in one place, take effect in multiple places", which completely avoids The duplication of work caused by "change once, modify everywhere" will also reduce the redundancy of the pipeline.

Benefit 2: Efficient construction, simple and convenient

Template can achieve multi-level nesting, that is, templates are referenced within templates. The advantage of this is that the content of the template can be fine-grained. It may be a stage or a job. For example, container image construction is a template, and container image security scanning is another template. If you want to build a pipeline for a new project, you can directly use multiple templates to "build building blocks" to quickly complete the construction of the pipeline, and then make some changes to the parameters or processes according to the actual construction process of the project.

Of course, in order to use templates efficiently, there is another issue that needs to be paid attention to, and that is the overwriting of variables in templates .

In order to use templates flexibly, use the same set of templates to build multiple different instances. The key lies in the use of variables in the template. For example, when building a container image, the tag may differ from version to version. In this case, the tag can be set as a variable:

variables:
  IMAGE_TAG: 1.0.0

build:
  image: docker:latest
  stage: build
  services:
    - docker:20.10.7-dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$IMAGE_TAG .
    - docker push $CI_REGISTRY_IMAGE:$IMAGE_TAG

Just overwrite the variable directly at the reference point:

variables:
  IMAGE_TAG: "2.0.0"

include: 
  - remote: 'https://jihulab.com/xiaomage/teamplates/raw/main/docker-image-build.gitlab-ci.yml'

1.0.0 After overwriting, the value of the image tag changes  from the default  2.0.0 , which can meet the requirements of different scenarios and is both efficient and flexible.

Component, creating a single source of trust for CI/CD Pipeline and simplifying CI/CD Pipeline construction

The use of templates greatly reduces the difficulty for users to build CI/CD Pipelines. Through the reference + parameter coverage mode, Pipelines corresponding to scenarios can be quickly built. However, there is currently no single trusted source of templates to facilitate users to find what they want. While using Pipeline, users who are willing to contribute cannot contribute available templates to jointly build a prosperous Pipeline ecosystem.

To this end, Jihu GitLab has launched the CI/CD Component feature, whose purpose is to create a single trusted source for CI/CD Pipeline by turning different pipelines (or separate jobs) into different components and then publishing them to component warehouse. Other users can search in this warehouse to find the component they want. They can directly reference it when building the pipeline. The reference of multiple components can quickly build the entire complete pipeline. This is a huge change. Improve the user experience of using CI/CD.

What's more important is that users can publish some excellent pipelines (or individual jobs) that they think have been practiced in the form of components to the component warehouse, and jointly create a prosperous CI/CD Pipeline ecosystem through the continuous contributions and iterations of different users. , and finally build a single trusted source for the enterprise's internal CI/CD Pipeline , which not only improves the efficiency of CI/CD Pipeline construction, but also greatly improves security.

Note: CI/CD Component is currently experimental.

CI/CD Component diagram

Therefore, the core of component is: the construction of component warehouse, the release of component and the reference of component .

Construction of Component warehouse

Build an initial component repository by creating a GitLab repository and marking it as a component repository. The repository requires at least two files  README.md and  template.yml:

• README.md The components contained in the warehouse can be described to facilitate users to learn and use;

• template.yml It is the specific content of the component.

Different components can be distinguished through different directory structures (branch, tag, etc.), such as:

├── template.yml
├── README.md
├── .gitlab-ci.yml
├── forntend/
│   └── template.yml
└── backend/
    └── template.yml

The above directory indicates that there are three components available in this component warehouse:

• template.yml The component represented in the root directory  ;

• template.yml The component represented in the frontend directory  ;

• template.yml The component represented in the backend directory  .

You can mark a warehouse as a component warehouse through Project → Settings → General → Visibility, Project Functions, General → Enable CI/CD Directory Resources.

Release of Component

If you need to publish a component, you need to write the corresponding content into a  template.yml file, and then push the file to the component warehouse. Taking the above image construction as an example, write the following content into a file  template.yml :

spec:  
  inputs:    
    stage:      
      default: test    
    image:      
      default: docker:latest    
    tags:      
      default: tags    
    image_tag:      
      default: 1.0.0
      
component-job-build-image:  
  image: $[[ inputs.image ]]  
  stage: $[[ inputs.stage ]]  
  tags:    
    - $[[ inputs.tags ]]  
  script:     
    - docker login -u "$REGISTRY_USER" -p "$REGISTRY_PWD" REGISTRY_URL    
    - docker build -t dllhb/cicd-component:$[[ inputs.image_tag ]] .    
    - docker push dllhb/cicd-component:$[[ inputs.image_tag ]]

Then push it  jh.instance.url/username/component-project to the directory. Parameter Description:

• jh.instance.url: Jihu GitLab private deployment instance address;

• username：Jihu GitLab username;

• component-project:component warehouse name.

The above component is located in the root directory of the warehouse and has a  component-job-build-image job.

Reference to Component

If you want to reference the component published above in other Pipelines,  .gitlab-ci.yml use the following syntax to reference it:

include:  
  - component: jh.instance.url/username/component-project@main    
  inputs:      
    stage: build      
    image: docker:latest      
    tags: cicd      
    image_tag: 2.0.0
    
stages: [build]

Things to note are:

• include Completely write the path of the component (that is,  tempate.yml the existing path) in  it, and use @ to make it clear which version of the component is being referenced (can be represented by branches, commit hash, tags, etc.);

• inputs Write specific parameters in . 

Trigger the CI/CD Pipeline and you can see that  component-job-build-image jobthe execution is successful:

Similarly, if you want to  dind change the build method  kaniko, you do not need to replace the content of the above component, you only need to publish a kaniko-themed component again.

There are many ways to achieve this, such as placing it  template.yml in another directory of the component warehouse (not in the root directory, because dind there ), branches, and tags to indicate that this is a different component; for example, for the above component , the main branch represents  dind the component, then you can create a new  kaniko branch to store the component corresponding to kaniko, and finally  .gitlab-ci.yml specify the branch when referencing it:

include:  
  - component: jh-jhma.gitlab.cn/cicd-component/cicd-component-demo@kaniko   
  inputs:      
    stage: build      
    image: gcr.io/kaniko-project/executor:debug      
    tags: cicd      
    image_tag: 2.0.0
    
stages: [build]

dind Of course, the image must also be changed  from the original one  kinako. This only requires modifying  inputs the parameters.

You can get the same results by running a CI/CD Pipeline.

The introduction of component kicked off a new paradigm for GitLab CI/CD. This approach of creating a single source of trust for CI/CD Pipeline through user contributions is of great help to users in building a complete Pipeline. It not only speeds up the construction of CI/CD Pipeline, but also greatly reduces the cost for users to learn complex YAML syntax.

The codes demonstrated above are all stored on the Jihu GitLab privatized deployment instance, and the address is  https://jh-jhma.gitlab.cn/cicd-component .

 Runner, a tool for efficient CI/CD construction

Runner is an important component of GitLab CI, which can help run jobs defined in the CI/CD pipeline . When developers submit code changes, Jihu GitLab will "notify" the Runner to  .gitlab-ci.yml complete the construction, testing, deployment, etc. of the changed code according to the defined pipeline steps. During this process, the Runner will follow the selected executor (such as the shell for PowerShell). , docker for containers, etc.) to run jobs in different environments. Regarding the selection of executors, please refer to Jihu GitLab Executor Network .

Runner is like an "agent" that accepts requests from the "server" side (Jihu GitLab instance). Therefore, in order to meet the needs of different scenarios, Runner must be able to run in different installation methods on different OS and different CPU architectures.

Exclusive + Shared, more flexible options

Runner is divided into two categories: proprietary and shared:

• Proprietary: means that the Runner is only used for designated projects, usually using some information of the project (Runner register token) to register the Runner to the corresponding project;

• Sharing: means that the Runner is for the entire GitLab instance, which means that all projects under the entire instance can use these Runners. As for how to use it, who uses it first, and who uses it later, it is implemented by the internal scheduling mechanism of GitLab. .

The biggest advantages of proprietary runners are:

• Save time : The dedicated Runner only runs the CI/CD pipeline for the corresponding project, so there is no need to queue up and wait for the shared Runner to execute the CI/CD pipeline. As the number of projects and pipelines increases, queuing will take a lot of time;

• Autonomous and controllable : The proprietary Runner is installed by the user on a server that is controllable by the user. During use, if you want to debug the pipeline process or modify the Runner configuration, or even want to obtain some data during the running process, you can Log in directly to the corresponding Runner to perform operations.

Configuration information of proprietary Runner

The benefits of sharing the Runner are also obvious: users do not need to know too much information about the Runner, nor do they need to install and operate it themselves . It is a relatively trouble-free way.

Therefore, users can choose different Runner methods according to their own needs to complete the corresponding CI/CD pipeline operation.

Dynamically expand and shrink capacity to improve resource utilization

Runner can be closely bound to the dynamic scaling feature of cloud resources to achieve dynamic scaling of Runner: when a CI/CD pipeline needs to be run, Runner uses some resources (CPU, memory, etc.) to execute all jobs. When the CI/CD pipeline After the operation ends (success or failure), the corresponding resources are released and the environment is restored.

For example, you can use containers to run Runner. The most typical one is to use Kubernetes to run Jihu GitLab Runner.

When executing CI/CD, Kubernetes will dynamically create a pod, and the pod will  .gitlab-ci.yml generate corresponding containers based on the stage and corresponding image described in the file (all containers are in one pod and share the internal resources of the pod). The running of the pipeline is Inside the container, when the pipeline ends, the pod will be deleted, and the data and resources required during the operation will be released.

In addition, you can also use serverless products provided by cloud vendors to dynamically expand the Runner's capacity and improve resource utilization.

Compliance assembly line, assisting the safe and compliant use of assembly lines

There is also a scenario encountered during the use of pipelines: a certain process requires all project pipelines to run. For example, an image security scan must be performed at the end of the image build. If there is a security vulnerability, the running of the pipeline needs to be terminated. The solution in this case is often to add a  include container image scanning link to the pipeline for all projects. However, as the number of projects increases (hundreds or even thousands), this means a huge amount of duplication of work, and the operation cannot be guaranteed. accuracy.

And the right solution is: a compliance pipeline.

The compliance pipeline is a security feature built into the GitLab CI/CD pipeline. It mainly ensures that all projects in the group can run specified compliance jobs . By configuring the compliance framework at the group level and selecting the compliance pipeline job that each project must run, all projects under this group will run this compliance job, even newly created projects under this group. Projects also perform this compliance job by default.

To use the compliance pipeline, you first need to configure the compliance framework at the group level. In Groups → Settings → General → Compliance Framework , select New Compliance Framework, then fill in the compliance framework name, description, compliance pipeline configuration (that is, where the compliance pipeline is stored), and finally select a background color That’s it.

If this compliance framework is set as the default compliance framework of the group, all new projects under this group will use this compliance framework by default, run this compliance pipeline by default, and there will be a compliance framework label on the project page. generate.

Then you need to write the compliance pipeline to a project under this group (such as Compliance-Pipeline). Taking the container image building and scanning pipeline as an example,  .gitlab-ci.yml write the following content in the file:

include: 
  - remote: 'https://jihulab.com/xiaomage/teamplates/raw/main/docker-image-build.gitlab-ci.yml'
  - template: Security/Container-Scanning.gitlab-ci.yml

All subsequent new projects will perform container image building and container image scanning, instead of the project's own pipeline.

If you want the project's own pipeline to be executed, you only need to merge the contents of the compliance pipeline with the content of the project's own pipeline. For example, the built-in pipeline needs to use cosgin to sign and verify the packaged container image to prevent the image from being tampered with:

stages:
  - singature
  - verfication

image-singature:
  stage: singature
  tags:
    - cosign
  image: 
    name: dllhb/cosign:1.0.0
    entrypoint: [""]
  before_script:
    - mkdir ~/.docker
    - cat "$DOCKER_CRED_FILE" > ~/.docker/config.json
    - cat "$COSIGN_KEY" > /tmp/cosign.key
    - export COSIGN_PASSWORD="$COSIGN_PASSWORD"
  script:
    - cosign sign --key /tmp/cosign.key $CI_REGISTRY_IMAGE:1.0.0

image-verfication:
  stage: verfication
  tags:
    - cosign
  image: 
    name: dllhb/cosign:1.0.0
    entrypoint: [""]
  before_script:
    - cat "$COSIGN_PUB" > /tmp/cosign.pub
    - export COSIGN_PASSWORD="$COSIGN_PASSWORD"
  script:
    - cosign verify --key /tmp/cosign.pub $CI_REGISTRY_IMAGE:1.0.0

The above pipeline needs to be introduced into the compliance pipeline:

include:
  - project: 'Compliance-Pipeline-Group/regular-pipeline'
    file: '.gitlab-ci.yml'

Eventually, the pipelines of other projects under this group will perform the four steps of packaging, scanning, signing, and verification of container images.

Therefore, the compliance framework is selected to mark certain projects that must meet certain compliance requirements or require additional supervision, and then complete the compliance work by executing the compliance pipeline.