[CI/CD practice based on GitLab] 04, GitLab Pipeline practice (middle)

Used to specify files or directories that need to be cached between jobs. Only paths within the project workspace can be used. Do not use the cache to pass artifacts between stages, as the cache is designed to store the runtime dependencies needed to compile the project.

If defined outside the job scope cache, it means it is a global setting and all jobs will use that definition. Disabled if not defined globally or per job.

cache:paths

Use paths the command to select the file or directory to be cached. The path is relative to the project directory and cannot be directly linked outside the project directory.

$CI_PROJECT_DIRproject directory.

Defining the cache in the job build will cache all .jar files in the target directory.

build:
  script: test
  cache:
    paths:
      - target/*.jar

When cache:paths is defined globally, it will be overridden by the job. The following example will cache the binaries directory.

cache:
  paths:
    - my/files

build:
  script: echo "hello"
  cache:
    key: build
    paths:
      - target/

Since the cache is shared between jobs, if different jobs use different paths, there will be a problem of cache coverage. How to make different jobs cache different caches? Can be set differently cache:key.

cache:key cache token

To mark the cache, you can configure the job and branch as the key to implement branch and job-specific caching. When different s are defined for different jobs cache:key, a separate cache will be allocated for each job. cache:key 变量可以使用任何预定义变量，默认 default , starting with GitLab 9.0, everything is shared between pipelines and jobs by default.

Set up caching by branch:

cache:
  key: ${CI_COMMIT_REF_SLUG}

cache:key:files file changes automatically create a cache

files: The cache is automatically regenerated when the file changes (files can specify up to two files), and the specified file is checked when submitting.

Generate a key to calculate the SHA checksum according to the specified file, if the file has not changed, the value is default.

cache:
  key:
    files:
      - Gemfile.lock
      - package.json
  paths:
    - vendor/ruby
    - node_modules

cache:key:prefix combination produces SHA checksum

prefix: Allows the value of the given prefix to be combined with the key generated by the specified file.

The global cache is defined here. If the file changes, the value is rspec-xxx111111111222222, and if it does not change, it is rspec-default.

cache:
  key:
    files:
      - Gemfile.lock
    prefix: ${CI_JOB_NAME}
  paths:
    - vendor/ruby

rspec:
  script:
    - bundle exec rspec

For example, adding $CI_JOB_NAME prefix will make the key look like: rspec-feef9576d21ee9b6a32e30c5c79d0a0ceb68d1e5, and the job cache is shared between different branches, and if the branch changes Gemfile.lock, that branch will cache:key:files have a new SHA checksum. A new cache key will be generated, and a new cache will be created for that key. If Gemfile.lock unchanged, the prefix is added default, so the key in the example is rspec-default.

cache:policy cache policy

Default: download the file at the beginning of the execution and reupload it at the end. called pull-push caching strategy.

policy: pullSkip the download step

policy: pushSkip the upload step

stages:
  - setup
  - test

prepare:
  stage: setup
  cache:
    key: gems
    paths:
      - vendor/bundle
  script:
    - bundle install --deployment

rspec:
  stage: test
  cache:
    key: gems
    paths:
      - vendor/bundle
    policy: pull
  script:
    - bundle exec rspec ...

Comprehensive example (1) global cache

before_script:
  - echo "before-script!!"

variables:
  DOMAIN: example.com

cache: 
  paths:
   - target/

stages:
  - build
  - test
  - deploy
  
build:
  before_script:
    - echo "before-script in job"
  stage: build
  tags:
    - build
  only:
    - master
  script:
    - ls
    - id
    - mvn clean package -DskipTests
    - ls target
    - echo "$DOMAIN"
    - false && true ; exit_code=$?
    - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;
    - sleep 2;
  after_script:
    - echo "after script in job"


unittest:
  stage: test
  tags:
    - build
  only:
    - master
  script:
    - echo "run test"
    - echo 'test' >> target/a.txt
    - ls target
  retry:
    max: 2
    when:
      - script_failure
  
deploy:
  stage: deploy
  tags:
    - build
  only:
    - master
  script:
    - echo "run deploy"
    - ls target
  retry:
    max: 2
    when:
      - script_failure
      

after_script:
  - echo "after-script"

Pipeline log analysis

When the build job runs, the project code will be packaged, and then the target directory will be generated. The job finishes creating the cache.

Start the second job test. At this time, the target directory in the current directory will be deleted (because of the git comparison).

Get the cache target directory generated by the first job.

Start the third job, also delete the target directory first, and then get the cache of the second job. Finally the current cache is generated.

Runner cache

When doing this experiment, I first cleared the working directory and history cache of the project on the local runner.

[root@run01 ~]# cd /home/gitlab-runner/builds/1Cxihk7-/0/demo/demo-maven-service/
[root@run01 demo-maven-service]# ls
Jenkinsfile  README.md  aaaaa  jenkins  pom.xml  src  target
[root@run01 demo-maven-service]# cd ..
[root@run01 demo]# ls
demo-maven-service  demo-maven-service.tmp
[root@run01 demo]# rm -fr demo-maven-service
[root@run01 demo]# rm -fr demo-maven-service.tmp/
[root@run01 demo]# cd
[root@run01 ~]# cd /home/gitlab-runner/cache/
[root@run01 cache]# ls
demo
[root@run01 cache]# rm -rf *

The project code will not be deleted by default, and it can be found that it is the cache of the second job. (Because the third job in the above example did not modify the cache content)

[root@run01 cache]# cd /home/gitlab-runner/builds/1Cxihk7-/0/demo/demo-maven-service/
[root@run01 demo-maven-service]# ls
Jenkinsfile  README.md  aaaaa  jenkins  pom.xml  src  target
[root@run01 demo-maven-service]# cd ..
[root@run01 demo]# ls
demo-maven-service  demo-maven-service.tmp
[root@run01 demo]# rm -fr *
[root@run01 demo]# ls
[root@run01 demo]# ls
demo-maven-service  demo-maven-service.tmp
[root@run01 demo]# cd demo-maven-service
[root@run01 demo-maven-service]# ls
Jenkinsfile  README.md  aaaaa  jenkins  pom.xml  src  target
[root@run01 demo-maven-service]# cat target/a.txt
test

Enter the runner cache directory to view the cache.

[root@run01 ~]# cd /home/gitlab-runner/cache/demo/demo-maven-service/default/
[root@run01 default]# ls
cache.zip
[root@run01 default]# unzip cache.zip
Archive:  cache.zip
   creating: target/
  inflating: target/a.txt
   creating: target/classes/
   creating: target/classes/com/
   creating: target/classes/com/mycompany/
   creating: target/classes/com/mycompany/app/
  inflating: target/classes/com/mycompany/app/App.class
   creating: target/maven-archiver/
  inflating: target/maven-archiver/pom.properties
   creating: target/maven-status/
   creating: target/maven-status/maven-compiler-plugin/
   creating: target/maven-status/maven-compiler-plugin/compile/
   creating: target/maven-status/maven-compiler-plugin/compile/default-compile/
  inflating: target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
  inflating: target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
   creating: target/maven-status/maven-compiler-plugin/testCompile/
   creating: target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/
  inflating: target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
  inflating: target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
  inflating: target/my-app-1.1-SNAPSHOT.jar
   creating: target/test-classes/
   creating: target/test-classes/com/
   creating: target/test-classes/com/mycompany/
   creating: target/test-classes/com/mycompany/app/
  inflating: target/test-classes/com/mycompany/app/AppTest.class
[root@run01 default]# ls
cache.zip  target
[root@run01 default]# cd target/
[root@run01 target]# ls
a.txt  classes  maven-archiver  maven-status  my-app-1.1-SNAPSHOT.jar  test-classes
[root@run01 target]# cat a.txt
test

At this point run the pipeline job again, the first job using the last cache generated by the previous job.

Enter the runner cache directory to view the changes in cache.zip time.

[root@run01 default]# ll
total 12
-rw------- 1 gitlab-runner gitlab-runner 9172 July 19 10:27 cache.zip
drwxrwxr-x 6 root          root           127 July 19 10:05 target

Conclusion: The global cache takes effect for all jobs that do not define the cache in the job. In this case, if each job changes the cache directory, the cache will be overwritten.

Comprehensive example (2)

Control the caching strategy: For example, in the build phase, we need to generate new target directory content, and we can optimize the settings to not download the cache when the job is running.

before_script:
  - echo "before-script!!"

variables:
  DOMAIN: example.com

cache: 
  paths:
   - target/

stages:
  - build
  - test
  - deploy
  
build:
  before_script:
    - echo "before-script in job"
  stage: build
  tags:
    - build
  only:
    - master
  script:
    - ls
    - id
    - cat target/a.txt
    - mvn clean package -DskipTests
    - ls target
    - echo "$DOMAIN"
    - false && true ; exit_code=$?
    - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;
    - sleep 2;
  after_script:
    - echo "after script in job"
  cache:
    policy: pull   #不下载缓存


unittest:
  stage: test
  tags:
    - build
  only:
    - master
  script:
    - echo "run test"
    - echo 'test' >> target/a.txt
    - ls target
    - cat target/a.txt
  retry:
    max: 2
    when:
      - script_failure
  
deploy:
  stage: deploy
  tags:
    - build
  only:
    - master
  script:
    - cat target/a.txt
    - echo "run deploy"
    - ls target
    - echo "deploy" >> target/a.txt
  retry:
    max: 2
    when:
      - script_failure
      

after_script:
  - echo "after-script"

artifacts

Used to specify a list of files or directories that should be appended to the job if the job succeeds or fails. When the job is complete, the artifacts are sent to GitLab and are available for download in the GitLab UI.

artifacts:paths artifact creation

Paths are relative to the project directory and cannot link directly outside of the project directory.

Set artifacts to target directory:

artifacts:
  paths:
    - target/

Disable artifact delivery

job:
  stage: build
  script: make build
  dependencies: []

You probably only want to create artifacts for tagged releases to avoid filling the build server store with temporary build artifacts. Create artifacts for tags only ( default-job no artifacts are created):

default-job:
  script:
    - mvn test -U
  except:
    - tags

release-job:
  script:
    - mvn package -U
  artifacts:
    paths:
      - target/*.war
  only:
    - tags

artifacts:expose_as Expose artifacts

Keywords expose_as can be used to expose job artifacts in the merge request UI.

For example, to match a single file:

test:
  script: 
    - echo 1
  artifacts:
    expose_as: 'artifact 1'
    paths: 
      - path/to/file.txt

With this configuration, GitLab will add a link in the relevant merge request pointed to file1.txt.

Product browsing:

Please note the following points:

Each merge request can expose up to 10 job artifacts.
If a directory is specified, the link will point to the job artifact browser if there are multiple files in the directory.
If GitlabPages is enabled, artifacts can be rendered for individual file artifacts with .html .htm .txt .json .log extensions.

artifacts:name artifact name

name Defines the name of the artifact archive created by the directive. It is possible to use a unique name for each archive. artifacts:name Variables can use any of the predefined variables. The default name is artifacts, download artifacts instead artifacts.zip.

Create an archive with the name of the current job:

job:
  artifacts:
    name: "$CI_JOB_NAME"
    paths:
      - binaries/

Create an archive with the name of an internal branch or tag (including only the binaries directory):

job:
  artifacts:
    name: "$CI_COMMIT_REF_NAME"
    paths:
      - binaries/

Create an archive with the name of the current job and the current branch or tag (only the binaries directory):

job:
  artifacts:
    name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
    paths:
      - binaries/

To create an archive with the current stage name and branch name:

job:
  artifacts:
    name: "$CI_JOB_STAGE-$CI_COMMIT_REF_NAME"
    paths:
      - binaries/

artifacts:when artifact creation conditions

Used to upload artifacts when or despite job failures. on_success uploads artifacts only when the job succeeds, which is the default; on_failure uploads artifacts only when the job fails; always uploads artifacts regardless of job status.

To upload artifacts only if the job fails:

job:
  artifacts:
    when: on_failure

artifacts:expire_in Artifact retention time

The validity period of the artifact is counted from the time when it is uploaded and stored in GitLab. If no expiration is defined, it defaults to 30 days.

expire_in The value is the elapsed time in seconds, unless a unit is provided. Examples of resolvable values:

‘42’
‘3 mins 4 sec’
‘2 hrs 20 min’
‘2h20min’
‘6 mos 1 day’
‘47 yrs 6 mos and 4d’
‘3 weeks and 2 days’

Expires in one week:

job:
  artifacts:
    expire_in: 1 week

artifacts:reports

Used to collect test reports, code quality reports and security reports from jobs. Display these reports in GitLab's UI.

Note: Test reports will be collected regardless of job outcome (success or failure).

`artifacts:reports:junit 单元测试报告`

Collect the junit unit test report, the collected JUnit report will be uploaded to GitLab as an artifact and will be automatically shown in the merge request.

build:
  stage: build
  tags:
    - build
  only:
    - master
  script:
    - mvn test
    - mvn cobertura:cobertura
    - ls target
  artifacts:
    name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
    when: on_success
    expose_as: 'artifact 1'
    paths:
      - target/*.jar
    reports:
      junit: target/surefire-reports/TEST-*.xml

Note: If you are using a JUnit tool that exports to multiple XML files, you can specify multiple test report paths in one job, and they will be automatically concatenated into one file. Use a filename pattern ( junit: rspec-*.xml), an array of filenames ( junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]), or a combination ( junit: [rspec.xml, test-results/TEST-*.xml]).

If this page cannot be displayed, system settings need to be changed. This option may increase resource usage, it is disabled by default and needs to be enabled.

登录gitlab
su -  git
$ gitlab-rails console
--------------------------------------------------------------------------------
 GitLab:       12.9.0 (9a382ff2c82) FOSS
 GitLab Shell: 12.0.0
 PostgreSQL:   10.12
--------------------------------------------------------------------------------
Feature.enable(:junit_pipeline_view)Loading production environment (Rails 6.0.2)
irb(main):001:0>
irb(main):002:0>
irb(main):003:0> Feature.enable(:junit_pipeline_view)
=> true
irb(main):004:0>

`artifacts:reports:cobertura 覆盖率`

Collected Cobertura coverage reports are uploaded to GitLab as artifacts and automatically displayed in merge requests.

build:
  stage: build
  tags:
    - build
  only:
    - master
  script:
    - mvn test
    - mvn cobertura:cobertura
    - ls target
  artifacts:
    name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
    when: on_success
    expose_as: 'artifact 1'
    paths:
      - target/*.jar
    reports:
      junit: target/surefire-reports/TEST-*.xml
      cobertura: target/site/cobertura/coverage.xml

$ gitlab-rails console
--------------------------------------------------------------------------------
 GitLab:       12.9.0 (9a382ff2c82) FOSS
 GitLab Shell: 12.0.0
 PostgreSQL:   10.12
--------------------------------------------------------------------------------


Loading production environment (Rails 6.0.2)
irb(main):001:0>
irb(main):002:0>
irb(main):003:0> Feature.enable(:coverage_report_view)
=> true

Maven integrates cobertura plugin

<plugins>       
  <!--  cobertura plugin start -->
  <plugin>
    <groupId>org.codehaus.mojo</groupId>  
    <artifactId>cobertura-maven-plugin</artifactId>  
    <version>2.7</version>  
    <configuration>  
      <formats>  
          <format>html</format>  
          <format>xml</format>  
      </formats>  
  	</configuration>  
  </plugin>       
  <!--  cobertura plugin end -->

</plugins>

Execute mvn cobertura:cobertura to run tests and generate Cobertura coverage reports.

dependencies

Defines a list of jobs to fetch artifacts from, jobs can only be defined from stages executed before the current stage. Defining an empty array will skip downloading any artifacts for that job without regard to the status of previous jobs, so if it fails or is a manual job that didn't run, no error will occur.

Dependent jobs will fail if the artifacts of the jobs that are set as dependencies are expired or deleted.

Comprehensive example (3)

before_script:
  - echo "before-script!!"

variables:
  DOMAIN: example.com


cache: 
  paths:
   - target/

stages:
  - build
  - test
  - deploy
  
build:
  before_script:
    - echo "before-script in job"
  stage: build
  tags:
    - build
  only:
    - master
  script:
    - ls
    - id
    - mvn test
    - mvn cobertura:cobertura
    - ls target
    - echo "$DOMAIN"
    - false && true ; exit_code=$?
    - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;
    - sleep 2;
  after_script:
    - echo "after script in job"
  artifacts:
    name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
    when: on_success
    #expose_as: 'artifact 1'
    paths:
      - target/*.jar
      #- target/surefire-reports/TEST*.xml
    reports:
      junit: target/surefire-reports/TEST-*.xml
      cobertura: target/site/cobertura/coverage.xml
  coverage: '/Code coverage: \d+\.\d+/'


unittest:
  dependencies:
    - build
  stage: test
  tags:
    - build
  only:
    - master
  script:
    - echo "run test"
    - echo 'test' >> target/a.txt
    - ls target
  retry:
    max: 2
    when:
      - script_failure
  
deploy:
  stage: deploy
  tags:
    - build
  only:
    - master
  script:
    - echo "run deploy"
    - ls target
  retry:
    max: 2
    when:
      - script_failure
      
      

after_script:
  - echo "after-script"

Previous article: [CI/CD Practice Based on GitLab] 03. GitLab Pipeline Practice (Part 1)_Stars.Sky's Blog-CSDN Blog

Next article: [CI/CD Practice Based on GitLab] 05. GitLab Pipeline Practice (Part 2)_Stars.Sky's Blog-CSDN Blog