If you're looking for an automated "Pull Request" or branch-based Jenkins CI/CD pipeline, this guide will help you get a solid understanding of how to do it using Jenkins multi-branch pipelines. Jenkins multi-branch pipeline is one of the best ways to design a CI/CD workflow because it is completely git-based pipeline as code. In this guide, I will discuss all the key concepts involved in Jenkins multi-branch pipeline setup
Let's start with multi-branch pipeline basics. Specifically, in this section, I'll describe what a multibranch pipeline is and why using it is essential for all Jenkins CI/CD pipelines. I'll also show you how multi-branch pipelines work with detailed workflow diagrams.
Multi-branch pipeline is a concept of automatically creating Jenkins Pipeline based on Git branches. That means, it can automatically discover new Git branches as they are created in SCM (Github), and automatically create a pipeline for that branch. When the pipeline build starts, Jenkins uses the Jenkinsfile in that branch for the build stage.
SCM can be a Github, Bitbucket or Gitlab repository.
If you don't want the selected branch to appear in the automatic pipeline with Java regular expressions, you can choose to exclude. Multi-branch pipelines support PR-based branch discovery. This means that if someone makes a PR (pull request) from a branch, the branch will be automatically discovered in the pipeline. If this configuration is enabled, a build will only be triggered when a PR is raised. So if you're looking for a PR based Jenkins build workflow, this is a good choice.
You can add conditional logic to your Jenkinsfile to build jobs based on branch requirements. For example, if you want your feature branch to only run unit tests and Sonar analysis, you can set a condition to skip the deployment phase using a when condition, as shown below.
So whenever a developer submits a PR from the feature branch to another branch, the pipeline will run the unit test and Sonar analysis stages, skipping the deploy stage. Also, multi-branch pipelines are not limited to continuous delivery applications. You can also use it to manage infrastructure code.
How do multi-branch pipelines work?
I'll walk you through a basic build and deploy workflow to understand how multibranch pipelines work.
Let's say I want the Jenkins pipeline to build and deploy the application under the following conditions.
-
Developers start on feature branches by committing code to them.
-
Whenever a developer pulls a PR from a feature branch to develop the branch, the Jenkins pipeline should trigger to run unit tests and static code analysis.
-
After successfully testing the code in the feature branch, the developer merges the PR into the develop branch.
-
When the code is ready to be released, the developer raises a PR from the develop branch to master. It should trigger a build pipeline that will run unit test cases, code analysis and deploy to dev/QA environment.
As you can see from the above conditions, there is no manual triggering of Jenkins jobs, and whenever there is a branch pull request, the pipeline needs to be triggered automatically and run the required steps for that branch. This workflow establishes a great feedback loop for engineers and avoids relying on DevOps teams to build and deploy in non-production environments. Developers can check the build status on Github and decide what to do next.
This workflow is easily achieved with Jenkins multi-branch pipelines. The image below shows what a multi-branch pipeline workflow for the above example build process would look like
This is how multi-branch pipelines work.
-
When a developer creates a PR from a feature branch to develop the branch, Github sends a webhook with the PR information to Jenkins.
-
Jenkins receives the PR, and finds the relevant multi-branch pipeline and automatically creates the branch pipeline. It then runs the job following the steps mentioned in the Jenkinsfile in the feature branch. During checkout, the source and target branches in the PR are merged. The PR merge will be blocked on Github until the build status is returned from Jenkins.
-
After the build is complete, Jenkins will update the status to a Github PR. Now you will be able to merge the code. Also, if you want to see the Jenkins build log, you can find the Jenkins build log link under the PR status.
Multi-branch Pipleline Jenkinsfile
Before getting started with the implementation, let's look at an example Jenkinsfile for a multi-branch pipeline Jenkins that can be used in a pipeline.
In order for multi-branch pipelines to work, you need to have a Jenkinsfile in your SCM repository.
If you are learning/testing, you can use the multibranch pipeline Jenkinsfile provided below. It has a checkout phase and other phases that echo messages.
Alternatively, you can clone and use the Github repository with this Jenkinsfile
NOTE : Replace the agent label "master" with your Jenkins agent name. master will also work, but it is not recommended to run it in a real project environment.
pipeline {
agent {
node {
label 'master'
}
}
options {
buildDiscarder logRotator(
daysToKeepStr: '16',
numToKeepStr: '10'
)
}
stages {
stage('Cleanup Workspace') {
steps {
cleanWs()
sh """
echo "Cleaned Up Workspace For Project"
"""
}
}
stage('Code Checkout') {
steps {
checkout([
$class: 'GitSCM',
branches: [[name: '*/main']],
userRemoteConfigs: [[url: 'https://github.com/spring-projects/spring-petclinic.git']]
])
}
}
stage(' Unit Testing') {
steps {
sh """
echo "Running Unit Tests"
"""
}
}
stage('Code Analysis') {
steps {
sh """
echo "Running Code Analysis"
"""
}
}
stage('Build Deploy Code') {
when {
branch 'develop'
}
steps {
sh """
echo "Building Artifact"
"""
sh """
echo "Deploying Code"
"""
}
}
}
}
Setting up a Jenkins multi-branch pipeline
Here I will walk you step by step to set up a multi-branch pipeline on Jenkins. The setup will be based on Github and the latest Jenkins 2.x version. You can also use Bitbucket or Gitlab as an SCM source for multibranch pipelines.
Step 1: Create a "New Project" on the Jenkins home page.
Step 2: Select Multi-Branch Pipeline from the options and click OK.
Step 3: Click "Add Source" and select Github.
Step 4: Under the Authentication field, select Jenkins and create an authentication using your Github username and password.
Step 5: Select the created credentials and provide your Github repository to verify the credentials as shown below.
If you're testing a multibranch pipeline, you can clone the demo Github repository and use it. https://github.com/devopscube/multibranch-pipeline-demo.
Step 6: Select the desired option to match your requirement. You can choose to discover all branches in the repository, or just the ones with Pull Requests.
Pipelines can also discover branches with PRs from forked repositories.
Selecting these options depends on the desired workflow.
You can choose additional behaviors from the "Add" button.
For example, if you choose not to discover all branches from the repository, you can choose the regular expression or wildcard method to discover branches from the repository, as shown below.
Here's a regex and wildcard example.
Step 7: If you choose to use a different name for your Jenkinsfile, you can do so by specifying the name in your build configuration. In the "Script Path" option, you can provide the desired name. Make sure you have a Jenkinsfile in the repository with the same name you gave it in your pipeline configuration.
Also, enable "Discard Old Versions" to keep only required build logs, as shown below.
Step 8: Save all job configurations. Jenkins scans the configured Github repository for all branches with PRs raised.
The image below shows a job that scans three branches, and since I haven't made any pull requests, Jenkins won't create any branch-based pipelines. I'll show how to test the automatic pipeline creation after setting up the webhook.
So far, we have configured Jenkins to scan branches based on PR requests. In order to have a complete workflow, we need to configure a webhook in Github to send all events (commits, PRs, etc.) to Jenkins, since pipelines can be triggered automatically.
Configure webhooks for multi-branch pipelines
Follow the steps below to setup a Jenkins webhook on a repository.
Step 1: Go to the Github repository and click on Settings.
Step 2: Select the webhook option on the left and click the "Add Webhook" button.
Step 3: Add your Jenkins URL followed by "/github-webhook/" under Payload URL. Select the content type as "application/json" and click "Add Webhook"
Note: You can choose the type of webhook you want to receive in Jenkins. For example, you only want to trigger a pipeline during a PR. You can then select only PR events from the "Let me select individual events" option.
You will see a green tick on successful webhook configuration as shown below.
If you don't see a green tick or warning sign, click the Webhooks link, then click the last webhook. You should be able to use the status code to see why the webhook delivery failed.
We now have all the required configuration for the multi-branch pipeline. The next step is to test the multibranch pipeline workflow triggers.
Test multi-branch pipeline
For demonstration purposes, I've selected the "Branch only as a PR's branch" option. With this option, only branches with PR requests are discovered.
To use multibranch pipelines, you can use this repo with a sample Jenkinsfile. https://github.com/devopscube/multibranch-pipeline-demo
This repository has three branches. Update something in the README in the feature branch and raise a PR for development. It will send a webhook to Jenkins, and Jenkins will send back the job details to Jenkins, and the PR will go into review status as shown below.
If you click "Details" it will take you to the Jenkins build log. You can write custom checks in your Jenkins file for use in build auditing.
Now, if you have selected Jenkins, you will find the pipeline for the feature branch in Jenkins as shown below.
If the build fails, the changes can be committed to the feature branch and it will trigger the feature pipeline whenever the PR is open.
In the Jenkinfile, I added a condition to skip the deploy phase if the branch is not developed. You can check in the Jenkins build log. Also, if you inspect the build process in the blue ocean dashboard, you can clearly see the skipped deployment stages as shown below.
Now merge the feature branch PR and promote the new PR from develop to master branch.
Jenkins will receive a webhook from Github for a new PR and create a development pipeline as shown below.
For the development branch, the deploy phase is enabled, and if you inspect Blue Ocean's build process, you can see that all phases are successfully triggered.
Troubleshoot multi-branch pipelines
I'll discuss some of the errors you might encounter in a multibranch pipeline, and how to fix them.
branch discovery problem
Sometimes, even after a new branch is created in SCM, it may not be reflected in the Jenkins pipeline. You can try running the "Scan repository now" option to scan the repository again. Also, check the repository scanning configuration in the pipeline.
Webhooks don't trigger pipelines
When the webhook doesn't trigger the pipeline, check the webhook delivery status code and errors in Github. Also, check that the Jenkins URL is correct.
Also check the Jenkins logs from Manage Jenkins->System Logs->All Jenkins logs. If Jenkins is able to receive the webhook, the logs should show why the job was not triggered.