Jenkins+Ansible+Gitlab automated deployment of the Three Musketeers

https://jenkins.io/index.html
https://www.ansible.com/tower




I have been learning how to write some playbooks in Ansible recently, so it has not been updated much. I am very interested in tools related to automated deployment, but I have no Chinese examples to learn. Here I will share with you the deployment experience that is currently popular in foreign countries that I have come into contact with in the past few months.

First of all, I will introduce Ansible to you . , Well, the important question is said three times, not Saltstack, Ansible, as an automated deployment tool written in python, does have some advantages over Chef, saltstack, and puppet that I have contacted before. First of all, it is agentless, no need for Linux client Install any service to seamlessly connect to the Linux default ssh port for deployment (windows needs to install winrm to enable the ssh service). In fact, I think this is very important. It is conceivable that many companies themselves are very strict with network management. When deploying a product At the same time, you need to consider a lot of time costs. The very difficult problem of using other deployment tools is to apply for opening ports. If the number of clients is small, we can wait. If there are more clients, you will go to request, waiting, unblock port, etc. for a long long process. ... it will take a long time in the end. This is fatal to many products. The reason why Saltstack is not recommended is also because it needs to install and test the client service on each agent one by one, which itself will take some time and cost.

What about the others? In fact, I think it is easy to use, the syntax is simple, there are ready-made templates for you to learn, and the python syntax we love very much, why not?

I don't need to say more about Jenkins, it is estimated that people who know it are using it. It is open source, lightweight, highly compatible and extensible, and intuitive GUI management. These are all its advantages. I think it will be very easy going with Ansible.

Finally, let's mention Gitlab, why use Gitlab? What does it have to do with deployment as a code version control system? In fact, this involves an issue of our Ansible playbook management. Imagine that we need to maintain a huge company server cluster, and we all need to deploy The machine or product will correspond to our relative deployment script. If the Ansible playbook we use is only saved in a specific directory of Ansible Server, it is not convenient for us to write, maintain and update (think about running to the remote to write every time. Every time the playbook is written locally or uploaded to the remote, I will make up tens of thousands of them.


Here Gitlab provides us with a very convenient and intuitive Playbook management. All we need to do is to build a playbook in Gitlab. Corresponding to the playbook warehouse of the product or server, and then we write it locally and commit it directly to the warehouse. Finally, when deploying, let Jenkins pull the playbook to its workspace and run the playbook as a Job. Standardized and easy to manage?

Of course, Ansible's own Enterprise Edition Tower will also provide a GUI management system similar to managing and maintaining playbooks and monitoring Ansible's own running process. It is also very good to use, but as a paid version, we will not do it here. More elaborated.


Here I recommend that Jenkins and Ansible can be installed in the same environment as a deployment server, and Gitlab can be deployed separately on another server as a version control system.

Summary:

Jenkins first grabs the playbook of the specific product we have written from Gitlab, and uses the Ansible related commands under virtualenv to ensure that we use the stable version to deploy our products to remote clients in batches in a clean environment.

Let's go.. ...

1. Installation environment

System: CentOS 6.7 x64 (deploy.example.com)

Jenkins: Jenkins ver. 1.650

Ansible: Ansible 2.1.0 Gitlab

: GitLab 7.14.3

2. Jenkins configuration

We create the deploy user as jenkins_user, and the workspace is The jenkins directory under the deploy home directory.

# su - root

# adduser deploy

# wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo

# rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key

# yum install jenkins -y

# vi /etc/sysconfig/jenkins

...
JENKINS_HOME="/home/deploy/jenkins"
JENKINS_USER="deploy"
...

# service jenkins start

The browser accesses the Jenkins page

http://deploy.example.com:8080

The installation is complete, the following are some Jenkins jobs that I have configured.

QQ20160311-0

Here we use a domestic The PHP website template phpcms is used as the product we need to deploy for this sample demonstration. Before the final Build, we need to do some preparatory work, and we will return to this interface later.

3. Ansible configuration

Here we need to configure virtualenv to isolate us The release version of ansible is the latest version 2.1.0, the default version 1.9 installed by pip or yum is not recommended here due to bugs and incompatibility with windows.

Configuration steps Portal: http://www.showerlee.com/archives /1862

Ansible-playbook example portal: http://www.showerlee.com/archives/1649

IV. Gitlab configuration

Deploy and use the portal: http://www.showerlee.com/archives/1285

We will eventually create a Ansible playbook repository [email protected]:showerlee/Ansible-showerlee.git, and write our rules locally, and finally commit to this repository so that Jenkins can call our deployment rules.

Here, the blogger clones a playbook repository for deploying phpcms alone, which is a benefit for everyone:

https://git.yanwenbo.cn/showerlee/leon-playbook-phpcms1.1

QQ20160311-2

QQ20160311-3

QQ20160311-45

. Final deployment After the

preparations are completed, we will introduce the Jenkins Job configuration.

1. Create a new item

QQ20160311-5

2. Create a freestyle Job with the naming rule "Product Name-Environment", here we are Phpcms-Dev

QQ20160311-6

3. Job configuration

1). Customize Build parameters.

Here Dynamic Choice Parameter is used to grab all the branches of this git repository through the Groovy script, and as a multi-option, it is convenient for us to choose the product Branch we need before the final Build Branch.

Groovy grabs Git branch code:

def gettags = ("git ls-remote -h [email protected]:showerlee/phpcms.git").execute() 
gettags.text.readLines().collect { it .split()[1].replaceAll('refs/heads/', '') }.unique()

QQ20160311-11

Choice Parameter is also used to customize the optional parameters before Build for our Job, but the parameters here can directly write

deploy_environment as our parameter name, define our deployment environment name, prod, qa are our specific options , define the two environments of our products.

QQ20160311-82). Source code management

We can use the built-in Source Code Management tool of Jenkins to grab the code of the remote Git or SVN warehouse to the local, here we grab the code stored on our Gitlab Playbook goes to the workspace directory of Jenkins for subsequent deployment work. If the warehouse needs to be authenticated, we can add the user account password of the warehouse in Credentials, and the rest can be kept by default (the default Jenkins default does not support Git, and needs to go to its background). Install Git plugin)

QQ20160311-93). Execute shell for final CLI deployment.

The Execute shell method under this Build module is a commonly used and very core function of Jenkins, which is used to execute the core commands in our deployment process.

Beginning and ending set +x, set -x is used to open and close the extended parameters and commands in this section.

Open virtualenv and load ansible environment variables

# source /home/deploy/.virtualenv/bin/activate
# . /home/deploy/.virtualenv/ansible /hacking/env-setup -q

Go to the warehouse subdirectory where the playbook is saved in the workspace directory of the job, check the ansible version, and execute the final deployment command.

cd $WORKSPACE/leon-playbook-phpcms1.1
ansible --version
ansible-playbook -i inventory/$ deploy_environment ./deploy.yml -e project=phpcms -e branch=$branch_selector -e env=$deploy_environment

Note: -i is used to customize the ansible host file path, ./deploy.yml is the ansible-playbook entry file, after -e You can follow the environment variables added to the current session.

Here $deploy_environment $branch_selector is an optional parameter defined for the Job, see 3-1) After the configuration of

QQ20160311-10

is completed, save it.

4. Execute the Job.

QQ20160311-12

selection The master branch and prod environment

QQ20160311-13

view the final console output of the Job, which is to display our actual output under the CLI.

QQ20160311-14

Console Output

Started by user Leon Li
Building in workspace /home/deploy/jenkins/workspace/Phpcms-Dev
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url [email protected]:showerlee/Ansible-showerlee.git # timeout=10
Fetching upstream changes from [email protected]:showerlee/Ansible-showerlee.git
> git --version # timeout=10
> git fetch --tags --progress [email protected]:showerlee/Ansible-showerlee.git +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 6bf787dcad68219d8eee09cecb83cbca36edbef1 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6bf787dcad68219d8eee09cecb83cbca36edbef1
> git rev-list 6bf787dcad68219d8eee09cecb83cbca36edbef1 # timeout=10
[Phpcms-Dev] $ /bin/sh -xe /tmp/hudson7452069223867148990.sh
+ set +x
ansible 2.1.0 (devel 6ddea3e915) last updated 2016/02/16 16:13:32 (GMT +800)
  lib/ansible/modules/core: (detached HEAD 8d126bd877) last updated 2016/02/16 16:19:09 (GMT +800)
  lib/ansible/modules/extras: (detached HEAD f6c5ed987f) last updated 2016/02/16 16:19:40 (GMT +800)
  config file = /home/deploy/jenkins/workspace/Phpcms-Dev/leon-playbook-phpcms1.1/ansible.cfg
  configured module search path = /home/deploy/active-ansible-modules/

PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [127.0.0.1]

TASK [deploy : Backup current source code] *************************************
changed: [127.0.0.1]

cmd: mv /data/deploy_dir/phpcms /data/deploy_dir/phpcms_master_1457681152

start: 2016-03-11 15:25:54.774716

end: 2016-03-11 15:25:54.927415

delta: 0:00:00.152699

TASK [deploy : Get new source code] ********************************************
changed: [127.0.0.1]

TASK [deploy : Check if caches/configs/database.php exists] ********************
ok: [127.0.0.1]

TASK [deploy : Check if test_dir exists] ***************************************
ok: [127.0.0.1]

TASK [deploy : debug] **********************************************************
ok: [127.0.0.1] => {
    "msg": "/data/deploy_dir/phpcms_master_1457681152/caches/configs/database.php exists"
}

msg: /data/deploy_dir/phpcms_master_1457681152/caches/configs/database.php exists

TASK [deploy : debug] **********************************************************
ok: [127.0.0.1] => {
    "msg": "/data/deploy_dir/phpcms_master_1457681152/test_dir exists"
}

msg: /data/deploy_dir/phpcms_master_1457681152/test_dir exists

TASK [deploy : Copy remote necessary original config to new release when Product env] ***
changed: [127.0.0.1] => (item={u'name': u'db_config', u'dir': u'caches/configs/database.php'})
changed: [127.0.0.1] => (item={u'name': u'version_config', u'dir': u'caches/configs/version.php'})

msg: All items completed

results: [
  {
    "src": "/data/deploy_dir/phpcms_master_1457681152/caches/configs/database.php",
    "changed": true,
    "group": "deploy",
    "uid": 606,
    "dest": "/data/deploy_dir/phpcms/caches/configs/database.php",
    "checksum": "91869c2faa244f8c5de8a586636c6b4f3c0a2817",
    "md5sum": "fd88a78a4629bca012a79d22fdcecadd",
    "owner": "deploy",
    "_ansible_no_log": false,
    "item": {
      "name": "db_config",
      "dir": "caches/configs/database.php"
    },
    "state": "file",
    "gid": 608,
    "mode": "0644",
    "invocation": {
      "module_args": {
        "src": "/data/deploy_dir/phpcms_master_1457681152/caches/configs/database.php",
        "directory_mode": null,
        "force": true,
        "remote_src": true,
        "dest": "/data/deploy_dir/phpcms/caches/configs/database.php",
        "selevel": null,
        "seuser": null,
        "setype": null,
        "group": null,
        "content": null,
        "serole": null,
        "original_basename": null,
        "delimiter": null,
        "mode": "0644",
        "regexp": null,
        "owner": null,
        "follow": false,
        "validate": null,
        "backup": false
      }
    },
    "size": 302
  },
  {
    "src": "/data/deploy_dir/phpcms_master_1457681152/caches/configs/version.php",
    "changed": true,
    "group": "deploy",
    "uid": 606,
    "dest": "/data/deploy_dir/phpcms/caches/configs/version.php",
    "checksum": "d0eaedb46a36303eb3f3e2a77cc2a623062eff3c",
    "md5sum": "7917d8199b7c6d5bc87ff3035a72670e",
    "owner": "deploy",
    "_ansible_no_log": false,
    "item": {
      "name": "version_config",
      "dir": "caches/configs/version.php"
    },
    "state": "file",
    "gid": 608,
    "mode": "0644",
    "invocation": {
      "module_args": {
        "src": "/data/deploy_dir/phpcms_master_1457681152/caches/configs/version.php",
        "directory_mode": null,
        "force": true,
        "remote_src": true,
        "dest": "/data/deploy_dir/phpcms/caches/configs/version.php",
        "selevel": null,
        "seuser": null,
        "setype": null,
        "group": null,
        "content": null,
        "serole": null,
        "original_basename": null,
        "delimiter": null,
        "mode": "0644",
        "regexp": null,
        "owner": null,
        "follow": false,
        "validate": null,
        "backup": false
      }
    },
    "size":127
  }
]

TASK [deploy : Copy dir test_dir to new release when Product env] **************
changed: [127.0.0.1]

cmd: cp -a /data/deploy_dir/phpcms_master_1457681152/test_dir /data/deploy_dir/phpcms/

start: 2016-03-11 15:26:16.966237

end: 2016-03-11 15:26:17.069705

delta: 0:00:00.103468

TASK [deploy : Get php version] ************************************************
changed: [127.0.0.1 -> localhost]

cmd: python /home/deploy/jenkins/workspace/Phpcms-Dev/leon-playbook-phpcms1.1/roles/deploy/files/get_php_version.py http://www.showerlee.com

start: 2016-03-11 15:26:17.468311

end: 2016-03-11 15:26:51.560313

delta: 0:00:34.092002

stdout: PHP/5.4.13

TASK [deploy : debug] **********************************************************
ok: [127.0.0.1] => {
    "msg": {
        "changed": true,
        "cmd": "python /home/deploy/jenkins/workspace/Phpcms-Dev/leon-playbook-phpcms1.1/roles/deploy/files/get_php_version.py http://www.showerlee.com",
        "delta": "0:00:34.092002",
        "end": "2016-03-11 15:26:51.560313",
        "rc": 0,
        "start": "2016-03-11 15:26:17.468311",
        "stderr": "",
        "stdout": "PHP/5.4.13",
        "stdout_lines": [
            "PHP/5.4.13"
        ],
        "warnings": []
    }
}

msg:{
  "changed": true,
  "end": "2016-03-11 15:26:51.560313",
  "stdout": "PHP/5.4.13",
  "cmd": "python /home/deploy/jenkins/workspace/Phpcms-Dev/leon-playbook-phpcms1.1/roles/deploy/files/get_php_version.py http://www.showerlee.com",
  "start": "2016-03-11 15:26:17.468311",
  "delta": "0:00:34.092002",
  "stderr": "",
  "rc": 0,
  "stdout_lines": [
    "PHP/5.4.13"
  ],
  "warnings": []
}

TASK [deploy : debug] **********************************************************
ok: [127.0.0.1] => {
    "msg": "PHP/5.4.13"
}

msg: PHP/5.4.13

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=12   changed=5    unreachable=0    failed=0  

Finished: SUCCESS

So we use Jenkins+Ansible+Gitlab to successfully deploy phpcms to the remote Client.