公司的CI/CD是使用Jenkins，开发、测试、预发的CI和CD都是在一起的，而生产环境的CI/CD我们是分开的
CI任务结束之后，开发可以选择发布哪个release版本。
可以看一下整体预览情况：
每个Job的Pipeline状态：
自定义发布机器、同时有发布及回滚功能：
我们都是基于maven的Java应用，进行编译打包其实比较简单，这里的CI较为简单，我这里只简单说明及其需要注意的点
maven编译打包时，可以去掉单元测试
扫描二维码关注公众号，回复： 11585024 查看本文章
若Jenkins的机器配置比较高，可以可以开启maven的并发编译（3.3版本以上，默认支持并发）
适当调整maven的JVM
下面着重介绍CD部分的配置
以： 我们的msg-server服务为例
添加参数化构建
a.添加PROJECT(字符参数):标识：应用名称
b.添加DEPLOY_TYPE(Active Choices Parameter ):标识：发布还是回滚
c.添加Version（ Active Choices Reactive Parameter的）：标识发布的版本
d.添加 HOTS（ Active Choices Reactive Parameter），表示发布的主机清单
e.添加 DEPLOYED_HOSTS（Active Choices Reactive Parameter ），标识：已经发布过的主机清单
2.基础共用脚本部分
cat get_versions.sh
#!/bin/bash #prod packageDir="/data/jenkins/repos/master/" #test #packageDir="/data/jenkins/repos/test" #dev #packageDir="/data/jenkins/repos/dev" project=$1 deploy_type=$2 if [[ "${deploy_type}" == "Rollback" ]] ; then lasted_version=`ls -lt ${packageDir}/${project}|grep ${project}|awk '{print $9}'|awk -F '.' '{print $1}'` echo ${lasted_version}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}' else lasted_version=`ls -lt ${packageDir}/${project}|grep ${project}|awk '{print $9}'|awk -F '.' '{print $1}'` echo ${lasted_version}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}' fi |
cat get_deploy_lists.sh
#!/bin/bash PACKAGE_DIR=/data/jenkins/repos/release_package project=$1 deploy_type=$2 version=$3 db_url='ops-db.xxxxxx.com' db_user='xxxxx' db_pwd="xxxxxxxx" db_port=3306 format_version=`echo ${version}|awk -F '##' '{print $1}'` #查询已经发布或回滚过的主机清单 query_sql="select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}' and version='${format_version}' and hostname like '%${project}%' " result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'` echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}' |
cat get_undeploy_lists.sh
#!/bin/bash PACKAGE_DIR=/data/jenkins/repos/release_package project=$1 deploy_type=$2 version=$3 db_url='ops-db.xxxxxxx.com' db_user='xxxxx' db_pwd="xxxxxxxx" db_port=3306 format_version=`echo ${version}|awk -F '##' '{print $1}'` #查询待发布的主机 #if [ "${deploy_type}" == "Deploy" ] ; then #query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}' and version='${format_version}' and hostname like '%${project}%' ) " #result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'` #echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}' #fi #if [ "${deploy_type}" == "Rollback" ] ; then #if rollback , # query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='Rollback' and version='${format_version}' and hostname like '%${project}%' ) " # result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'` # echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}' #fi query_sql="select host_ip from hosts where host_ip not in (select host_ip from deploy_history where status=1 and deploy_type='${deploy_type}' and version='${format_version}' and hostname like '%${project}%' ) and hostname like '%${project}%' " result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${query_sql}" |awk 'NR>1'` echo ${result}|awk -F " " '{for(i=1;i<=NF;i++) a[i,NR]=$i}END{for(i=1;i<=NF;i++) {for(j=1;j<=NR;j++) printf a[i,j] " ";print ""}}' |
cat deploy_success_update.sh
#!/bin/bash project=$1 host_ip=$2 deploy_type=$3 version=$4 db_url='ops-db.xxxxxxxx.com' db_user='xxxxx' db_pwd="xxxxxx" db_port=3306 insert_sql="insert into deploy_history(hostname,host_ip,deploy_type,status,version) values(\"${project}\",\"${host_ip}\",\"${deploy_type}\",1,\"${version}\" )" echo ${insert_sql} count_result=`mysql -u${db_user} -p${db_pwd} -h${db_url} -P${db_port} -B jenkins -e "${insert_sql}" ` |
3.创建数据库及定义表结构
/* Navicat MySQL Data Transfer Source Server : ops-db Source Server Version : 50728 Source Host : ops-db. xxxxx.com:3306 Source Database : jenkins Target Server Type : MYSQL Target Server Version : 50728 File Encoding : 65001 Date: 2020-08-13 14:25:52 */ Create Database: CREATE DATABASE `jenkins` /*!40100 DEFAULT CHARACTER SET utf8mb4 */ SET FOREIGN_KEY_CHECKS=0; -- ---------------------------- -- Table structure for deploy_history -- ---------------------------- CREATE TABLE `deploy_history` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `hostname` varchar(255) NOT NULL COMMENT '发布的主机名', `host_ip` varchar(255) NOT NULL COMMENT '发布主机IP地址', `deploy_type` varchar(255) NOT NULL, `status` tinyint(1) NOT NULL COMMENT '0 发布失败; 1发布成功', `version` varchar(255) NOT NULL, `ctm_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `udx_deploy_type_version` (`deploy_type`,`version`) USING BTREE, KEY `idx_host_ip` (`host_ip`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8mb4; -- ---------------------------- -- Table structure for hosts -- ---------------------------- CREATE TABLE `hosts` ( `id` int(11) NOT NULL AUTO_INCREMENT, `hostname` varchar(255) NOT NULL, `host_ip` varchar(255) NOT NULL, `comment` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), KEY `idx_host_ip` (`host_ip`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4; -- ---------------------------- -- Table structure for hosts-test -- ---------------------------- CREATE TABLE `hosts-test` ( `id` int(11) NOT NULL AUTO_INCREMENT, `hostname` varchar(255) NOT NULL, `host_ip` varchar(255) NOT NULL, `comment` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), KEY `idx_host_ip` (`host_ip`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=19 DEFAULT CHARSET=utf8mb4; |
#可以查看以往的历史发布记录：
4.Pipelin流水线代码
#Groovy代码如下：
/** * Description:Master环境JavaSpringBoot的Pipeline脚本 * Author: ledi * Date:2020-03-01 */ pipeline { agent any environment { packageDir="/data/jenkins/repos/master" backupDir="/data/release_package" deployDir="/data/www" ansible_Dir="/etc/ansible/roles/masterv2" port="20881" } stages { stage("推包至远端"){ steps{ script{ if ( "${DEPLOY_TYPE}" == 'Deploy' ){ sh ''' HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g` source /etc/profile &> /dev/null #/bin/mv ${WORKSPACE}/${PROJECT}/target/${PROJECT}.jar ${packageDir}/${PROJECT}_${BUILD_NUMBER}.jar old_version=`echo "${VERSION}"` new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'` ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/copy_jar.yaml -e "backupDir=${backupDir} packageDir=${packageDir} old_version=${old_version} new_version=${new_version} project=${PROJECT}" --limit ${HOSTS} ''' }else{ echo "执行回滚操作,无需推包至远端" } } // end script } // end steps } //end stage stage("停应用") { steps { script { sh ''' HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g` ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/stop_app.yaml -e "project=${PROJECT} " --limit ${HOSTS} ''' } } } stage("启应用"){ steps{ script { sh ''' HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g` new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'` ansible-playbook -i ${ansible_Dir}/hosts ${ansible_Dir}/start_app.yaml -e "project=${PROJECT} new_version=${new_version} " --limit ${HOSTS} ''' } } } stage("健康检查"){ steps{ script{ sh ''' HOSTS=`echo ${HOSTS}|sed s/[[:space:]]//g` hosts_ip=`echo ${HOSTS} |sed "s/,/ /g" ` new_version=`echo "${VERSION}"|awk -F '##' '{print $1}'` function check_health(){ for (( i=1;i<="$#";i++ )); do ansible ${!i} -u www -i ${ansible_Dir}/hosts -m shell -a "tail -1000 /data/logs/${PROJECT}/nohup.out" for (( j=1;j<=60;j++ )) do echo "ip is ${!i}" if [[ `(echo "status -l ";sleep 1;exit)|telnet ${!i} ${port} |grep "server"| grep -o "OK"` == "OK" ]]; then echo " ^_^^_^ IP ${!i} ${port} 端口检查成功 ^_^^_^" sh /data/jenkins/scripts/deploy_success_update.sh ${PROJECT} ${!i} ${DEPLOY_TYPE} ${new_version} break 1; else echo "==== IP ${!i} ${port} 端口异常,继续探测 ==== " sleep 3 fi done
if [[ `(echo "status -l ";sleep 1;exit)|telnet ${!i} ${port} |grep "server"| grep -o "OK"` != "OK" ]] ; then echo "==== IP ${!i} ${port} 端口异常,启动失败,请检查应用 === " exit 1 fi done
}
check_health ${hosts_ip} ''' }// end script } // end steps } // end stage heal } //end stages } |
说明：
这里也可以将此Groovy代码和应用放在一起，放入GitLab中，这样更容易维护及管理。
5.Ansible 公用部分
cat copy_jar.yaml
--- - hosts: all gather_facts: False remote_user: root vars: backupDir: {backupDir} packageDir: {packageDir} project: {PROJECT} old_version: {old_version} new_version: {new_version} tasks: - name: Copy Jar to Remoute Machine copy: src={{ packageDir }}/{{project }}/{{ old_version }}.jar dest={{ backupDir }}/{{ new_version }}.jar owner=www group=www mode=0664 force=yes |
cat stop_app.yaml
--- - hosts: all gather_facts: False remote_user: root vars: project: {PROJECT} tasks: - name: Stop Java App shell: sh /home/www/scripts/stop_springboot.sh {{ project }} owner=www group=www |
cat start_app.yaml
--- - hosts: all gather_facts: False remote_user: www vars: project: {PROJECT} new_version: {new_version} tasks: - name: Start Java App shell: /bin/bash /home/www/scripts/start_springboot.sh {{ project }} {{ new_version }} owner=www group=www |
最后，开发们可以自定义发布主机，假设一个微服务有20台机器，可以先选择5台进行发布，通过后，再选择5台或以上，然后慢慢类似滚动发布。
存在不足的点：
若一个微服务有20台机器，若采用每5台发布，则开发需要手工进行4次操作，改进的地方：若第一次5台成功后，则自动执行后面的操作
在使用Ansible推包到阿里云ECS时，很慢；这个是我们的网络机房问题，因为打包机器在公司内部，和阿里云ECS通过***过去的，所以比较慢，改进的地方：
一是将打包机器也迁移到云端；二是通过专线将公司网络与阿里云***形成物理线路；不过上面两个都是需要￥的哈,运维同学不单单考虑系统可用性、稳定性，也要为公司节约一定成本哈。（不知道领导看到了，会不会给我加个鸡腿，哈哈）