Recently, I sorted out the complete process of the entire requirement development process before, as well as the secondary development of the DevOps system. The points covered in this process are mainly the entire process from receiving requirements to completing the online requirements.
The whole process is shown in the figure:
Explanation of key points:
1. Code is associated with requirements/tasks/bugs
At that time, Tencent’s TAPD was used as the project management and agile development platform. This platform has a feature that supports associated gitlab submission:
during the code submission during the development process, fill in the comment (including requirement ID, BugID, etc.) in the format of TAPD, and Submit to TAPD's API via webhook. Note: You can configure the submission rules in Gitlab, and it is mandatory that each submission must conform to this format, that is, configure the hook
under the git address of each project, and refer to the end of the article for the content of the hook file.custom_hooks/pre-receivepre-receive
2. Generate a release application
According to 1, we have the submission relationship between the requirements and the code, then in the DevOps system, if we select the requirements or bugs to be released, we can automatically associate them, which Git projects are involved and need to be released, then we can:
- Automatically generate release requisitions, including all associated Git projects
- Avoid missing a project, or a feature branch when releasing
- Automatically generate MR application for each Git project
- Automatically push MR notification messages to the code reviewers of these GIt projects for review
- Automatically push messages to be released to the development leaders of these Git projects for attention
- Automatically generate a list of test and pre-configuration differences for each Git project
- Automatically generate a list of table structure differences for each Git project corresponding to the database, and brush database SQL
- The person in charge of development confirms the above two items, whether it needs to be pushed to the person in charge of operation and maintenance for processing
- Automatically push the news to be released to the QA and product manager corresponding to the demand for attention
3. Daily code quality scan
Because the code scan takes a long time, we have optimized the code scan from the initial release to scan the changed projects once every morning, and only scan the two branches of test and prod. If the scan fails, the corresponding project will be banned next time. Published, and Dingding notification was sent to the person in charge to request rectification.
The code scanning tool used is Sonarqube, and the corresponding quality index description reference: https://docs.sonarqube.org/latest/user-guide/metric-definitions/
Related script reference
pre-receiveDocument content reference
#!/bin/bash
while read oldVersion newVersion branch; do
# read oldVersion newVersion branch
echo "do pre-recievt: ${oldVersion} ${newVersion} ${branch}"
# oldrev=`git rev-parse $oldVersion` # 获取该版本的sha1值，貌似多余，传入的参数就是sha1
# newrev=`git rev-parse $newVersion`
commitList=`git rev-list $oldVersion..$newVersion` # 获取新旧版本之间，提交了几个版本
echo $commitList
# 按空格分割字符串，并遍历
for commitSha1 in $commitList
do
# 获取每次提交的log，正常情况下，log格式如下：
# $ git cat-file commit 51ffa547167535fffb0f4b87d09022938f8d404b
# tree ffd5876a90a555eedd605b0e2df606c83840437f
# parent b78c620aa64b75e6f312a0541e59df3ad8bfca57
# author youbl <[email protected]> 1557478996 +0800
# committer youbl <[email protected]> 1557478996 +0800
#
# 删除验证
# sed '1,/^$/d'表示从第1行删除到第一个空行为止，剩下的就是log了
msg=`git cat-file commit $commitSha1 | sed '1,/^$/d'`
# 分支Merge，跳过
match=`echo "$msg" | egrep 'Merge branch ' `
if [ -n "$match" ];then continue; fi
match=`echo "$msg" | egrep 'Merge remote-tracking branch ' `
if [ -n "$match" ];then continue; fi
match=`echo "$msg" | egrep '合并分支 ' ` # gitlab上发起的合并请求，会以“合并分支”开头
if [ -n "$match" ];then continue; fi
# 验证log是否匹配规则
match=`echo "$msg" | egrep '\-\-story=[0-9]{7,}' `
if [ -n "$match" ];then continue; fi
match=`echo "$msg" | egrep '\-\-bug=[0-9]{7,}' `
if [ -n "$match" ];then continue; fi
match=`echo "$msg" | egrep '\-\-task=[0-9]{7,}' `
if [ -n "$match" ];then continue; fi
echo "!!! Your submit msg: $msg !!!($commitSha1)"
echo !!! This submit must be related to tapd with id !!!
exit 1
done
# ===========================================================
# 如果要遍历提交的文件，用下面的代码
function eachFiles () {
diff=`git diff --name-status $oldVersion $newVersion` # 获取2个版本之间，所有版本的差异文件列表，格式: M README.md D db/20181018.sql M db/full.sql A test.md
echo $diff
isDel=false
# 按空格分割字符串，并遍历
for file in $diff
do
if [ "$file" = "D" ];then
isDel=true # 删除的文件，要置标志位，忽略下一个遍历
fi
# echo ${#file} # 输出字符串长度
if [ "${#file}" -lt "2" ];then # 字符串长度小于1时跳过
continue
fi
if [ "$isDel" = "true" ];then # 当前文件前面的标志是D，表示删除，不操作
isDel=false
continue
fi
echo $file
# fileDiff=`git show $newVersion:$file` # 获取当前文件的内容
# echo $fileDiff
done
}
done
# echo !!!test prevent push!!!
exit 0
Timing tasks: sh script reference for adding hooks to new projects
Because each git project must add the above hook file, then when the user creates a new git project, there will be no such hook, so a script is needed to traverse the new project and add the hook. In the crontab of linux,
specify Execute the following script every hour.
#!/bin/bash
function eachGitDir(){
for ele in `ls $1`
do
subdir="$1/$ele"
if [ ! -d $subdir ];then continue; fi # 不是目录，跳过
if [[ $subdir == "/data/gitlab/data/git-data/repositories/Frontend" ]]; then continue; fi # 跳过前端内部项目
if [[ $ele == "boot" ]]; then continue; fi # 跳过boot项目
if [[ $ele == "ops" ]]; then continue; fi
if [[ $ele == "tests" ]]; then continue; fi
if [[ $ele == *".wiki.git" ]]; then continue; fi
if [[ $ele == "ToolsProject.git" ]]; then continue; fi # 跳过Tools项目
if [[ $ele != *".git" ]]; then
eachGitDir $subdir
continue
fi # 不是git目录，递归
hookdir=$subdir/custom_hooks
if [ ! -d $hookdir ];then mkdir $hookdir; fi
hookfile=$hookdir/pre-receive
if [ -e $hookfile ];then continue; fi # 文件存在，不处理
`/usr/bin/cp /root/hooks/pre-receive $hookfile`
echo $hookdir
done
}
# repository="/root/tmp"
repository="/data/gitlab/data/git-data/repositories"
eachGitDir $repository
Hook to clear all items
Sometimes, you need to clean all project hooks and reconfigure, you can use this script:
#!/bin/bash
function eachGitDir(){
for ele in `ls $1`
do
subdir="$1/$ele"
if [ ! -d $subdir ];then continue; fi # 不是目录，跳过
if [[ $ele != *".git" ]]; then
eachGitDir $subdir
continue
fi # 不是git目录，递归
hookdir=$subdir/custom_hooks
if [ ! -d $hookdir ];then continue; fi
hookfile=$hookdir/pre-receive
if [ ! -e $hookfile ];then
`rmdir $hookdir`
continue
fi # 文件不存在，不处理
`rm -f $hookfile`
`rmdir $hookdir`
echo "deleted $hookfile"
done
}
# repository="/root/tmp"
repository="/data/gitlab/data/git-data/repositories"
eachGitDir $repository
Clear the hook for the specified item
Sometimes, you need to clean up the hook of the specified item, and you are too lazy to find the directory and delete it. You can use this script:
#!/bin/bash
function eachGitDir(){
for ele in `ls $1`
do
subdir="$1/$ele"
if [ ! -d $subdir ];then continue; fi # 不是目录，跳过
if [[ $ele != *".git" ]]; then
eachGitDir $subdir
continue
fi # 不是git目录，递归
processHook $subdir
done
}
function processHook(){
subdir=$1
hookdir=$subdir/custom_hooks
if [ ! -d $hookdir ];then continue; fi
hookfile=$hookdir/pre-receive
if [ ! -e $hookfile ];then
`rmdir $hookdir`
echo "rmdir $hookdir"
else
`rm -f $hookfile`
`rmdir $hookdir`
echo "deleted $hookfile"
fi
}
ele=$1
if [ "${#ele}" -lt "1" ];then # 字符串长度小于1时跳过
echo 参数不能为空
exit 1
fi
# 移除前面的http协议和域名
if [[ $ele = http* ]]; then
ele=`echo $ele | cut -d/ -f4-`
fi
ele="/data/gitlab/data/git-data/repositories/$ele"
if [ ! -d $ele ];then
ele=$ele".git";
if [ ! -d $ele ];then
echo "目录不存在: $ele"
exit 1
fi
fi
# echo $ele
if [[ $ele != *".git" ]]; then
eachGitDir $ele
else
processHook $ele
fi