Docker搭建jenkins
DevOps概念
DevOps 一词的来自于 Development 和 Operations 的组合，突出重视软件开发人员和运维人员的沟通合作，通过自动化流程来使得软件构建、测试、发布更加快捷、频繁和可靠。DevOps 其实包含了三个部分：开发、测试和运维。换句话 DevOps 希望做到的是软件产品交付过程中IT工具链的打通，使得各个团队减少时间损耗，更加高效地协同工作。
DevOps 强调的是高效组织团队之间如何通过自动化的工具协作和沟通来完成软件的生命周期管理，从而更快、更频繁地交付更稳定的软件
如果从字面上来理解，DevOps 只是Dev（开发人员）+Ops（运维人员），实际上，它是一组过程、方法与系统的统称，其概念从2009 年首次提出发展到现在，内容非常丰富，有理论也有实践，包括组织文化、自动化、精益、反馈和分享等不同方面。
Docker部署Jenkins
Jenkins 项目产生两个发行线, 长期支持版本 (LTS) 和每周更新版本
两个版本都以 .war 文件, 原生包, 安装程序, 和 Docker 容器的形式分发
最新下载地址：https://jenkins.io/zh/download/
制作Jenkins镜像
Dockerfile及所依赖的脚本
Dockerfile
FROM centos:centos7.9.2009
MAINTAINER 笑起来真好看
ARG TARGETARCH
RUN yum update -y && yum install wget vim fontconfig fontconfig-devel fontconfig.i686 git gcc gcc-c++ \
autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel python3 python3-pip rpm-build \
snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib zlib-devel lz4 lz4-devel yasm help2man epel-release -y && \
mkdir -p /opt/software && mkdir -p /opt/compile && \
wget -O /opt/software/cmake.tar.gz https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz --no-check-certificate && \
wget -O /opt/software/protobuf.tar.gz https://hub.nuaa.cf/protocolbuffers/protobuf/archive/refs/tags/v3.7.1.tar.gz --no-check-certificate && \
wget -O /opt/software/snappy.tar.gz https://hub.nuaa.cf/google/snappy/archive/refs/tags/1.1.3.tar.gz --no-check-certificate && \
wget -O /opt/software/zstd.tar.gz https://codeload.github.com/facebook/zstd/tar.gz/refs/tags/v1.4.10 --no-check-certificate && \
wget -O /opt/software/nasm.tar.gz https://www.nasm.us/pub/nasm/releasebuilds/2.14.02/nasm-2.14.02.tar.gz --no-check-certificate && \
wget -O /opt/software/isa-l.tar.gz https://codeload.github.com/intel/isa-l/tar.gz/refs/tags/v2.30.0 --no-check-certificate && \
wget -O jenkins.war https://get.jenkins.io/war-stable/2.346.1/jenkins.war --no-check-certificate && \
wget -O /opt/software/scala.tgz https://downloads.lightbend.com/scala/2.12.17/scala-2.12.17.tgz && \
mkdir -p /usr/share/jenkins/ && mv jenkins.war /usr/share/jenkins/ && \
yum clean all && rm -rf /var/cache/yum/*
# 双架构的 jdk 版本
ADD jdk-8u371-linux-${TARGETARCH}.tar.gz /opt/software/
ADD nodejs.tar.gz /opt/software/
ADD https://github.com/krallin/tini/releases/download/v0.19.0/tini-${TARGETARCH} /usr/bin/tini
COPY install_soft.sh /opt/software/
COPY check_soft.sh /opt/software/
COPY build_hadoop.sh /opt/compile/
COPY jenkins.sh /usr/local/bin/
COPY jenkins-support /usr/local/bin/
RUN /bin/bash -c /opt/software/install_soft.sh && yum clean all && rm -rf /var/cache/yum/* && chmod +x /usr/bin/tini && mkdir -p /usr/share/jenkins/ref
ENV LANG=C.UTF-8
ENV JENKINS_HOME=/var/jenkins_home
ENV JENKINS_SLAVE_AGENT_PORT=50000
ENV REF=/usr/share/jenkins/ref
ENV JENKINS_VERSION=2.405
ENV JENKINS_UC=https://updates.jenkins.io
ENV JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental
ENV JENKINS_INCREMENTALS_REPO_MIRROR=https://repo.jenkins-ci.org/incrementals
ENV COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log
ENV NODE_HOME=/opt/software/nodejs
ENV JAVA_HOME=/opt/software/jdk1.8.0_371
ENV SCALA_HOME=/opt/software/scala
ENV MAVEN_HOME=/opt/software/maven
ENV PATH=$PATH:$MAVEN_HOME/bin:$JAVA_HOME/bin:$NODE_HOME/bin:$SCALA_HOME/bin
Entrypoint ["/usr/bin/tini", "--", "/usr/local/bin/jenkins.sh"]
其中 install_soft.sh 是自动化安装hadoop编译所需要的依赖软件包.需要可以私信我
jenkins.sh 是镜像启动的入口脚本
扫描二维码关注公众号，回复：
15749499 查看本文章
#! /bin/bash -e
: "${JENKINS_WAR:="/usr/share/jenkins/jenkins.war"}"
: "${JENKINS_HOME:="/var/jenkins_home"}"
: "${COPY_REFERENCE_FILE_LOG:="${JENKINS_HOME}/copy_reference_file.log"}"
: "${REF:="/usr/share/jenkins/ref"}"
touch "${COPY_REFERENCE_FILE_LOG}" || {
echo "Can not write to ${COPY_REFERENCE_FILE_LOG}. Wrong volume permissions?"; exit 1; }
echo "--- Copying files at $(date)" >> "$COPY_REFERENCE_FILE_LOG"
find "${REF}" \( -type f -o -type l \) -exec bash -c '. /usr/local/bin/jenkins-support; for arg; do copy_reference_file "$arg"; done' _ {
} +
# if `docker run` first argument start with `--` the user is passing jenkins launcher arguments
if [[ $# -lt 1 ]] || [[ "$1" == "--"* ]]; then
# shellcheck disable=SC2001
effective_java_opts=$(sed -e 's/^ $//' <<<"$JAVA_OPTS $JENKINS_JAVA_OPTS")
# read JAVA_OPTS and JENKINS_OPTS into arrays to avoid need for eval (and associated vulnerabilities)
java_opts_array=()
while IFS= read -r -d '' item; do
java_opts_array+=( "$item" )
done < <([[ $effective_java_opts ]] && xargs printf '%s\0' <<<"$effective_java_opts")
readonly agent_port_property='jenkins.model.Jenkins.slaveAgentPort'
if [ -n "${JENKINS_SLAVE_AGENT_PORT:-}" ] && [[ "${effective_java_opts:-}" != *"${agent_port_property}"* ]]; then
java_opts_array+=( "-D${agent_port_property}=${JENKINS_SLAVE_AGENT_PORT}" )
fi
readonly lifecycle_property='hudson.lifecycle'
if [[ "${JAVA_OPTS:-}" != *"${lifecycle_property}"* ]]; then
java_opts_array+=( "-D${lifecycle_property}=hudson.lifecycle.ExitLifecycle" )
fi
if [[ "$DEBUG" ]] ; then
java_opts_array+=( \
'-Xdebug' \
'-Xrunjdwp:server=y,transport=dt_socket,address=*:5005,suspend=y' \
)
fi
jenkins_opts_array=( )
while IFS= read -r -d '' item; do
jenkins_opts_array+=( "$item" )
done < <([[ $JENKINS_OPTS ]] && xargs printf '%s\0' <<<"$JENKINS_OPTS")
exec java -Duser.home="$JENKINS_HOME" "${java_opts_array[@]}" -jar "${JENKINS_WAR}" "${jenkins_opts_array[@]}" "$@"
fi
# As argument is not jenkins, assume user wants to run a different process, for example a `bash` shell to explore this image
exec "$@"
jenkins-support 内容如下：
#!/bin/bash -eu
: "${REF:="/usr/share/jenkins/ref"}"
# compare if version1 < version2
versionLT() {
local v1; v1=$(echo "$1" | cut -d '-' -f 1 )
local q1; q1=$(echo "$1" | cut -s -d '-' -f 2- )
local v2; v2=$(echo "$2" | cut -d '-' -f 1 )
local q2; q2=$(echo "$2" | cut -s -d '-' -f 2- )
if [ "$v1" = "$v2" ]; then
if [ "$q1" = "$q2" ]; then
return 1
else
if [ -z "$q1" ]; then
return 1
else
if [ -z "$q2" ]; then
return 0
else
[ "$q1" = "$(echo -e "$q1\n$q2" | sort -V | head -n1)" ]
fi
fi
fi
else
[ "$v1" = "$(echo -e "$v1\n$v2" | sort -V | head -n1)" ]
fi
}
# returns a plugin version from a plugin archive
get_plugin_version() {
local archive; archive=$1
local version; version=$(unzip -p "$archive" META-INF/MANIFEST.MF | grep "^Plugin-Version: " | sed -e 's#^Plugin-Version: ##')
version=${version%%[[:space:]]}
echo "$version"
}
# Copy files from /usr/share/jenkins/ref into $JENKINS_HOME
# So the initial JENKINS-HOME is set with expected content.
# Don't override, as this is just a reference setup, and use from UI
# can then change this, upgrade plugins, etc.
copy_reference_file() {
f="${1%/}"
b="${f%.override}"
rel="${b#"$REF/"}"
version_marker="${rel}.version_from_image"
dir=$(dirname "${rel}")
local action;
local reason;
local container_version;
local image_version;
local marker_version;
local log; log=false
if [[ ${rel} == plugins/*.jpi ]]; then
container_version=$(get_plugin_version "$JENKINS_HOME/${rel}")
image_version=$(get_plugin_version "${f}")
if [[ -e $JENKINS_HOME/${version_marker} ]]; then
marker_version=$(cat "$JENKINS_HOME/${version_marker}")
if versionLT "$marker_version" "$container_version"; then
if ( versionLT "$container_version" "$image_version" && [[ -n $PLUGINS_FORCE_UPGRADE ]]); then
action="UPGRADED"
reason="Manually upgraded version ($container_version) is older than image version $image_version"
log=true
else
action="SKIPPED"
reason="Installed version ($container_version) has been manually upgraded from initial version ($marker_version)"
log=true
fi
else
if [[ "$image_version" == "$container_version" ]]; then
action="SKIPPED"
reason="Version from image is the same as the installed version $image_version"
else
if versionLT "$image_version" "$container_version"; then
action="SKIPPED"
log=true
reason="Image version ($image_version) is older than installed version ($container_version)"
else
action="UPGRADED"
log=true
reason="Image version ($image_version) is newer than installed version ($container_version)"
fi
fi
fi
else
if [[ -n "$TRY_UPGRADE_IF_NO_MARKER" ]]; then
if [[ "$image_version" == "$container_version" ]]; then
action="SKIPPED"
reason="Version from image is the same as the installed version $image_version (no marker found)"
# Add marker for next time
echo "$image_version" > "$JENKINS_HOME/${version_marker}"
else
if versionLT "$image_version" "$container_version"; then
action="SKIPPED"
log=true
reason="Image version ($image_version) is older than installed version ($container_version) (no marker found)"
else
action="UPGRADED"
log=true
reason="Image version ($image_version) is newer than installed version ($container_version) (no marker found)"
fi
fi
fi
fi
if [[ ! -e $JENKINS_HOME/${rel} || "$action" == "UPGRADED" || $f = *.override ]]; then
action=${action:-"INSTALLED"}
log=true
mkdir -p "$JENKINS_HOME/${dir}"
cp -pr "${f}" "$JENKINS_HOME/${rel}";
# pin plugins on initial copy
touch "$JENKINS_HOME/${rel}.pinned"
echo "$image_version" > "$JENKINS_HOME/${version_marker}"
reason=${reason:-$image_version}
else
action=${action:-"SKIPPED"}
fi
else
if [[ ! -e $JENKINS_HOME/${rel} || $f = *.override ]]
then
action="INSTALLED"
log=true
mkdir -p "$JENKINS_HOME/${dir}"
cp -pr "$(realpath "${f}")" "$JENKINS_HOME/${rel}";
else
action="SKIPPED"
fi
fi
if [[ -n "$VERBOSE" || "$log" == "true" ]]; then
if [ -z "$reason" ]; then
echo "$action $rel" >> "$COPY_REFERENCE_FILE_LOG"
else
echo "$action $rel : $reason" >> "$COPY_REFERENCE_FILE_LOG"
fi
fi
}
# Retries a command a configurable number of times with backoff.
#
# The retry count is given by ATTEMPTS (default 60), the initial backoff
# timeout is given by TIMEOUT in seconds (default 1.)
#
function retry_command() {
local max_attempts=${ATTEMPTS-3}
local timeout=${TIMEOUT-1}
local success_timeout=${SUCCESS_TIMEOUT-1}
local max_success_attempt=${SUCCESS_ATTEMPTS-1}
local attempt=0
local success_attempt=0
local exitCode=0
while (( attempt < max_attempts ))
do
set +e
"$@"
exitCode=$?
set -e
if [[ $exitCode == 0 ]]
then
success_attempt=$(( success_attempt + 1 ))
if (( success_attempt >= max_success_attempt))
then
break
else
sleep "$success_timeout"
continue
fi
fi
echo "$(date -u '+%T') Failure ($exitCode) Retrying in $timeout seconds..." 1>&2
sleep "$timeout"
success_attempt=0
attempt=$(( attempt + 1 ))
timeout=$(( timeout ))
done
if [[ $exitCode != 0 ]]
then
echo "$(date -u '+%T') Failed in the last attempt ($*)" 1>&2
fi
return $exitCode
}
build镜像
docker build --platform=linux/arm64 -t centos-jenkins:v1 --no-cache .
利用docker-compose部署jenkins
version: '3'
services:
jenkins:
image: centos-jenkins:v1
container_name: jenkins
ports:
- "8080:8080"
- "50000:50000"
networks:
- back
volumes:
- ./jenkins:/var/jenkins_home
- ./maven:/opt/software/maven
privileged: true
environment:
- TZ="Asia/Shanghai"
restart: always
networks:
back:
driver: bridge
使用 docker-compose up -d 就可以启动了
启动成功后 就可以 本地 访问 jenkins访问地址
配置Jenkins
管理员密码
输入我们在控制台中看到的密钥，如果找不到可以到/root/.jenkins/secrets/initialAdminPassword中查看密钥
插件安装
网络好的情况下，直接选择安装推荐的插件。
必要的插件：
- 汉化插件 Chinese
- Git
- Maven Integration
安装完 插件后，重启jenkins
系统配置
全局工具配置
MAVEN 配置
配置 自定义的 settings 文件
JDK 配置
设置自定义的 JAVA_HOME 目录
GIT 配置
设置 git 执行文件路径
MAVEN 配置
设置 全局 maven 的home路径
Jenkins + Maven + Git 自动化编译
在2021年8月13日git移除了通过用户名和密码在外部访问源码仓库的方式。
问题描述如下：
Caused by: hudson.plugins.git.GitException: Command “/usr/bin/git fetch --tags --progress https://github.com/xxxxx/hadoop.git +refs/heads/:refs/remotes/origin/” returned status code 128:
目前想要在 jenkins 中拉取 github 中的源代码，需要在 github 网页中生成 TOKEN 信息
找到token生成界面
- 点击个人头像，点击 Settings 设置界面
- 侧边栏 点击 Developer settings 进入到开发者设置界面
- 进入到 Personal access tokens (classic) 界面
- 根据配置项 产生 新 的 token
生成git的token
根据提示，填写 token 的名称，过期时间和作用域，具体根据个人实际需求进行选择
token 仅显示一次，一定要记录下来
进行jenkins配置
创建 Username with Password 的类型的凭证
- 填写 用户名
- 填写 token 信息
配置 spark 编译任务
新建任务
选择 自由风格的 软件项目
配置任务
源码管理
构建
其中构建脚本如下：
#!/bin/sh
export BUILD_ID=dontKillMe
source ~/.bash_profile sh
${WORKSPACE}/dev/make-distribution.sh --pip --tgz -Phive -Phive-2.3 -Phive-thriftserver -Pscala-2.12 -Phadoop-3.2 -Pmesos -Pyarn -Pkubernetes
希望对正在查看文章的您有所帮助，记得关注、评论、收藏，谢谢您