Docker build jenkins
DevOps concept
The term DevOps comes from the combination of Development and Operations, which emphasizes the communication and cooperation between software developers and operation and maintenance personnel, and makes software construction, testing, and release faster, more frequent, and more reliable through automated processes. DevOps actually consists of three parts: development, testing, and operation and maintenance. In other words, what DevOps hopes to achieve is to open up the IT tool chain in the software product delivery process, so that each team can reduce time loss and work together more efficiently.
DevOps emphasizes how to complete software life cycle management through automated tool collaboration and communication between efficient organizational teams, so as to deliver more stable software faster and more frequently
If you understand it literally, DevOps is just Dev (developer) + Ops (operation and maintenance personnel). In fact, it is a collective term for a set of processes, methods and systems. Its concept has been developed since it was first proposed in 2009. The content is very rich , with theory and practice, including different aspects of organizational culture, automation, lean, feedback and sharing.
Docker deploy Jenkins
The Jenkins project produces two release lines, Long Term Support (LTS) and Weekly Updates
Both versions are distributed as .war files, native packages, installers, and Docker containers
Latest download address: https://jenkins.io/zh/download/
Create a Jenkins image
Dockerfile and dependent scripts
Dockerfile
FROM centos:centos7.9.2009
MAINTAINER 笑起来真好看
ARG TARGETARCH
RUN yum update -y && yum install wget vim fontconfig fontconfig-devel fontconfig.i686 git gcc gcc-c++ \
autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel python3 python3-pip rpm-build \
snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib zlib-devel lz4 lz4-devel yasm help2man epel-release -y && \
mkdir -p /opt/software && mkdir -p /opt/compile && \
wget -O /opt/software/cmake.tar.gz https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz --no-check-certificate && \
wget -O /opt/software/protobuf.tar.gz https://hub.nuaa.cf/protocolbuffers/protobuf/archive/refs/tags/v3.7.1.tar.gz --no-check-certificate && \
wget -O /opt/software/snappy.tar.gz https://hub.nuaa.cf/google/snappy/archive/refs/tags/1.1.3.tar.gz --no-check-certificate && \
wget -O /opt/software/zstd.tar.gz https://codeload.github.com/facebook/zstd/tar.gz/refs/tags/v1.4.10 --no-check-certificate && \
wget -O /opt/software/nasm.tar.gz https://www.nasm.us/pub/nasm/releasebuilds/2.14.02/nasm-2.14.02.tar.gz --no-check-certificate && \
wget -O /opt/software/isa-l.tar.gz https://codeload.github.com/intel/isa-l/tar.gz/refs/tags/v2.30.0 --no-check-certificate && \
wget -O jenkins.war https://get.jenkins.io/war-stable/2.346.1/jenkins.war --no-check-certificate && \
wget -O /opt/software/scala.tgz https://downloads.lightbend.com/scala/2.12.17/scala-2.12.17.tgz && \
mkdir -p /usr/share/jenkins/ && mv jenkins.war /usr/share/jenkins/ && \
yum clean all && rm -rf /var/cache/yum/*
# 双架构的 jdk 版本
ADD jdk-8u371-linux-${TARGETARCH}.tar.gz /opt/software/
ADD nodejs.tar.gz /opt/software/
ADD https://github.com/krallin/tini/releases/download/v0.19.0/tini-${TARGETARCH} /usr/bin/tini
COPY install_soft.sh /opt/software/
COPY check_soft.sh /opt/software/
COPY build_hadoop.sh /opt/compile/
COPY jenkins.sh /usr/local/bin/
COPY jenkins-support /usr/local/bin/
RUN /bin/bash -c /opt/software/install_soft.sh && yum clean all && rm -rf /var/cache/yum/* && chmod +x /usr/bin/tini && mkdir -p /usr/share/jenkins/ref
ENV LANG=C.UTF-8
ENV JENKINS_HOME=/var/jenkins_home
ENV JENKINS_SLAVE_AGENT_PORT=50000
ENV REF=/usr/share/jenkins/ref
ENV JENKINS_VERSION=2.405
ENV JENKINS_UC=https://updates.jenkins.io
ENV JENKINS_UC_EXPERIMENTAL=https://updates.jenkins.io/experimental
ENV JENKINS_INCREMENTALS_REPO_MIRROR=https://repo.jenkins-ci.org/incrementals
ENV COPY_REFERENCE_FILE_LOG=/var/jenkins_home/copy_reference_file.log
ENV NODE_HOME=/opt/software/nodejs
ENV JAVA_HOME=/opt/software/jdk1.8.0_371
ENV SCALA_HOME=/opt/software/scala
ENV MAVEN_HOME=/opt/software/maven
ENV PATH=$PATH:$MAVEN_HOME/bin:$JAVA_HOME/bin:$NODE_HOME/bin:$SCALA_HOME/bin
Entrypoint ["/usr/bin/tini", "--", "/usr/local/bin/jenkins.sh"]
Among them, install_soft.sh is the dependent software package required for automatic installation of hadoop compilation. If you need it, you can private message me
jenkins.sh is the entry script for mirror startup
#! /bin/bash -e
: "${JENKINS_WAR:="/usr/share/jenkins/jenkins.war"}"
: "${JENKINS_HOME:="/var/jenkins_home"}"
: "${COPY_REFERENCE_FILE_LOG:="${JENKINS_HOME}/copy_reference_file.log"}"
: "${REF:="/usr/share/jenkins/ref"}"
touch "${COPY_REFERENCE_FILE_LOG}" || {
echo "Can not write to ${COPY_REFERENCE_FILE_LOG}. Wrong volume permissions?"; exit 1; }
echo "--- Copying files at $(date)" >> "$COPY_REFERENCE_FILE_LOG"
find "${REF}" \( -type f -o -type l \) -exec bash -c '. /usr/local/bin/jenkins-support; for arg; do copy_reference_file "$arg"; done' _ {
} +
# if `docker run` first argument start with `--` the user is passing jenkins launcher arguments
if [[ $# -lt 1 ]] || [[ "$1" == "--"* ]]; then
# shellcheck disable=SC2001
effective_java_opts=$(sed -e 's/^ $//' <<<"$JAVA_OPTS $JENKINS_JAVA_OPTS")
# read JAVA_OPTS and JENKINS_OPTS into arrays to avoid need for eval (and associated vulnerabilities)
java_opts_array=()
while IFS= read -r -d '' item; do
java_opts_array+=( "$item" )
done < <([[ $effective_java_opts ]] && xargs printf '%s\0' <<<"$effective_java_opts")
readonly agent_port_property='jenkins.model.Jenkins.slaveAgentPort'
if [ -n "${JENKINS_SLAVE_AGENT_PORT:-}" ] && [[ "${effective_java_opts:-}" != *"${agent_port_property}"* ]]; then
java_opts_array+=( "-D${agent_port_property}=${JENKINS_SLAVE_AGENT_PORT}" )
fi
readonly lifecycle_property='hudson.lifecycle'
if [[ "${JAVA_OPTS:-}" != *"${lifecycle_property}"* ]]; then
java_opts_array+=( "-D${lifecycle_property}=hudson.lifecycle.ExitLifecycle" )
fi
if [[ "$DEBUG" ]] ; then
java_opts_array+=( \
'-Xdebug' \
'-Xrunjdwp:server=y,transport=dt_socket,address=*:5005,suspend=y' \
)
fi
jenkins_opts_array=( )
while IFS= read -r -d '' item; do
jenkins_opts_array+=( "$item" )
done < <([[ $JENKINS_OPTS ]] && xargs printf '%s\0' <<<"$JENKINS_OPTS")
exec java -Duser.home="$JENKINS_HOME" "${java_opts_array[@]}" -jar "${JENKINS_WAR}" "${jenkins_opts_array[@]}" "$@"
fi
# As argument is not jenkins, assume user wants to run a different process, for example a `bash` shell to explore this image
exec "$@"
The content of jenkins-support is as follows:
#!/bin/bash -eu
: "${REF:="/usr/share/jenkins/ref"}"
# compare if version1 < version2
versionLT() {
local v1; v1=$(echo "$1" | cut -d '-' -f 1 )
local q1; q1=$(echo "$1" | cut -s -d '-' -f 2- )
local v2; v2=$(echo "$2" | cut -d '-' -f 1 )
local q2; q2=$(echo "$2" | cut -s -d '-' -f 2- )
if [ "$v1" = "$v2" ]; then
if [ "$q1" = "$q2" ]; then
return 1
else
if [ -z "$q1" ]; then
return 1
else
if [ -z "$q2" ]; then
return 0
else
[ "$q1" = "$(echo -e "$q1\n$q2" | sort -V | head -n1)" ]
fi
fi
fi
else
[ "$v1" = "$(echo -e "$v1\n$v2" | sort -V | head -n1)" ]
fi
}
# returns a plugin version from a plugin archive
get_plugin_version() {
local archive; archive=$1
local version; version=$(unzip -p "$archive" META-INF/MANIFEST.MF | grep "^Plugin-Version: " | sed -e 's#^Plugin-Version: ##')
version=${version%%[[:space:]]}
echo "$version"
}
# Copy files from /usr/share/jenkins/ref into $JENKINS_HOME
# So the initial JENKINS-HOME is set with expected content.
# Don't override, as this is just a reference setup, and use from UI
# can then change this, upgrade plugins, etc.
copy_reference_file() {
f="${1%/}"
b="${f%.override}"
rel="${b#"$REF/"}"
version_marker="${rel}.version_from_image"
dir=$(dirname "${rel}")
local action;
local reason;
local container_version;
local image_version;
local marker_version;
local log; log=false
if [[ ${rel} == plugins/*.jpi ]]; then
container_version=$(get_plugin_version "$JENKINS_HOME/${rel}")
image_version=$(get_plugin_version "${f}")
if [[ -e $JENKINS_HOME/${version_marker} ]]; then
marker_version=$(cat "$JENKINS_HOME/${version_marker}")
if versionLT "$marker_version" "$container_version"; then
if ( versionLT "$container_version" "$image_version" && [[ -n $PLUGINS_FORCE_UPGRADE ]]); then
action="UPGRADED"
reason="Manually upgraded version ($container_version) is older than image version $image_version"
log=true
else
action="SKIPPED"
reason="Installed version ($container_version) has been manually upgraded from initial version ($marker_version)"
log=true
fi
else
if [[ "$image_version" == "$container_version" ]]; then
action="SKIPPED"
reason="Version from image is the same as the installed version $image_version"
else
if versionLT "$image_version" "$container_version"; then
action="SKIPPED"
log=true
reason="Image version ($image_version) is older than installed version ($container_version)"
else
action="UPGRADED"
log=true
reason="Image version ($image_version) is newer than installed version ($container_version)"
fi
fi
fi
else
if [[ -n "$TRY_UPGRADE_IF_NO_MARKER" ]]; then
if [[ "$image_version" == "$container_version" ]]; then
action="SKIPPED"
reason="Version from image is the same as the installed version $image_version (no marker found)"
# Add marker for next time
echo "$image_version" > "$JENKINS_HOME/${version_marker}"
else
if versionLT "$image_version" "$container_version"; then
action="SKIPPED"
log=true
reason="Image version ($image_version) is older than installed version ($container_version) (no marker found)"
else
action="UPGRADED"
log=true
reason="Image version ($image_version) is newer than installed version ($container_version) (no marker found)"
fi
fi
fi
fi
if [[ ! -e $JENKINS_HOME/${rel} || "$action" == "UPGRADED" || $f = *.override ]]; then
action=${action:-"INSTALLED"}
log=true
mkdir -p "$JENKINS_HOME/${dir}"
cp -pr "${f}" "$JENKINS_HOME/${rel}";
# pin plugins on initial copy
touch "$JENKINS_HOME/${rel}.pinned"
echo "$image_version" > "$JENKINS_HOME/${version_marker}"
reason=${reason:-$image_version}
else
action=${action:-"SKIPPED"}
fi
else
if [[ ! -e $JENKINS_HOME/${rel} || $f = *.override ]]
then
action="INSTALLED"
log=true
mkdir -p "$JENKINS_HOME/${dir}"
cp -pr "$(realpath "${f}")" "$JENKINS_HOME/${rel}";
else
action="SKIPPED"
fi
fi
if [[ -n "$VERBOSE" || "$log" == "true" ]]; then
if [ -z "$reason" ]; then
echo "$action $rel" >> "$COPY_REFERENCE_FILE_LOG"
else
echo "$action $rel : $reason" >> "$COPY_REFERENCE_FILE_LOG"
fi
fi
}
# Retries a command a configurable number of times with backoff.
#
# The retry count is given by ATTEMPTS (default 60), the initial backoff
# timeout is given by TIMEOUT in seconds (default 1.)
#
function retry_command() {
local max_attempts=${ATTEMPTS-3}
local timeout=${TIMEOUT-1}
local success_timeout=${SUCCESS_TIMEOUT-1}
local max_success_attempt=${SUCCESS_ATTEMPTS-1}
local attempt=0
local success_attempt=0
local exitCode=0
while (( attempt < max_attempts ))
do
set +e
"$@"
exitCode=$?
set -e
if [[ $exitCode == 0 ]]
then
success_attempt=$(( success_attempt + 1 ))
if (( success_attempt >= max_success_attempt))
then
break
else
sleep "$success_timeout"
continue
fi
fi
echo "$(date -u '+%T') Failure ($exitCode) Retrying in $timeout seconds..." 1>&2
sleep "$timeout"
success_attempt=0
attempt=$(( attempt + 1 ))
timeout=$(( timeout ))
done
if [[ $exitCode != 0 ]]
then
echo "$(date -u '+%T') Failed in the last attempt ($*)" 1>&2
fi
return $exitCode
}
build image
docker build --platform=linux/arm64 -t centos-jenkins:v1 --no-cache .
Deploy jenkins using docker-compose
version: '3'
services:
jenkins:
image: centos-jenkins:v1
container_name: jenkins
ports:
- "8080:8080"
- "50000:50000"
networks:
- back
volumes:
- ./jenkins:/var/jenkins_home
- ./maven:/opt/software/maven
privileged: true
environment:
- TZ="Asia/Shanghai"
restart: always
networks:
back:
driver: bridge
Use docker-compose up -d to start
After the startup is successful, you can access the jenkins access address locally
Configure Jenkins
administrator password
Enter the key we saw in the console, if you can't find it, you can view the key in /root/.jenkins/secrets/initialAdminPassword
plugin installation
When the network is good, directly choose to install the recommended plug-in.
Necessary plugins:
- Sinicization plug-in Chinese
- Git
- Maven Integration
After installing the plugin, restart jenkins
System Configuration
Global Tool Configuration
maven configuration
Configure a custom settings file
JDK configuration
Set a custom JAVA_HOME directory
Git configuration
Set the git execution file path
maven configuration
Set the home path of the global maven
Jenkins + Maven + Git automatic compilation
On August 13, 2021, git removed the way to access the source code warehouse externally through username and password.
The problem description is as follows:
Caused by: hudson.plugins.git.GitException: Command “/usr/bin/git fetch --tags --progress https://github.com/xxxxx/hadoop.git +refs/heads/:refs/remotes/origin/” returned status code 128:
At present, if you want to pull the source code in github in jenkins, you need to generate TOKEN information in the github webpage
Find the token generation interface
- Click on the profile picture, click on the Settings interface
- Click Developer settings on the sidebar to enter the developer settings interface
- Go to the Personal access tokens (classic) interface
- Generate new tokens based on configuration items
Generate git token
According to the prompt, fill in the token name, expiration time and scope, and choose according to your actual needs
The token is only displayed once, so be sure to record it
Perform jenkins configuration
Create credentials of type Username with Password
- fill in username
- Fill in the token information
Configure spark compilation tasks
new task
Choose a Freestyle Software Project
configuration tasks
source code management
Construct
The build script is as follows:
#!/bin/sh
export BUILD_ID=dontKillMe
source ~/.bash_profile sh
${WORKSPACE}/dev/make-distribution.sh --pip --tgz -Phive -Phive-2.3 -Phive-thriftserver -Pscala-2.12 -Phadoop-3.2 -Pmesos -Pyarn -Pkubernetes
I hope it will be helpful to you who are viewing the article, remember to pay attention, comment, and favorite, thank you