【k8s实战一】Jenkins 部署应用到 Kubernetes

01 本文主旨

目标是演示整个Jenkins从源码构建镜像到部署镜像到Kubernetes集群过程。

为了简化流程与容易重现文中效果，做出如下操作：

使用 VBox 创建两台虚拟机
使用 Minikube 初始化简单的k8s集群
使用 GitHub 作为代码仓库，本文内容所有配置文件均会上传至此仓库 https://github.com/hellxz/cicd-demo.git

02 CI/CD流程

根据我画的这张图，Jenkins任务整体的流程分为五个步骤：

获取源码
构建制品
构建镜像并推送
发起部署请求
k8s调度pod，拉取镜像，部署完成

03 准备虚拟机

准备两个虚拟机，均安装好docker，参考Ubuntu 16.04及以上安装/卸载 Docker-CE

VM用途	系统版本	IP
Jenkins	Ubuntu 18.04.2 LTS	192.168.87.128
k8s master	Ubuntu 18.04.2 LTS	192.168.87.129

设置Jenkins机器免密登录k8s master主机

ssh-keygen -t rsa
ssh-copy-id [email protected]

验证免密效果

ssh [email protected]
exit

04 部署registry

部署registry之前，可以先将私有仓库配置到 /etc/docker/daemon.json，两台虚拟机均需要配置

sudo mkdir -p /etc/docker
sudo vim /etc/docker/daemon.json

内容如下：

{
    "insecure-registries": ["192.168.87.129:5000"]
}

接着在 192.168.87.129 虚拟机上部署registry，生产环境推荐使用Harbor

docker run -d --name registry -p 5000:5000 registry

05 配置maven

在jenkins虚拟机安装maven

wget https://mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar zxvf apache-maven-3.6.3-bin.tar.gz
mv apache-maven-3.6.3 maven-3.6.3

修改 /etc/profile，添加 M2_HOME 环境变量映射到 maven 目录，重写 PATH 添加 maven 的 bin 目录，如图31、32行

06 Tomcat部署Jenkins

在jenkins虚拟机 192.168.87.128上执行命令下载 tomcat 与 jenkins.war

#下载tomcat
wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz
#解压并重命名
tar -zxf apache-tomcat-9.0.41.tar.gz
mv apache-tomcat-9.0.41 /home/hellxz/jenkins
#进入jenkins安装目录
cd ~/jenkins/webapps
#删除webapps下示例项目
rm -rf *
#下载最新版jenkins.war
wget https://mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war
#安装openjdk、git
sudo apt-update
sudo apt-get install -y default-jdk default-jdk-headless git

设置Jenkins家目录，编辑 /etc/profile，添加 JENKINS_HOME 环境变量

刷新配置

source /etc/profile

修改tomcat下 conf/context.xml，加大静态资源缓存，提前解决启动 Jenkins 时的报错

<Resources cachingAllowed="true" cacheMaxSize="9999999" />

启动Jenkins

cd ~/jenkins/bin
./startup.sh

访问 http://192.168.87.129:8080，安装推荐插件，安装加速可参考 Jenkins安装插件提速

07 部署Kubernetes集群

部署一个可以通过 kubectl 命令部署应用的k8s环境即可，部署k8s环境不是本文重点，以下为参考

使用kubeadm部署Kubernetes集群，点击展开查看参考我之前的博客吧，差异应该不大【k8s学习笔记】使用 kubeadm 部署 v1.18.5 版本 Kubernetes集群

使用Minikube部署单节点测试k8s环境，点击展开查看


#添加阿里apt kubernetes源
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 
sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
#安装kubectl
sudo apt-get install -y kubectl
#下载minikube二进制文件
curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
#部署k8s环境，默认使用docker驱动。读者也可使用vbox
minikube start -–insecure-registry=192.168.87.129:5000

本文使用Minikube做测试，坑比较多，想趟坑可以先参考下 https://www.kubernetes.org.cn/6388.html
关于Minikube的吐槽
写文时，用的Win10创建Linux无桌面虚拟机，在虚拟机基础上装的minikube，minikube默认使用Docker在虚拟机中又创建了个Docker容器，里边才是k8s环境，所以想在虚拟机外访问套了两层的内网应用服务，可能还需要在虚拟机中加Nginx，通过它转发请求到虚拟机中的内网地址……虽然疯狂套娃挺坑的，但为了演示下效果还是可以的。 PS：在Linux/MacOS环境直接装Minikube，是可以通过浏览器直接看的。由于未直接在Win环境使用Docker Desktop，所以未确定Win下是否可用。

08 准备Demo

Demo分五部分：测试代码、Jenkins构建脚本、docker镜像构建脚本、k8s部署配置yaml、部署脚本，均处在同一仓库中

仓库地址：https://github.com/hellxz/cicd-demo.git

目录结构：

demo

使用start.spring.io生成的只添加spring-boot-starter-web依赖，在启动类上加了个 /test 的接口

Dockerfile

FROM openjdk:8-jdk
#设置工作目录
WORKDIR /root
#复制制品jar到/root/app.jar位置
ADD target/*.jar app.jar
#JVM参数，给后期调优使用
ENV JVM_OPTS=""
#启动服务
CMD java ${JVM_OPTS} \
    -Djava.security.egd=file:/dev/./urandom \
    -jar app.jar

shell

artifact2image.sh 制品转镜像Shell脚本，本文为了简单未安装Jenkins关于docker的插件

#!/bin/bash
# -- 构建镜像并推送私有仓库 --
set -eu #如有报错或取不到变量情况停止执行

#声名常量
IMG_REGISTRY="192.168.87.129:5000" #镜像仓库
IMG_NAME="cicd-demo"
IMG_TAG=`date "+%Y%M%d_%H%M"`      #镜像标签，如 20201223_1351
IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #镜像上传与拉取的名称

#构建镜像
docker build -t ${IMG_FULL_NAME} .

#推送镜像
docker push ${IMG_FULL_NAME}

#删除本地镜像
docker rmi ${IMG_FULL_NAME}

#修改deploy.yaml的镜像标签
sed -i "s#{
    
    {IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml

deploy2k8s.sh 部署镜像到k8s环境脚本

#!/bin/bash
# -- 部署image到k8s --
# !注意：需要提前做ssh免密登录
set -eu

#定义常量
PROJECT_NAME="cicd-demo"
UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}"
FILE_NAME="${UPLOAD_DIR}/deploy.yaml"
SSH_USER="hellxz"
SSH_IP="192.168.87.129"

#首先删除待上传目录的同名文件
ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}"

#确保部署文件目录存在
ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}"

#远程复制部署文件
scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME}

#远程执行部署命令
ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}"

Jenkinsfile

pipeline{
    agent any
    stages {
        //由于源码和Jenkinsfile处于同一仓库，在Jenkins项目执行时，会先将Jenkinsfile所在的仓库克隆下来，为了简单，这里就不重复添加拉取源码的操作了
        //其它情况，如使用多个仓库一些构建的，这种就需要额外添加拉取代码的stage了。
        stage('Build Artifact') {
            steps{
                sh label:'maven building', script: 'mvn clean package -DskipTests'
            }
        }
        stage('Build Image'){
            steps{
                sh label:'image building', script: '/bin/bash artifact2image.sh'
            }
        }
        stage('Deploy k8s'){
            steps{
                sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh'
            }
        }
    }
    post {
        success{
            //成功清理工作空间，失败保留现场
            cleanWs()
        }
    }
}

deploy.yaml

apiVersion: apps/v1   #api版本
kind: Deployment      #对象类型
metadata:             #元信息标识
  name: cicd-demo
  namespace: default
  labels:             #自定义Deployment绑定的标签
    app: cicd-demo
spec:                 #期待运行状态
  replicas: 1         #部署实例数
  selector:
    matchLabels:
      app: cicd-demo
  template:           #pod
    metadata:
      labels:         #与spec.selector.matchLabels相同
        app: cicd-demo
    spec:             #pod容器期望的状态
      containers:
        - name: cicd-demo
          image: {
    
    {IMAGE_NAME}}
          imagePullPolicy: IfNotPresent
          env:
            - name: JVM_OPTS
              value: "-Xms128m -Xmx256m"
          ports:
            - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: cicd-demo-svc
  namespace: default
  labels:
    service: cicd-demo-svc
spec:
  selector:            #匹配pod标签
    app: cicd-demo
  type: NodePort     #svc类型，ExternalName, ClusterIP, NodePort, and LoadBalancer
  ports:
    - name: cicd-demo-port
      protocol: TCP
      port: 8080
      nodePort: 30000

09 创建Jenkins流水线项目

访问 http://192.168.87.128:8080/jenkins 并登录，以下创建新流水线demo构建项目

部署与验证

执行构建demo程序

日志输出：

Started by user admin
Obtained Jenkinsfile from git https://github.com/hellxz/cicd-demo.git
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/hellxz/jenkins/home/workspace/demo
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/hellxz/cicd-demo.git
 > git init /home/hellxz/jenkins/home/workspace/demo # timeout=10
Fetching upstream changes from https://github.com/hellxz/cicd-demo.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > git fetch --tags --progress -- https://github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/hellxz/cicd-demo.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
Commit message: "update"
 > git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build Artifact)
[Pipeline] sh (maven building)
+ mvn clean package -DskipTests
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[INFO] Scanning for projects...

[INFO] 
[INFO] ----------------------< online.hellxz:cicd-demo >-----------------------
[INFO] Building cicd-demo 0.0.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo ---
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo ---

[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes

[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo ---
[INFO] Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.248 s
[INFO] Finished at: 2020-12-23T11:51:20Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build Image)

[Pipeline] sh (image building)
+ /bin/bash artifact2image.sh
Sending build context to Docker daemon  17.26MB

Step 1/5 : FROM openjdk:8-jdk
 ---> 89f100fa8f9f
Step 2/5 : WORKDIR /root
 ---> Using cache
 ---> 17670bfa01f1
Step 3/5 : ADD target/*.jar app.jar

 ---> b7e4292e4f80
Step 4/5 : ENV JVM_OPTS=""
 ---> Running in 0be8c16c9c8a
Removing intermediate container 0be8c16c9c8a
 ---> 3ef66b100082
Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom     ${JVM_OPTS}     -jar app.jar
 ---> Running in c8209a642a2d
Removing intermediate container c8209a642a2d
 ---> bc7859f9b4fe
Successfully built bc7859f9b4fe
Successfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151

The push refers to repository [192.168.87.129:5000/cicd-demo]
44cbb90cc2c3: Preparing
f85e383859a1: Preparing
ffb4778f8a52: Preparing
e528f2c31deb: Preparing
c5f4367d4a59: Preparing
ceecb62b2fcc: Preparing
193bc1d68b80: Preparing
f0e10b20de19: Preparing
ceecb62b2fcc: Waiting
193bc1d68b80: Waiting
f0e10b20de19: Waiting
ffb4778f8a52: Layer already exists
e528f2c31deb: Layer already exists
f85e383859a1: Layer already exists
193bc1d68b80: Layer already exists
c5f4367d4a59: Layer already exists
ceecb62b2fcc: Layer already exists
f0e10b20de19: Layer already exists
44cbb90cc2c3: Pushed

20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006
Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151
Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4
Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291
Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4
Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9
Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Deploy k8s)
[Pipeline] sh (deploy image to k8s)
+ /bin/bash deploy2k8s.sh

deployment.apps/cicd-demo created
service/cicd-demo-svc created
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)

[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

可以看到以下输出，说明程序已部署到k8s集群中了

deployment.apps/cicd-demo created
service/cicd-demo-svc created

ssh登录k8s的机器验证

总结

简单总结下，Jenkins与Kubernetes的集成，更多地是Jenkins如何控制Kubernetes部署的流程，文中例子是通过 构建并上传镜像 和 ssh发送配置和命令 实现的。

本文是参考慕课网上的教程，快速扫了下视频实践所得，是了解其思路后自行实现的。

实际操作上才发现，脑子说会了，然后双手却没有跟上，还是要动手啊！

操作了一天半左右，最开始还想着用vagrant初始化VM，由于工作上有个虚拟机是VBox5的，升级新版不能用。搞了半天放弃了Vagrant(VBox6.1升级了命令参数不兼容旧版，导致Vagrant为兼容它把几乎所有版本Vagrant都改了……)。

文中难免因为安装部分的描述而过于冗长，感谢大家能看到这里，最后再次感谢慕课网的鹿哥

参考

[1]: https://www.imooc.com/learn/1112

[2]: https://cloud.tencent.com/developer/article/1394657

[3]: https://www.jianshu.com/p/3f13850d66ea

[4]: https://www.cnblogs.com/hellxz/p/use-kubeadm-init-kubernetes-cluster.html

[5]: https://www.cnblogs.com/hellxz/p/how-kubernetes-deploy-application.html

[6]: https://developer.aliyun.com/article/221687?spm=a2c6h.12873639.0.0.755c6235EZuEht

[7]: https://minikube.sigs.k8s.io/docs/start

[8]: https://www.cnblogs.com/hellxz/p/install_jenkins.html

[9]: https://www.cnblogs.com/hellxz/p/jenkins_install_plugins_faster.html