jenkins发布k8s应用

因为还没有完成控制台开发,我们k8s应用的发布目前只能直接在jenkins中实现了。

1. 基础镜像

按层次制作基础镜像

centos -> centos-sshd -> centos74-jre[678] -> centos74-jre[678]-[tomcat6|tomcat7|tomcat8|springboot]

发布应用时,根据jre\middleware参数,选择基础镜像

2. 资源申请参数说明

必填

project

项目的名称,纯数字前要加'p'

app

应用名称,以项目名称作为前缀,app在k8s集群是唯一的,作为应用的标识

domain

域名,无填'null'

sessionSticky

会话保持,类似nginx的ip_hash,'true'或’false’

replicate

副本数,即节点数

javaVersion

jre版本,支持jre6\jre7\jre8

middleware

中间件,支持tomcat6\tomcat7\tomcat8\springboot

rwFileList

Tomcat项目文件如需要写权限,多个用逗号分隔,无填’null’

codeRepo

git地址

warDir

war包相对路径,如’target’

mvnCommand

mvn打包命令

sbStartCommand

Springboot项目启动文件指定profile的命令

dingToken

如需要发dingding群通知,提供token,无则’null’

可选(运维有默认配置)

cpuReq

Cpu初始分配核数目

cpuLmt

Cpu分配核数目上限

memReq

内存初始分配大小(M)

memLmt

内存分配大小上限(M),一般和memReq相同

jvmSize

Jvm大小,默认’null’,公式根据memReq自动计算

directMemSize

对外内存最大大小,默认’null’,公式根据memReq自动计算

mvnVersion

Mvn版本

mvnJavaVersion

Mvn环境的jdk版本

imageVersion

基础镜像版本(研发不需了解)

3. jenkins pipeline

所有变量抽出至environment下,便于配置,也便于后续自研平台调用;

整个过程分为checkout、mvn、build、deploy、post action几个部分;

checkout:拉取代码

mvn:调用mvn命令打包

build:执行make_dockerfile.sh,生成dockerfile,调用docker命令build镜像、push镜像

deploy:执行make_deploymentfile.sh,make_ingressfile.sh,调用kubectl命令创建deployment、ingress(含service)

post action:发送钉钉通知完整jenkinsfile如下

pipeline {
    environment {
        project = 'myproj'
        app = 'myproc-myapp'
        domain = 'mydomain.ali.com'
        replicate = '4'
        sessionSticky = 'false'
        javaVersion = 'jre8'
        middleware = 'springboot'
        rwFileList = 'null'
        codeRepo = 'http://git.mydomain.com/usergroup/myapp.git'
        warDir = 'target'
        mvnCommand = 'clean install -DskipTests=true -Ptest'
        sbStartCommand = '-Dspring.profiles.active=test'
        dingToken = '8826e85c02c796aa5aea0a39d292e1111111111111111111111'
        
        cpuReq = '0.2'
        cpuLmt = '2'
        memReq = '1024'
        memLmt = '1024'
        jvmSize = 'null'
        directMemSize = 'null'
        mvnVersion = 'maven-3.6.0'
        mvnJavaVersion = 'jdk1.8.0_221'
        imageVersion = 'v1'

    }
    
    agent any
    
    options {
        timeout(time: 10, unit: 'MINUTES') 
    }
    
    stages {
        
        stage('checkout') {
            steps {
                script {
                    def codeRepoCred
                    def reg1 = /.+git\.mydomain\.com.+/
                    if ("${codeRepo}".matches(reg1)) {
                        codeRepoCred = 'mycred'
                    }
                    checkout([$class: 'GitSCM', branches: [[name: '$tag']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${codeRepoCred}", url: "${codeRepo}"]]])
                }
            }
        }
        stage('mvn') {
            environment {
                PATH = "/usr/local/java/${mvnJavaVersion}/bin:$PATH"
            }
            steps {
                sh "/usr/local/maven/${mvnVersion}/bin/mvn -f pom.xml ${mvnCommand}"
            }
        }
        stage('build') {
            steps {
                sh "/data1/template/make_dockerfile.sh ${WORKSPACE} ${javaVersion} ${middleware} ${cpuLmt} ${memLmt} ${rwFileList} ${imageVersion} ${sbStartCommand} ${jvmSize} ${directMemSize}"
                script {
                    tagnew = "${tag}".replaceAll("/","_");
                    sh "/usr/bin/docker build -f ${WORKSPACE}/deploy/dockerfile -t harbor.mydomain.com/$project/${app}:${tagnew} ${WORKSPACE}/${warDir}"
                    sh "/usr/bin/docker push harbor.mydomain.com/$project/${app}:${tagnew}"
                    sh "/usr/bin/docker rmi harbor.mydomain.com/$project/${app}:${tagnew}"
                }
            }
        }
        stage('deploy') {
            steps {
                script {
                    tagnew = "${tag}".replaceAll("/","_");
                    sh "/data1/template/make_deploymentfile.sh ${WORKSPACE} ${app} ${project} ${replicate} ${BUILD_NUMBER} ${cpuReq} ${cpuLmt} ${memReq} ${memLmt} ${tagnew}"
                    sh "/usr/bin/kubectl apply -f ${WORKSPACE}/deploy/deployment.yaml"
                    sh "/usr/bin/kubectl delete ingress ${app} > /dev/null 2>&1 || true"
                    sh "/usr/bin/kubectl delete service ${app} > /dev/null 2>&1 || true"
                    if ("${domain}" != "null") {
                        sh "/data1/template/make_ingressfile.sh ${WORKSPACE} ${app} ${domain} ${sessionSticky}"
                        sh "/usr/bin/kubectl apply -f ${WORKSPACE}/deploy/ingress.yaml"
                    }
                    sh "/usr/bin/kubectl rollout status deployment/${app}"
                }

            }
        }
    }
    post {
		success {
			dingTalk accessToken:"https://oapi.dingtalk.com/robot/send?access_token=${dingToken}", 
			imageUrl:'', 
			jenkinsUrl:'http://10.40.16.212/', 
			message:' deploy success', 
			notifyPeople:''
		}
		failure {
			dingTalk accessToken:'https://oapi.dingtalk.com/robot/send?access_token=${dingToken}', 
			imageUrl:'', 
			jenkinsUrl:'http://10.40.16.212/', 
			message:' deploy failed', 
			notifyPeople:''
		}
    }
}

4. shell与模板文件

jenkins服务器上的模板相关文件:*.template

make_*.sh传递pipeline中的变量,根据template文件生成各个项目特定的dockerfile、deployment.yaml、ingress.yaml

dockerfile.template 

将应用启动需要的参数,输出至用户.bash_profile中,生成环境变量;后续脚本直接调用环境变量

FROM harbor.mydomain.com/baseimage/imageTag
LABEL type="appimage"
COPY --chown=appuser:appuser *.war /apps/deploy/
RUN echo "export CPU_LIMIT=cpuLmtTag" >> /home/appuser/.bash_profile \
    && echo "export MEM_LIMIT=memLmtTag" >> /home/appuser/.bash_profile \
    && echo "export JVM_SIZE=jvmSizeTag" >> /home/appuser/.bash_profile \
    && echo "export DIRECT_MEM_SIZE=directMemSizeTag" >> /home/appuser/.bash_profile \
    && echo "export PERM_NAME=permNameTag" >> /home/appuser/.bash_profile \
    && echo "export FILE_LIST=rwFileListTag" >> /home/appuser/.bash_profile \
    && echo "export SB_START_COMMAND=sbStartCommandTag" >> /home/appuser/.bash_profile \
    && echo -e "LIBSYSCONFCPUS=\${CPU_LIMIT}" >> /home/appuser/.bash_profile \                
    && echo -e "if [ \${LIBSYSCONFCPUS} -lt 2 ]; then LIBSYSCONFCPUS=2; fi" >> /home/appuser/.bash_profile \
    && echo "export LIBSYSCONFCPUS" >> /home/appuser/.bash_profile \      
    && echo -e "export LD_PRELOAD=/usr/local/lib/libsysconfcpus.so:\${LD_PRELOAD}" >> /home/appuser/.bash_profile
CMD ["/apps/script/pod_start.sh"]

deployment.template

apiVersion: apps/v1
kind: Deployment
metadata:
  name: appTag
  labels:
    app: appTag
spec:
  replicas: replicateTag
  selector:
    matchLabels:
      app: appTag
  template:
    metadata:
      labels:
        app: appTag
        project: projectTag
        buildnum: "buildnumTag"
    spec:
      dnsPolicy: Default
      containers:
      - name: appTag
        image: harbor.mydomain.com/projectTag/appTag:tagTag
        imagePullPolicy: Always
        ports:
        - containerPort: 9999
        securityContext:
          privileged: false
        resources:
          requests:
            cpu: cpuReqTag
            memory: memReqTagMi
            ephemeral-storage: 2Gi
          limits:
            cpu: cpuLmtTag
            memory: memLmtTagMi
            ephemeral-storage: 5Gi
        readinessProbe:
          #httpGet:
          #  path: /healthProbe.jsp
          tcpSocket:
            port: 9114
          initialDelaySeconds: 30
          periodSeconds: 3
        livenessProbe:
          #httpGet:
          #  path: /healthProbe.jsp
          tcpSocket:
            port: 9114
          initialDelaySeconds: 120
          periodSeconds: 3
        lifecycle:
          preStop:
            exec:
              command: ["/apps/script/pod_pre_stop.sh"]
        volumeMounts:
        - mountPath: /apps/heapdump
          name: dump-vol
      volumes:
      - name: dump-vol
        hostPath:
          path: /var/lib/docker/dumpvol
          type: Directory

ingress.template

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: appTag
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: domainTag
    http:
      paths:
      - path: /
        backend:
          serviceName: appTag
          servicePort: http
---
kind: Service
apiVersion: v1
metadata:
  name: appTag
  annotations:
    traefik.ingress.kubernetes.io/affinity: "sessionStickyTag"
    traefik.ingress.kubernetes.io/session-cookie-name: "lbcookie"
spec:
  clusterIP: None
  selector:
    app: appTag
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 9999

make_dockerfile.sh

#!/bin/bash

workspace=$1
jre=$2
middleware=$3
cpulmt=$4
memlmt=$5
rwfilelist=$6
imageVersion=$7
sbstartCommand=$8
jvmSize=$9
directMemSize=${10}

if [ $# -ne 10 ]; then
    echo "Usage: $0 [workspace] [jre] [middlewre] [cpulmt] [memlmt] [rwfilelist] [imageVersion] [sbstartCommand] [jvmSize] [directMemSize]"
    exit 1
fi

if [ $9 == "null" ]; then
    if [ $memlmt -le 3072 ]; then
        jvmSize=`expr $memlmt / 2`
    else
        jvmSize=`expr $memlmt - 1500`
    fi
fi

if [ ${10} == "null" ]; then
    if [ $memlmt -le 2048 ]; then
        directMemSize=256
    else
        directMemSize=512
    fi
fi

if [ $jre == "jre6" || $jre == "jre7" ]; then
    permName="PermSize"
else
    permName="MetaspaceSize"
fi

image="centos74-"${jre}"-"${middleware}":"${imageVersion}

mydockerfile=$workspace/deploy/dockerfile
mkdir -p $workspace/deploy
cp /data1/template/dockerfile.template $mydockerfile

/usr/bin/sed -i "s/imageTag/${image}/g" $mydockerfile
/usr/bin/sed -i "s/cpuLmtTag/${cpulmt}/g" $mydockerfile
/usr/bin/sed -i "s/memLmtTag/${memlmt}/g" $mydockerfile
/usr/bin/sed -i "s/rwFileListTag/${rwfilelist}/g" $mydockerfile
/usr/bin/sed -i "s/sbStartCommandTag/${sbstartCommand}/g" $mydockerfile
/usr/bin/sed -i "s/jvmSizeTag/${jvmSize}/g" $mydockerfile
/usr/bin/sed -i "s/directMemSizeTag/${directMemSize}/g" $mydockerfile
/usr/bin/sed -i "s/permNameTag/${permName}/g" $mydockerfile

make_deploymentfile.sh 

#!/bin/bash

workspace=$1
app=$2
project=$3
replicate=$4
buildnum=$5
cpuReq=$6
cpuLmt=$7
memReq=$8
memLmt=$9
tag=${10}

if [ $# -ne 10 ]; then
    echo "Usage: $0 [workspace] [app] [project] [replicate] [buildnum] [cpuReq] [cpuLmt] [memReq] [memLmt] [tag]"
    exit 1
fi

mydeploymentfile=$workspace/deploy/deployment.yaml
cp /data1/template/deployment.template $mydeploymentfile

/usr/bin/sed -i "s/appTag/${app}/g" $mydeploymentfile
/usr/bin/sed -i "s/projectTag/${project}/g" $mydeploymentfile
/usr/bin/sed -i "s/replicateTag/${replicate}/g" $mydeploymentfile
/usr/bin/sed -i "s/buildnumTag/${buildnum}/g" $mydeploymentfile
/usr/bin/sed -i "s/cpuReqTag/${cpuReq}/g" $mydeploymentfile
/usr/bin/sed -i "s/cpuLmtTag/${cpuLmt}/g" $mydeploymentfile
/usr/bin/sed -i "s/memReqTag/${memReq}/g" $mydeploymentfile
/usr/bin/sed -i "s/memLmtTag/${memLmt}/g" $mydeploymentfile
/usr/bin/sed -i "s/tagTag/${tag}/g" $mydeploymentfile

make_ingressfile.sh

#!/bin/bash

workspace=$1
app=$2
domain=$3
sessionSticky=$4

if [ $# -ne 4 ]; then
    echo "Usage: $0 [workspace] [app] [domain] [sessionSticky]"
    exit 1
fi

myingressfile=$workspace/deploy/ingress.yaml
cp /data1/template/ingress.template $myingressfile

/usr/bin/sed -i "s/appTag/${app}/g" $myingressfile
/usr/bin/sed -i "s/domainTag/${domain}/g" $myingressfile
/usr/bin/sed -i "s/sessionStickyTag/${sessionSticky}/g" $myingressfile
发布了24 篇原创文章 · 获赞 25 · 访问量 2万+

猜你喜欢

转载自blog.csdn.net/sdmei/article/details/101309616
今日推荐