Operation and maintenance (39) Deploy SpringBoot to K8S case DevOps through KubeSphere

Article directory

DevOps

See demo source code: https://gitee.com/zhengqingya/java-workspace
based onkubesphere 3.2.1

Automatically check out code, test, analyze, build, deploy and publish

insert image description here

1. Create`DevOps项目`

insert image description here

2. DevOps Credentials

1. gitee warehouse certification`gitee-auth`

insert image description here

2. Ali cloud docker warehouse certification`aliyun-docker-registry-auth`

insert image description here

3. k8s certificate`kubeconfig-auth`

# k8s权限配置文件
cat /root/.kube/config

insert image description here

https://lb.kubesphere.local:6443Put -> in the content https://指定IP:6443, otherwise there may be problems in later deployment...
insert image description here

finally
insert image description here

3. Maven configures Alibaba Cloud Central Warehouse

平台管理 -> 集群管理 -> default -> 配置 -> 配置字典 -> ks-devops-agent

insert image description here

edit settings

insert image description here

<mirrors>
    <!-- 国内中央仓库的配置-阿里云中央仓库 -->
    <mirror>
        <id>nexus-aliyun</id>
        <mirrorOf>central</mirrorOf>
        <name>Nexus aliyun</name>
        <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </mirror>
</mirrors>

insert image description here

Four, k8s-project configuration

Alibaba cloud docker warehouse certification`aliyun-docker-registry-auth`

insert image description here

# 也可通过命令查看凭证
kubectl get secrets -n my-project

5. Create a pipeline

insert image description here

After entering, you can click 编辑流水线and provide some templates

insert image description here

ex: The first step is to pull the code

insert image description here

Let's play here by yourself, it's very simple... Just customize it according to your own needs...

insert image description here

6. Others

`Jenkinsfile`

pipeline {
    agent {
        node {
            label 'maven'
        }

    }

    environment {
        DOCKER_REGISTRY_AUTH = "aliyun-docker-registry-auth"
        DOCKER_REGISTRY = 'registry.cn-hangzhou.aliyuncs.com'
        DOCKER_REGISTRY_NAMESPACE = 'zhengqingya'
        APP_DOCKER_IMAGE = "${DOCKER_REGISTRY}/${DOCKER_REGISTRY_NAMESPACE}/${APP_NAME}:${BRANCH_NAME}"
        PROJECT_GIT_URL = 'https://gitee.com/zhengqingya/test.git'
        APP_NAME = 'test'
        BRANCH_NAME = 'master'
        IS_SKIP_BUILD = 'false'
        JAVA_OPTS = "-XX:+UseG1GC -Xms100m -Xmx100m -Dserver.port=8080"
    }

//    parameters {
//        string(name: 'BRANCH_NAME', defaultValue: 'master', description: 'git分支名')
//        choice(name: 'IS_SKIP_BUILD', choices: ['false', 'true'], description: '是否跳过构建,直接部署')
//        choice(name: 'SERVICE_NAMES', choices: ['test', 'system' ,'all'], description: '请选择要构建的服务,支持单个服务发布或全部服务发布')
//    }

    stages {

        stage('参数验证') {
            agent none
            steps {
                container('maven') {
                    sh """
                        echo "分支: ${BRANCH_NAME}"
                        echo "是否跳过构建，直接部署(tips:适用于之前已经进行过构建打包的情景)：${IS_SKIP_BUILD}"
                        echo "app镜像: ${APP_DOCKER_IMAGE}"
                        echo "构建运行ID: ${BUILD_NUMBER}"
                        echo "JAVA_OPTS: ${JAVA_OPTS}"
                    """
                }
            }
        }

        stage('拉取代码') {
            agent none
            steps {
                container('maven') {
                    git(credentialsId: 'gitee-auth', url: "${PROJECT_GIT_URL}", branch: "${BRANCH_NAME}", changelog: true, poll: false)
                    sh 'ls -al'
                }
            }
        }

        stage('项目编译') {
            agent none
            steps {
                container('maven') {
                    sh 'mvn clean package -Dmaven.test.skip=true'
                    sh 'ls -al'
                }
            }
        }

        stage('docker镜像构建&推送') {
            agent none
            steps {
                container('maven') {
                    sh 'cp target/*.jar docker'
                    sh """
                        cd docker
                        ls
                        echo "app镜像: ${APP_DOCKER_IMAGE}"
                        docker build -f Dockerfile  -t ${APP_DOCKER_IMAGE} . --no-cache
                    """
                    withCredentials([usernamePassword(credentialsId: "${DOCKER_REGISTRY_AUTH}", passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME',)]) {
                        sh 'echo "$DOCKER_PASSWORD" | docker login $DOCKER_REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push ${APP_DOCKER_IMAGE}'
                        sh "echo 镜像推送成功：${APP_DOCKER_IMAGE}"
                        sh 'ls -al'
                    }

                }

            }
        }

        stage('发布到k8s') {
            agent none
            steps {
                container('maven') {
                    sh 'ls -al'
                    withCredentials([kubeconfigFile(credentialsId: 'kubeconfig-auth', variable: 'KUBECONFIG')]) {
                        // envsubst: 将相关参数传给该yml文件
                        sh 'envsubst < k8s/k8s-deploy.yml | kubectl apply -f -'
                    }
                }
            }
        }

    }
}

`k8s-deploy.yml`

---
# 定义工作负载
apiVersion: apps/v1
kind: Deployment  # 无状态部署
metadata:
  name: ${
    
    APP_NAME}
  namespace: my-project   # TODO 命名空间
  labels:
    app: ${
    
    APP_NAME}
spec:
  replicas: 3 # TODO 3个副本
  strategy:
    rollingUpdate: # 由于replicas为3,则整个升级,pod个数在2-4个之间
      maxSurge: 1        # 滚动升级时会先启动1个pod
      maxUnavailable: 1  # 滚动升级时允许的最大Unavailable的pod个数
  selector:
    matchLabels:
      app: ${
    
    APP_NAME}
  template:
    metadata:
      labels:
        app: ${
    
    APP_NAME}
    spec:
      imagePullSecrets:
        - name: aliyun-docker-registry-auth  # TODO 提前在项目下配置访问阿里云仓库的账号密码
      containers:
        - name: ${
    
    APP_NAME}
          image: ${
    
    APP_DOCKER_IMAGE} # TODO 镜像地址
          imagePullPolicy: Always
          env: # 环境变量
            - name: JAVA_OPTS
              value: ${
    
    JAVA_OPTS}
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          # CPU内存限制
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
          # 就绪探针
#          readinessProbe:
#            httpGet:
#              path: /actuator/health
#              port: 8080
#            timeoutSeconds: 10
#            failureThreshold: 30
#            periodSeconds: 5

---
# 定义服务
apiVersion: v1
kind: Service
metadata:
  name: ${
    
    APP_NAME}  # TODO 服务名
  namespace: my-project   # TODO 命名空间
spec:
  selector:
    app: ${
    
    APP_NAME} # TODO  label selector配置，将选择具有label标签的Pod作为管理
  type: ClusterIP # 访问方式  ClusterIP/NodePort
  ports:
    - name: http            # 端口名称
      port: 8080
      protocol: TCP    # 端口协议，支持TCP和UDP，默认TCP
      targetPort: 8080
      # nodePort: 666  # TODO 当`type = NodePort`时 对外开放端口
  sessionAffinity: None  # 是否支持session

k8s yaml online writing tool

https://k8syaml.com

report error`ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: java.net.UnknownHostException: lb.kubesphere.local: Name or service not known`

insert image description here

Deploy to Kubernetes14.66 s失败
Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/devops-testp5hsh/test/k8s/k8s-deploy.yml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: java.net.UnknownHostException: lb.kubesphere.local: Name or service not known
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: java.net.UnknownHostException: lb.kubesphere.local: Name or service not known
	at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)
	at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:213)
	at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:201)
	at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)
	at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
	at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
	at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
	at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
	at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.70.143/10.233.70.143:51962
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at hudson.FilePath.act(FilePath.java:1160)
		at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
		at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
		at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
		at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
		at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
		at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
		at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
		at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
		at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		... 1 more
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: java.net.UnknownHostException: lb.kubesphere.local: Name or service not known
	at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:898)
	at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeploymentWithHttpInfo(AppsV1Api.java:7299)
	at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeployment(AppsV1Api.java:7275)
	at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:210)
	... 16 more
Caused by: hudson.remoting.ProxyException: java.net.UnknownHostException: lb.kubesphere.local: Name or service not known
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at okhttp3.Dns.lambda$static$0(Dns.java:39)
	at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:171)
	at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:135)
	at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
	at okhttp3.RealCall.execute(RealCall.java:81)
	at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:894)
	... 19 more
Api call failed with code 0, detailed message: null
Kubernetes deployment ended with HasError

solve:

kubesphere 3.2.1Pipeline adjustment

Step 1: Replace `kubernetesDeploy`the deployment method

https://github.com/kubesphere/website/pull/2098

stage('发布到k8s') {
    agent none
    steps {
        container('maven') {
            // 废弃...
            //     kubernetesDeploy(enableConfigSubstitution: true, deleteResource: false, kubeconfigId: 'kubeconfig-auth', configs: 'k8s/**')
            // 改为下面这种方式
            withCredentials([kubeconfigFile(credentialsId: 'kubeconfig-auth', variable: 'KUBECONFIG')]) {
                // envsubst: 将相关参数传给该yml文件
                sh 'envsubst < k8s/k8s-deploy.yml | kubectl apply -f -'
            }
        }
    }
}

Step Two: Modify DevOps Credentials`kubeconfig`

https://lb.kubesphere.local:6443Put the -> in the contenthttps://指定IP:6443

insert image description here

The final release was successful

insert image description here

Today's sharing sentence:
No matter how high the sky is, stand on tiptoe and get closer to the sun.