An article to understand automated containerized deployment (git+jenkins+harbor+kubernetes)

1. What is automated container deployment

  Simply put, we only need to click (or trigger through a gitlab hook), and the code will automatically complete the build -> image generation -> containerized deployment. Compared with traditional deployment, many steps can be saved, and it is especially suitable for agile development projects with frequent changes (in fact, it is best to use automation for containerized deployment projects, which is quite trouble-free).
  This article is mainly about the understanding and use of each component, the specific installation steps will not be expanded. It is a simple understanding of individuals, I hope you can correct me.
  The following figure is the flowchart and tools used for automated container deployment.

2. What is jenkins

  Jenkins is an open source continuous integration (CI) tool that provides a friendly operation interface. It is mainly used to continuously and automatically build/test software projects and monitor the operation of external tasks. Simply put, it is responsible for the entire process of our automated deployment. Whether it is building or producing images, or deploying, it is controlled by it. It can automatically complete many things through different configurations.
  Jenkins commonly used version control tools include SVN and GIT, and commonly used build tools include Maven, Ant, and Gradle.

3. What is mirror and dockerfile

  Traditional projects, such as jar packages, need to be placed on the host to run java -jar, such as war packages, need to be placed on the host, a tomcat is installed, and then startup.sh starts tomcat to run. One disadvantage of these traditional deployments is that the environment and deployment packages are separated. Versions like JDK and tomcat are fixed on the host. If we want to migrate to a new host, we must install the new environment and ensure that the version of the environment is consistent. , Which is both troublesome and error-prone, leading to deployment failures.
  So the Docker image came into being, it is actually equivalent to a runnable deployment package and the environment in which the deployment package runs, operating system, configuration parameters and other things that are needed at runtime. The image can be started and run on any host, and the environment remains consistent, which solves some of the pain points of traditional deployment. The container is actually equivalent to a running image.
  Dockerfile is a file that describes how the image is generated. It generally contains the basic image (such as the operating system or tomcat, etc.), startup parameters, and startup commands.

4. What is harbor

  Harbor is the mirror warehouse, which can access the mirror. Kubernetes can pull the mirror through the docker pull mirror warehouse/project name/mirror name: tag command, and jenkins can push the built mirror to the mirror warehouse. Other commonly used mirror warehouses include Docker hub and so on.

5. What are kubernetes and rancher

  Kubernetes is a portable and extensible open source platform for managing containerized workloads and services, which can promote declarative configuration and automation, referred to as k8s for short. Simply put, it is a tool that can perform container orchestration and realize automatic container operation, maintenance and management. As mentioned above, the container is the running image. The image is the deployable unit that contains the environment. The container is not enough to run alone. Functions such as load balancing, service discovery, health check, failed restart, etc. are needed. k8s is such a platform.
  Rancher is an open source enterprise-level container management platform. It includes kubernetes, which is the upper layer of the container orchestration tool. K8s has very powerful container management functions, but because it is managed through the command line, the visual display ability is relatively poor. Rancher has a UI interface, through which various functions of k8s can be conveniently executed, and various functions such as log collection, multi-k8s cluster management and monitoring can be added. It can be considered an enhanced version of k8s.

6. Start configuration

1. First create a jenkins project

  First is the project description. Discard old builds is to discard old builds. Generally, it is recommended to configure (because each project is built at a time of hundreds of mb, and the disk pressure is high). Days to keep builds is to save the builds within a few days, Max # of builds to keep is the maximum number of builds.

2. Configure git

  Configure the url for gitlab to pull the code. Note that if it reports red, it is generally due to network failure or incorrect credentials. Authentication failed means that the credentials are incorrect. After clicking add, you can enter the gitlab account password to configure a credential.

3. Configuration hook

  Let me mention that the hook is webhook, which is to provide an interface to gitlab and let it push events to tell jenkins to start automated deployment.

  Check Build when a change is pushed to GitLab. GitLab webhook URL, click the advanced option, then generate the key, and you can configure the hook on gitlab with the URL and key.

  Enter the gitlab project, click setting on the left to enter the hook configuration page, fill in the URL and key obtained by jenkins, check Trigger and click save, and the hook is configured. It should be noted that push events triggers the hook every time the code is pushed, and the tag push event triggers the hook only for tag pushes. Generally speaking, tag push is selected to prevent too frequent construction, and the tag can be used as a mirror tag at the same time. Distinguish the images that are built each time.

4. Configuration build and image generation

#!/bin/bash
#harbor环境变量配置
export REGISTRY=harbor域名
export PROJECT=harbor项目名
export BUILD_IMAGE_NAME=镜像名称
#git_tag_name是gitlab钩子推送时的标签，一般也用来作为镜像的标签
export BUILD_IMAGE_TAG=${
    
    GIT_TAG_NAME}
export HORBOR_USERNAME=harbor账号
export HORBOR_PASSWORD=harbor密码
#镜像名称
export IMAGE_ADDRESS=${
    
    REGISTRY}/${
    
    PROJECT}/${
    
    BUILD_IMAGE_NAME}:${
    
    BUILD_IMAGE_TAG}

#根据Dockerfile制作镜像，并推送镜像到harbor
#开启docker服务
service docker start
#登录镜像仓库
docker login ${
    
    REGISTRY} -u ${
    
    HORBOR_USERNAME} -p ${
    
    HORBOR_PASSWORD}
#根据-f指定dockerfile，并根据dockerfile打包镜像，-t指定镜像名称
docker build -f ${
    
    WORKSPACE}/scripts/apollo-on-kubernetes/apollo-portal-server/Dockerfile -t ${
    
    IMAGE_ADDRESS} .
#镜像推送至仓库
docker push ${
    
    IMAGE_ADDRESS}
#删除镜像（为了减轻jenkins磁盘压力）
docker rmi -f  ${
    
    IMAGE_ADDRESS}

  Click add post step and select execute shell to execute the shell script to package the image.

5. Configure k8s push

  Jenkins needs plug-ins to support k8s. Generally, there are mainly the following:
  Kubernetes Cli Plugin : This plug-in can be directly operated in Jenkins using the kubernetes command line.
  Kubernetes plugin : To use kubernetes, you need to install the plug-in.
  Kubernetes Continuous Deploy Plugin : kubernetes deployment plug-in can be used as needed.
  Enter jenkins settings by clicking Manage jenkins, and then click manage plugins to perform plug-in management. Most plug-ins can be searched and installed directly. Not supported, you can also download the plugin through the plugin website, such as https://mirrors.tuna.tsinghua.edu.cn/jenkins/ (mirror of Tsinghua University) or the official website https://plugins.jenkins.io/ and then install the plugin .
  Then click add post step and select deploy to kubernetes (plug-in support is required).

  If there is no certificate, you need to obtain the kubernetes config file from the kubernetes cluster, click the cluster and then click the kubeconfig file, and then copy the content.


  Click add in the jenkins credential, select kubeconfig for kind, check enter directly, paste the previously copied kubeconfig content into content, enter id and description for easy identification, and then save.

  At this point, the configuration of jenkins is complete. You can automate container deployment with one click by hooking or clicking Build Now.
  Note : You must first create the corresponding namespaces and projects in the harbor and k8s clusters. You can refer to the following example for the yaml file, or manually deploy the service in Rancher, export the yaml, and delete some time and unique id parameters.

Seven. dockerfile example

General springBoot project example

# 基础镜像，这里一般是基础操作系统镜像（带JDK的centos之类）
FROM xxxxx/xxxx/docker-base-os-centos-jdk8:1.0.0
# 时区
ENV TZ Asia/Shanghai
# 将原项目的jar复制过来改个名字
ADD ${
    
    WORKSPACE}/src/webapp/target/webapp-1.0.0.jar server.jar
# 启动命令
ENTRYPOINT ["java","-jar","-Xms4000m","-Xmx4000m","-Xss256k","-Xmn200m","/server.jar"]

Example of tomcat project

# 基础镜像，这里是tomcat基础镜像，地址从镜像仓库获取
FROM xxxxx/xxxx//tomcat
# 时区
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#将webapp下的文件全部删除
RUN rm -rf /usr/local/tomcat/webapps/*
#将target下的xx.war拷贝到/usr/local/tomcat/webapps/下
ADD ./src/target/javafw-1.0.war /usr/local/tomcat/webapps/ROOT.war
#端口
EXPOSE 8080
#设置启动命令
ENTRYPOINT ["/usr/local/tomcat/bin/catalina.sh","run"]

8. yaml file example

Sample workload configuration, deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    workload.user.cattle.io/workloadselector: deployment-web-admin-web
  name: admin-web
  namespace: web
spec:
  replicas: 1
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-uac-web-admin-web
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        workload.user.cattle.io/workloadselector: deployment-web-admin-web
    spec:
      containers:
      - args:
        - --spring.profiles.active=test
# 镜像名，注意，git_tag_name为jenkins从gitlab获取的推送时的标签
        image: XXXXXXXXX/XXXXX/image-name:${
    
    GIT_TAG_NAME}
        imagePullPolicy: Always
        name: admin-web
        ports:
#容器端口
        - containerPort: 8081
          name: web-8081
          protocol: TCP
      restartPolicy: Always

Cluster service (for access within the cluster), clusterlp-service.yaml:

# 对内暴露的端口
apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/targetWorkloadIds: '["deployment:web:admin-web"]'  # 配置命名空间和工作负载
  name: admin-web
  namespace: web
spec:
  ports:
  - name: web-8081-admin-web
    port: 8081
    protocol: TCP
    targetPort: 8081
  selector:
    workload.user.cattle.io/workloadselector: uac-web-admin-web
  type: ClusterIP

Public service, port exposure (for external access to the cluster), nodeport-service.yaml:

# 对外暴露的端口
apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/targetWorkloadIds: '["deployment:web:admin-web"]'    # 配置命名空间和工作负载
  name: admin-web-nodeport
  namespace: web
spec:
  ports:
  - name: uac-web-8081
#对外暴露的端口
    nodePort: 8555
#容器内部端口
    port: 8081
    protocol: TCP
    targetPort: 8081
  selector:
    workload.user.cattle.io/workloadselector: deployment-web-admin-web
  type: NodePort

IX. Common problem record

Generally, when the build fails, you can click single build and click console output to view the build log and the reason for the failure.

1. jenkins用spring-boot-maven-plugin构建出错：repackage failed: Unable to find main class
   https://blog.csdn.net/qq_35530005/article/details/109333356