Jenkins发布Kubernetes(K8s)集群(基于containerd)

物联网 2023-09-13 17:55:39 阅读次数: 0

上一篇文章> Jenkins入门与安装

一、实验环境

1、k8s环境

版本v1.26.5，容器为containerd
二进制安装Kubernetes(K8s)集群(基于containerd)—从零安装教程（带证书）

主机名 IP 系统版本 安装服务
master01 10.10.10.21 rhel7.5 nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master02 10.10.10.22 rhel7.5 nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master03 10.10.10.23 rhel7.5 nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
node01 10.10.10.24 rhel7.5 nginx、kubelet、proxy
node02 10.10.10.25 rhel7.5 nginx、kubelet、proxy

2、jenkins环境

jenkins入门与安装
容器为docker

主机 IP 系统版本
jenkins 10.10.10.10 rhel7.5

二、docker-compose安装

jenkins服务器上面安装

1、下载

https://github.com/docker/compose/releases/
下载版本：v2.18.0

在这里插入图片描述

2、安装

[root@jenkins ~]# cp docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@jenkins ~]# chmod +x /usr/local/bin/docker-compose

3、查看版本

[root@jenkins ~]# docker-compose --version
Docker Compose version v2.18.0

三、cfssl证书生成

此处记录使用cfssl工具生成harbor私有证书，并使用证书搭建Harbor仓库，此证书使用按照kubernetes时使用的ca证书安装

1、安装cfssl

https://imroc.cc/kubernetes/trick/certs/sign-certs-with-cfssl.html
安装包下载地址：https://github.com/cloudflare/cfssl/releases

[root@jenkins ~]# ls cfssl*
cfssl_1.6.2_linux_amd64  cfssl-certinfo_1.6.2_linux_amd64  cfssljson_1.6.2_linux_amd64
[root@jenkins ~]# mv cfssl_1.6.2_linux_amd64 /usr/bin/cfssl
[root@jenkins ~]# mv cfssl-certinfo_1.6.2_linux_amd64  /usr/bin/cfssl-certinfo
[root@jenkins ~]# mv cfssljson_1.6.2_linux_amd64 /usr/bin/cfssljson
[root@jenkins ~]# chmod +x /usr/bin/cfssl*

2、ca生成证书

[root@jenkins ~]# mkdir -p pki &&  cd pki
[root@jenkins pki]# cat > ca-csr.json   << EOF 
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

[root@jenkins pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@jenkins pki]# ls
ca.csr  ca-csr.json  ca-key.pem  ca.pem

3、过期时间查看

[root@jenkins pki]# openssl x509 -noout -text -in ca.pem|grep -A 5 Validity
        Validity
            Not Before: Jun  4 12:32:00 2023 GMT
            Not After : May 11 12:32:00 2123 GMT
        Subject: C=CN, ST=Beijing, L=Beijing, O=Kubernetes, OU=Kubernetes-manual, CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption

4、创建Harbor证书

[root@jenkins pki]# cat > ca-config.json << EOF 
{
  "signing": {
    "default": {
      "expiry": "438000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "438000h"
      }
    }
  }
}
EOF

[root@jenkins pki]# cat > harbor-csr.json  << EOF 
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "10.10.10.10",  
    "harbor.wielun.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}
EOF

[root@jenkins pki]# cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=kubernetes harbor-csr.json | cfssljson -bare harbor

[root@jenkins pki]# mkdir -p /etc/harbor/pki
[root@jenkins pki]# cp harbor.pem harbor-key.pem /etc/harbor/pki

四、安装harbor

jenkins服务器上面安装

1、下载

下载地址：https://github.com/goharbor/harbor/releases#install
安装官网：https://goharbor.io/docs/2.8.0/install-config/

在这里插入图片描述

2、安装

（1）解压文件

[root@jenkins ~]#  tar xf harbor-offline-installer-v2.8.1.tgz -C /usr/local

（2）修改harbor.yml

[root@jenkins ~]# cd /usr/local/harbor/
[root@jenkins harbor]# cp harbor.yml.tmpl harbor.yml
[root@jenkins harbor]# vim harbor.yml

在这里插入图片描述
（3）启动

[root@jenkins harbor]# docker load -i harbor.v2.8.1.tar.gz
[root@jenkins harbor]# ./prepare
[root@jenkins harbor]# ./install.sh
[root@jenkins harbor]# docker-compose up -d     #手动启动命令

3、创建登录证书

[root@jenkins ~]# mkdir -p /etc/docker/certs.d/10.10.10.10
[root@jenkins ~]# mkdir -p /etc/docker/certs.d/harbor.wielun.com
[root@jenkins ~]# cp pki/ca.pem /etc/docker/certs.d/10.10.10.10
[root@jenkins ~]# cp pki/ca.pem /etc/docker/certs.d/harbor.wielun.com

4、修改daemon.json

[root@jenkins ~]# cat /etc/docker/daemon.json
{
 "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"],
  "insecure-registries": [
    "10.10.10.10",
    "harbor.wielun.com"
  ]
}

[root@jenkins ~]# systemctl restart docker

5、添加hosts

[root@jenkins ~]# vim /etc/hosts
10.10.10.10 harbor.wielun.com

6、登录验证

账号密码：admin/Harbor12345

（1）docker login

[root@jenkins ~]# docker login 10.10.10.10
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[root@jenkins ~]# docker login harbor.wielun.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

（2）浏览器登录

https://10.10.10.10/

在这里插入图片描述

7、docker上传镜像测试

（1）拉取镜像

[root@jenkins ~]# docker pull nginx
[root@jenkins ~]# docker images|grep nginx
nginx                           latest              f9c14fe76d50        10 days ago         143MB
goharbor/nginx-photon           v2.8.1              cea1bb2450ee        3 weeks ago         127MB

（2）打包上传

[root@jenkins ~]# docker tag nginx:latest harbor.wielun.com/library/nginx:latest
[root@jenkins ~]# docker push harbor.wielun.com/library/nginx

（3）浏览器中查看
在这里插入图片描述

五、K8s（containerd）拉取镜像（每台机器）

选择一种即可，这边我使用得是跳过证书

1、K8s（containerd）拉取镜像（跳过证书）

（1）删除之前containerd配置

[root@master01 ~]# rm -rf /etc/containerd/config.toml
[root@master01 ~]# containerd config default | tee /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml

（2）配置hosts.toml

[root@master01 ~]# mkdir -p /etc/containerd/certs.d/harbor.wielun.com
[root@master01 ~]# cat > /etc/containerd/certs.d/harbor.wielun.com/hosts.toml << EOF
server = "https://harbor.wielun.com"
[host."https://harbor.wielun.com"]
  capabilities = ["pull", "resolve", "push"]
  skip_verify = true
EOF

（3）重启containerd

[root@master01 ~]# systemctl restart containerd

（4）添加hosts

[root@master01 ~]# cat /etc/hosts
10.10.10.10 harbor.wielun.com

2、K8s（containerd）拉取镜像（通过证书）

（1）证书配置

[root@master01 ~]# mkdir -p /etc/containerd/certs.d/harbor.wielun.com

[root@jenkins ~]# cd pki/
[root@jenkins pki]# scp ca.pem harbor.pem harbor-key.pem [email protected]:/etc/containerd/certs.d/harbor.wielun.com

（2）删除之前containerd配置

[root@master01 ~]# rm -rf /etc/containerd/config.toml
[root@master01 ~]# containerd config default | tee /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml

（3）配置config.toml

[root@master01 ~]# vim /etc/containerd/config.toml
    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugin."io.containerd.grpc.v1.cri".registry.configs."harbor.wielun.com".tls]
          ca_file = "/etc/containerd/certs.d/harbor.wielun.com/ca.pem"
          cert_file = "/etc/containerd/certs.d/harbor.wielun.com/harbor.pem"
          key_file = "/etc/containerd/certs.d/harbor.wielun.com/harbor-key.pem"
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.wielun.com".auth]
          username = "admin"
          password = "Harbor12345"

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.wielun.com"]
          endpoint = ["https://harbor.wielun.com"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

在这里插入图片描述

（4）重启containerd

[root@master01 ~]# systemctl restart containerd

（5）添加hosts

[root@master01 ~]# cat /etc/hosts
10.10.10.10 harbor.wielun.com

3、测试拉取镜像

（1）拉取镜像

# -k：跳过证书认证
[root@master01 ~]# ctr -n harbor.wielun.com  images  pull harbor.wielun.com/library/nginx:latest -k
harbor.wielun.com/library/nginx:latest:                                           resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:831f51541d386c6d0d86f6799fcfabb48e91e9e5aea63c726240dd699179f495:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f03b40093957615593f2ed142961afb6b540507e0b47e3f7626ba5e02efbbbf1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eed12bbd64949353649476b59d486ab4c5b84fc5ed2b2dc96384b0b892b6bf7e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa7eb8c8eee8792b8db1c0043092b817376f096e3cc8feeea623c6e00211dad1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ff3b2b12318a41d4b238b643d7fcf1fe6da400ca3e02aa61766348f90455354:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f67c7de5f2c7e0dc408ce685285419c1295f24b7a01d554517c7a72374d4aeb:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6...

[root@master01 ~]# ctr -n harbor.wielun.com images pull harbor.wielun.com/library/nginx:latest --tlscacert  /etc/containerd/certs.d/harbor.wielun.com/ca.pem --tlscert   /etc/containerd/certs.d/harbor.wielun.com/harbor.pem  --tlskey  /etc/containerd/certs.d/harbor.wielun.com/harbor-key.pem
harbor.wielun.com/library/nginx:latest:                                           resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:831f51541d386c6d0d86f6799fcfabb48e91e9e5aea63c726240dd699179f495:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f03b40093957615593f2ed142961afb6b540507e0b47e3f7626ba5e02efbbbf1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eed12bbd64949353649476b59d486ab4c5b84fc5ed2b2dc96384b0b892b6bf7e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa7eb8c8eee8792b8db1c0043092b817376f096e3cc8feeea623c6e00211dad1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ff3b2b12318a41d4b238b643d7fcf1fe6da400ca3e02aa61766348f90455354:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f67c7de5f2c7e0dc408ce685285419c1295f24b7a01d554517c7a72374d4aeb:    done           |++++++++++++++++++++++++++++++++++++++|

（2）查看镜像

[root@master01 ~]# ctr -n harbor.wielun.com images ls
REF                                    TYPE                                                 DIGEST                                                                  SIZE     PLATFORMS   LABELS
harbor.wielun.com/library/nginx:latest application/vnd.docker.distribution.manifest.v2+json sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6 54.5 MiB linux/amd64 -

（3）crictl拉取镜像

[root@master01 ~]# crictl  pull harbor.wielun.com/library/nginx:latest
Image is up to date for sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda

[root@master01 ~]# crictl images ls|grep harbor.wielun.com/library/nginx
harbor.wielun.com/library/nginx                                                latest              f9c14fe76d502       57.2MB

六、jenkins发布到K8S

1、推送tomcat到harbor

这里用Java项目做演示，一般项目打包成jar包，我们这里打包成war包

[root@jenkins ~]# docker pull tomcat:8.5.59
[root@jenkins ~]# docker tag tomcat:8.5.59 harbor.wielun.com/library/tomcat:8.5.59
[root@jenkins ~]# docker push harbor.wielun.com/library/tomcat:8.5.59

2、创建项目

这里我们使用Jenkinsfile

在这里插入图片描述
在这里插入图片描述

3、查看项目文件

在这里插入图片描述
在这里插入图片描述

（1）配置Dockerfile

FROM harbor.wielun.com/library/tomcat:8.5.59
MAINTAINER Wielun
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war

（2）配置Jenkinsfile

pipeline {
    agent any
	environment {
		harborUser = 'admin'
		harborPasswd = 'Harbor12345'
		HarborAddress = 'harbor.wielun.com'
		harborRepo = 'library'
	}
    stages {
        stage('git拉取代码') {
            steps {
				git credentialsId: '0c71c0f9-8277-493b-xxxx-540a9324cf08', url: 'https://jihulab.com/xxxx/java-demo.git'
            }
        }
    
        stage('maven编译') { 
           steps {
                    sh '''JAVA_HOME=/usr/local/jdk
                    PATH=$PATH:$JAVA_HOME/bin
                    /usr/local/maven/bin/mvn clean package -Dmaven.test.skip=true'''
                }
        }
        stage('生成自定义镜像') { 
           steps {
                    sh '''docker build -t ${JOB_NAME}:latest .'''
                }
        }
        stage('上传自定义镜像到harbor') { 
           steps {
                    sh '''docker login -u ${harborUser} -p ${harborPasswd} ${HarborAddress}
                    docker tag ${JOB_NAME}:latest ${HarborAddress}/${harborRepo}/${JOB_NAME}:latest
                    docker push ${HarborAddress}/${harborRepo}/${JOB_NAME}:latest'''
                }
        }
        stage('发送yaml到k8s-master并部署') { 
           steps {
					sshPublisher(publishers: [sshPublisherDesc(configName: 'k8s-master', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: '''
					/usr/local/bin/kubectl apply -f /tmp/${JOB_NAME}/pipeline.yaml''', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '${JOB_NAME}', remoteDirectorySDF: false, removePrefix: '', sourceFiles: 'pipeline.yaml')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
                }
        } 
    }
}

（3）配置pipeline.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: test
  name: pipeline
  labels:
    app: pipeline
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pipeline
  template:
    metadata:
      labels:
        app: pipeline
    spec:
      containers:
      - name: pipeline
        image: harbor.wielun.com/library/java-k8s:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  namespace: test
  name: pipeline
  labels:
    app: pipeline
spec:
  ports:
  - port: 8081
    targetPort: 8080
  selector:
    app: pipeline
  type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: test
  name: pipeline
spec:
  ingressClassName: nginx
  rules:
  - host: "harbor.wielun.com"
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: pipeline
            port:
              number: 8081

4、创建namespace

[root@master01 ~]# kubectl create ns test

5、构建并查看结果

[root@master01 ~]# kubectl get pod -n test
NAME                        READY   STATUS    RESTARTS   AGE
pipeline-556759f7b4-7x8ml   1/1     Running   0          11s
pipeline-556759f7b4-zwdgr   1/1     Running   0          11s

下一篇文章> Jenkins集成SonarQube代码质量检测

猜你喜欢

转载自blog.csdn.net/Dream_ya/article/details/131036183
今日推荐
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
周排行
Metasploit文件目录与入侵基本概念
跨域(CORS)请求问题[No 'Access-Control-Allow-Origin' header is present on the requested resource]常见解决方案
CodeIgniter 源码解读之 CodeIgniter.php（二）
SAS入门之（四）改变数据类型
初识元组
[数学建模]数学建模算法和模型（B站视频）（二）
Nginx 服务器源码安装配置流程
C#实现语音视频录制 【基于MCapture + MFile】
开发进度4
下载安装vue的方法网址
每日归档
更多
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022