上一篇文章> Démarrer et installer Jenkins
1. Environnement expérimental
1. Environnement K8s
Version
v1.26.5, le conteneur est
un cluster d'installation binaire Containerd Kubernetes (K8s) (basé sur ContainerD) - tutoriel d'installation à partir de zéro (avec certificat)
| Nom du processeur | PI | version du système | service d'installation |
|---|---|---|---|
| maître01 | 10.10.10.21 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| maître02 | 10.10.10.22 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| maître03 | 10.10.10.23 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| noeud01 | 10.10.10.24 | rhel7.5 | nginx、kubelet、proxy |
| noeud02 | 10.10.10.25 | rhel7.5 | nginx、kubelet、proxy |
2. environnement Jenkins
Démarrer avec Jenkins et installer
des conteneurs en tant que Docker
| Hôte | PI | version du système |
|---|---|---|
| jenkins | 10.10.10.10 | rhel7.5 |
2. Installation de Docker-composer
Installé sur le serveur jenkins
1. Télécharger
https://github.com/docker/compose/releases/Version
téléchargée : v2.18.0
2.Installation
[root@jenkins ~]# cp docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@jenkins ~]# chmod +x /usr/local/bin/docker-compose
3. Vérifiez la version
[root@jenkins ~]# docker-compose --version
Docker Compose version v2.18.0
3. Génération de certificat cfssl
Il est enregistré ici que l'outil cfssl est utilisé pour générer un certificat privé du port, et le certificat est utilisé pour construire l'entrepôt Harbour. Ce certificat est installé à l'aide du certificat ca utilisé dans Kubernetes.
1. Installez cfssl
https://imroc.cc/kubernetes/trick/certs/sign-certs-with-cfssl.html
Adresse de téléchargement du package d'installation : https://github.com/cloudflare/cfssl/releases
[root@jenkins ~]# ls cfssl*
cfssl_1.6.2_linux_amd64 cfssl-certinfo_1.6.2_linux_amd64 cfssljson_1.6.2_linux_amd64
[root@jenkins ~]# mv cfssl_1.6.2_linux_amd64 /usr/bin/cfssl
[root@jenkins ~]# mv cfssl-certinfo_1.6.2_linux_amd64 /usr/bin/cfssl-certinfo
[root@jenkins ~]# mv cfssljson_1.6.2_linux_amd64 /usr/bin/cfssljson
[root@jenkins ~]# chmod +x /usr/bin/cfssl*
2. CA génère un certificat
[root@jenkins ~]# mkdir -p pki && cd pki
[root@jenkins pki]# cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "Kubernetes",
"OU": "Kubernetes-manual"
}
],
"ca": {
"expiry": "876000h"
}
}
EOF
[root@jenkins pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@jenkins pki]# ls
ca.csr ca-csr.json ca-key.pem ca.pem
3. Vérifiez la date d'expiration
[root@jenkins pki]# openssl x509 -noout -text -in ca.pem|grep -A 5 Validity
Validity
Not Before: Jun 4 12:32:00 2023 GMT
Not After : May 11 12:32:00 2123 GMT
Subject: C=CN, ST=Beijing, L=Beijing, O=Kubernetes, OU=Kubernetes-manual, CN=kubernetes
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
4. Créer un certificat Harbour
[root@jenkins pki]# cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "438000h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "438000h"
}
}
}
}
EOF
[root@jenkins pki]# cat > harbor-csr.json << EOF
{
"CN": "harbor",
"hosts": [
"127.0.0.1",
"10.10.10.10",
"harbor.wielun.com"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "Kubernetes",
"OU": "Kubernetes-manual"
}
]
}
EOF
[root@jenkins pki]# cfssl gencert \
-ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=kubernetes harbor-csr.json | cfssljson -bare harbor
[root@jenkins pki]# mkdir -p /etc/harbor/pki
[root@jenkins pki]# cp harbor.pem harbor-key.pem /etc/harbor/pki
4. Installer le port
Installé sur le serveur jenkins
1. Télécharger
Adresse de téléchargement : https://github.com/goharbor/harbor/releases#install
Site officiel d'installation : https://goharbor.io/docs/2.8.0/install-config/
2.Installation
(1) Décompressez le fichier
[root@jenkins ~]# tar xf harbor-offline-installer-v2.8.1.tgz -C /usr/local
(2) Modifier harbour.yml
[root@jenkins ~]# cd /usr/local/harbor/
[root@jenkins harbor]# cp harbor.yml.tmpl harbor.yml
[root@jenkins harbor]# vim harbor.yml
(3) Démarrer
[root@jenkins harbor]# docker load -i harbor.v2.8.1.tar.gz
[root@jenkins harbor]# ./prepare
[root@jenkins harbor]# ./install.sh
[root@jenkins harbor]# docker-compose up -d #手动启动命令
3. Créer un certificat de connexion
[root@jenkins ~]# mkdir -p /etc/docker/certs.d/10.10.10.10
[root@jenkins ~]# mkdir -p /etc/docker/certs.d/harbor.wielun.com
[root@jenkins ~]# cp pki/ca.pem /etc/docker/certs.d/10.10.10.10
[root@jenkins ~]# cp pki/ca.pem /etc/docker/certs.d/harbor.wielun.com
4. Modifier démon.json
[root@jenkins ~]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"],
"insecure-registries": [
"10.10.10.10",
"harbor.wielun.com"
]
}
[root@jenkins ~]# systemctl restart docker
5. Ajouter des hôtes
[root@jenkins ~]# vim /etc/hosts
10.10.10.10 harbor.wielun.com
6. Vérification de connexion
Mot de passe du compte : admin/Harbor12345
（1）Connexion Docker
[root@jenkins ~]# docker login 10.10.10.10
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@jenkins ~]# docker login harbor.wielun.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
(2) Connexion au navigateur
https://10.10.10.10/
7. Test d'image de téléchargement Docker
(1) Tirez l'image
[root@jenkins ~]# docker pull nginx
[root@jenkins ~]# docker images|grep nginx
nginx latest f9c14fe76d50 10 days ago 143MB
goharbor/nginx-photon v2.8.1 cea1bb2450ee 3 weeks ago 127MB
(2) Package et téléchargement
[root@jenkins ~]# docker tag nginx:latest harbor.wielun.com/library/nginx:latest
[root@jenkins ~]# docker push harbor.wielun.com/library/nginx
(3) Afficher dans le navigateur
5. K8s (containerd) extrait l'image (chaque machine)
Choisissez-en un, ici j'utilise le certificat de saut.
1. K8s (containerd) extrait l'image (en sautant le certificat)
(1) Supprimer la configuration précédente du conteneur
[root@master01 ~]# rm -rf /etc/containerd/config.toml
[root@master01 ~]# containerd config default | tee /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
(2) Configurer hosts.toml
[root@master01 ~]# mkdir -p /etc/containerd/certs.d/harbor.wielun.com
[root@master01 ~]# cat > /etc/containerd/certs.d/harbor.wielun.com/hosts.toml << EOF
server = "https://harbor.wielun.com"
[host."https://harbor.wielun.com"]
capabilities = ["pull", "resolve", "push"]
skip_verify = true
EOF
(3) Redémarrer le conteneur
[root@master01 ~]# systemctl restart containerd
(4) Ajouter des hôtes
[root@master01 ~]# cat /etc/hosts
10.10.10.10 harbor.wielun.com
2. K8s (containerd) extrait l'image (via un certificat)
(1) Configuration du certificat
[root@master01 ~]# mkdir -p /etc/containerd/certs.d/harbor.wielun.com
[root@jenkins ~]# cd pki/
[root@jenkins pki]# scp ca.pem harbor.pem harbor-key.pem [email protected]:/etc/containerd/certs.d/harbor.wielun.com
(2) Supprimer la configuration précédente du conteneur
[root@master01 ~]# rm -rf /etc/containerd/config.toml
[root@master01 ~]# containerd config default | tee /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
[root@master01 ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml
(3) Configurer config.toml
[root@master01 ~]# vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugin."io.containerd.grpc.v1.cri".registry.configs."harbor.wielun.com".tls]
ca_file = "/etc/containerd/certs.d/harbor.wielun.com/ca.pem"
cert_file = "/etc/containerd/certs.d/harbor.wielun.com/harbor.pem"
key_file = "/etc/containerd/certs.d/harbor.wielun.com/harbor-key.pem"
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.wielun.com".auth]
username = "admin"
password = "Harbor12345"
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.wielun.com"]
endpoint = ["https://harbor.wielun.com"]
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
(4) Redémarrer le conteneur
[root@master01 ~]# systemctl restart containerd
(5) Ajouter des hôtes
[root@master01 ~]# cat /etc/hosts
10.10.10.10 harbor.wielun.com
3. Testez l'extraction de l'image
(1) Tirez l'image
# -k：跳过证书认证
[root@master01 ~]# ctr -n harbor.wielun.com images pull harbor.wielun.com/library/nginx:latest -k
harbor.wielun.com/library/nginx:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:831f51541d386c6d0d86f6799fcfabb48e91e9e5aea63c726240dd699179f495: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f03b40093957615593f2ed142961afb6b540507e0b47e3f7626ba5e02efbbbf1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eed12bbd64949353649476b59d486ab4c5b84fc5ed2b2dc96384b0b892b6bf7e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa7eb8c8eee8792b8db1c0043092b817376f096e3cc8feeea623c6e00211dad1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ff3b2b12318a41d4b238b643d7fcf1fe6da400ca3e02aa61766348f90455354: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f67c7de5f2c7e0dc408ce685285419c1295f24b7a01d554517c7a72374d4aeb: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6...
[root@master01 ~]# ctr -n harbor.wielun.com images pull harbor.wielun.com/library/nginx:latest --tlscacert /etc/containerd/certs.d/harbor.wielun.com/ca.pem --tlscert /etc/containerd/certs.d/harbor.wielun.com/harbor.pem --tlskey /etc/containerd/certs.d/harbor.wielun.com/harbor-key.pem
harbor.wielun.com/library/nginx:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:831f51541d386c6d0d86f6799fcfabb48e91e9e5aea63c726240dd699179f495: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f03b40093957615593f2ed142961afb6b540507e0b47e3f7626ba5e02efbbbf1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:eed12bbd64949353649476b59d486ab4c5b84fc5ed2b2dc96384b0b892b6bf7e: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fa7eb8c8eee8792b8db1c0043092b817376f096e3cc8feeea623c6e00211dad1: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ff3b2b12318a41d4b238b643d7fcf1fe6da400ca3e02aa61766348f90455354: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f67c7de5f2c7e0dc408ce685285419c1295f24b7a01d554517c7a72374d4aeb: done |++++++++++++++++++++++++++++++++++++++|
(2) Voir l'image
[root@master01 ~]# ctr -n harbor.wielun.com images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
harbor.wielun.com/library/nginx:latest application/vnd.docker.distribution.manifest.v2+json sha256:6b06964cdbbc517102ce5e0cef95152f3c6a7ef703e4057cb574539de91f72e6 54.5 MiB linux/amd64 -
(3) image d'extraction critique
[root@master01 ~]# crictl pull harbor.wielun.com/library/nginx:latest
Image is up to date for sha256:f9c14fe76d502861ba0939bc3189e642c02e257f06f4c0214b1f8ca329326cda
[root@master01 ~]# crictl images ls|grep harbor.wielun.com/library/nginx
harbor.wielun.com/library/nginx latest f9c14fe76d502 57.2MB
6. Publier Jenkins sur K8S
1. Poussez Tomcat vers le port
Un projet Java est utilisé pour la démonstration ici. Généralement, le projet est emballé dans un package jar. Ici, nous l'emballons dans un package war.
[root@jenkins ~]# docker pull tomcat:8.5.59
[root@jenkins ~]# docker tag tomcat:8.5.59 harbor.wielun.com/library/tomcat:8.5.59
[root@jenkins ~]# docker push harbor.wielun.com/library/tomcat:8.5.59
2. Créez un projet
Ici, nous utilisons Jenkinsfile
3. Afficher les fichiers du projet
(1) Configurer le fichier Docker
FROM harbor.wielun.com/library/tomcat:8.5.59
MAINTAINER Wielun
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
(2) Configurer le fichier Jenkins
pipeline {
agent any
environment {
harborUser = 'admin'
harborPasswd = 'Harbor12345'
HarborAddress = 'harbor.wielun.com'
harborRepo = 'library'
}
stages {
stage('git拉取代码') {
steps {
git credentialsId: '0c71c0f9-8277-493b-xxxx-540a9324cf08', url: 'https://jihulab.com/xxxx/java-demo.git'
}
}
stage('maven编译') {
steps {
sh '''JAVA_HOME=/usr/local/jdk
PATH=$PATH:$JAVA_HOME/bin
/usr/local/maven/bin/mvn clean package -Dmaven.test.skip=true'''
}
}
stage('生成自定义镜像') {
steps {
sh '''docker build -t ${JOB_NAME}:latest .'''
}
}
stage('上传自定义镜像到harbor') {
steps {
sh '''docker login -u ${harborUser} -p ${harborPasswd} ${HarborAddress}
docker tag ${JOB_NAME}:latest ${HarborAddress}/${harborRepo}/${JOB_NAME}:latest
docker push ${HarborAddress}/${harborRepo}/${JOB_NAME}:latest'''
}
}
stage('发送yaml到k8s-master并部署') {
steps {
sshPublisher(publishers: [sshPublisherDesc(configName: 'k8s-master', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: '''
/usr/local/bin/kubectl apply -f /tmp/${JOB_NAME}/pipeline.yaml''', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '${JOB_NAME}', remoteDirectorySDF: false, removePrefix: '', sourceFiles: 'pipeline.yaml')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
}
}
}
}
(3) Configurer pipeline.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: test
name: pipeline
labels:
app: pipeline
spec:
replicas: 2
selector:
matchLabels:
app: pipeline
template:
metadata:
labels:
app: pipeline
spec:
containers:
- name: pipeline
image: harbor.wielun.com/library/java-k8s:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
namespace: test
name: pipeline
labels:
app: pipeline
spec:
ports:
- port: 8081
targetPort: 8080
selector:
app: pipeline
type: NodePort
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: test
name: pipeline
spec:
ingressClassName: nginx
rules:
- host: "harbor.wielun.com"
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: pipeline
port:
number: 8081
4. Créer un espace de noms
[root@master01 ~]# kubectl create ns test
5. Créez et affichez les résultats
[root@master01 ~]# kubectl get pod -n test
NAME READY STATUS RESTARTS AGE
pipeline-556759f7b4-7x8ml 1/1 Running 0 11s
pipeline-556759f7b4-zwdgr 1/1 Running 0 11s
下一篇文章> Jenkins intègre la détection de la qualité du code SonarQube