gitlabrunner使用k8s executor
相关的文章有:gitlab-runner使用docker executor
两者主要区别在于,一个部署在docker服务器上,一个部署在k8s的pod中。
所需镜像
基础镜像
- gitlab-runner:latest
- gitlab-runner打包成的镜像,需要在k8s上长期驻留
- gitlab-runner-helper:x86_64-f100a208
- 这个是gitlab-runner用来启动pod的,他的版本由gitlab-runner版本和服务器版本决定
- maven:3-jdk-8
- 如果处理的是java代码,那此镜像既可以做为编译的镜像,也可以作为发布时的基础镜像。
- busybox:latest
- 默认镜像,如果不指定runner启动哪个镜像,就选择busybox
- docker:dind
- 用于编译业务镜像,基于maven:3-jdk-8
- kubectl
- 将业务镜像部署到k8s上
第一步:在K8S中部署gitlab-runner
在gitlab上找到url和token
- 打开你想要配置CICD的项目或项目组
- runner配置地址:
settings->CI/CD->Runners->expand
- 找到设置runner需要的url和token,拿小本本记下来
创建一个可用的runner
- 找一台安装了docker的机器,使用以下命令生成toml:
docker run -it --name runner gitlab/gitlab-runner:v11.10.0 register
- 在
settings->CI/CD->Runners->expand
下找到新增的runner,获得新tokenkg8sKRsXs9kYWsMp5PJ7
,记在小本本上。而上一步启动的runner已经不再需要了。
创建k8s上的gitlab-runner的配置文件
注意:这里的token使用我们新获得的token。
kind: ConfigMap
apiVersion: v1
metadata:
name: gitlab-runner
namespace: gitlab-runner
data:
config.toml: |
concurrent = 4
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "first test"
url = "http://${
gitlabIP}"
token ="......"
executor = "kubernetes"
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = true
privileged = true
#这是runner的默认镜像;具体镜像maven:3-jdk-8在.gitlab-ci.yml中配置
image = "${
registryIP}/gitlab-runner/busybox:latest"
namespace = "gitlab-runner"
namespace_overwrite_allowed = ""
cpu_limit = "1"
memory_limit = "2Gi"
cpu_request = "1"
memory_request = "2Gi"
service_cpu_limit = "1"
service_memory_limit = "2Gi"
service_cpu_request = "1"
service_memory_request = "2Gi"
helper_cpu_limit = "1"
helper_memory_limit = "2Gi"
helper_cpu_request = "1"
helper_memory_request = "2Gi"
service_account = "gitlab"
helper_image = "${
registryIP}/gitlab-runner/gitlab-runner-helper:x86_64-f100a208"
[[runners.kubernetes.volumes.config_map]]
name = "gitlab-runner-maven"
mount_path = "/usr/share/maven/configmap/"
创建configmap:gitlab-runner-maven,保存maven的setting文件
kind: ConfigMap
apiVersion: v1
metadata:
name: gitlab-runner-maven
namespace: gitlab-runner
data:
settings.xml: "此处是maven的settings.xml的内容"
创建serviceaccount.yml 实现权限,否则提示无法在namespace中创建pod
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab
namespace: gitlab-runner
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab
namespace: gitlab-runner
rules:
- apiGroups: [""]
resources: ["pods","pods/exec"]
verbs: ["get","list","watch","create","update","patch","delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab
namespace: gitlab-runner
subjects:
- kind: ServiceAccount
name: gitlab
apiGroup: ""
roleRef:
kind: Role
name: gitlab
apiGroup: rbac.authorization.k8s.io
通过deployment.yml部署gitlab-runner
- 注意写入serviceaccountname gitlab
- 注意导入configmap,即gitlab-runner的/etc/gitlab-runner/config.toml
- configmap通过
[[runners.kubernetes.volumes.config_map]]
,将maven的settings挂载到了maven:3-jdk-8容器 - 注意config.toml中token和cicd/runner的token并不一样
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gitlab-runner
namespace: gitlab-runner
spec:
replicas: 1
selector:
matchLabels:
name: gitlab-runner
template:
metadata:
labels:
name: gitlab-runner
spec:
serviceAccountName: gitlab
containers:
- args:
- '--debug'
- run
image: ${
registryIP}/gitlab-runner/gitlab-runner:latest
imagePullPolicy: IfNotPresent
name: gitlab-runner
volumeMounts:
- mountPath: /etc/gitlab-runner
name: config
- mountPath: /etc/ssl/certs
name: cacerts
readOnly: true
resources:
limits:
cpu: "2"
memory: "4Gi"
requests:
cpu: "2"
memory: "4Gi"
restartPolicy: Always
volumes:
- configMap:
name: gitlab-runner
name: config
- hostPath:
path: /usr/share/ca-certificates/mozilla
name: cacerts
第二步:配置.gitlab-ci.yml
- 选择maven模板
- 在 MVAEN_CLI_OPTS 中加入"–settings /usr/share/maven/configmap/settings.xml"
- 简单示例,打JAR包、打Docker镜像、发布到k8s:
variables:
MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true --settings /usr/share/maven/configmap/settings.xml"
## 编译并打包
compile:
stage: build
script:
- 'mvn $MAVEN_CLI_OPTS package'
image: ${
registryIP}/gitlab-runner/maven:3-jdk-8
artifacts:
name: "jars"
expire_in: 60 mins
paths:
- target/*.jar
when: on_success
## 拿到jar包,并编译docker镜像,这里使用一些伪脚本
build_docker_image:
stage: build_docker_image
image: ${
registryIP}/gitlab-runner/docker:1.12.6
services:
- name: ${
registryIP}/gitlab-runner/docker:1.12.6-dind
command: ["--insecure-registry=${registryIP}"]
dependencies:
- compile
variables:
GIT_STRATEGY: none
DOCKER_HOST: "tcp:127.0.0.1:2375"
script:
- 'docker build -t mybiz-${branch}:${CI_PIPELINE_ID} .'
- 'docker login xxxx '
- 'docker push xxxx '
## 部署到k8s,这里使用一些伪脚本
deploy_k8s:
stage: deploy_k8s
image: ${
registryIP}/gitlab-runner/kubectl:latest
script:
- 'create deployment.yml'
- 'kubectl apply -f deployment.yml'
only:
refs:
- tags
variables:
- ${
CI_PROJECT_NAMESPACE} =~ /mygroup/
variables:
GIT_STRATEGY: none
dependencies: []
第三步:测试
打开runner配置地址,可以看到刚才创建的runner,以及连接状态,修改为无需指定tag。
- 如果你的commit信息中包含 [ci skip] 或者 [skip ci] ,不论大小写,将会跳过jobs。
- gitlab -> 项目(组)xxxxx -> CI/CD -> pipelines
- 直接create pipeline就可以了
- 提示"exceeded quota",需要在config.toml中设定limit、request等参数
- 提示"pod limits=0",serviceaccount没设置好
第四步:Auto-Devops
- 修改gitlab-CI模板
- 找到模板目录,进行相应的修改。
- 建议修改所有的模板,并在通用模板中提示用户改变模板。不同gitlab版本,地址略有差异:
vi /opt/gitlab/embedded/service/gitlab-rails/vendor/gitlab-ci-yml/Auto-DevOps.gitlab-ci.yml
。
vi /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml
第五步:优化
上面只是比较简单的一个流程,后面还会把一些代码质量、单元测试、自动测试、甚至可以把其他的一些第三方平台也集成进来,更可以做成一个管理平台,将ci.yml都管理起来。