kubeadm-config 使用说明

版本列表

kubeadm-config 的版本列表

命令	版本
kubeadm v1.11	v1alpha2
kubeadm v1.12	v1alpha3
kubeadm v1.13 or v1.14	v1beta1
kubeadm v1.15	v1beta2

基础知识

kubeadm-config 支持以下配置类型

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration

init 和join 的默认配置可以使用以下命令打印出来

kubeadm config print init-defaults
kubeadm config print join-defaults

Kubeadm init configuration 配置说明

执行kubeadm init --config, 有以下类型可以配置

• InitConfiguration
• ClusterConfiguration
• KubeProxyConfiguration
• KubeletConfiguration

现在详细讲解配置文件

InitConfiguration

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
- token: "9a08jv.c0izixklcxtmnze7"           
  description: "kubeadm bootstrap token"          #初始化集群使用的token
  ttl: "24h"
- token: "783bde.3f89s0fje9f38fhf"
  description: "another bootstrap token"          #用于添加node的token
  usages:
  - authentication
  - signing
  groups:
  - system:bootstrappers:kubeadm:default-node-token
nodeRegistration:                                   
  criSocket: /var/run/dockershim.sock
  name: rancher.local
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
localAPIEndpoint:                                    # 指定master的监听ip和端口
  advertiseAddress: 1.2.3.4
  bindPort: 6443

• bootstrapTokens ：集群初始化需要的token 设置

• nodeRegistration ：包含集群节点的配置相关字段

• LocalAPIEndpoint ： API server部署在当前节点的监听地址和端口号

ClusterConfiguration

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
etcd:
  # 新建单个etcd或者连接外部etcd集群
  local:
    imageRepository: "k8s.gcr.io"
    imageTag: "3.2.24"
    dataDir: "/var/lib/etcd"
    extraArgs:
      listen-client-urls: "http://10.100.0.1:2379"
    serverCertSANs:
    -  "ec2-10-100-0-1.compute-1.amazonaws.com"
    peerCertSANs:
    - "10.100.0.1"
  # external:
    # endpoints:
    # - "10.100.0.1:2379"
    # - "10.100.0.2:2379"
    # caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
    # certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
    # keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
networking:
  # 设置集群网络
  serviceSubnet: "10.96.0.0/12"
  podSubnet: "10.100.0.1/24"
  dnsDomain: "cluster.local"
#集群版本号
kubernetesVersion: "v1.12.0"
#masterip和端口，这里也可以设置域名或者VIP
controlPlaneEndpoint: "10.100.0.1:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
  extraVolumes:
  - name: "some-volume"
    hostPath: "/etc/some-path"
    mountPath: "/etc/some-pod-path"
    readOnly: false
    pathType: File
  certSANs:
  # 设置证书，如果是多个master就把master的ip和主机名写入，还可以配置域名和VIP
  - "10.100.1.1"
  - "ec2-10-100-0-1.compute-1.amazonaws.com"
  timeoutForControlPlane: 4m0s
controllerManager:
  extraArgs:
    "node-cidr-mask-size": "20"
  extraVolumes:
  - name: "some-volume"
    hostPath: "/etc/some-path"
    mountPath: "/etc/some-pod-path"
    readOnly: false
    pathType: File
scheduler:
  extraArgs:
    address: "10.100.0.1"
  extraVolumes:
  - name: "some-volume"
    hostPath: "/etc/some-path"
    mountPath: "/etc/some-pod-path"
    readOnly: false
    pathType: File
# 指定证书存放路径
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"

• Networking ： 集群的网络配置，比如node和 service的子网
• Etcd configurations： etcd的相关配置，以及指定自建etcd 集群
• kube-apiserver, kube-scheduler, kube-controller-manager ： master组件的相关配置

KubeProxyConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
# 启用 ipvs 模式
mode: "ipvs"
ipvs:
  # 如果node提供lvs服务，排除以下CIDR 不受kube-proxy管理，避免刷掉lvs规则
  excludeCIDRs: [1.1.1.0/24,2.2.2.0/24]
  minSyncPeriod: 5s
  syncPeriod: 5s
  # ipvs 负载策略
  scheduler: "wrr"

更多的配置参数查看官方文档

https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or https://godoc.org/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration

KubeletConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
#禁用swap检测
cgroupDriver: systemd
#修改driver为systemd
rotateCertificates: true
# 开启证书轮询

更多的配置参数查看官方文档

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration

关于这个配置文件的文档还是很不完善，对于不懂 golang 的人来说很难知道具体怎么配置，以下做一下简要说明(请确保你已经拉取了 kubernetes 源码

kubeadm 配置中每个配置段都会有个 kind 字段，kind 实际上对应了 go 代码中的 struct 结构体；同时从 apiVersion 字段中能够看到具体的版本，比如 v1alpha1 等；有了这两个信息事实上你就可以直接在源码中去找到对应的结构体

关于数据类型，如果是 string 的类型，那么意味着你要在 yaml 里写 "xxxx" 带引号这种，当然有些时候不写能兼容，有些时候不行比如 extraArgs 字段是一个 map[string]string 如果 value 不带引号就报错；如果数据类型为 metav1.Duration(实际上就是 time.Duration)，那么你看着它是个 int64 但实际上你要写 1h2m3s 这种人类可读的格式

init 完整示例

本示例基于1.17,基本上所有能配的东西都写里面了。请结合你的实际情况进行取舍。

apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
bootstrapTokens:
- token: "9a08jv.c0izixklcxtmnze7"
  description: "kubeadm bootstrap token"
  ttl: "24h"
- token: "783bde.3f89s0fje9f38fhf"
  description: "another bootstrap token"
  usages:
  - authentication
  - signing
  groups:
  - system:bootstrappers:kubeadm:default-node-token
nodeRegistration:
  taints:
  - key: "kubeadmNode"
    value: "master"
    effect: "NoSchedule"
  kubeletExtraArgs:
    cgroup-driver: "systemd"
certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
etcd:
  # one of local or external
  local:
    imageRepository: "k8s.gcr.io"
    imageTag: "3.2.24"
    dataDir: "/var/lib/etcd"
    extraArgs:
      listen-client-urls: "http://10.100.0.1:2379"
    serverCertSANs:
    -  "ec2-10-100-0-1.compute-1.amazonaws.com"
    peerCertSANs:
    - "10.100.0.1"
  # external:
    # endpoints:
    # - "10.100.0.1:2379"
    # - "10.100.0.2:2379"
    # caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
    # certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
    # keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
networking:
  serviceSubnet: "10.96.0.0/12"
  podSubnet: "10.100.0.1/24"
  dnsDomain: "cluster.local"
kubernetesVersion: "v1.17.0"
controlPlaneEndpoint: " k8s.foxchan.com:8443"
apiServer:
  extraArgs:
    v: "2"
    logtostderr: "false"
    log-dir: "/var/log/kubernetes"
    # 审计日志相关配置
    audit-log-maxage: "20"
    audit-log-maxbackup: "10"
    audit-log-maxsize: "100"
    audit-log-path: "/var/log/kube-audit/audit.log"
    audit-policy-file: "/etc/kubernetes/audit-policy.yaml"
    authorization-mode: "Node,RBAC"
    event-ttl: "720h"
    runtime-config: "api/all=true"
    service-node-port-range: "30000-50000"
    service-cluster-ip-range: "10.25.0.0/16"    
  extraVolumes:
  - name: "k8s-log"
    hostPath: "/var/log/kubernetes"
    mountPath: "/var/log/kubernetes"
    pathType: "DirectoryOrCreate"
  - name: "audit-config"
    hostPath: "/etc/kubernetes/audit-policy.yaml"
    mountPath: "/etc/kubernetes/audit-policy.yaml"
    readOnly: true
    pathType: "File"
  - name: "audit-log"
    hostPath: "/var/log/kube-audit"
    mountPath: "/var/log/kube-audit"
    pathType: "DirectoryOrCreate"  
  - name: "timezone"
    hostPath: "/etc/localtime"
    mountPath: "/etc/localtime"
    readOnly: true
    pathType: "File"
  certSANs:
  - "10.100.1.1"
  - "10.100.1.2"
  - "k8s.foxchan.com"
  timeoutForControlPlane: 4m0s
controllerManager:
  extraArgs:
    "node-cidr-mask-size": "20"
  extraVolumes:
  - name: "k8s-log"
    hostPath: "/var/log/kubernetes"
    mountPath: "/var/log/kubernetes"
    pathType: "DirectoryOrCreate"
  - name: "timezone"
    hostPath: "/etc/localtime"
    mountPath: "/etc/localtime"
    readOnly: true
    pathType: "File"
scheduler:
  extraArgs:
    address: "0.0.0.0"
    bind-address: "0.0.0.0"
  extraVolumes:
  - name: "k8s-log"
    hostPath: "/var/log/kubernetes"
    mountPath: "/var/log/kubernetes"
    pathType: "DirectoryOrCreate"
  - name: "timezone"
    hostPath: "/etc/localtime"
    mountPath: "/etc/localtime"
    readOnly: true
    pathType: "File"
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"
clusterName: "example-cluster"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# kubelet specific options here
# 此配置保证了 kubelet 能在 swap 开启的情况下启动
failSwapOn: false
nodeStatusUpdateFrequency: 5s
# 一些驱逐阀值，具体自行查文档修改
evictionSoft:
  "imagefs.available": "15%"
  "memory.available": "512Mi"
  "nodefs.available": "15%"
  "nodefs.inodesFree": "10%"
evictionSoftGracePeriod:
  "imagefs.available": "3m"
  "memory.available": "1m"
  "nodefs.available": "3m"
  "nodefs.inodesFree": "1m"
evictionHard:
  "imagefs.available": "10%"
  "memory.available": "256Mi"
  "nodefs.available": "10%"
  "nodefs.inodesFree": "5%"
evictionMaxPodGracePeriod: 30
imageGCLowThresholdPercent: 70
imageGCHighThresholdPercent: 80
kubeReserved:
  "cpu": "500m"
  "memory": "512Mi"
  "ephemeral-storage": "1Gi"
rotateCertificates: true
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
# kube-proxy specific options here
clusterCIDR: "10.244.0.0/16"
# 启用 ipvs 模式
mode: "ipvs"
ipvs:
  # 如果node提供lvs服务，排除以下CIDR 不受kube-proxy管理，避免kube-proxy刷掉lvs规则
  excludeCIDRs: [1.1.1.0/24,2.2.2.0/24]
  minSyncPeriod: 5s
  syncPeriod: 5s
  # ipvs 负载策略
  scheduler: "wrr"

Kubeadm join configuration 配置说明

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
   ...

• NodeRegistration ：同上，node的相关配置
• APIEndpoint ： 指定要加入的集群 apiserver

join 完整示例

apiVersion: kubeadm.k8s.io/v1beta2
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: k8sgs.foxchan.com:8443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: 1.1.1.1
  taints: null