suse 12 二进制部署 Kubernetets 1.19.7 - 第06章 - 部署kube-apiserver组件

1.6、部署kube-apiserver

• 所有master节点需要kube-apiserver
• kube-apiserver是无状态服务，需要通过kube-nginx进行代理访问，从而保证服务可用性
• 部署kubectl的时候已经下载了完整的kubernetes二进制文件，因此kube-apiserver就无须下载了，等下脚本分发即可

1.6.0、创建kubernetes证书和私钥

k8s-01:~ # cd /opt/k8s/ssl/
k8s-01:/opt/k8s/ssl # source /opt/k8s/bin/k8s-env.sh
k8s-01:/opt/k8s/ssl # cat > kubernetes-csr.json <<EOF
{
    
    
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.72.39",
    "192.168.72.40",
    "192.168.72.41",
    "192.168.72.42",
    "192.168.72.43",
    "${CLUSTER_KUBERNETES_SVC_IP}",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    
    
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
    
    
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "k8s",
      "OU": "bandian"
    }
  ]
}
EOF

• 需要将集群的所有IP添加到证书内

1.6.1、生成kubernetes证书和私钥

k8s-01:/opt/k8s/ssl # cfssl gencert -ca=/opt/k8s/ssl/ca.pem \
-ca-key=/opt/k8s/ssl/ca-key.pem \
-config=/opt/k8s/ssl/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

1.6.2、创建metrics-server证书和私钥

k8s-01:/opt/k8s/ssl # cat > metrics-server-csr.json <<EOF
{
    
    
  "CN": "aggregator",
  "hosts": [
  ],
  "key": {
    
    
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
    
    
      "C": "CN",
      "ST": "ShangHai",
      "L": "ShangHai",
      "O": "k8s",
      "OU": "bandian"
    }
  ]
}
EOF

1.6.3、生成metrics-server证书和私钥

k8s-01:/opt/k8s/ssl # cfssl gencert -ca=/opt/k8s/ssl/ca.pem \
-ca-key=/opt/k8s/ssl/ca-key.pem \
-config=/opt/k8s/ssl/ca-config.json \
-profile=kubernetes metrics-server-csr.json | cfssljson -bare metrics-server

1.6.4、配置kube-apiserver为systemctl管理

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # source /opt/k8s/bin/k8s-env.sh
k8s-01:/opt/k8s/conf # cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\
  --v=2 \\
  --advertise-address=##NODE_IP## \\
  --secure-port=6443 \\
  --bind-address=##NODE_IP## \\
  --etcd-servers=${ETCD_ENDPOINTS} \\
  --allow-privileged=true \\
  --service-cluster-ip-range=${SERVICE_CIDR} \\
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
  --authorization-mode=RBAC,Node \\
  --enable-bootstrap-token-auth=true \\
  --token-auth-file=/etc/kubernetes/cert/token.csv \\
  --service-node-port-range=${NODE_PORT_RANGE} \\
  --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\
  --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\
  --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\
  --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\
  --client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --service-account-key-file=/etc/kubernetes/cert/ca.pem \\
  --etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\
  --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\
  --audit-log-maxage=15 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-truncate-enabled \\
  --audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\
  --proxy-client-cert-file=/etc/kubernetes/cert/metrics-server.pem \\
  --proxy-client-key-file=/etc/kubernetes/cert/metrics-server-key.pem \\
  --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\
  --requestheader-allowed-names=aggregator \\
  --requestheader-extra-headers-prefix="X-Remote-Extra-" \\
  --requestheader-group-headers=X-Remote-Group \\
  --requestheader-username-headers=X-Remote-User

Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

• --v 日志等级
• --etcd-servers etcd集群地址
• --bind-address 监听地址
• --secure-port https安全端口
• --advertise-address 集群通告地址
• --allow-privileged 启用授权
• --service-cluster-ip-range Service虚拟IP地址段
• --enable-admission-plugins 准入控制模块
• --authorization-mode 认证授权，启用RBAC授权和节点自管理
• --enable-bootstrap-token-auth 启用TLS bootstrap机制
• --token-auth-file bootstrap token文件
• --service-node-port-range Service nodeport类型默认分配端口范围
• --kubelet-client-xxx apiserver访问kubelet客户端证书
• --tls-xxx-file apiserver https证书
• --etcd-xxxfile 连接Etcd集群证书 --audit-log-xxx:审计日志
• --requestheader-xxx-xxx 开启kube-apiserver的aggregation（hpa和metrics依赖aggregation）
• --proxy-client-xxx 同上

1.6.5、配置bootstrap token文件

k8s-01:~ # cd /opt/k8s/ssl/
k8s-01:/opt/k8s/ssl # cat > token.csv <<EOF
404a083c42f5d39979fd731a24774b83,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

• bootstrap token文件格式
  • token，用户名，UID，用户组
• token生成方式
  • head -c 16 /dev/urandom | od -An -t x | tr -d ' '

1.6.6、分发kube-apiserver命令和秘钥等文件到其他节点

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

# 替换模板文件
for (( i=0; i < 3; i++ ))
do
    sed -e "s/##NODE_IP##/${MASTER_IPS[i]}/" /opt/k8s/conf/kube-apiserver.service.template > \
           /opt/k8s/conf/kube-apiserver-${MASTER_IPS[i]}.service 
done

# 分发到master节点
for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
	scp /opt/k8s/packages/kubernetes/server/bin/{
    
    apiextensions-apiserver,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubelet,mounter} ${host}:/opt/k8s/bin/
	scp /opt/k8s/ssl/{
    
    kubernetes*.pem,token.csv} ${host}:/etc/kubernetes/cert/
	scp /opt/k8s/ssl/metrics-server*.pem ${host}:/etc/kubernetes/cert/
	scp /opt/k8s/conf/kube-apiserver-${host}.service ${host}:/etc/systemd/system/kube-apiserver.service
done

# 分发到所有节点
for host_node in ${NODE_IPS[@]}
do
    printf "\e[1;34m${host_node}\e[0m\n"
	scp /opt/k8s/packages/kubernetes/server/bin/{
    
    kubelet,kube-proxy} ${host_node}:/opt/k8s/bin/
done

1.6.7、启动kube-apiserver服务

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "mkdir -p ${K8S_DIR}/kube-apiserver/"
    ssh root@${host} "systemctl daemon-reload && \
                      systemctl enable kube-apiserver --now && \
                      systemctl status kube-apiserver | grep Active"
done

• 注：返回的如果是Active: activating (auto-restart)，可以稍等一下，然后再次执行systemctl status kube-apiserver | grep Active，出现running就可以了，否则的话，需要查看日志journalctl -xeu kube-apiserver

1.6.8、查看kube-apiserver写入etcd的数据

k8s-01:~ # source /opt/k8s/bin/k8s-env.sh
k8s-01:~ # etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--cacert=/opt/k8s/ssl/ca.pem \
--cert=/opt/k8s/ssl/etcd.pem \
--key=/opt/k8s/ssl/etcd-key.pem \
get /registry/ --prefix --keys-only

1.6.9、检查kubernetes集群信息

k8s-01:~ # kubectl cluster-info
Kubernetes master is running at https://192.168.72.39:8443

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

k8s-01:~ # kubectl get all --all-namespaces
NAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.254.0.1   <none>        443/TCP   38s

k8s-01:~ # kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
etcd-1               Healthy     {
    
    "health":"true"}                                                                        
etcd-2               Healthy     {
    
    "health":"true"}                                                                        
etcd-0               Healthy     {
    
    "health":"true"}

• 注：如果有报错，检查一下~/.kube/config 的配置，以及证书是否正确

1.6.10、授权kubelet-bootstrap用户允许请求证书

k8s-01:~ # kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes