A mounting docker
Second, the installation kubelet
Third, the installation kube-proxy
four, verify the availability of the cluster
five, described on Bootstrap
First, install docker
[root@master1 yaml]# docker version
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:20:36 2017
OS/Arch: linux/amd64Not be described here mounting method docker
1. Create a service
In practice it is recommended to backup service files
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl daemon-reload;systemctl stop docker"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "rm -rf /usr/lib/systemd/system/docker.service /etc/docker/daemon.json" #清理默认的参数配置文件
[root@master1 service]# vim docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
[Service]
WorkingDirectory=/data/k8s/docker
Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m copy -a "src=./docker.service dest=/etc/systemd/system" Note; $ DOCKER_NETWORK_OPTIONS will apply flannel environment variable added to the startup parameters docker's
2. Create a docker-daemon configuration file
[root@master1 service]# vim docker-daemon.json
{
"registry-mirrors": ["192.168.192.234:888"],
"max-concurrent-downloads": 20,
"live-restore": true,
"max-concurrent-uploads": 10,
"debug": true,
"data-root": "/data/k8s/docker/data",
"exec-root": "/data/k8s/docker/exec",
"log-opts": {
"max-size": "100m",
"max-file": "5"
}
}
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m copy -a "src=./docker-daemon.json dest=/etc/docker/daemon.json " 3, start-up and inspection
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl daemon-reload;systemctl restart docker;systemctl status docker"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl status docker" |grep -i active #确保都是running
[root@master1 service]# ansible master -i /root/udp/hosts.ini -m shell -a "systemctl enable docker"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "ip addr show docker0 ; ip addr show flannel.1"
success => 192.168.192.225 => rc=0 =>
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1450 qdisc noqueue state DOWN
link/ether 02:42:28:1d:d2:69 brd ff:ff:ff:ff:ff:ff
inet 172.30.56.1/21 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether 96:6c:a6:5a:01:21 brd ff:ff:ff:ff:ff:ff
inet 172.30.56.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "docker info" #查看参数是否生效Ensure that the network segment and docker0 flannel.1 in the same network segment
Registry Reference: https://blog.51cto.com/hmtk520/2422947 # Measured in building registry at 192.168.192.234
Second, the installation kubelet
kubelet run on all node, and receive api-server registration request to the management of the pod api-server, if the master node at the same time play the role, master also need to deploy kubelet, flannel, docker, kube-proxy and other roles
1. Create a file kubelet bootstrap kubeconfig
[root@master1 service]# NODE_NAMES=(master1 master2 master3 node1 node2)
[root@master1 service]# source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do
export BOOTSTRAP_TOKEN=$(kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:${node_name} --kubeconfig ~/.kube/config) # 创建 token
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/cert/ca.pem --embed-certs=true --server=https://127.0.0.1:8443 --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置集群参数
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置客户端认证参数
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置上下文参数
kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig # 设置默认上下文
done
[root@master1 service]# ls |grep bootstrap
kubelet-bootstrap-master1.kubeconfig
kubelet-bootstrap-master2.kubeconfig
kubelet-bootstrap-master3.kubeconfig
kubelet-bootstrap-node1.kubeconfig
kubelet-bootstrap-node2.kubeconfig
[root@master1 service]# kubeadm token list --kubeconfig ~/.kube/config
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
5vk286.d0r0fjv1hnfmnpmp 23h 2019-07-22T13:01:00+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node1
6b3u55.99wxedj52ldsl2d7 23h 2019-07-22T13:00:59+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:master2
h6cex2.g3fc48703ob47x0o 23h 2019-07-22T13:00:59+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:master1
losmls.877d58c6rd9g5qvh 23h 2019-07-22T13:01:00+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:master3
xn0i2f.lvdf3vy5aw3d4tl0 23h 2019-07-22T13:01:01+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node2kubeconfig include token, bootstrap and other information, kube-controller-manager to create client and server certificates for kubelet after certification by the
token is valid for one day, it can no longer be used after the expiration boostrap kubelet, and will be tokencleaner kube-controller-manager of cleanup;
Kube-apiserver kubelet after receiving the bootstrap token, the user requests to system: bootstrap: <token ID> , group set system: bootstrappers, this group is provided for the subsequent ClusterRoleBinding;
[root@master1 yaml]# kubectl explain secret.type #Secret共有多种类型的，常见类型如下
kubernetes.io/service-account-token
bootstrap.kubernetes.io/token
Opaque
[root@master1 service]# kubectl get secrets -n kube-system|grep bootstrap-token #查看token关联的Secret：
bootstrap-token-5vk286 bootstrap.kubernetes.io/token 7 6m30s
bootstrap-token-6b3u55 bootstrap.kubernetes.io/token 7 6m31s
bootstrap-token-h6cex2 bootstrap.kubernetes.io/token 7 6m31s
bootstrap-token-losmls bootstrap.kubernetes.io/token 7 6m30s
bootstrap-token-xn0i2f bootstrap.kubernetes.io/token 7 6m29s
[root@master1 yaml]# NODE_NAMES=(master1 master2 master3 node1 node2)
[root@master1 service]# for node_name in ${NODE_NAMES[@]} ; do scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig ;done2, kubelet profile
[root@master1 yaml]# vim kubelet-config.yaml.template
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "##NODE_IP##"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
authentication:
anonymous:
enabled: false
webhook:
enabled: true
x509:
clientCAFile: "/etc/kubernetes/cert/ca.pem"
authorization:
mode: Webhook
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "##NODE_IP##"
clusterDomain: "cluster.local"
clusterDNS:
- "10.244.0.2"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: cgroupfs
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "172.30.0.0/16"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]Parameter Description:
address: kubelet secure port (HTTPS, 10250)
readOnlyPort = 0: turn off the read-only port (default 10255)
authentication.anonymous.enabled: set to false, does not allow anonymous access port 10250;
authentication.x509.clientCAFile: Specifies the signature client certificate CA certificate, open HTTP certificate authentication;
authentication.webhook.enabled = to true: open HTTPs bearer token authentication; x509 certificate for authentication requests and webhook (kube-apiserver or other client) does not pass, will be rejected, Unauthorized the tips;
authroization.mode = Webhook: kubelet use SubjectAcce *** eview API query kube-apiserver a user, group whether the operating rights to the resource (RBAC);
eatureGates.RotateKubeletClientCertificate, featureGates.RotateKubeletServerCertificate: automatically rotate the certificate, the certificate is valid depends kube -controller-manager of --experimental-cluster-signing-duration parameter;
[root@master1 yaml]# NODE_IPS=(192.168.192.222 192.168.192.223 192.168.192.224 192.168.192.225 192.168.192.226)
[root@master1 yaml]# NODE_NAMES=(master1 master2 master3 node1 node2)
[root@master1 yaml]# for i in `seq 0 4`;do sed "s@##NODE_IP##@${NODE_IPS["$i"]}@g" ./kubelet-config.yaml.template &> ./kubelet-config.${NODE_NAMES[$i]};done
[root@master1 yaml]# for i in `seq 0 4`;do scp ./kubelet-config.${NODE_NAMES[$i]} root@${NODE_IPS[$i]}:/etc/kubernetes/kubelet-config.yaml; done 3, service configuration
[root@master1 service]# vim kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/data/k8s/k8s/kubelet
ExecStart=/opt/k8s/bin/kubelet \
--allow-privileged=true \
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
--cert-dir=/etc/kubernetes/cert \
--cni-conf-dir=/etc/cni/net.d \
--container-runtime=docker \
--container-runtime-endpoint=unix:///var/run/dockershim.sock \
--root-dir=/data/k8s/k8s/kubelet \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet-config.yaml \
--hostname-override==##NODE_NAME## \
--pod-infra-container-image=192.168.192.234:888/pause:latest \
--image-pull-progress-deadline=15m \
--volume-plugin-dir=/data/k8s/k8s/kubelet/kubelet-plugins/volume/exec/ \
--logtostderr=true \
--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
[root@master1 service]# bootstrap-kubeconfig: pointing to bootstrap kubeconfig file, kubelet use the file in the user name and token request to send TLS Bootstrapping-apiserver Kube
CERT-dir # K8S after signing the certificate store certificates and private key files and directories, then writes --kubeconfig file
[root@master1 service]# NODE_NAMES=(master1 master2 master3 node1 node2)
[root@master1 service]# for i in ${NODE_NAMES[@]} ; do sed "s@##NODE_NAME##@$i@g" ./kubelet.service.template &> ./kubelet.service.$i ;done
[root@master1 service]# for i in ${NODE_NAMES[@]} ;do scp ./kubelet.service.$i root@$i:/etc/systemd/system/kubelet.service; done 4, start the service and inspection
[root@master1 docker]# ansible all -i /root/udp/hosts.ini -m shell -a "mkdir /data/k8s/k8s/kubelet/kubelet-plugins/volume/exec/ -pv "
[root@master1 docker]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl daemon-reload;systemctl restart kubelet"
[root@master1 docker]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl status kubelet"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl enable kubelet" 5, examination revealed two node can not join the cluster
[root@master2 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready <none> 179m v1.14.3
master2 Ready <none> 4h10m v1.14.3
master3 Ready <none> 4h10m v1.14.3On node1: systemctl status kubelet error
Jul 21 23:06:08 node1 kubelet[21844]: E0721 23:06:08.081728 21844 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://127.0.0.1:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Jul 21 23:06:08 node1 kubelet[21844]: E0721 23:06:08.113704 21844 kubelet.go:2244] node "node1" not foundResolution:
Method 1: The kube-nginx also mounted on the node1 and node2
Method 2: Modify the /etc/kubernetes/kubelet-bootstrap.kubeconfig node1 and node2 Server 6443 is the port master1 / master2 / master3 of
at https: revisions have been made in //blog.51cto.com/hmtk520/2423306
6, Boostrap Token Auth and grant permissions
1) lookup file (user certificate information) whether there is a corresponding time parameter --kubeletconfig kubelet start, if there is no use --bootstrap-kubeconfig kubeconfig specified file is sent to kube-apiserver certificate signing request (the CSR)
kube-apiserver after receiving the CSR request, the Token wherein the authentication is requested by the user will set system: bootstrap: <Token ID> , group set system: bootstrappers, a process called Bootstrap Token Auth.
By default, this user and group permissions do not create the CSR, kubelet failed to start
The solution is: Create a clusterrolebinding, the group system: bootstrappers and clusterrole system: node-bootstrapper Binding:
[root @ Master1 ~] # kubectl the Create clusterrolebinding kubelet-Bootstrap --clusterrole = System: the Node-bootstrapper --group = System : bootstrappers
2) After the start kubelet sent using --bootstrap-kubeconfig to kube-apiserver CSR request, and when the CSR is approve, kube-controller-manager to create a client certificate for TLS kubelet, private and --kubeletconfig file
Note: kube- controller-manager to configure --cluster-signing-cert-file and --cluster-signing-key-file parameters, will create a certificate and private key for the TLS Bootstrap.
3) automatically signed csr
create three ClusterRoleBinding, it is used to automatically approve client, renew client, renew server certificate
[root@master1 service]# kubectl get csr #有很多pending状态的
[root@master1 yaml]# vim csr-crb.yaml
# Approve all CSRs for the group "system:bootstrappers"
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: auto-approve-csrs-for-group
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.io
---
# To let a node of the group "system:nodes" renew its own credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-client-cert-renewal
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests/selfnodeserver"]
verbs: ["create"]
---
# To let a node of the group "system:nodes" renew its own server credentials
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: node-server-cert-renewal
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: approve-node-server-renewal-csr
apiGroup: rbac.authorization.k8s.io
[root@master1 yaml]# kubectl apply -f csr-crb.yaml
auto-approve-csrs-for-group：自动 approve node 的第一次 CSR； 注意第一次 CSR 时，请求的 Group 为 system:bootstrappers；
node-client-cert-renewal：自动 approve node 后续过期的 client 证书，自动生成的证书 Group 为 system:nodes;
node-server-cert-renewal：自动 approve node 后续过期的 server 证书，自动生成的证书 Group 为 system:nodes;7, inspection and certification
[root@master1 yaml]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready <none> 18h v1.14.3
master2 Ready <none> 19h v1.14.3
master3 Ready <none> 19h v1.14.3
node1 Ready <none> 15h v1.14.3
node2 Ready <none> 15h v1.14.3
kube-controller-manager 为各 node 生成了 kubeconfig 文件和公私钥：
[root@master1 yaml]# ansible all -i /root/udp/hosts.ini -m shell -a "ls /etc/kubernetes/kubelet.kubeconfig"
[root@master1 yaml]# ansible all -i /root/udp/hosts.ini -m shell -a "ls -l /etc/kubernetes/cert/ |grep kubelet"
[root@master1 yaml]# kubectl get csr |grep Pending #获取pending状态的csr
[root@master1 yaml]# kubectl certificate approve csr-zzhxb #手动签署证书
[root@node1 ~]# ls -l /etc/kubernetes/cert/kubelet-*
-rw------- 1 root root 1269 Jul 21 23:16 /etc/kubernetes/cert/kubelet-client-2019-07-21-23-16-26.pem
lrwxrwxrwx 1 root root 59 Jul 21 23:16 /etc/kubernetes/cert/kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2019-07-21-23-16-26.pem
-rw------- 1 root root 1305 Jul 22 14:49 /etc/kubernetes/cert/kubelet-server-2019-07-22-14-49-22.pem
lrwxrwxrwx 1 root root 59 Jul 22 14:49 /etc/kubernetes/cert/kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-2019-07-22-14-49-22.pem
kubelet监听情况
[root@node1 ~]# netstat -tunlp |grep kubelet
tcp 0 0 192.168.192.225:10248 0.0.0.0:* LISTEN 8931/kubelet #10248: healthz http服务
tcp 0 0 127.0.0.1:19529 0.0.0.0:* LISTEN 8931/kubelet
tcp 0 0 192.168.192.225:10250 0.0.0.0:* LISTEN 8931/kubelet #10250: https服务，访问该端口时需要认证和授权（即使访问 /healthz 也需要）8, kubelet api authentication and authorization
kubelet: 10250 listens port https request, the interface provides:
/pods、/runningpods
/metrics、/metrics/cadvisor、/metrics/probes
/spec
/stats、/stats/container
/logs
/run/、/exec/, /attach/, /portForward/, /containerLogs/Reference: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go#L434:3
Due to the closure of the anonymous authentication, and authorization to open a webhook, all requests to access port 10250 https API will need to be authenticated and authorized.
Predefined ClusterRole system: kubelet-api-admin privileges granted access kubelet all the API (kubernetes certificate kube-apiserver use User granted that permission):
[root@master1 yaml]# kubectl describe clusterrole system:kubelet-api-admin
Name: system:kubelet-api-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
nodes/log [] [] [*]
nodes/metrics [] [] [*]
nodes/proxy [] [] [*]
nodes/spec [] [] [*]
nodes/stats [] [] [*]
nodes [] [] [get list watch proxy]kubelet authorization parameter is configured as follows:
- authentication.anonymous.enabled: set to false, does not allow anonymous access port 10250;
- authentication.x509.clientCAFile: Specifies the client certificate signing CA certificate, open HTTPs certificate authentication;
- authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；
- authroization.mode = Webhook: open RBAC authorization;
kubelet receipt of the request, using clientCAFile to sign the certificate for authentication, or query bearer token is valid. If they did not pass, the request is denied, suggesting Unauthorized:[root@master1 yaml]# curl -s --cacert /etc/kubernetes/cert/ca.pem https://172.27.137.240:10250/metrics Unauthorized [root@master1 yaml]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer 123456" https://172.27.137.240:10250/metrics UnauthorizedOnce certified, kubelet SubjectAcce *** eview API used to send a request kube-apiserver, certificate or token corresponding to the query user, whether the group have rights to resources (RBAC)
9, certificate authentication and authorization
[root@master1 yaml]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://192.168.192.222:10250/metrics
Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)[root@master1 yaml]#
[root@master1 yaml]#
[root@master1 yaml]# curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/cert/admin.pem --key /opt/k8s/work/cert/admin-key.pem https://192.168.192.222:10250/metrics|head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 010, bear token authentication and authorization
[root@master1 yaml]# kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
clusterrolebinding.rbac.authorization.k8s.io/kubelet-api-test created
[root@master1 yaml]# SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
[root@master1 yaml]# TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
[root@master1 yaml]# echo ${TOKEN}
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Imt1YmVsZXQtYXBpLXRlc3QtdG9rZW4tc2hnNnAiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia3ViZWxldC1hcGktdGVzdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImVhZDI4ZjMzLWFlZjAtMTFlOS05MDMxLTAwMTYzZTAwMDdmZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Omt1YmVsZXQtYXBpLXRlc3QifQ.Ej1D4q-ekLNi_1QEDXq4RxvNdPBWHNPNwUk9NE-O6-4a003yHZtU019ykqBu7WEc8yi1MB75D1in3I9oXh5bL1-bpJbDcK6z7_M8qGl2-20yFnppprP99tUJUQo7d6G3XS7jtmMbpwPgJyVrwvIX_ex2pvRkWqijR0jzGLWbWt7oEmY206o5ePF1vycpHdKznkEh4lXmJt-Nx8kzeuOLcBrc8MM2luIi26E3Tec793cQWsf6uMCgqaivkI0PlYY_jLmUJyg3pY8bsFUSCj6C6_5SqFMv2NjiMOtX6AjBDdtq64EHZcfaQd3O7hlXNgK4txlNlaqwE1vO8viexk1ONQ
[root@master1 yaml]# curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://192.168.192.222:10250/metrics|head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 011、cadvisor和metric
cadvisor is embedded in kubelet in binary, each container resources of the node where statistics (CPU, memory, disk, network cards) use of the service.
Browser access https://192.168.192.222:10250/metrics and https://192.168.192.222:10250/metrics/cadvisor return the metrics kubelet and cadvisor of.
kubelet.config.json set authentication.anonymous.enabled visit https service 10250 is false, does not allow anonymous certificate;
need to manually install the CA certificate kubernetes
this case: Before metric added a proxy, the proxy address is: https: //10.10.110.103:10250/metrics and https://10.10.110.103:10250/metrics/cadvisor
1) the certificate is downloaded to the local ca.pem
rename ca.crt
2) install the certificate
root certificate file, double-click the exported (in ca.crt) -> install a certificate -> memory location: a local computer -> 3 
) View certificate status
4) Export Personal Certificates
https kubelet port requires two-way authentication
[root @ master1 ~] # openssl pkcs12 -export -out admin.pfx -inkey admin-key.pem -in admin.pem -certfile ca.pem
5)导入个人证书
chrome->设置->高级->管理证书->个人->导入[按照提示导入该证书]
12、获取kubelet配置
[root@master1 yaml]# curl -sSL --cacert /etc/kubernetes/cert/ca.pem --cert /opt/k8s/work/cert/admin.pem --key /opt/k8s/work/cert/admin-key.pem https://127.0.0.1:8443/api/v1/nodes/master1/proxy/configz | jq '.kubeletconfig|.kind=KubeletConfiguration|.apiVersion=kubelet.config.k8s.io/v1beta1'三、安装kube-proxy
kube-proxy运行在所有worker节点上，监听 apiserver 中 service 和 endpoint 的变化情况，创建路由规则以提供服务 IP 和负载均衡功能。
本文讲解使用 ipvs 模式的 kube-proxy 的部署过程。
1、创建证书
[root@master1 cert]# vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "HangZhou",
"L": "HangZhou",
"O": "k8s",
"OU": "FirstOne"
}
]
}
[root@master1 cert]# cfssl gencert -ca=./ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy2、配置kubeconfig
Master的地址信息和认证信息
[root@master1 cert]# kubectl config set-cluster kubernetes --certificate-authority=./ca.pem --embed-certs=true --server=https://127.0.0.1:8443 --kubeconfig=kube-proxy.kubeconfig
[root@master1 cert]# kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
[root@master1 cert]# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
[root@master1 cert]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
[root@master1 cert]# ansible all -i /root/udp/hosts.ini -m copy -a "src=./kube-proxy.kubeconfig dest=/etc/kubernetes/" 3、创建kube-proxy配置文件
[root@master1 cert]# vim kube-proxy-config.yaml.template
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
burst: 200
kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"
qps: 100
bindAddress: ##NODE_IP##
healthzBindAddress: ##NODE_IP##:10256
metricsBindAddress: ##NODE_IP##:10249
enableProfiling: true
clusterCIDR: 172.30.0.0/16
hostnameOverride: ##NODE_NAME##
mode: "ipvs"
portRange: ""
kubeProxyIPTablesConfiguration:
masqueradeAll: false
kubeProxyIPVSConfiguration:
scheduler: rr
excludeCIDRs: []
[root@master1 cert]# NODE_IPS=(192.168.192.222 192.168.192.223 192.168.192.224 192.168.192.225 192.168.192.226)
[root@master1 cert]# NODE_NAMES=(master1 master2 master3 node1 node2)
[root@master1 cert]# for i in `seq 0 4 `;do sed "s@##NODE_NAME##@${NODE_NAMES[$i]}@g;s@##NODE_IP##@${NODE_IPS[$i]}@g" ./kube-proxy-config.yaml.template &> ./kube-proxy-config.yaml.${NODE_NAMES[$i]};done
[root@master1 cert]# for i in ${NODE_NAMES[@]};do scp ./kube-proxy-config.yaml.$i root@$i:/etc/kubernetes/kube-proxy-config.yaml ;done 4、配置systemd文件
[root@master1 service]# vim kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/data/k8s/k8s/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy-config.yaml \
--logtostderr=true \
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.targe
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m copy -a "src=./kube-proxy.service dest=/etc/systemd/system/"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "mkdir /data/k8s/k8s/kube-proxy"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "modprobe ip_vs_rr"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "lsmod |grep vs"
[root@master1 service]# ansible all -i /root/udp/hosts.ini -m shell -a "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy" 四、验证集群可用性
1、检查node状态
[root@master1 cert]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready <none> 20h v1.14.3
master2 Ready <none> 21h v1.14.3
master3 Ready <none> 21h v1.14.3
node1 Ready <none> 17h v1.14.3
node2 Ready <none> 17h v1.14.32、创建测试yaml
[root@master1 yaml]# cat dp.yaml
apiVersion: v1
kind: Service
metadata:
name: baseservice
namespace: default
spec:
ports:
- name: ssh
port: 22
targetPort: 22
selector:
name: base
version: v1
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: base
namespace: default
spec:
replicas: 5
selector:
matchLabels:
name: base
version: v1
template:
metadata:
labels:
name: base
version: v1
spec:
containers:
- name: base-v1
image: 192.168.192.234:888/base:v1
ports:
- name: ssh
containerPort: 22
[root@master1 yaml]# kubectl apply -f dp.yaml
service/baseservice created
deployment.apps/base created
[root@master1 yaml]# kubectl get pods
NAME READY STATUS RESTARTS AGE
base-775b9cd6f4-6hg4k 1/1 Running 0 3s
base-775b9cd6f4-hkhfm 1/1 Running 0 3s
base-775b9cd6f4-hvsrm 1/1 Running 0 3s
base-775b9cd6f4-mmkgj 1/1 Running 0 3s
base-775b9cd6f4-rsn79 0/1 CrashLoopBackOff 1 3s
[root@master1 yaml]# kubectl describe pods base-775b9cd6f4-rsn79
Name: base-775b9cd6f4-rsn79
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: node1/192.168.192.225
Start Time: Mon, 22 Jul 2019 23:08:15 +0800
Labels: name=basev1
pod-template-hash=775b9cd6f4
version=v1
Annotations: <none>
Status: Running
IP:
Controlled By: ReplicaSet/base-775b9cd6f4
Containers:
base:
Container ID: docker://4d0ac70c18989fb1aefd80ae5202449c71e9c63690dc3da44de5ee100b3d7959
Image: 192.168.192.234:888/base:latest
Image ID: docker-pullable://192.168.192.234:888/base@sha256:1fff451e7371bbe609536604a17fd73920e1644549c93308b9f4d035ea4d8740
Port: 22/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: cannot join network of a non running container: 8df53e746bb9e80c9249a08eb6c4da0e9fcf0a8435f2f26a7f02dca6e50ba77b
Exit Code: 128
Started: Mon, 22 Jul 2019 23:08:26 +0800
Finished: Mon, 22 Jul 2019 23:08:26 +0800
Ready: False
Restart Count: 2
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-lf2bh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-lf2bh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lf2bh
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 360s
node.kubernetes.io/unreachable:NoExecute for 360s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27s default-scheduler Successfully assigned default/base-775b9cd6f4-rsn79 to node1
Normal Pulling 26s (x2 over 27s) kubelet, node1 Pulling image "192.168.192.234:888/base:latest"
Normal Pulled 26s (x2 over 26s) kubelet, node1 Successfully pulled image "192.168.192.234:888/base:latest"
Normal Created 26s (x2 over 26s) kubelet, node1 Created container base
Warning Failed 26s kubelet, node1 Error: failed to start container "base": Error response from daemon: cannot join network of a non running container: 177fae54ba52402deb52609f98c24177695ef849dd929f5166923324e431997e
Warning Failed 26s kubelet, node1 Error: failed to start container "base": Error response from daemon: cannot join network of a non running container: 1a3d1c9f9ec7cf6ae2011777c60d7347d54efe4e1a225f1e75794bc51ec55556
Normal SandboxChanged 18s (x9 over 26s) kubelet, node1 Pod sandbox changed, it will be killed and re-created.
Warning BackOff 18s (x8 over 25s) kubelet, node1 Back-off restarting failed container
[root@master1 yaml]# 尝试了各种方法，暂时没有找到根本原因 //最后通过对该node重新初始化操作系统并重新安装配置docker、kubelet、kube-proxy解决
3、问题记录：
kueblet启动报错
failed to start ContainerManager failed to initialize top level QOS containers: failed to update top level Burstable QOS cgroup : failed to set supported cgroup subsystems for cgroup [kubepods burstable]: Failed to find subsystem mount for required subsystem: pids解决方法：docker.service 配置中增加 --exec-opt native.cgroupdriver=systemd 选项。
解法1：重启主机
解法2：手动挂载pid：mount -t cgroup -o pids,rw,nosuid,nodev,noexec,relatime,pids pids /sys/fs/cgroup/pids
五、bootstrap说明
1、bootstrap
node节点默认只安装了docker和kubelet，kubelet要和apiserver通信，但是双方谁都不认识谁。所以，kubelet bootstrap要以自动化的方式解决如下几个问题：
1)在只知道 api server IP 地址的情况下，node 如何获取 api server 的 CA 证书？
2)如何让 api server 信任 worker？因为在 api server 信任 worker 之前，worker 没有途径拿到自己的证书，有点鸡生蛋蛋生鸡的感觉
2、bootstrap token创建
kubelet要让apiserver信任自己，worker 得需要先过 master 认证鉴权这一关
k8s支持的认证方式：Bootstrap Token Authentication、token文件、x509证书、service Account等等
使用 Bootstrap Token Authentication时，只需告诉kubelet一个特殊的token，kubelet自然就能通过api server的是认证
[root@master1 yaml]# cat token.yaml
apiVersion: v1
kind: Secret
metadata:
name: bootstrap-token-abcdef #token 的 name 必须是 bootstrap-token-<token-id> 的格式
namespace: kube-system
type: bootstrap.kubernetes.io/token #token 的 type 必须是 bootstrap.kubernetes.io/token
stringData:
description: "The bootstrap token for testing."
token-id: abcdef #token 的 token-id 和 token-secret 分别是6位和16位数字和字母的组合
token-secret: 0123456789abcdef
expiration: 2019-09-16T00:00:00Z #token的有效期，token 失效，kubelet 就无法使用 token 跟 api server 通信
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
auth-extra-groups: system:bootstrappers:test-nodes #定义了 token 代表的用户所属的额外的 group，而默认 group 名为 system:bootstrappers
[root@master1 yaml]# kubectl explain secret.type
secret的type主要有2中kubernetes.io/service-account-token、bootstrap.kubernetes.io/token、Opaque
[root@master1 bootstrap]# kubectl describe secret bootstrap-token-abcdef -n kube-system
[root@master1 bootstrap]# kubectl get secret bootstrap-token-abcdef -n kube-system -o yaml3、生成kubelet证书
token定义有expiration过期时间，过了有效期后kubelet就无法使用token和api server通信。所以这个token只能作为kubelet初始化时跟api server的临时通信
kubelet最终还是要使用证书和apiserver通信，如果让运维人员为每个node维护一个证书，太过于繁琐。
bootstrap解决了这个问题：kubelet 使用低权限的 bootstrap token 跟 api server 建立连接后，要能够自动向 api server 申请自己的证书，并且 api server 要能够自动审批证书
4、证书
[root@master1 bootstrap]# name=vip ;group=newlands
[root@master1 bootstrap]# cat vip.json
{
"hosts": [],
"CN": "vip",
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "NZ",
"ST": "Wellington",
"L": "Wellington",
"O": "newlands",
"OU": "Test"
}
]
}
[root@master1 bootstrap]# cat vip.json | cfssl genkey - | cfssljson -bare $name
生成：vip.csr vip-key.pem5、创建 csr
[root@master1 bootstrap]# cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: $name
spec:
groups:
- $group
request: $(cat ${name}.csr | base64 | tr -d '\n')
usages:
- key encipherment
- digital signature
- client auth
[root@master1 bootstrap]# kubectl get csr vip -o yaml
[root@master1 bootstrap]# kubectl get csr vip
NAME AGE REQUESTOR CONDITION
vip 14m admin Pending6、为vip用户绑定clusterrole
目标：让vip用户发起的csr能够被自动审批，那么vip 用户需要访问csr api(授权给vip用户该clusterrole即可)
[root@master1 cert]# kubectl describe clusterrole system:node-bootstrapper
Name: system:node-bootstrapper
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
certificatesigningrequests.certificates.k8s.io [] [] [create get list watch]
[root@master1 cert]# kubectl create clusterrolebinding csr-vip --clusterrole system:node-bootstrapper --user vip其次，我们需要给 vip 用户另一个特殊的 clusterrole
[root@master1 cert]# kubectl describe clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient
Name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
certificatesigningrequests.certificates.k8s.io/nodeclient [] [] [create]
[root@master1 cert]# kubectl create clusterrolebinding nodeclient-vip --clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient --user vip7、vip用户测试
因为自动审批目前只针对 kubelet，所以 vip 申请的 csr 用户名必须是 system:node:<name> 的形式，group 必须是 system:nodes，并且 usages 也必须是命令中所示：
然后查看 csr，可以看到 csr 的状态已经是 Approved,Issued，实验结束：
$ kubectl get csr
NAME AGE REQUESTOR CONDITION
system:node:test-node 3s vip Approved,Issued
[root@master1 cert]# kubectl describe clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient
Name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
certificatesigningrequests.certificates.k8s.io/nodeclient [] [] [create]
Reference document:
https://feisky.gitbooks.io/kubernetes/content/troubleshooting/cluster.html
https://k8s-install.opsnull.com/07-2.kubelet.html
https://docs.docker.com / Engine / Reference / the CommandLine / dockerd / #Dockerd more parameters
https://docs.docker.com/engine/reference/commandline/dockerd//#daemon-configuration-file # daemon.json profile
https: // kubernetes .io / docs / Reference / the Command-Line-Tools-Reference / kubelet-authentication-the authorization / #kubelet authentication and authorization
https://k8s-install.opsnull.com/08.%E9%AA%8C%E8%AF . 9B% E9% 81%% 86% E7% BE% A4% E5% 8A% 9F% E8% 83% BD.html
https://lingxiankong.github.io/2018-09-18-kubelet-bootstrap-process. HTML
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/types.go #kubelet profile