检查etcd集群的状态：
[root@master ~]# etcdctl cluster-health
member 8e9e05c52164694d is healthy: got healthy result from http://192.168.2.178:2379
cluster is healthy
[root@master ~]# vi /etc/sysconfig/flanneld
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://192.168.2.178:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
#FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_ETCD_PREFIX="/k8s/network"
# Any additional options that you want to pass
FLANNEL_OPTIONS="--iface=ens33"
此时启动flanneld服务，半天起不来，其实报错了：
systemctl restart flanneld
查看系统日志/var/log/messages
Dec 18 22:06:03 k8s01 flanneld-start: E1218 22:06:03.423708 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:04 k8s01 flanneld-start: E1218 22:06:04.428807 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:05 k8s01 flanneld-start: E1218 22:06:05.431055 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:06 k8s01 flanneld-start: E1218 22:06:06.434662 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:07 k8s01 flanneld-start: E1218 22:06:07.440863 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:08 k8s01 flanneld-start: E1218 22:06:08.444012 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:09 k8s01 flanneld-start: E1218 22:06:09.447245 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:10 k8s01 flanneld-start: E1218 22:06:10.452601 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
Dec 18 22:06:11 k8s01 flanneld-start: E1218 22:06:11.453889 1735 network.go:102] failed to retrieve network config: 100: Key not found (/k8s) [3]
会发现大量这样的报错。
[root@master ~]# etcdctl set /k8s/network/config '{"Network":"10.255.0.0/16"}'
{"Network":"10.255.0.0/16"}
[root@master ~]# etcdctl get /k8s/network/config
{"Network":"10.255.0.0/16"}
执行一下上面的命令，设置一下/k8s/network/config即可。
[root@master ~]# systemctl restart flanneld
[root@master ~]# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-18 22:17:52 CST; 21s ago
Process: 1854 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
Main PID: 1843 (flanneld)
Memory: 7.0M
CGroup: /system.slice/flanneld.service
└─1843 /usr/bin/flanneld -etcd-endpoints=http://192.168.2.178:2379...
Dec 18 22:17:52 master systemd[1]: Starting Flanneld overlay address etcd a.....
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.454158 1843 m...s
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.454556 1843 m...8
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.454568 1843 m...)
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.457115 1843 l...0
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.467605 1843 m...4
Dec 18 22:17:52 master flanneld-start[1843]: I1218 22:17:52.467830 1843 n...s
Dec 18 22:17:52 master systemd[1]: Started Flanneld overlay address etcd agent.
Hint: Some lines were ellipsized, use -l to show in full.
[root@master ~]# systemctl restart kube-apiserver kube-controller-manager kube-scheduler
[root@master ~]# systemctl status kube-apiserver kube-controller-manager kube-scheduler
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-18 22:24:13 CST; 21s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 1981 (kube-apiserver)
Memory: 36.4M
CGroup: /system.slice/kube-apiserver.service
└─1981 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://192.168.2.178:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0....
Dec 18 22:24:12 master systemd[1]: Starting Kubernetes API Server...
Dec 18 22:24:12 master kube-apiserver[1981]: I1218 22:24:12.900368 1981 config.go:562] Will report 192.168.2.178 as public IP address.
Dec 18 22:24:12 master kube-apiserver[1981]: W1218 22:24:12.900898 1981 handlers.go:50] Authentication is disabled
Dec 18 22:24:12 master kube-apiserver[1981]: [restful] 2019/12/18 22:24:12 log.go:30: [restful/swagger] listing is available at https://192.168.2.178:6443/swaggerapi/
Dec 18 22:24:12 master kube-apiserver[1981]: [restful] 2019/12/18 22:24:12 log.go:30: [restful/swagger] https://192.168.2.178:6443/swaggerui/ is mapped to folder /swagger-ui/
Dec 18 22:24:13 master kube-apiserver[1981]: I1218 22:24:13.028655 1981 serve.go:95] Serving securely on 0.0.0.0:6443
Dec 18 22:24:13 master kube-apiserver[1981]: I1218 22:24:13.028712 1981 serve.go:109] Serving insecurely on 0.0.0.0:8080
Dec 18 22:24:13 master systemd[1]: Started Kubernetes API Server.
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-18 22:24:12 CST; 21s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 1980 (kube-controller)
Memory: 20.2M
CGroup: /system.slice/kube-controller-manager.service
└─1980 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://192.168.2.178:8080
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.539626 1980 pet_set.go:146] Starting statefulset controller
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.551133 1980 controllermanager.go:544] Attempting to start certificates, full resource map map[autoscaling/v1:&APIReso...orizontalPod
Dec 18 22:24:16 master kube-controller-manager[1980]: ationcontrollers true ReplicationControllerDummy} {replicationcontrollers/scale true Scale} {thirdpartyresources false ThirdPartyResour...gets/status
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.551305 1980 controllermanager.go:546] Starting certificates.k8s.io/v1alpha1 apis
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.551322 1980 controllermanager.go:548] Starting certificate request controller
Dec 18 22:24:16 master kube-controller-manager[1980]: E1218 22:24:16.551544 1980 controllermanager.go:558] Failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such...or directory
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.553035 1980 attach_detach_controller.go:235] Starting Attach Detach Controller
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.553089 1980 serviceaccounts_controller.go:120] Starting ServiceAccount controller
Dec 18 22:24:16 master kube-controller-manager[1980]: I1218 22:24:16.562581 1980 garbagecollector.go:766] Garbage Collector: Initializing
Dec 18 22:24:26 master kube-controller-manager[1980]: I1218 22:24:26.563231 1980 garbagecollector.go:780] Garbage Collector: All monitored resources synced. Proceeding to collect garbage
● kube-scheduler.service - Kubernetes Scheduler Plugin
Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2019-12-18 22:24:12 CST; 21s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 1979 (kube-scheduler)
Memory: 15.4M
CGroup: /system.slice/kube-scheduler.service
└─1979 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://192.168.2.178:8080 --address=0.0.0.0
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000603 1979 reflector.go:199] pkg/controller/informers/factory.go:89: Failed to list *api.PersistentVolumeClaim: Get http://1...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000633 1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:473: Failed to list *api.ReplicationC...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000658 1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470: Failed to list *api.Service: Get...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000686 1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:466: Failed to list *api.PersistentVo...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000718 1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:463: Failed to list *api.Node: Get ht...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000757 1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:460: Failed to list *api.Pod: Get http://192.168.2...
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000807 1979 leaderelection.go:228] error retrieving resource lock kube-system/kube-scheduler: Get http://192.168.2.178:8080/a...ction refused
Dec 18 22:24:13 master kube-scheduler[1979]: E1218 22:24:13.000894 1979 reflector.go:199] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:457: Failed to list *api.Pod: Get http://192.168.2...
Dec 18 22:24:16 master kube-scheduler[1979]: I1218 22:24:16.457251 1979 leaderelection.go:188] sucessfully acquired lease kube-system/kube-scheduler
Dec 18 22:24:16 master kube-scheduler[1979]: I1218 22:24:16.457483 1979 event.go:217] Event(api.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"12...became leader
Hint: Some lines were ellipsized, use -l to show in full.
[root@master ~]#
停止服务master节点：
systemctl stop kube-apiserver kube-controller-manager kube-scheduler flanneld
systemctl status kube-apiserver kube-controller-manager kube-scheduler flanneld
systemctl start kube-apiserver kube-controller-manager kube-scheduler flanneld
