截止目前（2020年1月2日），最新的kubernetes版本是v1.17，那我就整理一个最新版本的安装文档，方便日后查阅。
官方提供的三种Kubernetes部署方式：
1、minikube Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，尝试Kubernetes或日常开发的用户使用。不能用于生产环境。官方地址：https://kubernetes.io/docs/setup/minikube/
2、kubeadm Kubeadm也是一个工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。官方地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
3、二进制包 从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。
kubeadm的安装文档
没有特殊说明的情况下，以下操作默认在所有主机执行。
准备工作
- 配置ip、dns、hostname、hosts文件
- 关闭防火墙、selinux、swap分区
- 安装依赖包
- 同步时间
- 内核参数优化
环境信息
操作系统：CentOS Linux release 7.6.1810 (Core)
docker：19.03.5
kubernetes：v1.17.0
主机名和ip：
| hostname | ip |
master01 |
192.168.1.230 |
node01 |
192.168.1.241 |
node02 |
192.168.1.242 |
同步所有主机的hosts文件：
cat <<EOF >>/etc/hosts 192.168.1.230 master01 192.168.1.241 node01 192.168.1.242 node02 EOF
关闭 防火墙&selinux&swap
systemctl stop firewalld systemctl stop firewalld setenforce 0 sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
安装依赖包
在每台机器上安装依赖包：
yum install -y epel-release
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget
同步时间
在每台机器上执行同步时间：
ntpdate time1.aliyun.com
加载内核模块
modprobe ip_vs_rr
modprobe br_netfilter
优化内核参数
cat > /etc/sysctl.d/kubernetes.conf << EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它 vm.overcommit_memory=1 # 不检查物理内存是否够用 vm.panic_on_oom=0 # 开启 OOM fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576 fs.file-max=52706963 fs.nr_open=52706963 net.ipv6.conf.all.disable_ipv6=1 net.netfilter.nf_conntrack_max=2310720 EOF
sysctl -p /etc/sysctl.d/kubernetes.conf
安装kubernetes和docker
- 安装k8s和docker
- 所有节点添加k8s和docker的yum源
- yum安装docker，启动docker
- yum安装kubeadm，kubelet和kubectl
- 部署主节点，部署网络插件，工作节点注册到主节点
在每台机器上都需要操作
添加kubernetes的yum源
cat <<EOF >>/etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes repo baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg enabled=1 EOF
添加docker的yum源
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
安装docker
yum -y install docker-ce
启动docker
systemctl enable docker systemctl start docker cat <<EOF >>/etc/docker/daemon.json { "registry-mirrors": ["https://dlbpv56y.mirror.aliyuncs.com"] } EOF systemctl restart docker
安装kubeadm,kubelet和kubectl（kubectl可以不必在所有节点上安装）
yum -y install kubelet kubeadm kubectl
systemctl enable kubelet
部署Kubernetes Master
此操作在master节点执行（注意ip请更换成自己环境中的主节点ip；这步等的时间长一些，我用了10分钟）
kubeadm init --apiserver-advertise-address=192.168.1.230 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.17.0 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16
输出内容：
[root@master01 ~]# kubeadm init --apiserver-advertise-address=192.168.1.230 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.17.0 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16 W0102 10:56:48.971892 16147 validation.go:28] Cannot validate kube-proxy config - no validator is available W0102 10:56:48.972021 16147 validation.go:28] Cannot validate kubelet config - no validator is available [init] Using Kubernetes version: v1.17.0 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.1.230] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master01 localhost] and IPs [192.168.1.230 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master01 localhost] and IPs [192.168.1.230 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0102 11:05:25.166035 16147 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0102 11:05:25.167026 16147 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 36.502566 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: h21v01.ca56fof5m8myjy3e [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.1.230:6443 --token h21v01.ca56fof5m8myjy3e \ --discovery-token-ca-cert-hash sha256:4596521eed7d2daf11832be58b03bee46b9c248829ce31886d40fe2e997b1919
根据输出的提示，还需要做以下几个动作：
1、开始使用集群前，需要在主节点上执行：
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
2、还需要部署一个pod 网络，我们选择flannel
安装网络插件
一般的网络无法访问quay.io，可以曲线救国，找国内的镜像源，或者从docker hub上拉取flannel的镜像，此处选择第2种方式。
手动拉取flannel镜像
在集群的所有机器上操作
# 手动拉取flannel的docker镜像 docker pull easzlab/flannel:v0.11.0-amd64 # 修改镜像名称 docker tag easzlab/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
下载并安装flannel资源配置清单（此操作在master节点上进行）
[root@master01 ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml [root@master01 ~]# kubectl apply -f kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created
3、工作节点需要注册到master
node节点加入集群
使用kubeadm join 注册Node节点到Matser
（kubeadm join 的内容，在上面kubeadm init (kubeadm init输出结果的最后写明了) 已经生成好了）
此操作在node节点上进行操作：
kubeadm join 192.168.1.230:6443 --token h21v01.ca56fof5m8myjy3e \
--discovery-token-ca-cert-hash sha256:4596521eed7d2daf11832be58b03bee46b9c248829ce31886d40fe2e997b1919
查看集群的node状态，安装完网络工具之后，只有显示如下状态，所有节点全部都Ready好了之后才能继续后面的操作
[root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready master 10m v1.17.0 node01 Ready <none> 4m44s v1.17.0 node02 Ready <none> 4m41s v1.17.0 [root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-9d85f5447-279k7 1/1 Running 0 10m coredns-9d85f5447-lz8d8 0/1 ContainerCreating 0 10m etcd-master01 1/1 Running 0 10m kube-apiserver-master01 1/1 Running 0 10m kube-controller-manager-master01 1/1 Running 0 10m kube-flannel-ds-amd64-5f769 1/1 Running 0 36s kube-flannel-ds-amd64-gl5lm 1/1 Running 0 36s kube-flannel-ds-amd64-ttbdk 1/1 Running 0 36s kube-proxy-tgs9j 1/1 Running 0 5m11s kube-proxy-vpgng 1/1 Running 0 10m kube-proxy-wthxn 1/1 Running 0 5m8s kube-scheduler-master01 1/1 Running 0 10m
至此使用kubeadm的方式安装k8s v1.17完毕
测试一下kubernetes集群
##创建一个镜像为nginx的容器
[root@master01 ~]# kubectl create deployment nginx --image=nginx deployment.apps/nginx created ##查看pod的详细信息，events部分可以看到创建过程
[root@master01 ~]# kubectl describe pod nginx-86c57db685-9xbn6 Name: nginx-86c57db685-9xbn6 Namespace: default Priority: 0 Node: node02/192.168.1.242 Start Time: Thu, 02 Jan 2020 11:49:52 +0800 Labels: app=nginx pod-template-hash=86c57db685 Annotations: <none> Status: Running IP: 10.244.2.2 IPs: IP: 10.244.2.2 Controlled By: ReplicaSet/nginx-86c57db685 Containers: nginx: Container ID: docker://baca9e4f096278fbe8851dcb2eed794aefdcebaa70509d38df1728c409e73cdb Image: nginx Image ID: docker-pullable://nginx@sha256:b2d89d0a210398b4d1120b3e3a7672c16a4ba09c2c4a0395f18b9f7999b768f2 Port: <none> Host Port: <none> State: Running Started: Thu, 02 Jan 2020 11:51:49 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-4ghv8 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-4ghv8: Type: Secret (a volume populated by a Secret) SecretName: default-token-4ghv8 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m43s default-scheduler Successfully assigned default/nginx-86c57db685-9xbn6 to node02 Normal Pulling 3m42s kubelet, node02 Pulling image "nginx" Normal Pulled 106s kubelet, node02 Successfully pulled image "nginx" Normal Created 106s kubelet, node02 Created container nginx Normal Started 106s kubelet, node02 Started container nginx
##查看pod的ip [root@master01 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-86c57db685-9xbn6 1/1 Running 0 2m18s 10.244.2.2 node02 <none> <none> ##访问nginx
[root@master01 ~]# curl 10.244.2.2 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
补充内容：
1、kubectl命令自动补全
##安装包
yum install -y bash-completion*
##手工执行
source <(kubectl completion bash)
##写入环境变量
echo "source <(kubectl completion bash)" >> ~/.bashrc
##需要手工执行一下，否则tab补全时会提示“-bash: _get_comp_words_by_ref: command not found ”
sh /usr/share/bash-completion/bash_completion
##加载环境变量
source /etc/profile
##再次使用kubectl命令进行tab补全就ok了
2、后续有nodes节点想加入集群的话，由于默认token的有效期为24小时，当过期之后，该token就不可用了，解决方法如下：
重新生成新的token ==> kubeadm token create # 1.查看当前的token列表 [root@K8S00 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 7mjtn4.9kds6sabcouxaugd 23h 2019-12-24T15:44:58+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token # 2.重新生成新的token [root@K8S00 ~]# kubeadm token create 369tcl.oe4punpoj9gaijh7 # 3.再次查看当前的token列表 [root@K8S00 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 369tcl.oe4punpoj9gaijh7 23h 2019-12-24T16:05:18+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token 7mjtn4.9kds6sabcouxaugd 23h 2019-12-24T15:44:58+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token # 4.获取ca证书sha256编码hash值 [root@K8S00 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 7ae10591aa593c2c36fb965d58964a84561e9ccd416ffe7432550a0d0b7e4f90 # 5.节点加入集群 [root@k8s-node03 ~]# kubeadm join --token 369tcl.oe4punpoj9gaijh7(新的token) --discovery-token-ca-cert-hash sha256:7ae10591aa593c2c36fb965d58964a84561e9ccd416ffe7432550a0d0b7e4f90(ca证书sha256编码hash值) 172.22.34.31:6443 --skip-preflight-chec
参考文章：