Ubuntu20.04部署安装Kubernetes1.23＜最新尝试，无坑版＞

节点安排：

name	IP
master	172.16.10.21
node1	172.16.10.22
node2	172.16.10.23

如果接下来的步骤中没有特殊指明是哪台机器要做的话，就都要执行

安装部署过程

1.修改基本配置

1.1.首先安装ubuntu的虚拟机，配置静态IP地址，使其能够正常上网，更换为国内镜像源，并且能够被xshell正常连接上（如果这步出现问题可以查看我的之前博客：Ubuntu系统配置静态IP地址、更换国内源以及连接xshell
1.2.修改主机名，配置hosts文件，禁用防火墙跟selinux以及swap交换分区(ubuntu默认没有selinux这个功能模块，可以忽略）
例如master节点上就是：

hostnamectl set-hostname master
vim /etc/hosts
cat /etc/hosts
172.16.10.21    master
172.16.10.22    node1
172.16.10.23    node2
ufw disable
swapoff -a && sed -i '/swap/d' /etc/fstab     

为什么需要关闭交换分区：在集群中，我们通常是希望如果出现OOM（内存溢出）的情况，就直接终止这个进程，然后kubernetes进行故障转移，把这个进程在其他节点上重启起来。而不是，出现OOM的时候，通过交换分区来延长使用，看似没有问题也不会有报错提示给我们，但是会导致节点hang住（没有响应，卡死）。更可怕的是有一些集群的swap位于机械硬盘阵列上，大量动用swap基本可以等同于死机，你甚至连root都登录不上，不用提杀掉问题进程了，往往结局就是硬盘重启

1.3.修改配置

# Enable kernel modules
sudo modprobe overlay && \
sudo modprobe br_netfilter

# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# Reload sysctl
sudo sysctl --system

2.安装docker

2.1.安装docker

sudo apt update && \
sudo apt install apt-transport-https ca-certificates curl software-properties-common && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" && \
apt-cache policy docker-ce && \
sudo apt install -y containerd.io docker-ce docker-ce-cli && \
sudo systemctl status docker

2.2配置docker

# Create required directories
sudo mkdir -p /etc/systemd/system/docker.service.d

# Create daemon json config file
mkdir /etc/docker
sudo tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"], 
  "registry-mirrors": ["https://8i185852.mirror.aliyuncs.com"],         
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# Start and enable Services
sudo systemctl daemon-reload && \
sudo systemctl restart docker && \
sudo systemctl enable docker

kubernetes的cgroup驱动默认是system的，而docker的cgroup驱动默认是cgroupfs，所以建议将docker的cgroup驱动改为system的，与kubernetes保持一致，否则易导致kubeadm
init失败

3.安装k8s

3.1 安装阿里源以及一些证书之类的

sudo apt update && \
sudo apt -y install curl apt-transport-https \
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -  \
echo "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee   /etc/apt/sources.list.d/kubernetes.list

3.2 更新下载

#查看kubeadm kubelet kubectl有哪些版本，以及版本安装时的具体名称，例如1.23.6是错误的，要是1.23.6-00
apt-cache madison kubeadm kubelet kubectl
#安装指定版本的kubeadm kubelet kubectl
sudo apt update && \
sudo apt-get -y install kubelet=1.23.6-00 kubeadm=1.23.6-00 kubectl=1.23.6-00 && \      # 安装的时候需要指定版本，否则会安装最新版本，node节点可以不需要安装kubectl
sudo apt-mark hold kubelet kubeadm kubectl        #阻止软件自动更新
systemctl start kubelet 
systemctl enable kubelet

查看安装的情况以及版本
kubectl version --client && kubeadm version

3.2 拉取k8s的镜像

root@master:~# kubeadm config images list
I0207 17:52:14.976303   56639 version.go:255] remote version is much newer: v1.26.1; falling back to: stable-1.23
k8s.gcr.io/kube-apiserver:v1.23.16
k8s.gcr.io/kube-controller-manager:v1.23.16
k8s.gcr.io/kube-scheduler:v1.23.16
k8s.gcr.io/kube-proxy:v1.23.16
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

#kubeadm config images pull #本来应该直接拉镜像，但是因为这个需要翻墙，所以此命令无法执行

#拉取阿里镜像
root@master:~# kubeadm config print init-defaults > kubeadm.conf
root@master:~# sed -i 's/k8s.gcr.io/registry.aliyuncs.com\/google_containers/g' kubeadm.conf
root@master:~# kubeadm config images list --config kubeadm.conf
root@master:~# kubeadm config images list --config kubeadm.conf    

#更改镜像名字
registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.23.0
registry.aliyuncs.com/google_containers/pause:3.6
registry.aliyuncs.com/google_containers/etcd:3.5.1-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6
#修改镜像名字为阿里镜像名字 前者为阿里云镜像名字 后者为谷歌镜像名字 
#前者：sudo kubeadm config images list --config kubeadm.conf
#后者：sudo kubeadm config images list

root@master:~# kubeadm config images pull --config kubeadm.conf  #拉取阿里镜像
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.6
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.1-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6

4.kubeadm建立集群

4.1 首先是在master上的操作

#如果初始化集群失败后要记得先kubeadm reset，再继续kubeadm init
#初始化集群配置

kubeadm init \
 --image-repository registry.aliyuncs.com/google_containers \
 --kubernetes-version v1.23.6 \
 --pod-network-cidr=192.168.0.0/16 \
 --service-cidr=10.96.0.0/12 \
 --apiserver-advertise-address=172.16.10.50
 
 # 初始化命令参数说明：--apiserver-advertise-addres=172.16.10.21 这个参数就是master主机的IP地址，例如我的Master主机的IP是：172.16.10.21
--image-repository=registry.aliyuncs.com/google_containers  这个是选择拉取 control plane images 的镜像repo这个是镜像地址，由于国外地址无法访问，故使用的阿里云仓库地址：registry.aliyuncs.com/google_containers，
--kubernetes-version=v1.23.6   这个参数是下载的k8s软件版本号
--service-cidr=10.96.0.0/12       这个参数后的IP地址直接就套用10.96.0.0/12 ,以后安装时也套用即可，不要更改
--pod-network-cidr=10.10.0.0/16   k8s内部的pod节点之间网络可以使用的IP段，不能和service-cidr写一样，如果不知道怎么配，就先用这个10.244.0.0/16，最好是和docker0处于同一个网段，如果后续安装的网络插件是Calico，那么kubeadm init时必须添加此参数

#在kubeadm init命令之前，会执行一系列被称为pre-flight checks的系统与检查，以确保主机环境符合安装要求，如果检查失败就直接终止，不再进行init操作；用户可以通过kubeadm init phase preflight命令执行预检查操作，确保系统就绪后再执行init操作；或者也可以在执行kubeadm init命令时添加--ignore-preflight-errors参数关闭预检查
 
 # 输出如下   
 ......
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.10.21:6443 --token 0t6kt4.e312po5zfm7xpt9y \
	--discovery-token-ca-cert-hash sha256:0f43ed44e4b628e96b27166b97b3b41a9fcc382b53a544e67e219addb25f5571 

# 在master上执行下面操作(初始化成功才执行）
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

#查看集群状态
kubectl cluster-info
#查看节点状态，此时都是NotReady
root@master:~# kubectl get nodes
NAME     STATUS   ROLES                  AGE    VERSION
master   NotReady    control-plane,master   2d3h   v1.23.6
node1    NotReady    <none>                 2d3h   v1.23.6
node2    NotReady    <none>                 2d3h   v1.23.6

#master产生的token会失效，重新产生token的命令为
root@master:/lianxi/2.13# kubeadm token create --print-join-command
kubeadm join 172.16.10.21:6443 --token txj249.qbxqb4w5ro2lhgy8 --discovery-token-ca-cert-hash sha256:0f43ed44e4b628e96b27166b97b3b41a9fcc382b53a544e67e219addb25f5571 

4.2 所有node节点分别上执行

kubeadm join 172.16.10.21:6443 --token 0t6kt4.e312po5zfm7xpt9y \
	--discovery-token-ca-cert-hash sha256:0f43ed44e4b628e96b27166b97b3b41a9fcc382b53a544e67e219addb25f5571 
 
 # 如果重新加入master，也需要先kubeadm reset

5.安装网络插件

# 第一种是安装calico插件，但是我安装之后有报错，所以我后面改成了flannel
# 只需要在master节点上执行
kubectl apply -f https://docs.projectcalico.org/v3.21/manifests/calico.yaml    #，创建过程会要几分钟，完成后所有node节点的状态变为ready
kubectl get nodes    #所有节点的状态会变成ready

# 第二种是安装flannel插件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml    #记得修改配置文件里的net-conf.json下的Network地址，修改为kubeadm init时设置的--pod-network-cidr，然后再apply

#检查集群
root@master:~# kubectl get nodes        #安装了网络插件后就都是Ready状态
NAME     STATUS   ROLES                  AGE    VERSION
master   Ready    control-plane,master   2d3h   v1.23.6
node1    Ready    <none>                 2d3h   v1.23.6
node2    Ready    <none>                 2d3h   v1.23.6

6.部署dashboard

wget  https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
mv recommended.yaml kubernetes-dashboard.yaml
vim kubernetes-dashboard.yaml
cat kubernetes-dashboard.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  selector:
    k8s-app: kubernetes-dashboard
    
root@master:~# kubectl apply -f kubernetes-dashboard.yaml 
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19, non-functional in v1.25+; use the "seccompProfile" field instead
deployment.apps/dashboard-metrics-scraper created
                            

root@master:~# kubectl get pods,svc -n kubernetes-dashboard -o wide
NAME                                             READY   STATUS    RESTARTS   AGE   IP              NODE    NOMINATED NODE   READINESS GATES
pod/dashboard-metrics-scraper-577dc49767-tzjls   1/1     Running   0          25m   10.10.166.131   node1   <none>           <none>
pod/kubernetes-dashboard-6bd77794f-4pqbh         1/1     Running   0          25m   10.10.104.2     node2   <none>           <none>

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE   SELECTOR
service/dashboard-metrics-scraper   ClusterIP   10.111.188.114   <none>        8000/TCP        25m   k8s-app=dashboard-metrics-scraper
service/kubernetes-dashboard        NodePort    10.110.197.85    <none>        443:30001/TCP   25m   k8s-app=kubernetes-dashboard



root@master:~# kubectl create serviceaccount dashboard-admin -n kube-system
serviceaccount/dashboard-admin created
root@master:~# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created
root@master:~# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Name:         dashboard-admin-token-4545w
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 37c0d3a6-2c21-4cd7-aa7b-2709be5fd424

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Ikp2OUktLTZXWlpUV3g5aTBlOEwyckxWX3dPelc0RktaTnU3RFhKN1kzcUEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tNDU0NXciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMzdjMGQzYTYtMmMyMS00Y2Q3LWFhN2ItMjcwOWJlNWZkNDI0Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.QGfm8T1fNWvO85-SEYSwdRNTEnLOuvCrwetI8o6OO_PXYg1Ka_0vp4M6knlPB9a_c2SrMbLFfoBd_6NBj5E1ovywOtrKyM14vBPkRrbmbbWpA0niDNrEYq-OHo_8XWv3q5w3hLfX_K5_GKZNhxLBYiNL_4R7crqILiFK0_vPxm6QPrguGNomNl4PjpOuYSEvQUxYbKnx37Lxc1EW2ZuYnKLvohEs3Ib22znusug3CF624e8Rnh7fGShSh_BaO_nAn54nKl3PsCgUlprlkq7N9JiOoLsWEmXNaWh-Y9RglomFoJOUrE-CnkZX9gODgcJziIZQPzMIpbbd9DWbygUHVA

#因为kubernetes-dashboard部署在node2上，所以通过https://node2IP:30001可以访问dashboard，token为上面所示