The second part of building a container cloud platform through kubernetes, just recently officially released V1.19.0. This article uses the latest version to introduce the installation of a highly available kubernetes cluster through kubeadm.
There are many tools for installing k8s on the market, but for learning, it is recommended to install them step by step to understand the components running inside the entire cluster, so that later learning and troubleshooting are more convenient. . .
The environment of this article is as follows:
Server: 3
operating systems: CentOS 7
topology diagram is not drawn, directly copy the official website
###Overview
Briefly speaking in this picture, three servers are used as master nodes, and keepalive+haproxy is used to load balance the apiserver. The communication between the node node and the apiserver is carried out through VIP. As mentioned in the first article, all the information of the cluster is stored in the ETCD cluster.
Next, go ahead. . .
###Configuration source
There are three sources configured here, all of which are replaced from domestic mirror sources to speed up the installation package.
# 系统源
curl -O http://mirrors.aliyun.com/repo/Centos-7.repo
# docker源
curl -O https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's/download.docker.com/mirrors.ustc.edu.cn\/docker-ce/g' docker-ce.repo
# kubernetes源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF###Configure system related parameters After the
system has configured the source, some parameters need to be set, which are all official recommendations. More optimization will be introduced later.
# 临时禁用selinux
# 永久关闭 修改/etc/sysconfig/selinux文件设置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
setenforce 0
# 临时关闭swap
# 永久关闭 注释/etc/fstab文件里swap相关的行
swapoff -a
# 开启forward
# Docker从1.13版本开始调整了默认的防火墙规则
# 禁用了iptables filter表中FOWARD链
# 这样会引起Kubernetes集群中跨Node的Pod无法通信
iptables -P FORWARD ACCEPT
# 配置转发相关参数，否则可能会出错
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system
# 加载ipvs相关内核模块
# 如果重新开机，需要重新加载
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
lsmod | grep ip_vs###Install kubeadm and related software
yum install -y kubelet kubeadm kubectl ipvsadm###Configure the
main configuration of docker to accelerate the download of public images and allow downloading of images from insecure private warehouses. You hub.xxx.omneed to change it to your own private warehouse address. If not, please delete insecure-registriesthis linevim /etc/docker/daemon.json
{
"registry-mirrors": ["https://ci7pm4nx.mirror.aliyuncs.com","https://registry.docker-cn.com","http://hub-mirror.c.163.com"],
"insecure-registries":["hub.xxx.om"]
}Write the configuration, restart docker
systemctl restart docker
systemctl enable docker.serviceView docker info, the output is as follows
Insecure Registries:
hub.xxx.com
127.0.0.0/8
Registry Mirrors:
https://ci7pm4nx.mirror.aliyuncs.com/
https://registry.docker-cn.com/
http://hub-mirror.c.163.com/###Start kubelet
systemctl enable --now kubeletkubelet now restarts every few seconds because it is stuck in an endless loop waiting for kubeadm commands.
##Install and configure haproxy and keepalive (all three machines must be installed and configured)
installation packageyum install -y haproxy keepalived
Configure haproxy
Need to pay attention to, manually create the /var/log/haproxy.log file
[root@k8s-master001 ~]# cat /etc/haproxy/haproxy.cfg
# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log /var/log/haproxy.log local0
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
retries 1
timeout http-request 10s
timeout queue 20s
timeout connect 5s
timeout client 20s
timeout server 20s
timeout http-keep-alive 10s
timeout check 10s
listen admin_stats
mode http
bind 0.0.0.0:1080
log 127.0.0.1 local0 err
stats refresh 30s
stats uri /haproxy-status
stats realm Haproxy\ Statistics
stats auth admin:admin
stats hide-version
stats admin if TRUE
#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
bind *:8443
mode tcp
option tcplog
default_backend apiserver
#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
option httpchk GET /healthz
http-check expect status 200
mode tcp
option ssl-hello-chk
balance roundrobin
server k8s-master001 10.26.25.20:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
server k8s-master002 10.26.25.21:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
server k8s-master003 10.26.25.22:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3####启动haproxy
systemctl start haproxy
systemctl enable haproxy
###Configure keepalived
[root@k8s-master001 ~]# cat /etc/keepalived/keepalived.conf
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_K8S
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens18
virtual_router_id 51
priority 100
authentication {
auth_type PASS
auth_pass kubernetes
}
virtual_ipaddress {
10.26.25.23
}
track_script {
check_apiserver
}
}Add keepalive check script
[root@k8s-master001 ~]# cat /etc/keepalived/check_apiserver.sh
#!/bin/sh
errorExit() {
echo "*** $*" 1>&2
exit 1
}
curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null || errorExit "Error GET https://localhost:8443/"
if ip addr | grep -q 10.26.25.23; then
curl --silent --max-time 2 --insecure https://10.26.25.23:8443/ -o /dev/null || errorExit "Error GET https://10.26.25.23:8443/"
fi
chmod +x /etc/keepalived/check_apiserver.sh####Start keepalived
systemctl start keepalived
systemctl enable keepalivedNow you can IP:1080/aproxy-statusaccess the haproxy management interface by visiting the master . The username and password are in the configuration file. This article is admin/admin, you can modify it yourself.
At the beginning, the lines of apiserver are all red, which means that the service has not been started yet. My picture here is post-cut, so it is green
Next, start to initialize the kubernetes cluster
###Initialize the first control node master001
[root@k8s-master001 ~]# kubeadm init --control-plane-endpoint 10.26.25.23:8443 --upload-certs --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr 10.244.0.0/16
W0910 05:09:41.166260 29186 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.1
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
........忽略了部分信息
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
............忽略了部分信息
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 10.26.25.23:8443 --token f28iti.c5fgj45u28332ga7 \
--discovery-token-ca-cert-hash sha256:81ec8f1d1db0bb8a31d64ae31091726a92b9294bcfa0e2b4309b9d8c5245db41 \
--control-plane --certificate-key 93f9514164e2ecbd85293a9c671344e06a1aa811faf1069db6f678a1a5e6f38b
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.26.25.23:8443 --token f28iti.c5fgj45u28332ga7 \
--discovery-token-ca-cert-hash sha256:81ec8f1d1db0bb8a31d64ae31091726a92b9294bcfa0e2b4309b9d8c5245db41See the output as above, which means the initialization is successful and the
initialization command description:kubeadm init --control-plane-endpoint 10.26.25.23:8443 --upload-certs --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr 10.244.0.0/16
- --control-plane-endpoint 10.26.25.23:8443 where 10.26.25.23 is the VIP configured by keepalived
- --image-repository registry.aliyuncs.com/google_containers changed the default download mirror address, the default is k8s.gcr.io, it can't be downloaded in China, or you can climb the wall by yourself~~~
- --pod-network-cidr 10.244.0.0/16 defines the network segment of the pod, which needs to be the same as the network segment defined by flannel, otherwise the pod of flannel may be restarted all the time when installing flannel, which will be mentioned later when installing flannel
Introduction to the initialization process:
- Download the required image
- Create a certificate
- Create a yaml configuration file for the service
- Start static pod
After the initialization is complete, you can now configure the kubectl client and use kubernetes according to the prompts, although there is only one master node.
Start using the cluster
[root@k8s-master001 ~]# mkdir -p $HOME/.kube
[root@k8s-master001 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master001 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master001 ~]# kubectl get no
NAME STATUS ROLES AGE VERSION
k8s-master001 NotReady master 105s v1.19.0Now you can see that there is only one node in the cluster, and the status is NotReady. This is because the network plug-in has not been installed.
Next, install the network plug-in Flannel
###Flannel Installation
Download the yalm files needed for installation: wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel.yml
Because the latest version of kubernetes is installed now rbac的api版本需要修改为rbac.authorization.k8s.io/v1，DaemonSet的api版本改为 apps/v1, and the selector is added, only part of the configuration is posted here.
[root@k8s-master001 ~]# cat kube-flannel.yml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
tier: node
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
Next, install Flannel through kubectl, and check whether the status of the flannel pod is running through kubectl.
kubectl apply -f kube-flannel.yaml
[root@k8s-master001 ~]# kubectl get no
NAME STATUS ROLES AGE VERSION
k8s-master001 Ready master 6m35s v1.19.0
[root@k8s-master001 ~]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d56c8448f-9cr5l 1/1 Running 0 6m51s
coredns-6d56c8448f-wsjwx 1/1 Running 0 6m51s
etcd-k8s-master001 1/1 Running 0 7m
kube-apiserver-k8s-master001 1/1 Running 0 7m
kube-controller-manager-k8s-master001 1/1 Running 0 7m
kube-flannel-ds-nmfwd 1/1 Running 0 4m36s
kube-proxy-pqrnl 1/1 Running 0 6m51s
kube-scheduler-k8s-master001 1/1 Running 0 7mYou can see a pod named kube-flannel-ds-nmfwd, the status is running, which means that flannel has been installed
because there is only one node, and only one pod of flannel is seen. Add another two nodes later, it will See more pods and
continue to add master nodes
###Add another control node master002, master003
because there is now a control node, the cluster already exists, just add the remaining machines to the cluster, the added information can be seen in the output when the node is just initialized , The command is as follows.
Because there is too much output, some unimportant output information will be deleted
and operated on master002:
[root@k8s-master002 ~]# kubeadm join 10.26.25.23:8443 --token f28iti.c5fgj45u28332ga7 --discovery-token-ca-cert-hash sha256:81ec8f1d1db0bb8a31d64ae31091726a92b9294bcfa0e2b4309b9d8c5245db41 --control-plane --certificate-key 93f9514164e2ecbd85293a9c671344e06a1aa811faf1069db6f678a1a5e6f38b
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
..............
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.Seeing this output indicates that the addition is successful.
Now let's check the cluster node information
[root@k8s-master002 ~]# kubectl get no
NAME STATUS ROLES AGE VERSION
k8s-master001 Ready master 21m v1.19.0
k8s-master002 Ready master 6m5s v1.19.0From the output, you can see two master nodes. The operation of adding the master003 node is the same as that of master002, so I won’t say more
After all the last three nodes are added, the specific information of the cluster can be seen through kubectl
[root@k8s-master003 ~]# kubectl get no
NAME STATUS ROLES AGE VERSION
k8s-master001 Ready master 25m v1.19.0
k8s-master002 Ready master 10m v1.19.0
k8s-master003 Ready master 26s v1.19.0Finally check all pods currently running
[root@k8s-master003 ~]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d56c8448f-9cr5l 1/1 Running 0 27m
coredns-6d56c8448f-wsjwx 1/1 Running 0 27m
etcd-k8s-master001 1/1 Running 0 27m
etcd-k8s-master002 1/1 Running 0 8m19s
etcd-k8s-master003 1/1 Running 0 83s
kube-apiserver-k8s-master001 1/1 Running 0 27m
kube-apiserver-k8s-master002 1/1 Running 0 12m
kube-apiserver-k8s-master003 1/1 Running 0 85s
kube-controller-manager-k8s-master001 1/1 Running 1 27m
kube-controller-manager-k8s-master002 1/1 Running 0 12m
kube-controller-manager-k8s-master003 1/1 Running 0 81s
kube-flannel-ds-2lh42 1/1 Running 0 2m31s
kube-flannel-ds-nmfwd 1/1 Running 0 25m
kube-flannel-ds-w276b 1/1 Running 0 11m
kube-proxy-dzpdz 1/1 Running 0 2m39s
kube-proxy-hd5tb 1/1 Running 0 12m
kube-proxy-pqrnl 1/1 Running 0 27m
kube-scheduler-k8s-master001 1/1 Running 1 27m
kube-scheduler-k8s-master002 1/1 Running 0 12m
kube-scheduler-k8s-master003 1/1 Running 0 76sNow you can see that the core service apiserver, -controller-manager, and scheduler of kubernetes are all three pods.
Above, kubernetes master high-tech is deployed.
Now you can use the web management interface of haproxy to see that the three masters are available.
###Troubleshooting
If the master fails to initialize or fails to add a node, you can use kubeadm reset to reset and then reinstall
#####Reset the node
[root@k8s-node003 haproxy]# kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W0910 05:31:57.345399 20386 reset.go:99] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get node registration: node k8s-node003 doesn't have kubeadm.alpha.kubernetes.io/cri-socket annotation
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0910 05:31:58.580982 20386 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.There is too much content in one article. For the follow-up content, see the next article. . .