Cloud Computing and Big Data - Deploy Kubernetes Cluster + Complete nginx Deployment (Super Detailed!)

Cloud Computing and Big Data - Deploy Kubernetes Cluster + Complete nginx Deployment (Super Detailed!)

The basic idea of ​​deploying a Kubernetes cluster is as follows:

1. Prepare the environment:

   • Select a suitable operating system: Select a suitable Linux distribution as the operating system according to your needs, and make sure to make the same selection on all nodes.
   • Install Docker: Install Docker on all nodes, which will be used to containerize applications and components.
   • Install Kubernetes tools: install kubectl, kubeadmand kubelettools, which will be used for cluster management and configuration.

2. Set up the master node (Master Node):

   • Select a node as the master node: usually one of the slave nodes is selected as the master node, which can be any machine with sufficient resources.
   • Initialize the master node: Use kubeadm initthe command to initialize the master node and get the generated join token.
   • Set up a network plugin: Select and install an appropriate network plugin, such as Flannel, Calico, or Weave, to enable network communication between nodes.

3. Add slave nodes (Worker Nodes):

   • Run the join command on each slave node: Using the previously generated join token, run the command on each slave node kubeadm jointo join it to the cluster.
   • Confirm Node Joining: Run the command on the master node kubectl get nodesto ensure that all nodes have successfully joined the cluster.

4. Deploy the web plugin:

   • Depending on the selected network plug-in, follow its specific deployment and configuration. This will ensure network communication between the nodes and provide the networking capabilities of the Kubernetes cluster.

5. Deploy other components and applications:

   • Deploy other required core components such as kube-proxy and kube-dns/coredns.
   • Deploy your applications or services, which can be managed using Kubernetes' Deployment, Service, or other resource types.

6. Verify cluster status:

   • Run kubectl get nodesand other kubectlcommands to ensure that the nodes and components in the cluster are functioning properly.
   • Test applications: Run tests on the applications deployed in the cluster to ensure they function properly and interact with other components.

The above is the basic idea of ​​deploying a Kubernetes cluster. The exact steps and details may vary depending on the environment and needs, but this brief description can help you understand the general flow of the deployment process.

Next, we will deploy the kubernetes cluster and complete the nginx service in practice.

Start deploying the Kubernetes cluster

Run the following commands as root:

1. Install and configure docker on all nodes

1. Install the tools required for docker

yum install -y yum-utils device-mapper-persistent-data lvm2

2. Configure the docker source of Alibaba Cloud

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3. Install docker-ce, docker-ce-cli, containerd.io

yum install -y docker-ce docker-ce-cli containerd.io

4. Start docker

systemctl enable docker	#设置开机自启
systemctl start docker		#启动docker

5. Set mirror accelerator
#Set mirror accelerator, create a new daemon.json file (reference 1)

cat <<EOF > /etc/docker/daemon.json
{
    
    
    "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
}

6. Generate and modify the default configuration file /etc/containerd/config.toml of containerd (reference 2)

containerd config default > /etc/containerd/config.toml

Change sandbox_image="registry.k8s.io/pause:3.6"
to sandbox_image="k8simage/pause:3.6"

Restart the containerd service

systemctl daemon-reload  
systemctl restart containerd.service 

Note: This step can solve failed to pull image \"registry.k8s.io/pause:3.6\"; Failed to create sandbox for pod: pull registry.k8s.io/pause:3.6 image and other problems (specific error issues can View the logs by running the following command: journalctl -xeu kubelet)
7. Turn off the firewall

systemctl disable firewalld
systemctl stop firewalld

8. Close selinux
# Temporarily disable selinux

setenforce 0

#Or close permanently, modify the /etc/sysconfig/selinux file settings

sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

9. Disable swap partition

swapoff -a

#Or permanently disable, comment out the line of swap in the /etc/fstab file

sed -i ‘s/.*swap.*/#&/’ /etc/fstab

10. Modify the kernel parameters
to pass the bridged IPv4 traffic to the iptables chain (some ipv4 traffic cannot go through the iptables chain, because a filter in the linux kernel, each traffic will pass through him, and then match whether it can enter the current application process To process, so it will cause traffic loss), configure the k8s.conf file (the k8s.conf file does not exist, you need to create it yourself)

cat<<EOF> /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
vm.swappiness=0                  
EOF

sysctl --system 

#Reload all system parameters, or use sysctl -p


Document 1: https://yebd1h.smartapps.cn/pages/blog/index?blogId=123605246&_swebfr=1&_swebFromHost=bdlite

2. Install and configure Kubernetes on all nodes (except those specified separately)

1. Configure Kubernetes Alibaba Cloud source

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2. Install kubeadm, kubectl, and kubelet (kubeadm and kubectl are both tools, and kubelet is a system service, reference 1)

yum install -y kubelet-1.14.2
yum install -y kubeadm-1.14.2

3. Start the kubelet service

systemctl enable kubelet && systemctl start kubelet

4. Generate the current version of the initialization configuration file to the /etc/kubernetes directory

kubeadm config print init-defaults > /etc/kubernetes/init-default.yaml

1) Specify the IP address that the kube-apiserver broadcasts to other components
This parameter needs to be set to the IP address of the master node to ensure that other nodes can access the kube-apiserver
: advertiseAddress: 1.2.3.4 -> advertiseAddress: [host ip( Intranet)]
This item is set according to the IP address of its master node. The setting of this machine is:

advertiseAddress:192.168.95.20

2) Specify the warehouse source for installing the image.
It is recommended to use a domestic image such as Alibaba Cloud:
imageRepository: registry.aliyuncs.com/google_containers
Note: We will encounter the following initialization problems at the beginning:

failed to pull image registry.k8s.io/kube-apiserver:v1.26.3

This setting combined with the following parameters of the kubeadm init command can solve the problem:

--image-repository=registry.aliyuncs.com/google_containers

That is, during initialization: kubeadm init --image-repository=registry.aliyuncs.com/google_containers
is initialized later
3) Edit /etc/hosts and add a line:

192.168.95.20 k8s.cnblogs.com #需根据自己主机的IP地址进行修改

In general, step 4 can solve the complicated problem of [kubelet-check] Initial timeout of 40s passed.
Reference 2:
https://blog.csdn.net/weixin_52156647/article/details/129765134
4) Unify the Cgroup Driver of Kubernetes and docker as systemd,
modify the /etc/docker/daemon.json file, and add the following parameters:

vim /etc/docker/daemon.json 

#In order to keep the docker configuration of all nodes consistent, the dockers of other nodes have also been changed

{
    
    
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],	
"exec-opts": ["native.cgroupdriver=systemd"]
}

reload docker

systemctl daemon-reload
systemctl restart docker

5. Before initializing Kubernetes initialization on the master node, run:

systemctl restart docker

kubelet command completion

echo "source <(kubectl completion bash)" >> ~/.bash_profile

source .bash_profile

Pull image
List the list of images required for Kubernetes cluster startup. These images include control plane components (such as kube-apiserver, kube-controller-manager, and kube-scheduler) and other necessary components (such as etcd, CoreDNS, etc.), and modify the tag to be consistent with the required version
.

kubeadm config images list

Set mirror source and script program

vim image.sh

#!/bin/bash
url=registry.cn-hangzhou.aliyuncs.com/google_containers
version=v1.14.2
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
  docker pull $url/$imagename
  docker tag $url/$imagename k8s.gcr.io/$imagename
  docker rmi -f $url/$imagename
done

run script

chmod u+x image.sh
./image.sh
docker images

docker images Check the images in the docker warehouse, and find that all the images start with registry.aliyuncs.com/google_containers/, and some of them are different from the image names required in the kubeadm config images list. We want to modify the image name, that is, re-tag the image

docker images

Show results:

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.10 k8s.gcr.io/kube-apiserver:v1.14.10

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.10 k8s.gcr.io/kube-controller-manager:v1.14.10

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.10 k8s.gcr.io/kube-scheduler:v1.14.10

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.10 k8s.gcr.io/kube-proxy:v1.14.10

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1

After modifying the tag, check again and find that the image name and version number are consistent with the list of images required for Kubernetes cluster startup listed in the "kubeadm config images list" command.

Another way we can also pull the images one by one

kubeadm config images list

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.40.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.40.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.40.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.40.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1

View the docker image again:

docker images

Reset the k8s cluster

kubeadm reset

To release the port occupation, delete the configuration file generated during the previous initialization, etc.
Then start to formally execute the cluster initialization:

kubeadm init --apiserver-advertise-address 192.168.95.20 --pod-network-cidr=10.244.0.0/16

Execution message appears: Your Kubernetes control-plane has initialized successfully!Cluster initialization completed successfully.


6. Configure the node
The following 3 commands use ordinary users:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

7. Pay attention to the command and secret key when joining from the node

kubeadm join 192.168.95.20:6443 --token 7em598.2cwgsvdgga5fohae \
    --discovery-token-ca-cert-hash sha256:9fca7635ebe04c5fe7eccb8c30974ff0e4f7cb08785d1132956be9a800ded442

Keep this key safe.

8. Check the running status of the node (NotReady status)

kubectl get nodes

3. Install and configure slave node Kubernetes

1. Install the slave node software and configure it according to the first and second steps above.
Basic configuration of host1:

docker version

Kubectl version
kubeadm version
kubelet version

2. Join the cluster from the node (use the root user)

kubeadm join 192.168.95.20:6443 --token 7em598.2cwgsvdgga5fohae \
--discovery-token-ca-cert-hash sha256:9fca7635ebe04c5fe7eccb8c30974ff0e4f7cb08785d1132956be9a800ded442

Note: This step generally encounters the following problems:

[ERROR CRI]: container runtime is not running:

This is because the containerd installed with an subpackage will disable it as a container runtime by default:

Solution:
1) Use systemctl status containerdView Status
Active: active (running) to indicate that the container is running normally

2) Check /etc/containerd/config.toml file, this is the configuration file when the container is running
3)

vim /etc/containerd/config.toml

If you see this line ：disabled_plugins : ["cri"],
comment this line with # or delete "cri":
#disabled_plugins : [“cri”] or
disabled_plugins : []
4) Restart the container runtime

systemctl restart containerd

Reference 3: https://blog.csdn.net/weixin_52156647/article/details/129758753

4. View the joined slave nodes on the master node and solve subsequent problems

kubectl get nodes

1. At this time, STATUS shows NotReady
solution:
1) Reinstall kubernetes-cni on all cluster nodes:

yum reinstall -y kubernetes-cni

2) Install the network on the master node:
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml, where 185.199.108.133 raw.githubusercontent.com Add etc/hosts
(Reference 5: https://www.cnblogs.com/sinferwu/p/12726833.html )
3. Problems when running kubectl get nodes
1) couldn't get current server API group list:
solved :
The command cannot be run as root.
2) kubernetes-admin problem
K8S input kubectl get nodes shows The connection to the server localhost:8080 was refused - did you specify the right host or port? The
reason for this problem is that the kubectl command needs to be run with kubernetes-admin. It may be caused by an unclean system environment, such as not completely clearing the configuration before reinstalling k8s.
Solution:
(1) Copy the /etc/kubernetes/admin.conf file generated after the master node is initialized to the corresponding directory of the slave node.

scp /etc/kubernetes/admin.conf  host1:/etc/kubernetes/

(2) Set environment variables on all nodes and update

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile 
source ~/.bash_profile

Check the real IP of raw.githubusercontent.com at https://www.ipaddress.com/.

run again

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

Install flannel successfully!

When the cluster node list is obtained through the kubectl get nodes command again, it is found that the host1 cluster status is always in the NotReady state, and an error is reported by checking the log:

journalctl -f -u kubelet

According to the log information, the reason for the error is that the kubelet configuration cannot be downloaded from /var/llib/kubelet/config.yaml.

The reason for the error is probably that I ran it without doing kubeadm init before systemctl start kubelet.
We can try to update the token and regenerate the token, the code is as follows:

kubeadm token create --print-join-command

Update the content of the token copy output on the master node g

, and run it in hsot1

to solve the problem successfully!
The clusters are all in the Ready state!

Reference Six: https://www.cnblogs.com/eastwood001/p/16318644.html

5. Test Kubernetes

1. Run on the master node:

kubectl create deployment nginx --image=nginx #创建一个httpd服务测试

kubectl expose deployment nginx --port=80 --type=NodePort #端口就写80,如果你写其他的可能防火墙拦截了

kubectl get svc,pod   #对外暴露端口

2. Access the Nginx homepage using the IP address of the master node and the reserved port:
for example:

192.168.95.20:21729

show connection failed
using the command

kubectl describe pod nginx-77b4fdf86c-krqtk

Show results:

open /run/flannel/subnet.env: no such file or directory 

Found that I was missing the relevant cni network configuration files.
We need to check each node carefully, including whether the main node exists /run/flannel/subnet.env. The content should be similar to the following:

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

By checking the error log, I found that I am missing the relevant cni network configuration file.

Create cni network related configuration files:

mkdir -p /etc/cni/net.d/

cat <<EOF> /etc/cni/net.d/10-flannel.conf
{
    
    "name":"cbr0","type":"flannel","delegate": {
    
    "isDefaultGateway": true}}
EOF

Here we use the cat command and the redirection operator (<<) to write {"name": "cbr0", "type": "flannel", "delegate": {"isDefaultGateway": true}} to / etc/cni/net.d/10-flannel.conf file.

mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/

cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=10.199.0.0/16
FLANNEL_SUBNET=10.199.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF

Here, between <<EOF and EOF, there are multiple lines of text, and each line contains the definition of an environment variable. Specifically:
FLANNEL_NETWORK=10.199.0.0/16 defines an environment variable named FLANNEL_NETWORK and sets it to 10.199.0.0/16.
FLANNEL_SUBNET=10.199.1.0/24 defines an environment variable named FLANNEL_SUBNET and sets it to 10.199.1.0/24.
FLANNEL_MTU=1450 defines an environment variable called FLANNEL_MTU and sets it to 1450.
FLANNEL_IPMASQ=true defines an environment variable called FLANNEL_IPMASQ and sets it to true.

If any node does not have the file, copy it and deploy it again. This error should not be reported.
Let's check it on the host1 node:

cat /run/flannel/subnet.env
cat  /etc/cni/net.d/10-flannel.conf

Through the command, you can see that the slave node already has related configuration files related to the cni network.
If these important configuration files are missing, errors will also be reported in the cluster log:

cni config uninitialized
5月 06 12:44:06 master kubelet[48391]: W0506 12:44:06.599700   48391 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
5月 06 12:44:07 master kubelet[48391]: E0506 12:44:07.068343   48391 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready

If there is no problem with the above configuration,
nginx is successfully displayed in the end.

Kubectl get nodes

The kubectl get nodes command here will return a table that contains information about all nodes in the cluster, such as node name, status, role (master/worker), etc.

browser url input

192.168.95.25:30722

Or enter CLUSTER-IP:

10.100.184.180

In this way, we have successfully deployed the Kubernetes cluster and completed the nginx deployment!

These are all personally tested, and can be successfully deployed according to the above normal operations. I wish you all the best.
Finally, I wish you all the best in the summer vacation, study solidly and thoroughly, and have fun.