Flink on k8s environment construction (1)

Flink on k8s environment construction (2) - wangqiaoowq's blog - CSDN blog

In the process of setting up the Flink on Yarn environment, a lot of configuration needs to be done, and related components such as zookeeper Hadoop Yarn need to be built. The installation process is relatively complicated, and the reinstallation process is also complicated when there is a problem with the cluster, and the three nodes of Yarn can only Three resource managers and one node manager have been set up. When Flink jobs apply for resources, they can only apply for resources from NodeManager nodes. There is a hidden danger of resource bottlenecks overall (there will be more and more subsequent flink jobs). Now try to build a Flink on k8s environment .

Flink on Kubernetes (also known as Flink on K8s) refers to running the Apache Flink distributed stream processing framework on a Kubernetes cluster.

Kubernetes is an open source container orchestration platform that helps manage containerized applications and provides an elastic, scalable, and reliable deployment environment. Combined with Flink and Kubernetes, efficient large-scale data stream processing can be achieved.

Flink on K8s provides the following advantages:

1. Elastic scaling: Kubernetes can automatically expand and contract the task resources required by Flink jobs according to the load.
2. Containerized management: Flink jobs can run as containers and benefit from Kubernetes' containerized management functions, such as version management, lifecycle management, and monitoring.
3. Failure recovery: Kubernetes can automatically detect and replace failed nodes, provide high availability and fault tolerance, and ensure continuous operation of jobs.

To run Flink on Kubernetes, you can use the Kubernetes deployment tool officially provided by Apache Flink or other third-party tools such as Helm Chart.

By deploying Flink through Kubernetes, you can use the API and resource management functions of Kubernetes to effectively manage and deploy your Flink jobs. In this way, you can easily expand the scale of the Flink cluster and realize dynamic and automatic resource allocation and job scheduling.

K8s environment construction

 Deploy 1master and 2node nodes first, and then add a master node later

1. Preliminary work (all nodes)

1. Modify the host name

     

     

 2. Configure ssh mutual trust   

# just keep pressing
ssh-keygen

ssh-copy-id -i ~/.ssh/id_rsa.pub root@bigData05
ssh-copy-id -i ~/.ssh/id_rsa.pub root@bigData06
ssh-copy-id -i ~/.ssh/id_rsa.pub root@bigData08

3. Time synchronization

yum install chrony -y

systemctl start chronyd

systemctl enable chronyd

chronyc sources

3) Ensure the uniqueness of MAC address and product_uuid on each node

• You can use the command ip link to get the MAC address of a network interface
• You can use the sudo cat /sys/class/dmi/id/product_uuid command to verify product_uuid

Generally speaking, hardware devices will have unique addresses, but the addresses of some virtual machines may be repeated. Kubernetes uses these values ​​to uniquely identify nodes in the cluster. If these values ​​are not unique on each node, it may cause the installation to fail.

 

 4. Turn off the firewall

systemctl stop firewalld
systemctl disable firewalld

5. Close swap

#Temporarily closed; closing swap is mainly for performance considerations
swapoff -a
# You can use this command to check whether swap is closed
free

# Permanently close
sed -ri 's/.*swap.*/#&/' /etc/fstab

 6. Disable SELinux

# temporarily close
setenforce 0
# permanently disable
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

7. Allow iptables to inspect bridged traffic (optional)

• Make sure the br_netfilter module is loaded. This can be done by running lsmod | grep br_netfilter. To load this module explicitly, execute sudo modprobe br_netfilter.

• In order for iptables on your Linux nodes to properly see bridge traffic, you need to make sure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl configuration. All nodes execute the following commands:

cat /etc/modules-load.d/k8s.conf

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat /etc/sysctl.d/k8s.conf

# Set the required sysctl parameters, which persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge. bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF

# Apply sysctl parameters without restarting
sudo sysctl --system

Check: 

cat /etc/sysctl.d/k8s.conf

Repeat the above operation on the other 3 nodes

5) Check required ports

netstat -tuln | grab 10250

= "master node

protocol	direction	port range	effect	user
TCP	inbound	6443	Kubernetes API server	all components
TCP	inbound	2379-2380	etcd	server client API
TCP	inbound	10250	Kubelet API	kubelet itself, control plane components
TCP	inbound	10251	kube-scheduler	kube-scheduler itself
TCP	inbound	10252	kube-controller-manager	kube-controller-manager itself

= "node (work) node

protocol	direction	port range	effect	user
TCP	inbound	10250	Kubelet API	kubelet itself, control plane components
TCP	inbound	30000-32767	NodePort service	all components

 2) Install container docker (all nodes)

# 实方源源
cd /etc/yum.repos.d ; look a lot; See mv CentOS-*.repo/

# centos7 (use centos7 this time)  
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# centos8 (if you use centos8, you can download this)
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo

# Install yum-config-manager configuration tool
yum -y install yum-utils
# Set yum source
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker- ce.repo

# Install docker-ce version
yum install -y docker-ce

This is because docker has already been installed in the system. It may be that you have typed yum -y install docker before or installed it in other ways to cause conflicts. You can use yum list installed | grep docker to quickly check whether there are other installed docker programs , if there is any, uninstall all and then install it. 

yum list installed | grep docker

yum -y remove docker-client

 Installing docker-ce through yum reports an error because the machine has been installed many times, as shown below, you need to resolve the conflicting dependencies and reinstall

rpm -e docker-common.x86_64  或 yum -y remove docker-common

re-install

yum install -y docker-ce

  yum list installed | grep docker

 # Start
systemctl start docker
# Start
systemctl enable docker
# View version number
docker --version
# View version specific information
docker version

 # Docker mirror source settings
# Modify the file /etc/docker/daemon.json, create without this file
# After adding the following content, restart the docker service:
cat >/etc/docker/daemon.json<<EOF
{    "registry-mirrors ": ["http://hub-mirror.c.163.com"] } EOF

# load
systemctl reload docker 

# View
systemctl status docker containerd

 [Reminder] dockerd actually calls the api interface of containerd, and containerd is an intermediate communication component between dockerd and runC. So when the docker service is started, the containerd service will also be started.

3) Configure k8s yum source (all nodes)

cat > /etc/yum.repos.d/kubernetes.repo << EOF

[k8s]
name=k8s
enabled=1
gpgcheck=0
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
EOF
 

4) Set the sandbox_image mirror source to Alibaba Cloud google_containers mirror source (all nodes)

# Export the default configuration, the file config.toml does not exist by default
containerd config default > /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.toml

sed -i "s#registry.k8s.io/pause#registry.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.toml

 5) Configure containerd cgroup driver systemd (all nodes)

 Since kubernets v 1.24.0, docker.shim is no longer used, and containerd is used instead as the container runtime endpoint. Therefore, containerd needs to be installed (installed on the basis of docker), and containerd is automatically installed when docker is installed above. The docker here is just as a client. The container engine is still containerd.

If you use Docker as the K8S container runtime, kubelet needs to call Docker through dockershim first, and then call containerd through Docker.

If you use containerd as the K8S container runtime, since containerd has a built-in CRI (Container Runtime Interface: container runtime interface) plug-in, kubelet can directly call containerd.

grep SystemdCgroup /etc/containerd/config.toml

sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml

# After applying all changes, restart containerd
systemctl restart containerd

 6) Start installing kubeadm, kubelet and kubectl (master node)

• kubeadm : Commands used to initialize the cluster.

• kubelet : Used to start Pods and containers on each node in the cluster.

• kubectl : A command-line tool used to communicate with the cluster.

# Do not specify the version is the latest version, the latest version is 1.24.1

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Or specify the version
yum install -y kubelet-1.24.1 kubeadm-1.24.1 kubectl-1.24.1 --disableexcludes=kubernetes
# disableexcludes=kubernetes: ban other warehouses except this kubernetes

Install yum install -y kubernetes-cni

Systemctl restart containerd.service takes effect after restarting containerd

# Set to boot up and start the service immediately --now: start the service immediately

systemctl start kubelet

systemctl enable --now kubelet

# Check the status, here you need to wait for a while to check the service status, the startup will be a bit slow
systemctl status kubelet -l

Frequently asked questions are as follows:

1. It was found that it could not be started, and the k8s yum source was replaced

 2 These errors may be due to a Pod with the same name already existing and the creation failed

1. Delete existing Pod: If  pods "already exists" the error means that a Pod with the same name already exists in the cluster, you can try to delete the duplicate Pod, and then create the mirror Pod you want again.

   • Use  kubectl delete pod <pod名称> the command to delete an existing Pod. For example, kubectl delete pod my-pod a Pod named "my-pod" would be deleted.

   • Note that deleting a Pod may cause service disruption or application downtime for a period of time. Please choose the appropriate time to perform the delete operation according to your specific situation

kubectl delete pod kube-apiserver-bigdata07 -n kube-system --grace-period=0 --force

kubectl delete pod kube-scheduler-bigdata07-n kube-system --grace-period=0 --force

kubectl delete pod etcd-bigdata07 -n kube-system --grace-period=0 --force

kubectl get pods -o wide -n kube-system

3. The server has not been initialized

 This error message "GenericPLEG: Unable to retrieve pods" is a problem encountered by the Kubernetes Pod Lifecycle Event Generator (PLEG). "rpc error: code = Unknown desc = server is not initialized yet" in the error message means that the server has not been initialized yet.

This error can occur because Kubernetes control plane components such as kube-apiserver, kube-scheduler, or kube-controller-manager have not fully started or are in an unavailable state.

   

 kubectl get pod -n kube-system

view version

kubectl version

yum info kubeadm

7) Use kubeadm to initialize the cluster (master node)

It is best to download the image in advance, so that the installation is fast

docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.1
docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.1
docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.1
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.24.1
docker pull registry.aliyuncs.com/google_containers/pause:3.7
docker pull registry.aliyuncs.com/google_containers/etcd:3.5.3-0
docker pull registry.aliyuncs.com/google_containers/coredns:v1.8.6

Cluster initialization

kubeadm reset

rm -rf /etc/cni/net.d

rm -rf $HOME/.kube/config

kubeadm init \
--apiserver-advertise-address=192.168.1.247 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.27.3 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.100.0.0/16\
--ignore-preflight-errors=all

or

kubeadm init --apiserver-advertise-address=192.168.1.244 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.27.4 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16

#Indicates --service-cidr=10.96.0.0/12that the Kubernetes service will use IP addresses ranging from 10.96.0.0 to 10.111.255.255, and a total of 4096 subnets are available. Service CIDR is usually used to assign IP addresses to various internal services in the Kubernetes cluster.

# –image-repository string: This is used to specify where to pull the image (only available in version 1.13). The default value is k8s.gcr.io, which we specify as the domestic image address: registry.aliyuncs.com/ google_containers
# –kubernetes-version string: Specify the kubenets version number, the default value is stable-1, which will result in downloading the latest version number from https://dl.k8s.io/release/stable-1.txt, we can set it Specify a fixed version (v1.22.1) to skip network requests.
# –apiserver-advertise-address indicates which interface of the Master is used to communicate with other nodes of the Cluster. If the Master has multiple interfaces, it is recommended to specify them explicitly. If not specified, kubeadm will automatically select the interface with the default gateway. The ip here is the master node ip, remember to replace it.
# –pod-network-cidr specifies the range of the Pod network. Kubernetes supports a variety of network solutions, and different network solutions have their own requirements for –pod-network-cidr, here is set to 10.244.0.0/16 because we will use the flannel network solution, which must be set to this CIDR. --pod-network-cidr=10.100.0.0/16 Specifies that the CIDR range for the Pod network is 10.100.0.0 to 10.100.255.255. There are 65536 IP addresses in this range. The range of VIP addresses that can be selected is 10.100.0.1 to 10.111.255.254.

For example, use the following VIP

• IP address: 10.96.0.12
• Subnet mask: /24 (or 255.255.255.0)
• This means that the virtual IP address belongs to a host on the 10.96.0.0 network, where the available IP address range is 10.96.0.1 to 10.96.0.254 with a subnet mask of 255.255.255.0.

# --control-plane-endpoint cluster-endpoint is the custom DNS name mapped to this IP, here configure hosts mapping: 192.168.0.113 cluster-endpoint. This will allow you to pass --control-plane-endpoint=cluster-endpoint to kubeadm init and the same DNS name to kubeadm join. Later you can modify the cluster-endpoint to point to the address of the load balancer in the high availability scenario. 

Configure environment variables

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

vim /etc/profile

export KUBECONFIG=/etc/kubernetes/admin.conf

source /etc/profile

8) Install the Pod network plug-in (CNI: Container Network Interface) (master)

You must deploy a container network interface (CNI) based on the pod network plugin so that your pods can communicate with each other.

flannel failed to install successfully and changed to install calico  

# It is best to download the image in advance (all nodes)
docker pull quay.io/coreos/flannel:v0.14.0
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

 kubectl get pods -A

kubectl get nodes

Change to 10.100.0.0/16

kubectl apply -f kube-flannel.yml

kubectl get pods -A

kubectl get nodes

kubectl describe pods -n kube-system coredns-7bdc4cb885-b75bp

remove stain

Since it is a single node, the master does not accept task scheduling by default, and the taint needs to be deleted.

kubectl describe node bigdata08 |grep Taints

It is found that the master node has a stain (that is, the word NoSchedule), delete the stain

kubectl taint node bigdata08 node-role.kubernetes.io/master-

kubectl taint node bigdata08 node-role.kubernetes.io/control-plane-

modified calico  

Download calico wget https://docs.projectcalico.org/manifests/calico.yaml

Disable IPIP mode

Calico network, the default is ipip mode. A tunl0 network port will be created on each node host. This tunnel links all node container networks. The official website recommends different ip network segments.

We turn off IPIP mode here.

In the pod variable of calico-node in the DaemonSet part, modify the value of CALICO_IPV4POOL_IPIP to off

            - name: CALICO_IPV4POOL_IPIP

              #value: "Always"

              value: "off"

Modify the network segment of the pod

Modify CALICO_IPV4POOL_CIDR to the pod network segment of the k8s cluster

Also in DaemonSet, in the pod variable of calico-node, modify it as follows:

            - name: CALICO_IPV4POOL_CIDR

              value: "10.244.0.0/16"

kubectl apply -f calico.yam

kubectl get pods -A

9) The node node joins the k8s cluster

Install kubelet on the work node first

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Install yum install -y kubernetes-cni

It will take effect after restarting containerd systemctl restart containerd.service
# Set it to boot up and start the service immediately --now: start the service immediately
systemctl start kubelet

systemctl enable --now kubelet
systemctl status kubelet

on the master node

kubeadm token create --print-join-command
can generate:

kubeadm join 192.168.1.244:6443 --token kc0md9.e6s96yyukvuquque --discovery-token-ca-cert-hash sha256:6246e998f96106f9aea4509c1b162d7ffa578838e086c8bc6941390448191597

Add worker node (worker) 

Execute the following command on other nodes to join the k8s cluster:

#Run this command on all worker nodes to add the node to the cluster (if installed, you can first kubeadm reset)

kubeadm reset

rm -rf /etc/cni/net.d

rm -rf $HOME/.kube/config

kubeadm join 192.168.1.244:6443 --token kc0md9.e6s96yyukvuquque --discovery-token-ca-cert-hash sha256:6246e998f96106f9aea4509c1b162d7ffa578838e086c8bc6941390448191597

kubectl get nodes

 #The worker node cannot run the kubectl command, because the worker node does not have an admin.conf file
# If you want to use the kubectl command on the worker node, you need to copy the admin.conf configuration file to the worker node, and then execute the following command:
scp root@master: /etc/kubernetes/admin.conf /etc/kubernetes/
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile
source /etc/profile

Here you need to wait for a while, and then check the node node status, because you need to install kube-proxy and flannel.

on other nodes

kubeadm reset

kubeadm join 192.168.1.247:6443 --token no32d3.k1zmaqlnjoswitvz --discovery-token-ca-cert-hash sha256:6246e998f96106f9aea4509c1b162d7ffa578838e086c8bc6941390448191597
kubectl get nodes

scp /etc/kubernetes/admin.conf root@bigData08:/etc/kubernetes/

scp /etc/kubernetes/admin.conf root@bigData06:/etc/kubernetes/

scp /etc/kubernetes/admin.conf root@bigData05:/etc/kubernetes/

10) Configure IPVS

[Problem] Unable to ping ClusterIP (or ServiceName) in the cluster

1. Load ip_vs related kernel modules

 modprobe -- ip_vs
 modprobe -- ip_vs_sh
 modprobe -- ip_vs_rr
 modprobe -- ip_vs_wrr

All nodes verify that ipvs is enabled:

lsmod |grep ip_vs

 3. Edit the kube-proxy configuration file, and change the mode to ipvs

kubectl edit configmap -n kube-system kube-proxy

 

4. Restart kube-proxy

# First check
kubectl get pod -n kube-system | grep kube-proxy

# Then delete it to pull it up
kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'

# Check again
kubectl get pod -n kube-system | grep kube-proxy

5. View ipvs forwarding rules

ipvsadm -Ln

Look again at kubectl get nodes

11) Cluster high availability configuration (non-high availability can be ignored)

 Two options for configuring high availability (HA) Kubernetes clusters:

• Use stacked control plane nodes, where etcd nodes coexist with control plane nodes (used in this chapter), the architecture diagram is as follows:

Add a new machine here as another master node: 192.168.0.116 The configuration is the same as the above master node. Just don't need the last step of initialization.

 1. Modify the host name and configure hosts All nodes are configured as follows:

2. Configure ssh mutual trust

3. Time synchronization

 7. Turn off the firewall

4. Close swap

5. Disable SELinux
6. Allow iptables to inspect bridged traffic (optional, all nodes)

7. Install container docker (all nodes)

8. Configure k8s yum source (all nodes)

9. Set the sandbox_image mirror source to Alibaba Cloud google_containers mirror source (all nodes)

10. Configure the containerd cgroup driver

The above operations are the same as the previous master node, and should have been configured

12. Join the k8s cluster

k8s-add master node

Run the following command on the current only master node

first step:

kubeadm init phase upload-certs --upload-certs

7d1d592d13c91fb41be143558bd2ce7e0bf33a80e0233c8ca2582139762a8782
 

 Step two:

 kubeadm token create --print-join-command

kubeadm join 192.168.1.247:6443 --token keyixf.jr0ovq2zqz09tvur --discovery-token-ca-cert-hash sha256:6246e998f96106f9aea4509c1b162d7ffa578838e086c8bc6941390448191597

third step:

On the current master node:

View kubeadm-config.yaml
kubectl -n kube-system get cm kubeadm-config -oyaml
found no controlPlaneEndpoint
Add controlPlaneEndpoint
kubectl -n kube-system edit cm kubeadm-config
probably in such a position:
kind: ClusterConfiguration
kubernetesVersion: v1.18.0
controlPlaneEndpoint : 172.16.0.56:6443
and then execute the kubeadm join command on the node to be added as master

Step 2
Copy the certificate from master1 to master2

scp /etc/kubernetes/pki/ca. [email protected]:/etc/kubernetes/

scp /etc/kubernetes/pki/ca.* [email protected]:/etc/kubernetes/pki/

scp /etc/kubernetes/pki/sa.* [email protected]:/etc/kubernetes/pki/

scp /etc/kubernetes/pki/front-proxy-ca.* [email protected]:/etc/kubernetes/pki/

mkdir -p /etc/kubernetes/pki/etcd on master2

scp /etc/kubernetes/pki/etcd/ca.* [email protected]:/etc/kubernetes/pki/etcd/

scp /etc/kubernetes/pki/etcd/ca.* [email protected]:/etc/kubernetes/pki/etcd/

scp /etc/kubernetes/admin.conf [email protected]:/etc/kubernetes/

the fourth step:

Concatenate the obtained token and key to get the following command:

kubeadm join 192.168.1.247:6443 --token keyixf.jr0ovq2zqz09tvur --discovery-token-ca-cert-hash sha256:6246e998f96106f9aea4509c1b162d7ffa578838e086c8bc6941390448191597 --control-plane --certificate-key 7d1d592d13c91fb41be143558bd2ce7e0bf33a80e0233c8ca2582139762a8782

Execute spliced ​​commands on the new master node that needs to be added

kubeadm join 192.168.1.247:6443 --token keyixf.jr0ovq2zqz09tvur --discovery-token-ca-cert-hash sha256:6246e998f96106f9aea4509c1b162d7ffa578838e086c8bc6941390448191597 --control-plane --certificate-key 7d1d592d13c91fb41be143558bd2ce7e0bf33a80e0233c8ca2582139762a8782

 Notice: 

1. Add --control-plane --certificate-key, otherwise it will be added as node node instead of master

2. Do not deploy on the node when joining, if you deploy kubeadm reset and then join

       kubeadm reset

# The --control-plane flag tells kubeadm to join to create a new control plane. This flag must be added to join the master
# --certificate-key ... will cause the control plane certificate to be downloaded from the kubeadm-certs Secret in the cluster and decrypted using the given key. The value here is the key printed by the above command (kubeadm init phase upload-certs --upload-certs).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes

 kubectl get pods -A -owide

Although there are already two masters, there is only one entrance to the outside world, so a load balancer is required. If one master fails, it will automatically switch to the other master node.

 12) Deploy Nginx+Keepalived high availability load balancer

reference:

1. Install Nginx and Keepalived 

# Execute yum install nginx keepalived -y on the two master nodes

yum -y install epel-release

yum install nginx -y

2. Nginx configuration is configured on two master nodes

cat > /etc/nginx/nginx.conf << "EOF"

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {     worker_connections 1024; } # Layer 4 load Balanced, providing load balancing stream for two Master apiserver components {     log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';     access_log /var/log/nginx/k8s-access.log main;     upstream k8s-apiserver {     # Master APISERVER IP:PORT        server 192.168.1.247:6443;     # Master2 APISERVER IP:PORT        server 192.168.1.248:6443;     }     server {        listen 16443;














       proxy_pass k8s-apiserver;
    }
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    server {
        listen       80 default_server;
        server_name  _;

        location / {
        }
    }
}
EOF

3. Keepalived configuration (master)

cat > /etc/keepalived/keepalived.conf << EOF

Global_defs {    notification_email      {[email protected]      [email protected]      [email protected]    }    notification_from wangqiaowqo_07 @ 707 @707@ QQ.com SMTP_SERVER    127.0.0.1    SMTP_CONNECT_TIMEOUT 30    Router_id nginx_master } vRRP_Script Check_nginx {     script "/etc/keepal_nginx.sh" " } vrrp_instance VI_1 {     state MASTER     interface ens33     virtual_router_id 51 # VRRP routing ID instance, each instance is unique     priority 100 # Priority, standby server setting 90     advert_int 1 # Specify VRRP heartbeat packet notification interval, default 1 second     authentication {         auth_type PASS





















        auth_pass 1111
    }
    # 虚拟IP
    virtual_ipaddress {
        10.96.0.12/24
    }
    track_script {
        check_nginx
    }
}
EOF

• vrrp_script: Specify the script to check the working status of nginx (judging whether to failover according to the status of nginx)
• virtual_ipaddress: Virtual IP (VIP) Check nginx status script:

cat > /etc/keepalived/check_nginx.sh  << "EOF"
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
 
if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi
EOF
 

4. Keepalived configuration (backup)

cat > /etc/keepalived/keepalived.conf << EOF

Global_defs {    notification_email      {[email protected]      [email protected]      [email protected]    }    notification_from wangqiaowqo_07 @ 707 @707@ QQ.com SMTP_SERVER    127.0.0.1    SMTP_CONNECT_TIMEOUT 30    Router_id nginx_master } vRRP_Script Check_nginx {     script "/etc/keepal_nginx.sh" " } vrrp_instance VI_1 {     state MASTER     interface ens192 # Change to the network card of your own machine     virtual_router_id 51 # VRRP routing ID instance, each instance is unique     priority 100 # Priority, standby server setting 90     advert_int 1 # Specify VRRP heartbeat packet notification interval, default 1 second     authentication {




















        auth_type PASS
        auth_pass 1111
    }
    # 虚拟IP
    virtual_ipaddress {
        10.96.0.12/24
    }
    track_script {
        check_nginx
    }
}
EOF

cat > /etc/keepalived/check_nginx.sh  << "EOF"

#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    exit 1
else
    exit 0
fi

EOF

chmod +x /etc/keepalived/check_nginx.sh

5. Start and set the boot to start

systemctl daemon-reload

systemctl restart nginx && systemctl enable nginx && systemctl status nginx 

 1.nginx -V ensures that nginx is installed with –with -stream; if not, re-install with yum install nginx -y
2. Install yum -y install epel-release
3. yum -y install nginx-all-modules.noarch

4、nginx -t -c /etc/nginx/nginx.conf

systemctl restart nginx && systemctl enable nginx && systemctl status nginx

systemctl restart keepalived && systemctl enable keepalived && systemctl status keepalived

 ip a view Temporary VIP configuration is not successful, follow-up follow-up.

curl -k https://cluster-endpoint:16443/version

 2. Deployment of k8s management platform dashboard environment

But this can only be accessed internally, so to access externally, either deploy ingress or set the service NodePort type. Here select service to expose the port.

recommended.yaml is as follows:

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.6.0
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.8
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

Redeploy

kubectl delete -f recommended.yaml
kubectl apply -f recommended.yaml
kubectl get svc,pods -n kubernetes-dashboard -o wide
 create login user

cat >ServiceAccount.yaml<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

 kubectl apply -f ServiceAccount.yaml

Create and get a login token

kubectl -n kubernetes-dashboard create token admin-user
eyJhbGciOiJSUzI1NiIsImtpZCI6InhyNnZ4UmEwcG9uZFFidVFWR3VQWHRFSmxzUjNGTVppUS1mZ3F1NmtsdUkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjkwMjU2MTMzLCJpYXQiOjE2OTAyNTI1MzMsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiYTkxMGMyMGItNjZkYS00MmYzLWI5NTQtZTRhOGVmZmRhNTNiIn19LCJuYmYiOjE2OTAyNTI1MzMsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.vxsBgAdHTHkDbONYtop09bJGt2FQ2EOAjMEGj4U7VD6IkVjioh2xBdfQfglVD1q-r4qSdfB_d7URIOzkW1vqcckOPayfaDm0VNY2p3LmoklK0X1QtxQfiVr5zG_AfPfbv-D4r0zusMFOOjnicUThGIAIhGBS5GWWFjG3Dmz9tPtpekrqIFFbpk9JG8YvOecs7t-lb1iOidcU5PcpIl6wkgX0ERZ5Ye5B8CQvI7LIYXyXU9-EKTH6ZWrwWAGC5B4ZNYVb_m1IKx3X-k63rvRB4hgU39h11X5aZPi8tNCKiOPjeT49BWpmvFM_5q8A7yV-WGo2_ojG9UUS5il76Kc2qA

https://192.168.1.245:31443/

Click on the blank space of the page, and then enter thisisunsafe, the page will ask to enter the above token

----------------------------------------------------------------------------------------------------------