suse 12 Compile and deploy Keepalived + nginx to provide high availability for kube-apiserver

IP	SERVICES
192.168.72.55	keepalived+nginx
192.168.72.56	keepalived+nginx
192.168.72.57	keepalived+nginx
192.168.72.100	VIP

• If you need to provide high availability for kube-apiserver, it is recommended to deploy keepalived+nginx in advance when deploying the kubernetes cluster.
  • Otherwise, you need to modify the kube-apiserver certificate and add vip to the certificate.
  • And amend kube-configthe relevant keys ( kube-controller, , kube-scheduler, kubelet) kube-proxyneed to be regenerated, will modify the original kube-apiserver certification for the VIP address
  • Restart kube-apiserver, kube-controller, kube-scheduler, kubelet, kube-proxyservices, and ultimately you can achieve high availability
  • Therefore, it will be much more convenient to complete keepalived+nginx before deploying the kubernetes cluster
• Why choose keepalived?
  • Because keepalived has a back-end service 健康检测mechanism, it detects that the back-end nginx (nginx is upstreamimplemented using modules 负载均衡) service is faulty, it will shut itself (keepalived), and make VIP 漂移to one of the other two nodes to ensure and implement kube -High availability of apiserver service
• The deployment is based on the front of my blog suse 12 二进制部署 Kubernetets 1.19.7series, the final distribution scripts and start the service inside the array to modify their own (modified to own ip or host name to the host name of the hosts need to be resolved in advance)

Compile and deploy nginx

Download the nginx source package

k8s-01:~ # cd /opt/k8s/packages/
k8s-01:/opt/k8s/packages # wget http://nginx.org/download/nginx-1.16.1.tar.gz
k8s-01:/opt/k8s/packages # tar xf nginx-1.16.1.tar.gz

Compile nginx

k8s-01:~ # cd /opt/k8s/packages/nginx-1.16.1/
k8s-01:/opt/k8s/packages/nginx-1.16.1 # ./configure --prefix=$(pwd)/nginx-prefix \
--with-stream \
--without-http \
--without-http_uwsgi_module && \
make && \
make install

Configure nginx.conf

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > kube-nginx.conf <<EOF
worker_processes 1;
events {
    
    
    worker_connections  1024;
}
stream {
    
    
    upstream backend {
    
    
        hash \$remote_addr consistent;
        server 192.168.72.55:6443        max_fails=3 fail_timeout=30s;
        server 192.168.72.56:6443        max_fails=3 fail_timeout=30s;
        server 192.168.72.57:6443        max_fails=3 fail_timeout=30s;
    }
    server {
    
    
        listen *:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

Configure nginx for systemctl management

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/server/kube-nginx/sbin/nginx \
          -c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \
          -p /opt/k8s/server/kube-nginx -t
ExecStart=/opt/k8s/server/kube-nginx/sbin/nginx \
       -c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \
       -p /opt/k8s/server/kube-nginx
ExecReload=/opt/k8s/server/kube-nginx/sbin/nginx \
        -c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \
        -p /opt/k8s/server/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

Distribute nginx binaries and configuration files

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "mkdir -p /opt/k8s/server/kube-nginx/{conf,logs,sbin}"
    scp /opt/k8s/packages/nginx-1.16.1/nginx-prefix/sbin/nginx ${host}:/opt/k8s/server/kube-nginx/sbin/
    scp /opt/k8s/conf/kube-nginx.conf ${host}:/opt/k8s/server/kube-nginx/conf/
    scp /opt/k8s/conf/kube-nginx.service ${host}:/etc/systemd/system/
done

Start kube-nginx service

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "systemctl daemon-reload && \
                      systemctl enable kube-nginx --now && \
                      systemctl status kube-nginx | grep Active"
done

Compile and deploy keepalived

Download the keepalived source package

k8s-01:~ # cd /opt/k8s/packages/
k8s-01:/opt/k8s/packages # wget https://www.keepalived.org/software/keepalived-2.2.0.tar.gz
k8s-01:/opt/k8s/packages # tar xf keepalived-2.2.0.tar.gz

Compile keepalived

k8s-01:/opt/k8s/packages/keepalived-2.2.0 # ./configure --prefix=$(pwd)/keepalived-prefix && \
make && \
make install

Configure keepalived.conf

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > keepalived.conf.template <<EOF
! Configuration File for keepalived
global_defs {
    
    
   router_id 192.168.0.50
}
vrrp_script chk_nginx {
    
    
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    
    
    state BACKUP
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip ##NODE_IP##
    nopreempt
    authentication {
    
    
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
    
    
         chk_nginx
    }
    virtual_ipaddress {
    
    
        192.168.72.100
    }
}
EOF

• In order to avoid problems with the keepalived service, restart keepalived after repairing, and the IP drifting back occurs. Here, three modes are selected as BACKUP to reduce data loss.

Create a health check script

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > check_port.sh <<"EOF"
CHK_PORT=$1
 if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=$(ss -lt|grep $CHK_PORT|wc -l)
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
 else
        echo "Check Port Cant Be Empty!"
 fi
EOF

Configure keepalived as systemctl management

k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > keepalived.service <<EOF
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived \$KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP \$MAINPID

[Install]
WantedBy=multi-user.target
EOF

Distribute keepalived binaries and configuration files

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

for (( i=0; i < 3; i++ ))
do
    sed -e "s/##NODE_IP##/${MASTER_IPS[i]}/" /opt/k8s/conf/keepalived.conf.template > \
           /opt/k8s/conf/keepalived.conf-${MASTER_IPS[i]}.template
done

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "mkdir -p /etc/keepalived"
    scp /opt/k8s/packages/keepalived-2.2.0/keepalived-prefix/sbin/keepalived ${host}:/usr/sbin/
    scp /opt/k8s/packages/keepalived-2.2.0/keepalived-prefix/etc/sysconfig/keepalived ${host}:/etc/sysconfig/
    scp /opt/k8s/conf/keepalived.conf-${host}.template ${host}:/etc/keepalived/keepalived.conf
    scp /opt/k8s/conf/check_port.sh ${host}:/etc/keepalived/
    scp /opt/k8s/conf/keepalived.service ${host}:/etc/systemd/system/
done

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "systemctl daemon-reload && \
                      systemctl enable keepalived --now && \
                      systemctl status keepalived | grep Active"
done

Check the machine where the VIP is located and whether it is pinged

#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh

for host in ${MASTER_IPS[@]}
do
    printf "\e[1;34m${host}\e[0m\n"
    ssh root@${host} "ip a | grep 192.168.72.100"
done

ping 192.168.72.100 -c 1