008.Kubernetes Nginx binary deploy high availability

Nginx proxy to achieve a high availability kube-apiserver

1.1 Nginx high availability

kube-apiserver nginx proxy-based high-availability solutions.

The control node kube-controller-manager, kube-scheduler is to deploy multiple instances, so long as there is a normal example, high availability can be guaranteed;

Pod in the domain name service cluster uses K8S kubernetes access kube-apiserver, kube-dns will automatically parse out more kube-apiserver IP node, it is also highly available;

A nginx process, the back-end docking apiserver multiple instances from each node, nginx for them to do health checks and load balancing;

kubelet, kube-proxy, controller-manager, scheduler kube-apiserver access through a local Nginx (monitor 127.0.0.1), so the high availability kube-apiserver;

Thereby achieving K8S node (master node and worker nodes) availability based access kube-apiserver nginx 4 layer transparent proxy function.

1.2 download compiling Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# wget http://nginx.org/download/nginx-1.15.3.tar.gz
  3 [root@k8smaster01 work]# tar -xzvf nginx-1.15.3.tar.gz
  4 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
  5 [root@k8smaster01 nginx-1.15.3]# mkdir nginx-prefix
  6 [root@k8smaster01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module
  7 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
  8 [root@k8smaster01 nginx-1.15.3]# make && make install

Explanation:

• --with-stream: a transparent layer 4 turned forwards (TCP Proxy) function;
• --without-xxx: Close all other functions, dynamic link thus generated binary dependent minimum.
• [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
• [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v

Nginx after verifying compiler 1.3

  . 1 [k8smaster01 the root @ ~] /opt/k8s/work/nginx-1.15.3 CD #
   2 [@ k8smaster01 Nginx the root-1.15.3] -v # ./nginx-prefix/sbin/nginx
   . 3 Nginx Version: Nginx / 1.15.3
   . 4 [@ k8smaster01 nginx the root-1.15.3] # # LDD ./nginx-prefix/sbin/nginx view nginx dynamic link libraries
   . 5          Linux-vdso.so.1 => (0x00007ffdda980000)
   . 6          libdl.so. = 2> /lib64/libdl.so.2 (0x00007feb37300000)
   . 7          libpthread.so.0 => /lib64/libpthread.so.0 (0x00007feb370e4000)
   . 8          libc.so.6 => /lib64/libc.so.6 ( 0x00007feb36d17000)
   9          /lib64/ld-linux-x86-64.so.2 (0x00007feb37504000)

Note: Since only turned transparent layer 4 forwarding function, and the like in addition to the operating system dependent libc libraries outer core lib, lib no dependencies on other (e.g. libz, libssl etc.), in order to achieve the purpose of streamlining the compilation.

1.4 install and deploy Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  7   done						#创建Nginx目录
  8 [root@k8smaster01 ~]# cd /opt/k8s/work
  9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
 10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
 11   do
 12     echo ">>> ${master_ip}"
 13     scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
 14     ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
 15     ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
 16   done						#分发Nginx二进制

1.5 Configuration Nginx transparently forward the four

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# cat > kube-nginx.conf <<EOF
  3 worker_processes 1;
  4 
  5 events {
  6     worker_connections  1024;
  7 }
  8 
  9 stream {
 10     upstream backend {
 11         hash $remote_addr consistent;
 12         server 172.24.8.71:6443        max_fails=3 fail_timeout=30s;
 13         server 172.24.8.72:6443        max_fails=3 fail_timeout=30s;
 14         server 172.24.8.73:6443        max_fails=3 fail_timeout=30s;
 15     }
 16 
 17     server {
 18         listen 127.0.0.1:8443;
 19         proxy_connect_timeout 1s;
 20         proxy_pass backend;
 21     }
 22 }
 23 EOF
 24 [root@k8smaster01 ~]# cd /opt/k8s/work
 25 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
 26 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
 27   do
 28     echo ">>> ${master_ip}"
 29     Kube-nginx.conf the root @ SCP $ {} master_ip: / opt / K8S / Kube-Nginx / the conf / Kube-nginx.conf
 30    DONE # Nginx four transparent proxy distribution profile

1.6 Configuration Nginx system

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# cat > kube-nginx.service <<EOF
  3 [Unit]
  4 Description=kube-apiserver nginx proxy
  5 After=network.target
  6 After=network-online.target
  7 Wants=network-online.target
  8 
  9 [Service]
 10 Type=forking
 11 ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
 12 ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
 13 ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
 14 PrivateTmp=true
 15 Restart=always
 16 RestartSec=5
 17 StartLimitInterval=0
 18 LimitNOFILE=65536
 19 
 20 [Install]
 21 WantedBy=multi-user.target
 22 EOF

1.7 Distribution Nginx systemd

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     scp kube-nginx.service  root@${master_ip}:/etc/systemd/system/
  7   done

Two starts and verification

2.1 start Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
  7   done

2.2 Service check Nginx

  1 [root@k8smaster01 ~]# cd /opt/k8s/work
  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh
  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}
  4   do
  5     echo ">>> ${master_ip}"
  6     ssh root@${master_ip} "systemctl status kube-nginx |grep 'Active:'"
  7   done