An introduction to a cluster

1 Overview

According to the function is divided into two categories: high availability and load balancing
A high-availability cluster usually consists of two servers, one for work and the other for redundancy. When the machine providing the service goes down, the redundancy will take over and continue to provide services.
Open source software to achieve high availability: heartbeat, keepalived
A load balancing cluster requires a server as a distributor, which is responsible for distributing user requests to the back-end servers for processing. In this cluster, in addition to the distributor, it is the server that provides services to users. The number of these servers is at least 2
The open source software that implements load balancing includes LVS, keepalived, haproxy, and nginx, and the commercial ones include F5, Netscaler
in Alibaba, Tencent, and other big companies, which do not allow services to be unavailable, so high availability is used in some core businesses; heartbeat is in There are many bugs in centos6, and it has not been updated for a long time; keepalived not only has the function of high availability, but also the function of load balancing; the advantage of commercial load balancing software is that it has a high amount of concurrency. It can support, and its stability is very good; the stability of using open source software to build load balancing depends on the stability of the server

2. Introduction to keepalived

Here we use keepalived to achieve high availability cluster, because heartbeat has some problems on centos6, which affects the experimental effect
keepalived achieves high availability through VRRP (Virtual Router Redundancy Protocl).
In this protocol, multiple routers with the same function will be formed into a group, and there will be 1 master role and N (N>=1) backup roles in this group.
The master will send VRRP protocol packets to each backup through multicast. When the backup cannot receive the VRRP packets sent by the master, it will consider the master to be down. At this point, it is necessary to decide who becomes the new master according to the priority of each backup.
Keepalived has three modules, namely core, check and vrrp. The core module is the core of keepalived, which is responsible for the startup and maintenance of the main process and the loading and parsing of global configuration files. The check module is responsible for health checks, and the vrrp module is used to implement the VRRP protocol. VRRP is called Virtual Routing Redundancy Protocol

3. Configure a highly available cluster with keepalived

Prepare two machines 130 and 136, 136 as master and 130 as backup
Both machines execute yum install -y keepalived
nginx is installed on both machines, among which nginx has been compiled and installed on 136, and nginx needs to be installed with yum on 130: yum install -y nginx
uses keepalived to achieve high availability, in fact, to make a service on the server highly available, here we are Let nginx achieve high availability. The reason for using nginx is that many companies use nginx as a load balancer. Here we can use the following command to check whether niginx has been installed on the next 130

# rpm -qa |grep nginx         //查看是否存在nginx的rpm包
# sudo yum install epel-release         //如果系统中找不到nginx的rpm包，则需要执行这两行命令
# yum update
# yum install -y nginx

Set virtual IP that vip is 100
Edit the keepalived configuration file on 136.
In fact, there is a keepalived.conf configuration file in the /etc/keepalived/ directory. We need to clear the original content of the keepalived.conf file first, and use the following command to clear it:

# > /etc/keepalived/keepalived.conf

The result is as follows:

[root@aminglinux ~]# > !$
> /etc/keepalived/keepalived.conf
[root@aminglinux ~]# cat /etc/keepalived/keepalived.conf
[root@aminglinux ~]#

Here we need to use another configuration file template, the content is obtained from this link https://coding.net/u/aminglinux/p/aminglinux-book/git/blob/master/D21Z/master_keepalived.conf , the following Write the content to /etc/keepalived/keepalived.conf

global_defs { //Some global parameters will be defined here
notification_email { //If there is a problem, you need to send an email to the following mailbox [email protected] }
notification_email_from [email protected] //Define which mailbox sends the email
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_nginx { //Check whether the service is normal
script "/usr/local/sbin/check_ng.sh" //Use this script to detect, if it is detected that it is abnormal, it needs to be started
interval 3 //The interruption time of detection 3 seconds
}
vrrp_instance VI_1 { //Used to define some things related to the master
state MASTER //Define the role of this machine as master, if it is a slave, the value is backup
interface ens33 //Define the network card that broadcasts through vrrp
virtual_router_id 51 //Define the id of the router, the same as the value from above
priority 100 //Weight, the value from above is different
advert_int 1
authentication { //Authentication-related information
auth_type PASS //Define the authentication type as password authentication
auth_pass aminglinux>com //Define the password for password authentication
}
virtual_ipaddress { //Define a public virtual ip as the ip for domain name resolution. machine, bind this ip immediately from
192.168.75.100
}
track_script { //Load service
chk_nginx
}
}

136 Edit the monitoring script, the content is obtained from https://coding.net/u/aminglinux/p/aminglinux-book/git/blob/master/D21Z/master_check_ng.sh

#!/bin/bash #Time
variable for logging
d=`date --date today +%Y%m%d_%H:%M:%S` #Calculate
the number of nginx processes
n=`ps -C nginx --no-heading|wc -l` #If
the process is 0, start nginx, and check the number of nginx processes again, #If
it is still 0, it means that nginx cannot be started, and you need to close keepalived
if [ $n -eq " 0" ]; then
/etc/init.d/nginx start
n2=`ps -C nginx --no-heading|wc -l`
if [ $n2 -eq "0" ]; then
echo "$d nginx down, keepalived will stop" >> /var/log/check_ng.log
systemctl stop keepalived
fi
fi

Give the script 755 permission. If the script does not have this permission, the keepalived service will not start normally.

[root@aminglinux ~]# chmod 755 !$    
chmod 755 /usr/local/sbin/check_ng.sh

systemctl start keepalived 136 starts the service

[root@aminglinux ~]# systemctl start keepalived   
[root@aminglinux ~]# ps aux |grep keepalived  
root      1740  0.0  0.0 120740  1404 ?        Ss   01:31   0:00 /usr/sbin/keepalived -D   
root      1741  0.0  0.1 127476  3272 ?        S    01:31   0:00 /usr/sbin/keepalived -D   
root      1744  0.0  0.1 131780  3116 ?        S    01:31   0:00 /usr/sbin/keepalived -D   
root      1878  0.0  0.0 112676   988 pts/0    S+   01:32   0:00 grep --color=auto keepalived   
[root@aminglinux ~]# ps aux |grep nginx    
root      2365  0.0  0.0  20616   712 ?        Ss   01:33   0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf   
nobody    2366  0.2  0.1  23060  3292 ?        S    01:33   0:00 nginx: worker process   
nobody    2367  0.2  0.1  23060  3296 ?        S    01:33   0:00 nginx: worker process   
root      2381  0.0  0.0 112676   984 pts/0    S+   01:33   0:00 grep --color=auto nginx

At this time, if we stop nginx, the keepalived script will automatically restart nginx

[root@aminglinux ~]# /etc/init.d/nginx stop
Stopping nginx (via systemctl):                            [  确定  ]
[root@aminglinux ~]# ps aux |grep nginx
root     15301  0.0  0.0  20616   708 ?        Ss   02:41   0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
nobody   15302  0.0  0.1  23060  3292 ?        S    02:41   0:00 nginx: worker process
nobody   15303  0.0  0.1  23060  3280 ?        S    02:41   0:00 nginx: worker process
root     15308  0.0  0.0 112676   984 pts/0    S+   02:41   0:00 grep --color=auto nginx
[root@aminglinux ~]#

We can also use the ip addr command to see that there is one more IP 192.168.75.100 bound to the ens33 network card, which is the IP defined in keepalived.conf

[root@aminglinux ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.75.136/24 brd 192.168.75.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.150/24 brd 192.168.75.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::d652:b567:6190:8f28/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:ca brd ff:ff:ff:ff:ff:ff
[root@aminglinux ~]#

Next, let's configure the slave. Before configuring the slave, you need to check whether the firewall or selinux is configured on the master and the slave, and they need to be closed.

Edit the configuration file on 130, the content is obtained from https://coding.net/u/aminglinux/p/aminglinux-book/git/blob/master/D21Z/backup_keepalived.conf

global_defs {
notification_email {
[email protected]
} notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_nginx {
script "/usr/local/sbin/check_ng.sh"
interval 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass aminglinux>com
}
virtual_ipaddress {
192.168.188.100
}
track_script {
chk_nginx
}
}

Edit the monitoring script /usr/local/sbin/check_ng.sh on 130, the content is obtained from https://coding.net/u/aminglinux/p/aminglinux-book/git/blob/master/D21Z/backup_check_ng.sh

#Time variable for logging
d=`date --date today +%Y%m%d_%H:%M:%S` #Calculate
the number of nginx processes
n=`ps -C nginx --no-heading| wc -l` #If
the process is 0, start nginx and check the number of nginx processes again. #If
it is still 0, it means that nginx cannot be started, and you need to close keepalived
if [ $n -eq "0" ]; then
systemctl start nginx
n2=`ps -C nginx --no-headig|wc -l`
if [ $n2 -eq "0" ]; then
echo "$d nginx down,keepalived will stop" >> /var/log/check_ ng.log
systemctl stop keepalived
fi
fi

Give the script 755 permissions

[root@localhost ~]# chmod 755 !$
chmod 755 /usr/local/sbin/check_ng.sh
[root@localhost ~]#

Also start the service systemctl start keepalived on 130

[root@localhost ~]# systemctl start keepalived
[root@localhost ~]# ps aux |grep keep
root     10928  0.2  0.0 120740  1404 ?        Ss   02:20   0:00 /usr/sbin/keepalived -D
root     10929  0.2  0.1 120740  2756 ?        S    02:20   0:00 /usr/sbin/keepalived -D
root     10930  0.2  0.1 125104  2840 ?        S    02:20   0:00 /usr/sbin/keepalived -D
root     10945  0.0  0.0 112676   984 pts/1    S+   02:20   0:00 grep --color=auto keep
[root@localhost ~]#

After the configuration is complete, let's enter the IP 192.168.75.136 address to access the main server through the browser. At this time, the file /data/wwwroot/default/index.html is accessed, and then 192.168.75.130 is used to access the slave server and the default page accessed. It is /usr/share/nginx/html/index.html; we will use vip 192.168.75.100 to access again, and the result is the main server page

4. Test high availability

First determine the difference between nginx on the two machines, for example, you can check the nginx version by curl -I
Test 1: Shut down the nginx service
on the master Shutting down nginx on the master or slave, keepalived can automatically restart nginx, which depends on the monitoring script we wrote
Test 2: Add iptabls rules on the master
iptables -I OUTPUT -p vrrp -j DROP

[root@aminglinux ~]# iptables -I OUTPUT -p vrrp -j DROP
[root@aminglinux ~]# iptables -nvL
Chain INPUT (policy ACCEPT 165 packets, 12440 bytes)
 pkts bytes target     prot opt in     out     source               destination         
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
Chain OUTPUT (policy ACCEPT 105 packets, 11068 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   81  3240 DROP       112  --  *      *       0.0.0.0/0            0.0.0.0/0

Here we discard the packets sent by the master through the vrrp protocol, and the result is that the purpose of switching master and slave cannot be achieved.

Test 3: Shut down the keepalived service on the master

[root@aminglinux ~]# iptables -F
[root@aminglinux ~]# iptables -nvL
Chain INPUT (policy ACCEPT 12 packets, 840 bytes)
 pkts bytes target     prot opt in     out     source               destination         
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
Chain OUTPUT (policy ACCEPT 11 packets, 948 bytes)
 pkts bytes target     prot opt in     out     source               destination         
[root@aminglinux ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.75.136/24 brd 192.168.75.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.150/24 brd 192.168.75.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::d652:b567:6190:8f28/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:ca brd ff:ff:ff:ff:ff:ff
[root@aminglinux ~]# systemctl stop keepalived
[root@aminglinux ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.75.136/24 brd 192.168.75.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.150/24 brd 192.168.75.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::d652:b567:6190:8f28/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:ca brd ff:ff:ff:ff:ff:ff
[root@aminglinux ~]#

We can see that when the keepalived service on the master is turned off, similar to the host downtime, the IP 192.168.75.100 bound to the host is unbound, and now we go to the slave to check the ip addr, and we can see that the slave has been bound. 192.168.75.100 is the IP, use a browser to access this IP, and the page accessed is the slave page, indicating that the master and slave have been switched.

[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:0c:20:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.75.130/24 brd 192.168.75.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b44e:aca4:f738:7833/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost ~]# tail /var/log/messages
Apr 10 03:04:15 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:15 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:15 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:15 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:20 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:20 localhost Keepalived_vrrp[12554]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.75.100
Apr 10 03:04:20 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:20 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:20 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
Apr 10 03:04:20 localhost Keepalived_vrrp[12554]: Sending gratuitous ARP on ens33 for 192.168.75.100
[root@localhost ~]#

Test 4: Start the keepalived service on the master

[root@aminglinux ~]# systemctl start keepalived
[root@aminglinux ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.75.136/24 brd 192.168.75.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.75.150/24 brd 192.168.75.255 scope global secondary ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::d652:b567:6190:8f28/64 scope link 
       valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:21:5e:ca brd ff:ff:ff:ff:ff:ff
[root@aminglinux ~]#

Then we go to the slave machine to check the bound IP

[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:0c:20:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.75.130/24 brd 192.168.75.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::b44e:aca4:f738:7833/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

In the actual production environment, there may be one master and multiple slaves. At this time, we can configure different weight priorities for each slave in keepalived.conf. The higher the weight, the higher the priority.
In addition to the high availability of nginx, we also You can do high availability of mysql. If you want to do high availability of mysql, you must ensure that the master and slave data are consistent

Linux cluster architecture--configure high availability cluster with keepalived

An introduction to a cluster

1 Overview

2. Introduction to keepalived

3. Configure a highly available cluster with keepalived

4. Test high availability

Guess you like