MySQL----MHA High Availability

1. MHA theory

1.1 What is MHA

MHA (MasterHigh Availability) is an excellent set of software for failover and master-slave replication in a MySQL high-availability environment.
The emergence of MHA is to solve the problem of MySQL single point.
During the MySQL failover process, MHA can automatically complete the failover operation within 0-30 seconds.
MHA can ensure data consistency to the greatest extent during the failover process to achieve high availability in the true sense.

1.2 Composition of MHA

• MHA Node (data node)
  MHA Node runs on each MySQL server.

• MHA Manager (management node)
  MHA Manager can be deployed on an independent machine to manage multiple master-slave clusters; it can also be deployed on a slave node.
  MHA Manager will regularly detect the master node in the cluster. When the master fails, it can automatically promote the slave with the latest data as the new master, and then re-point all other slaves to the new master. The entire failover process is completely transparent to the application.

1.3 Features of MHA

• During the automatic failover process, MHA tries to save the binary log from the downtime master server to ensure that the data is not lost to the greatest extent
• Using semi-synchronous replication can greatly reduce the risk of data loss. If only one slave has received the latest binary log, MHA can apply the latest binary log to all other slave servers, thus ensuring data consistency of all nodes
• At present, MHA supports one master and multiple slaves architecture, at least three servers, that is, one master and two slaves

2. One master and two slave deployment of MHA

experimental design

Satisfied requirements: Build a high-availability read-write replication mysql cluster with one master and two slaves. When the master server goes down, the slave server with the most updated data will replace the master server and occupy the VIP to ensure normal operation.

Experimental components:

node server	system	CPU name	IP address	Installation Tools and Services
MHA manager server	CentOS7.9(64 bit)	manager	192.168.44.100	MHA node and manager components
Master server	CentOS7.9(64 bit)	mysql1	192.168.44.101	MHA node components
Slave1 server	CentOS7.9(64 bit)	mysql2	192.168.44.102	MHA node components
Slave2 server	CentOS7.9(64 bit)	mysql3	192.168.44.103	MHA node components
VIP address: 192.168.44.150

1. Time synchronization
Perform time synchronization on the master and slave servers (online time synchronization)

2. Modify the host names of Master, Slave1, Slave2, and manager nodes
hostnamectl set-hostname mysql1
hostnamectl set-hostname mysql2
hostnamectl set-hostname mysql3
hostnamectl set-hostname manager

3. Turn off the firewalls of all servers and modify the hosts files of the master and slave servers

4. Modify the Mysql master configuration file /etc/my.cnf of the Master, Slave1, and Slave2 nodes

##Master 节点##
vim /etc/my.cnf
[mysqld]
server-id = 1
log_bin = mysql-bin
binlog_format = mixed
log-slave-updates = true
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

systemctl restart mysqld

##Slave1、Slave2 节点##
vim /etc/my.cnf
server-id = 2 						#三台服务器的 server-id 不能一样
log_bin = mysql-bin
binlog_format = mixed
log-slave-updates = true
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

systemctl restart mysqld

5. Create two soft links on the Master, Slave1, and Slave2 nodes

ln -s /usr/local/mysql/bin/mysql /usr/sbin/
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/

6. Configure mysql, one master and two slaves
(1) All database nodes authorize mysql
mysql -uroot -p
grant replication slave on . to 'myslave'@'192.168.80.%' identified by '123'; #Synchronous use from the database
grant all privileges on . to 'manager'@'192.168.80.%' identified by 'mha123'; #manager use

grant all privileges on . to 'manager'@'mysql1' identified by 'mha123 ' ; #Prevent the slave library from being unable to connect to the main library through the host name
grant all privileges on . grant all privileges on . to 'manager'@'mysql3' identified by 'mha123'; flush privileges;

(2) View binary files and synchronization points on the Master node

show master status;

(3) Perform synchronization operations on Slave1 and Slave2 nodes

change master to master_host='192.168.44.100',master_user='myslave',master_password='123',master_log_file='mysql-bin.000002',master_log_pos=1761; 

start slave;

(4) Two slave libraries must be set to read-only mode:

set global read_only=1;

7. Install MHA software
(1) Install the MHA-dependent environment on all servers, first install the epel source

yum install epel-release --nogpgcheck -y

yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN

(2) To install the MHA software package, you must first install the node component on all servers
. For each operating system version is different, here CentOS7.4 must choose version 0.57.
The node component must be installed on all servers first, and finally the manager component must be installed on the MHA-manager node, because the manager depends on the node component.

cd /opt
tar zxvf mha4mysql-node-0.57.tar.gz
cd mha4mysql-node-0.57
perl Makefile.PL
make && make install

(3) Install the manager component on the MHA manager node

cd /opt
tar zxvf mha4mysql-manager-0.57.tar.gz
cd mha4mysql-manager-0.57
perl Makefile.PL
make && make install

After the #manager component is installed, several tools will be generated under /usr/local/bin, mainly including the following:
masterha_check_ssh checks the SSH configuration status of MHA
masterha_check_repl checks MySQL replication status
masterha_manger starts the manager script
masterha_check_status checks the current MHA running status
masterha_master_monitor detection Master is down or not
masterha_master_switch Control failover (automatic or manual)
masterha_conf_host Add or delete configured server information
masterha_stop Close manager
#node After the component is installed, several scripts will be generated under /usr/local/bin (these tools are usually provided by MHAManager The script triggers without manual operation) mainly as follows:
save_binary_logs Save and copy master’s binary log
apply_diff_relay_logs Identify difference relay log events and apply the difference events to other slave
filter_mysqlbinlog Remove unnecessary ROLLBACK events (MHA is no longer use this tool)
purge_relay_logs purge relay logs (does not block SQL thread)

8. Configure passwordless authentication on all servers
(1) Configure passwordless authentication to all database nodes on the manager node

ssh-keygen -t rsa 				#一路按回车键
yum -y install sshpass         #安装工具
ssh-copy-id 192.168.44.101
ssh-copy-id 192.168.44.102
ssh-copy-id 192.168.44.103

Modify the /etc/ssh/ssh_conf file

Modify the /etc/ssh/ssh_conf file

(2) Configure passwordless authentication to database nodes mysql2 and mysql3 on mysql1

ssh-keygen -t rsa
ssh-copy-id 192.168.44.102
ssh-copy-id 192.168.44.103

(3) Configure passwordless authentication to database nodes mysql1 and mysql3 on mysql2

ssh-keygen -t rsa
ssh-copy-id 192.168.44.101
ssh-copy-id 192.168.44.103

(4) Configure passwordless authentication to database nodes mysql1 and mysql2 on mysql3

ssh-keygen -t rsa
ssh-copy-id 192.168.44.101
ssh-copy-id 192.168.44.102

9. Configure MHA on the manager node

(1) Copy the relevant scripts to the /usr/local/bin directory on the manager node

cp -rp /opt/mha4mysql-manager-0.57/samples/scripts /usr/local/bin
//拷贝后会有四个执行文件
ll /usr/local/bin/scripts/

master_ip_failover #Script of VIP management during automatic switching
master_ip_online_change #Management of VIP during online switching
power_manager #Script to shut down the host after failure occurs
send_report #Script to send alarm after failover

(2) Copy the above-mentioned VIP management script during automatic switching to the /usr/local/bin directory, where the master_ip_failover script is used to manage VIP and failover

cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin

(3) The modified content is as follows: (Delete the original content, directly copy and modify vip related parameters. You can enter: set paste before copying to solve the problem of vim pasting out of order)
vim /usr/local/bin/master_ip_failover

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
 
use Getopt::Long;
 
my (
    $command, $orig_master_host, $orig_master_ip,$ssh_user,
    $orig_master_port, $new_master_host, $new_master_ip,$new_master_port,
    $orig_master_ssh_port,$new_master_ssh_port,$new_master_user,$new_master_password
);
 
# 这里定义的虚拟IP配置要注意，这个ip必须要与你自己的集群在同一个网段，否则无效
my $vip = '192.168.44.150/24';
my $key = '1';
# 这里的网卡名称 “ens33” 需要根据你机器的网卡名称进行修改
# 如果多台机器直接的网卡名称不统一，有两种方式，一个是改脚本，二是把网卡名称修改成统一
# 我这边实际情况是修改成统一的网卡名称
my $ssh_start_vip = "sudo /sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "sudo /sbin/ifconfig ens33:$key down";
my $ssh_Bcast_arp= "sudo /sbin/arping -I ens33 -c 3 -A $vip";

GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'orig_master_ssh_port=i' => \$orig_master_ssh_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
    'new_master_ssh_port' => \$new_master_ssh_port,
    'new_master_user' => \$new_master_user,
    'new_master_password' => \$new_master_password
 
);
 
exit &main();
 
sub main {
    $ssh_user = defined $ssh_user ? $ssh_user : 'root';
    print "\n\nIN SCRIPT TEST====$ssh_user|$ssh_stop_vip==$ssh_user|$ssh_start_vip===\n\n";
 
    if ( $command eq "stop" || $command eq "stopssh" ) {
 
        my $exit_code = 1;
        eval {
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn "Got Error: $@\n";
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {
 
        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
        &start_arp();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
        print "Checking the Status of the script.. OK \n";
        exit 0;
    }
    else {
        &usage();
        exit 1;
    }
}
 
sub start_vip() {
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub stop_vip() {
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
 
sub start_arp() {
    `ssh $ssh_user\@$new_master_host \" $ssh_Bcast_arp \"`;
}
sub usage {
    print
    "Usage: master_ip_failover --command=start|stop|stopssh|status --ssh_user=user --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

(4) Create the MHA software directory and copy the configuration file, here use the app1.cnf configuration file to manage the mysql node server

Created in the manager server

 mkdir -p /opt/mysql-mha/mha

Created in all servers

 mkdir -p /opt/mysql-mha/mha-node

Create a file in the manager server

vim /opt/mysql-mha/mysql-mha.cnf
[server default]
manager_log=/opt/mysql-mha/manager.log
manager_workdir=/opt/mysql-mha/mha
master_binlog_dir=/usr/local/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
user=manager
password=mha123
port=3306
ping_interval=1
remote_workdir=/opt/mysql-mha/mha-node
repl_user=myslave
repl_password=123456
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.44.102 -s 192.168.44.103
shutdown_script=""
ssh_user=root

[server1]
hostname=192.168.44.101
port=3306

[server2]
candidate_master=1
check_repl_delay=0
hostname=192.168.44.102
port=3306

[server3]
hostname=192.168.44.103
port=3306

[server default]
manager_log=/opt/mysql-mha/manager.log #Specify the manager log path
manager_workdir=/opt/mysql-mha/mha #Specify the manager working directory
master_binlog_dir=/usr/local/mysql/data #Specify the master to save The location of the binlog, the path here must be consistent with the path of the binlog configured in the master, so that MHA can find
master_ip_failover_script=/usr/local/bin/master_ip_failover #Setting the switching script for automatic failover, which is the script above
master_ip_online_change_script=/ usr/local/bin/master_ip_online_change #Set the switching script for manual switching
user=manager #Set the account password for mha to access the database
password=mha123 #Set the account password for mha to access the database
ping_interval=1 #Set the time to monitor the main library and send ping packets Interval, the default is 3 seconds, and failover will be performed automatically when there is no response after three attempts
remote_workdir=/opt/mysql-mha/mha-node #Specify the working directory of mha on the remote node
repl_user=myslave #Set the user for master-slave replication
repl_password= 123456 #Set the user password for master-slave replication
report_script=/usr/local/send_report #Setting to send an email reminder when a failover occurs
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.44.102 -s 192.168.44.103 #Specify the slave server IP address to check
shutdown_script="" #Set the script to close the failed host after the failure occurs (the main function of the script is to shut down the host to prevent split brain, which is not used here) ssh_user=root #Set the
login user name of ssh
[server1]
hostname=192.168.44.101
port=3306
[server2 ]
hostname=192.168.44.102
port=3306
candidate_master=1
#Set as a candidate master. After setting this parameter, the slave library will be promoted to the master library after the master-slave switch occurs, even if the slave library is not the latest slave in the cluster
check_repl_delay =0
#By default, if a slave lags behind the master by more than 100M relay logs, MHA will not select the slave as a new master, because it takes a long time to restore the slave; by setting check_repl_delay=0, MHA Trigger switchover will ignore the replication delay when selecting a new master. This parameter is very useful for hosts with candidate_master=1, because the candidate master must be the new master during the switchover process
[server3]
hostname=192.168.44.103
port=3306

10. Add vip to the main server

ifconfig ens33:1 192.168.44.150/24

11,10. Test the ssh passwordless authentication on the manager node. If it is normal, it will output successfully at last, as shown below.

masterha_check_ssh -conf=/opt/mysql-mha/mysql-mha.cnf

12. Test the master-slave connection of mysql on the manager node, and the words MySQL Replication Health is OK appear at the end, indicating that it is normal. As follows.

masterha_check_repl -conf=/opt/mysql-mha/mysql-mha.cnf

13, 12. Start MHA on the manager node

nohup masterha_manager \
--conf=/opt/mysql-mha/mysql-mha.cnf \
--remove_dead_master_conf \
--ignore_last_failover < /dev/null > /var/log/mha_manager.log 2>&1 &

–remove_dead_master_conf: This parameter means that when the master-slave switch occurs, the ip of the old master library will be removed from the configuration file.
–ignore_last_failover: By default, if MHA detects continuous downtimes and the interval between two downtimes is less than 8 hours, Failover will not be performed. The reason for this limitation is to avoid the ping-pong effect. This parameter means to ignore the file generated by the last MHA trigger switch. By default, after the MHA switch occurs, it will be recorded in the app1.failover.complete log file. If the file is found to exist in the directory when the switch is made again next time, it will not be allowed Trigger switchover, unless the file is deleted after the first switchover, for convenience, here is set to –ignore_last_failover.
●Use & run the program in the background: the result will be output to the terminal; use Ctrl+C to send the SIGINT signal, the program is immune; close the session and send the SIGHUP signal, the program is closed.
●Use nohup to run the program: the result will be output to nohup.out by default; use Ctrl+C to send a SIGINT signal, and the program will close; close the session and send a SIGHUP signal, and the program will be immune.
● Use nohup and & to start the program nohup ./test &: Simultaneously immune to SIGINT and SIGHUP signals.

14. Check the MHA status, you can see that the current master is the mysql1 node.

masterha_check_status --conf=/opt/mysql-mha/mysql_mha.cnf

15. Check the MHA log, and you can see that the current master is 192.168.80.10, as shown below.

cat /opt/mysql-mha/manager.log | grep "current master"

16. Check whether the VIP address 192.168.80.200 of mysql1 exists. This VIP address will not disappear because the manager node stops the MHA service.
ifconfig

//To close the manager service, you can use the following command.
masterha_stop --conf=/opt/mysql-mha/mysql_mha.cnf
or can be closed directly by killing the process ID.

Fault simulation

1. Monitor and observe log records on the manager node

2. Stop the mysql service on the Master node mysql1

systemctl stop mysqld

manager log:

slave1：

slave2：

#After a normal automatic switch, the MHA process will exit. HMA will automatically modify the content of the app1.cnf file and delete the downtime mysql1 node. Check whether mysql2 takes over the Algorithm of VIP
ifconfig
failover alternative master database:
1．Generally, the slave library is judged from (position/GTID) to judge whether it is good or bad, and the data is different. The slave closest to the master becomes the candidate master.
2. If the data is consistent, select an alternative main library according to the order of the configuration files.
3. Set the weight (candidate_master=1), and the candidate master is forced to be designated according to the weight.
(1) By default, if a slave lags behind the master's relay logs by 100M, it will fail even if it has weight.
(2) If check_repl_delay=0, even if there are many logs behind, it is forced to be selected as the backup master.

View /opt/mysql-mha/mysql-mha.cnf file

Troubleshooting steps:

1. repair mysql
systemctl restart mysqld

2. Repair the master-slave # View the binary file and synchronization point show master status
on the current master library server mysql2 ;

#Execute synchronous operation on the original master database server mysql1
change master to master_host='192.168.44.101', master_user='myslave', master_password='123456', master_log_file='mysql-bin.000001', master_log_pos=2647;

start slave;