Table of contents
concept
deploy
Install MHA
Configure SSH key pair verification
Install mysql
Detection
test failure
repair
concept
MHA (Master High Availability) is currently a relatively mature solution for MysaL high availability. It was developed by vousimaton employees of Japan's DeNA Company (now working at Facebook). It is an excellent set of failover solutions for MysaL high availability environments. High-availability software with enhanced master-slave roles. During the MysaL failover process, MHA can automatically complete the master-slave failover operation of the database within 0 to 30 seconds, and during the failover process, MHA can ensure data consistency to the greatest extent to ensure Achieve true high availability.
MHA consists of two parts: MHA Manager (management node) and MHA Node (data node). MHA Manager
can be deployed on an independent machine to manage multiple master-slave clusters, or it can be deployed on a slave node. MHA Node runs on each MysaL server and Manager server. MHA Manager will regularly detect the master node in the cluster. When the master fails, it can automatically promote the slave with the latest data to the new master, and then all other The slave repoints to the newly promoted master. The entire failover process is completely transparent to the application level.
931 Oral Zhema Scripture Examination·Oral Content Qualification
During the MHA automatic failover process, MHA will try to save the binary log from the main server of the container machine to ensure that data is not lost to the greatest extent, but this operation is probabilistic. For example, if the primary server hardware fails or is inaccessible via ssh, MHA cannot save the binary logs and only fails over, thereby losing the latest data. Using MvSaL
5.5's semi-synchronous replication, you can reduce the risk of data loss. MHA can be combined with semisynchronous replication. If only one slave has received the latest binary log, MHA can apply the latest binary log to all other slave servers, thus ensuring data consistency on all nodes.
Currently, MHA mainly supports a one-master, multiple-slave architecture. To build MHA, a MysaL replication cluster must have
at least three database servers, one master and two slaves, that is, one acts as the master, one acts as the backup master, and the other acts as The slave database requires at least three servers. Due to machine cost considerations, Taobao has also modified it on this basis. Currently, Taobao TMHA supports one master and one slave. In addition, for those who want to set up quickly, you can refer to: MHA Quick Setup. In fact, we can also use 1 master and 1 slave for our own use, but the master host cannot be switched after it is down, and the binlog
cannot be completed .
After the master's mysgld process crashes, the switch can still be successful and the binlog can be completed.
Workflow
(1) Try to save binary log events from the crashed master
(2) Identify the slave server with the latest update:
(3) Apply the difference relay log to other slaves;
(4) Apply binary log events (binlog events) saved from the master;
(5) Promote a slave to the new master server:
(6) Point other slave connections to the new master for master-slave replication:
MHA software consists of two parts, Manager tool kit and Node tool kit. The specific instructions are as follows
The Manager toolkit mainly includes the following tools:
masterha_check_ssh Check the SSH configuration status of MHA
masterha_check_rep Check the MysQL replication status
masterha_manger Start MHA
masterha_check_status Check the current MHA running status
masterha_master_monitor Check whether the master is down
masterha_master_switch Control failover (automatic or manual)
masterha_conf_host Add or delete configured server information
The Node toolkit (these tools are usually triggered by MHA Manager scripts and do not require human operation) mainly includes the following tools:
save binary_logs Save and copy the master's binary log
apply diff relay logs Identify differential relay log events and apply the differential events to other
slaves
filter_mysqlbinlog Remove unnecessary ROLLBACK events (MHA no longer uses this tool)
purge relay_ logs Clear relay log (does not block SQL thread)
deploy
Install MHA
Prepare environment
5 virtual machines
Five virtual machines were renamed to better divide them.
Instruction: hostnamectl set-hostname server
Refresh: bash
Manager 
main server 
From server 1 
From server 2 
From server 3 
All machine installation
Turn off firewall
View yum, view CD mounting
Download epel-release source
You can directly pull it into the virtual machine
Then use rpm -ivh to install
rpm -ivh epel-release-latest-7.noarch.rpm
Installing Alibaba source files is also directly pulled into the virtual machine.
You can also install it using the following command
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
Then install MHA
yum install -y perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker
Install MHA node
If there is a problem with rz transmission, you can directly pull it into the virtual machine.
cd to find the node software package and just unzip it to the current directory (all five virtual machines need to be operated)
tar xf mha4mysql-node-0.56.tar.gz
Enter the unzipped directory to execute
Configuration via perl source
perl Makefile.PL
Then compile and install
make && make install
After MHA Node is installed, the following scripts (4 script files) will be generated in /usr/local/bin
ls -l /usr/local/bin/
Install MHA Manger
Only one machine needs to be installed (server1)
Install the perl module that MHA Manger depends on
yum install -y perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes
Then you need to install the perl package to pull it into the virtual machine
Or open a window with rz and pull it into the virtual machine
perl-Config-Tiny-2.14-7.el7.noarch.rpm
ls view
Install using rpm -ivh
perl-Config-Tiny-2.14-7.el7.noarch.rpm
Install MHA Manger package
rz opens the window and pulls into the virtual machine
mha4mysql-manager-0.56.tar.gz
Then operate the next command
tar xf mha4mysql-manager-0.56.tar.gz unzip
cd mha4mysql-manager-0.56/ enter the directory
perl Makefile.PL compiles files in perl language
make && make install compile and install
After the installation is complete, there will be the following script files (13 script files)
ls -l /usr/local/bin/
Configure SSH key pair verification
[Note that each virtual machine must implement password-free login]! ! !
First analyze the configuration of each virtual machine
vim /etc/hosts
Generate key
ssh-keygen
Can be generated 4 times continuously in a loop
for i in 115 116 117 126;do ssh-copy-id 192.168.1.$i;done
Verify password-free login
ssh server1...5
Install mysql
server2....5
Four hosts need to be installed
yum -y install mariadb mariadb-server
Then start the system
systemctl start mariadb
After restarting
Enter the command to view the port
netstat -lnpt | grip :3306
input the command
mysqladmin -u root password 123456 sets the initial password for the database
Set the master-slave replication configuration file
vim /etc/my.cnf
【server2】Master
server-id = 1
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0
【server3】auxiliary master
server-id = 2
log-bin=master-bin
log-slave-updates=true
relay_log_purge=0
【server4】from
server-id=3
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0
【server5】from
server-id=4
log-bin=mysql-bin
relay-log=slave-relay-bin
log-slave-updates=true
relay_log_purge=0
Restart the server
systemctl restart mariadb
Authorize
Server2...4 databases all require authorization
Create authorized users
4 virtual machines log in to mysql
mysql -uroot -p123456
then authorize
grant replication slave on *.* to 'repl'@'192.168.1.%' identified by '123456';
Refresh authorization
flush privileges;
Check main library status
show master status;
Then three slave servers designate the master server
Shut down the slave server first
stop slave;
Enter the following
CHANGE MASTER TO
MASTER_HOST='192.168.1.115',
MASTER_USER='repl',
MASTER_PASSWORD='123456',
MASTER_LOG_FILE='mastar-binlog.000001',
MASTER_LOG_POS=472;
Enter the following command to view the replication status in the MySQL database
show slave status\G
【! Notice! ! ! If something goes wrong, try this command to troubleshoot]
stop slave;
reset slave;
set global sql_slave_skip_counter =1 ;
start slave
Set read_only status from server
After doing this, the slave server will no longer be able to write anything.
Writing outside the library
mysql -uroot -p123456 -e 'set global read_only=1;'
How to write Curry
set global read_only=1;
Create monitoring user server2....5 to serve the monitoring node with higher permissions
grant all privileges on *.* to 'root'@'192.168.1.%' identified by '123456';
Refresh permissions
flush privileges;
Next step is to authorize your own host name.
They are:
grant all privileges on *.* to 'root'@'server2' identified by '123456';
grant all privileges on *.* to 'root'@'server3' identified by '123456';
grant all privileges on *.* to 'root'@'server4' identified by '123456';
grant all privileges on *.* to 'root'@'server5' identified by '123456';
Return to server1
Create /etc/masterha directory Create configuration directory Copy template files
mkdir /etc/masterha
Then copy mha4mysql-manager-0.56 under root to the directory just created.
cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha
Modify the file configuration just copied
vim /etc/masterha/app1.cnf
[server default]
manager_workdir=/var/log/masterha/app1
manager_log=/var/log/masterha/app1/manager.log
master_binlog_dir=/var/lib/mysql
master_ip_failover_script= /usr/local/bin/master_ip_failoverpassword=123456
user=root
ping_interval=1
remote_workdir=/tmp
reql_password=123456
reql_user=repl
[server1]
hostname=server2
port=3306
[server2]
hostname=server3
candidate_master=1
port=3306
check_repl_delay=0
[server3]
hostname=server4
port=3306[server4]
hostname=server5
port=3306
Configure failover script
After opening, copy the script content into it to modify the VIP and modify the network card name.
vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port,
);
my $vip = '192.168.200.100';
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down";
$ssh_user = "root";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
#eval {
# print "Disabling the VIP on old master: $orig_master_host \n";
# &stop_vip();
# $exit_code = 0;
#};
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
#my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`;
#if ( $ping le "90.0%"&& $ping gt "0.0%" ){
#$exit_code = 0;
#}
#else {
&stop_vip();
# updating global catalog, etc
$exit_code = 0;
#}
};if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_ip \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --
new_master_host=host --new_master_ip=ip --new_master_port=port\n"; }
After the configuration is completed, give it x execution permissions
chmod +x /usr/local/bin/master_ip_failover
Detection
Check whether the MHA ssh communication status is normal
masterha_check_ssh --conf=/etc/masterha/app1.cnf
Returning successfully means there is no problem
Check the status of the entire cluster
masterha_check_repl --conf=/etc/masterha/app1.cnf
[If 'NET OK' appears, you can follow the above steps to check whether the commands are typed correctly or whether the letters are typed incorrectly]
Check manager status
masterha_check_status --conf=/etc/masterha/app1.cnf
If normal, "PING_OK" will be displayed.
NOT_RUNNING", indicating that MHA monitoring is not enabled.
Turn on monitoring and check to see if it is turned on.
Enable manager monitoring
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1 &
--remove_dead_master_conf When a master-slave switch occurs, the IP address of the old master library will be removed from the configuration file.
--manger_log log storage location
--ignore_last_failover
By default, if MHA detects continuous downtime and the interval between two downtimes is less than 8 hours, Failover will not be performed. The reason for this restriction is to avoid the ping-pong effect. This parameter means to ignore the files generated by the last MHA trigger switch. By default, after the MHA switch occurs, the app1.failover.complete file will be generated in the log directory, which is the /data I set above. If it is found when switching again next time The existence of this file in this directory will not allow the switch to be triggered unless the file is deleted after the first switch. For convenience, this is set to --ignore_last_failover.
Turn off monitoring
masterha_stop --conf=/etc/masterha/app1.cnf
test failure
After the monitoring host completes the configuration, you can check the main database network card to obtain VIP
ip a | grep ens33
Simulate main database failure
systemctl stop mariadb turns off the service
Check what each machine is doing and what role it plays
tail -f /etc/masterha/app1/manger
After completing the previous step, enter mysql from the library
show slave status\G
You will find that the IP specified from the library has changed.
repair
Just restart
systemctl restart mariadb
But the original IP will not go back
Use the command to view
ip a | grep ens33