MHA high-availability cluster
- 1. Introduction to MHA:
- 2. Deploy MHA:
- 3. Operation steps
-
- 1. Install mysql database on three servers
- 2. Modify the main configuration file of mysql: /etc/my.cnf, note that the server-id of the three servers cannot be the same
- Three, three servers start mysql service
- Fourth, configure Mysql master-slave synchronization (one master and two slaves)
- Five. Configure MHA
1. Introduction to MHA:
MHA（Master High Availability）
(1 Introduction
目前在MySQL高可用方面是一个相对成熟的解决方案，它由日本DeNA公司youshimaton（现就职于Facebook公司）开发，是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用。
(2) The software consists of two parts:
MHA Manager（管理节点）和MHA Node（数据节点）。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上，MHA Manager会定时探测集群中的master节点，当master出现故障时，它可以自动将最新数据的slave提升为新的master，然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。
(3) Working principle:
1、在MHA自动故障切换过程中，MHA试图从宕机的主服务器上保存二进制日志，最大程度的保证数据的不丢失，但这并不总是可行的。例如，如果主服务器硬件故障或无法通过ssh访问，MHA没法保存二进制日志，只进行故障转移而丢失了最新的数据。使用MySQL 5.5的半同步复制，可以大大降低数据丢失的风险。MHA可以与半同步复制结合起来。如果只有一个slave已经收到了最新的二进制日志，MHA可以将最新的二进制日志应用于其他所有的slave服务器上，因此可以保证所有节点的数据一致性。
2、顺序
① 从宕机崩溃的master保存二进制日志事件（binlog events）;
② 识别含有最新更新的slave；
③ 应用差异的中继日志（relay log）到其他的slave；
④ 应用从master保存的二进制日志事件（binlog events）；
⑤ 提升一个slave为新的master；
⑥ 使其他的slave连接新的master进行复制
2. Deploy MHA:
Services that the role needs to install
master(192.168.45.133) mha4mysql-node
slave1(192.168.45.130) mha4mysql-node
slave2(192.168.45.134) mha4mysql-node
manager(192.168.45.135) mha4mysql-manager、 mha4mysql-node
(1) Demand:
In this case, MHA is required to monitor the MySQL database to automatically switch in the event of a failure, without affecting the business.
(2) Ideas:
Install MySQL database
Configure MySQL one master and two slaves
Install MHA software
Configure passwordless authentication
Configure MySQL and MHA high availability
Simulate master failover
(3) Operating system:
Cent0S7…6 version, MHA version is 0.57 version
3. Operation steps
1. Install mysql database on three servers
(For MySQL version, please use 5.6.36; for cmake version, please use 2.8.6), because the installation operations of the three machines are the same, here is a demonstration of the installation sequence on the master.
For the convenience of the experiment, modify the host name
#修改服务器名称，将manster服务器的主机名改为manster
hostnamectl set-hostname manster
su
#修改服务器名称，将slave1服务器的主机名改为slaveq
hostnamectl set-hostname slave1
su
#修改服务器名称，将slave2服务器的主机名改为slave2
hostnamectl set-hostname slave2
su
#安装编译依懒环境
yum -y install gcc gcc-c++ ncurses ncurses-devel bison perl-Module-Install cmake
#挂载软件包
mount.cifs //192.168.100.3/mha /mnt
Password for root@//192.168.100.3/mha:
#安装gmake编译软件
cd /mnt
tar zxvf cmake-2.8.6.tar.gz -C /opt
cd /opt/cmake-2.8.6/
./configure
gmake && gmake install
#安装mysql数据库
[root@localhost cmake-2.8.6]# cd /mnt
[root@localhost mnt]# tar zxvf mysql-5.6.36.tar.gz -C /opt
#编译mysql
cd /opt/mysql-5.6.36/
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DWITH_EXTRA_CHARSETS=all \
-DSYSCONFDIR=/etc
#安装
make && make install
#设置环境变量
cp support-files/my-default.cnf /etc/my.cnf
cp support-files/mysql.server /etc/rc.d/init.d/mysqld
chmod +x /etc/rc.d/init.d/mysqld
chkconfig --add mysqld
echo "PATH=$PATH:/usr/local/mysql/bin" >> /etc/profile
source /etc/profile
#创建mysql数据库，并授权
groupadd mysql
useradd -M -s /sbin/NOlogin mysql -g mysql
chown -R mysql.mysql /usr/local/mysql
mkdir -p /data/mysql
#初始化数据库
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql
2. Modify the main configuration file of mysql: /etc/my.cnf, note that the server-id of the three servers cannot be the same
—Configure the main server:
vim /etc/my.cnf
[mysql]
server-id = 1
#开启二进制日志
log_bin = master-bin
#允许从服务器进行同步
log-slave-updates = true
—Configure slave server 1:
vim /etc/my.cnf
[mysql]
server-id = 2
#开启二进制日志
log_bin = master-bin
#使用中继日志进行同步
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index
—Configuration from the server 2:
vim /etc/my.cnf
[mysql]
server-id = 3
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index
Three, three servers start mysql service
#在三台服务器上分别创建这两个个软链接
ln -s /usr/local/mysql/bin/mysql /usr/sbin/
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
#启动mysql
/usr/local/mysql/bin/mysqld_safe --user=mysql &
#关闭防火墙和安全功能
systemctl stop firewalld.service
setenforce 0
Fourth, configure Mysql master-slave synchronization (one master and two slaves)
1. The mysql master-slave configuration is relatively simple. It should be noted that authorization
is to authorize two users on all database nodes, one is to use the user myslave synchronously from the database, and the other is to use the manager to monitor the user.
mysql -u root -p //进入数据库
mysql> grant replication slave on *.* to 'myslave'@'192.168.45.%' identified by '123';
mysql> grant all privileges on *.* to 'mha'@'192.168.45.%' identified by 'manager';
mysql> flush privileges; //刷新数据库
#在数据库上添加下列授权（理论上不需要）以主机名授权（MHA检查时是通过主机名的形式）
mysql> grant all privileges on *.* to 'mha'@'master' identified by 'manager';
mysql> grant all privileges on *.* to 'mha'@'slave1' identified by 'manager';
mysql> grant all privileges on *.* to 'mha'@'slave2' identified by 'manager';
View binary files and synchronization points on the master server
show master status;
Set up synchronization on two slave servers
#在两台从服务器上都执行下列的命令，同步主服务器的日志
mysql> change master to master_host='192.168.45.133',master_user='myslave',master_password='123',master_log_file='master-bin.000001',master_log_pos=1292;
mysql> start slave; //开启slave
mysql> show slave status\G; //查看slave
Set two slave libraries to read-only mode
mysql> set global read_only=1;
mysql> flush privileges; //刷新数据库
Install MHA to install MHA
on all servers in a lazy environment, first install epel source
yum install epel-release --nogpgcheck -y
yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN
Install node on all servers
#修改服务器名称，将manager服务器的主机名改为manager
hostnamectl set-hostname manager
su
#挂载软件包
mount.cifs //192.168.100.3/mha /mnt
#解压安装node
cd ~
tar zxvf /mnt/mha4mysql-node-0.57.tar.gz
cd mha4mysql-node-0.57/
perl Makefile.PL
make && make install
Install the manager on the manger server (note: the node component must be installed to install the manager component)
#关闭防火墙
systemctl stop firewalld.service
setenforce 0
#解压并安装manager
cd ~
tar zxvf /mnt/mha4mysql-manager-0.57.tar.gz
mha4mysql-manager-0.57/
perl Makefile.PL
make && make install
After the manager server is installed, several tools will be generated under the usr/local/bin directory:
masterha_check_repl 检查mysql复制状况
masterha_master_monitor 检查master是否宕机
masterha_check_ssh 检查MHA的SSH配置情况
masterha_master_switch 控制故障转移
masterha_check_status 检查当前MHA运行状态
masterha_conf_host 添加或删除配置的server信息
masterha_stop 关闭manager
masterha_manager 启动manager的脚本
Several scripts will be generated under /usr/local/bin after node is installed (usually triggered by the script of MHA Manager, without human operation)
apply_diff_relay_logs ：识别差异的中继日志事件并将其差异的事件应用于其他的 slave；
save_binary_logs：保存和复制 master 的二进制日志；
filter_mysqlbinlog ：去除不必要的 ROLLBACK 事件 （MHA 已不再使用这个工具）；
purge_relay_logs：清除中继日志（不会阻塞 SQL 线程）；
Configure passwordless access
1、在manager配置所有数据库节点的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.133
ssh-copy-id 192.168.45.130
ssh-copy-id 192.168.45.134
2、在master上配置到数据库节点slave1和slave2的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.130
ssh-copy-id 192.168.45.134
3、在slave1上配置到数据库节点master'和slave2的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.133
ssh-copy-id 192.168.45.134
4、在slave2上配置到数据库节点slave1和master的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.133
ssh-copy-id 192.168.45.130
Five. Configure MHA
1. Copy the relevant script on the manager node to the /usr/local directory
cp -ra /root/mha4mysql-manager-0.57/samples/scripts/ /usr/local/bin/
ls scripts/
master_ip_failover：自动切换时 VIP 管理的脚本；
master_ip_online_change：在线切换时 VIP 的管理；
power_manager：故障发生后关闭主机的脚本；
send_report：因故障切换后发送报警的脚本；
2. Copy the VIP management script during automatic switching to the /usr/local/bin/ directory:
cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin/
3. Rewrite the master_ip_failover script:
vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.45.100';
my $brdc = '192.168.45.255';
my $ifdev = 'ens33';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
my $exit_code = 0;
#my $ssh_start_vip = "/usr/sbin/ip addr add $vip/24 brd $brdc dev $ifdev label $ifdev:$key;/usr/sbin/arping -q -A -c 1 -I $ifdev $vip;iptables -F;";
#my $ssh_stop_vip = "/usr/sbin/ip addr del $vip/24 dev $ifdev label $ifdev:$key";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
4. Create MHA software directory and copy configuration files
mkdir /etc/masterha
cp /root/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha/
#编辑配置文件
vim /etc/masterha/app1.cnf
[server default]
#manager配置文件
manager_log=/var/log/masterha/app1/manager.log
#manager日志
manager_workdir=/var/log/masterha/app1
#master保存binlog的位置，这里的路径要与master里配置的bilog的相同
master_binlog_dir=/usr/local/mysql/data
#设置自动failover时候的切换脚本。也就是上边的那个脚本
master_ip_failover_script=/usr/local/bin/master_ip_failover
#设置手动切换时候的切换脚本
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
#这个密码是前文中创建监控用户的那个密码
password=manager
remote_workdir=/tmp
#设置复制用户密码
repl_password=123
#设置复制用户的用户
repl_user=myslave
#设置发生切换后发生报警的脚本
reporl_script=/usr/local/send_report
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.45.130 -s 192.168.45.134
#设置故障发生关闭故障脚本主机
shutdown_script=""
#设置ssh的登录用户名
ssh_user=root
#设置监控用户
user=mha
[server1]
hostname=192.168.45.133
port=3306
[server2]
candidate_master=1
#设置为候选master，如果设置该参数以后，发送主从切换以后将会从此从库升级为主库
hostname=192.168.45.130
check_repl_delay=0
port=3306
[server3]
hostname=192.168.45.134
port=3306
Test ssh without password authentication, if it is normal, it will output successful at the end:
masterha_check_ssh -conf=/etc/masterha/app1.cnf
Check health status
masterha_check_repl -conf=/etc/masterha/app1.cnf
Configure virtual ip on master
/sbin/ifconfig ens33:1 192.168.45.100/24
Start the manager
1. Start mha on the manager server
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
2. Check the MHA status, you can see that the current master is a mysql node
masterha_check_status --conf=/etc/masterha/app1.cnf
Fault simulation
1, start monitoring, observe, log, and record
tailf /var/log/masterha/app1/manager.log
2. Turn off the master server
pkill -9 mysql
You can see the status of the slave library, and vip will switch to one of the slave libraries:
At this point, the client can also connect to the database through the virtual ip:
mysql -h 192.168.45.100 -u root -p