目录
一、MHA介绍
MHA（Master High Availability）目前在 MySQL 高可用方面是一个相对成熟的解决方案，是一套优秀的作为 MySQL 高可用性环境下故障切换和主从提升的高可用软件。
在 MySQL 故障切换过程中，MHA 能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA 能在最大程度上保证数据的一致性，以达到真正意义上的高可用。
该软件由两部分组成：MHA Manager（管理节点）和 MHA Node（数据节点）。MHA Manager 可以单独部署在一台独立的机器上管理多个 master-slave 集群，也可以部署在一台 slave 节点上。MHA Node 运行在每台 MySQL 服务器上，MHA Manager 会定时探测集群中的 master 节点，当 master 出现故障时，它可以自动将最新数据的 slave 提升为新的 master，然后将所有其他的 slave 重新指向新的 master。整个故障转移过程对应用程序完全透明。
在 MHA 自动故障切换过程中，MHA 试图从宕机的主服务器上保存二进制日志，最大程度的保证数据的不丢失，但这并不总是可行的。例如，如果主服务器硬件故障或无法通过ssh访问，MHA 没法保存二进制日志，只进行故障转移而丢失了最新的数据。使用 MySQL 5.5 的半同步复制，可以大大降低数据丢失的风险。MHA 可以与半同步复制结合起来。如果只有一个 slave 已经收到了最新的二进制日志，MHA 可以将最新的二进制日志应用于其他所有的 slave 服务器上，因此可以保证所有节点的数据一致性。
目前 MHA 主要支持一主多从的架构，要搭建 MHA,要求一个复制集群中必须最少有三台数据库服务器，一主二从，即一台充当 master，一台充当备用 master，另外一台充当从库，因为至少需要三台服务器，出于机器成本的考虑，淘宝也在该基础上进行了改造，淘宝TMHA已经支持一主一从。
MHA原理：
MHA用于维持 MySQL Replication 中 master 库的高可用性，最大的特点是可以修复多个slave上的差异日志，最终使所有slave保持数据一致，然后从中选取一个充当新的 master，并让其他 slave 指向它。
当master出现故障时，通过对比 slave 之间的 I/O thread 读取主库的 binlog 的 position 号，选取最接近的slave作为备胎（被选主库），其它从库通过与备胎对比，生成差异的中继日志，在备胎上运用从原来的 master 保存的 binlog，同时将备胎提升为master。最后在其他 slave 上运用相应的差异中继日志，并从新的 master 开始复制。
MHA优点：
① 故障切换时，自动判断哪个从库与主库离的最近，并切换到该从库
② 支持binlog server，提高 binlog 的传送效率
③ 结合半同步功能，确保故障切换时数据不丢失
二、MHA搭建
1.环境准备
| IP |
Hostname |
OS Version |
Role |
| 192.168.16.18 |
centos8-min8 |
CentOS Linux release 8.2.2004 |
master |
| 192.168.16.19 |
centos8-min9 |
CentOS Linux release 8.2.2004 |
slave1 |
| 192.168.16.20 |
centos8-min10 |
CentOS Linux release 8.2.2004 |
slave2+ manager |
由于安装MHA-Node过程中发现centos8并不能安装MHA-Node，所以将操作系统重装换成centos7
| IP |
Hostname |
OS Version |
Role |
| 192.168.16.21 |
centos7-min-ha1 |
CentOS Linux release 7.8.2003 |
master |
| 192.168.16.22 |
centos7-min-ha2 |
CentOS Linux release 7.8.2003 |
slave1 |
| 192.168.16.23 |
centos7-min-ha3 |
CentOS Linux release 7.8.2003 |
slave2+ manager |
2.MHA下载与MySQL安装
> MHA下载地址
wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58.tar.gz
- Download MYSQL from MySQL official website: https://downloads.mysql.com/archives/community/
At present, the MYSQL official website does not provide the MYSQL product of the redhat8 version.
centos8-min8：
centos8-min9：
centos8-min10：
- MySQL installation
Use the mysql automated installation script to install mysql on the three servers. The automated installation script can be referred to as follows:
###采用二进制安装包安装MySQL###
###############################
#!/bin/bash
## The 3 messages you need to update according to your package
dir_package='/opt'
install_package=${dir_package}'/mysql-5.7.24-el7-x86_64.tar.gz'
package_version='mysql-5.7.24'
## The default linux system version is Redhat/CentOS7
clear
echo "=========================================================================="
echo "A tool to auto-compile & install $package_version on Redhat/CentOS7 Linux "
echo "=========================================================================="
## 1. check the integrity of your mysql package
echo ">>Step1: md5sum 验证包的完整性"
echo `md5sum $install_package` | awk '{print $1}'
echo "If the md5sum is the same with the one obtained from https://downloads.mysql.com/archives/community/ ?"
read -p "(Please input yes or no):" checkmysql
case "$checkmysql" in
yes|y|Y|YES|yeS|yES|Yes|YeS|yEs|YEs)
checkmysql='y'
echo "安装包完整，可以开始安装"
;;
*)
echo "安装包不完整，请重新获取安装包"
checkmysql='n'
exit
esac
## 2. preparations for mysql installation
echo ">>Step2: mysql安装准备中..."
package_tar_name=${install_package##*/}
package_name=${package_tar_name%%.tar*}
## decompress the install package
tar zxvf $package_tar_name -C ${dir_package}/
cd ${dir_package}/
mv ${package_name} mysql
## create user mysql
useradd mysql
## create some directories needed by this installation
cd ${dir_package}/mysql/
touch my.cnf
mkdir -p data
mkdir -p log
mkdir -p mylog
mkdir -p pid
mkdir -p socket
mkdir -p tmp
chown -R mysql:mysql ${dir_package}/mysql
## assign the port for mysql installation
read -p "Tell me a viable port you'd like to install mysql (the default port is 3306):" port
if [ ! $port ];then
mport=3306
elif [ !`$port -eq 3306` ];then
mport=$port
fi
echo "The port="$mport
## define the my.cnf
cat >>${dir_package}/mysql/my.cnf<<EOF
[client]
port = $mport
socket = ${dir_package}/mysql/socket/mysqld-T-prod-$mport.sock
default-character-set=utf8
[mysqld]
user = mysql
port = $mport
socket = ${dir_package}/mysql/socket/mysqld-T-prod-$mport.sock
datadir = ${dir_package}/mysql/data
basedir = ${dir_package}/mysql
server-id = 1
log-bin = ${dir_package}/mysql/mylog/mysql-bin-T-prod-$mport
sync-binlog=1
tmpdir = ${dir_package}/mysql/tmp
[mysqld_safe]
pid-file = ${dir_package}/mysql/pid/mysql-T-prod-$mport.pid
log-error = ${dir_package}/mysql/log/mysql-T-prod-$mport.err
EOF
## Update ${dir_package}/mysql/support-files/mysql.server for mysql start | stop
cd ${dir_package}/mysql/support-files
sed -i "/^datadir=/{s/$/\\${dir_package}\\/mysql\\/data/}" mysql.server ##datadir=${dir_package}/mysql/data
sed -i "/^basedir=/{s/$/\\${dir_package}\\/mysql/}" mysql.server ##basedir=${dir_package}/mysql
sed -i "/^mysqld_pid_file_path=/{s/$/\\${dir_package}\\/mysql\\/pid\\/mysql-T-prod-$mport.pid/}" mysql.server
#default_conf="conf=/etc/my.cnf"
#conf="conf=/opt/mysql57/my.cnf"
#sed -i "s/${default_conf}/${conf}/g" mysql.server
sed -i "s/\\/etc\\/my.cnf/\\${dir_package}\\/mysql\\/my.cnf/g" mysql.server ##conf=${dir_package}/mysql/my.cnf
## 3. initial and start mysql installation
###function installMysql(){
echo ">>Step3: 开始安装$package_version"
echo "We are about to install $package_version, please wait..."
echo ">>Step3-1: 数据库初始化"
${dir_package}/mysql/bin/mysqld --defaults-file=${dir_package}/mysql/my.cnf --initialize --user=mysql --basedir=${dir_package}/mysql --datadir=${dir_package}/mysql/data
echo "=======Very important!! Please remember the password of root if the initial is successful=========="
echo ">>Step3-2: 启动数据库"
## 使用mysql用户启动mysql服务（如果mysql没有启动，请手动启动哈）
su - mysql -s /bin/bash -c "${dir_package}/mysql/support-files/mysql.server start"
## 检查mysql启动的进程
if [ `ps -ef | grep mysql | grep -v grep | wc -l` -eq 0 ]
then
echo "Mysql started failed"
else
echo "Mysql started successfully"
fi
###}
##installMysql 2&>1 | tee ${dir_package}/mysql/installMysql.log
## 4. what you need to do after mysql installation
## ① 设置mysql环境变量
#echo `export MYSQL_HOME=${dir_package}/mysql` >> /home/mysql/.bashrc
#echo `export PATH=$PATH:$MYSQL_HOME/bin`>> /home/mysql/.bashrc
#source /home/mysql/.bashrc
## ② 使用临时root密码登录mysql并修改root密码
#su - mysql
#mysql -uroot -p
#mysql> alter user user() identified by '123456';
Customize my.cnf
[root@centos8-min8 mysql]# cat my.cnf
[client]
port = 3306
socket = /root/mysql/socket/mysqld-T-prod-3306.sock
default-character-set=utf8
[mysqld]
user = mysql
port = 3306
socket = /root/mysql/socket/mysqld-T-prod-3306.sock
datadir = /root/mysql/data
basedir = /root/mysql
tmpdir = /root/mysql/tmp
##server-id = 1618
##log-bin = /root/mysql/mylog/mysql-bin-T-prod-3306
##sync-binlog=1
##relay_log = /root/mysql/mylog/relay-log-T-prod-3306
###log_slave_updates=1
#GTID
gtid_mode=on
enforce_gtid_consistency=on
server-id=1618 #The server-id of each server should be different
#binlog
log-bin=/root/mysql/mylog/mysql-bin-T-prod-3306
sync-binlog=1
log_slave_updates=1
binlog_format=row
#relay log
skip_slave_start=1
[mysqld_safe]
pid-file = /root/mysql/pid/mysql-T-prod-3306.pid
log-error = /root/mysql/log/mysql-T-prod-3306.err
[root@centos8-min8 ~]# mysql -uroot -p
mysql: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory
[root@centos8-min8 ~]# yum -y install libncurses*
[root@centos8-min10 ~]# mysql -uroot -p
Enter password:
mysql> alter user user() identified by '123456';
3. Configure the basic information of the three servers
① Associate Hostname with IP
[root@centos7-min-ha1 ~]# cat /etc/hosts
192.168.16.21 centos7-min-ha1
192.168.16.22 centos7-min-ha2
192.168.16.23 centos7-min-ha3
[root@centos7-min-ha1 ~]# scp /etc/hosts [email protected]:/etc/hosts
[root@centos7-min-ha1 ~]# scp /etc/hosts [email protected]:/etc/hosts
② Configure SSH password-free login
[root@centos8-min8 ~]# ssh -V
OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS 28 May 2019 —— CentOS8.2 ssh默认版本
[root@centos7-min-ha1 ~]# ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 —— CentOS7.8 ssh默认版本
/root/.ssh
[root@centos8-min8 ~]# ssh-keygen ——生成公私钥对
[root@centos8-min8 .ssh]# ll
total 12
-rw-------. 1 root root 2602 Feb 2 17:35 id_rsa
-rw-r--r--. 1 root root 571 Feb 2 17:35 id_rsa.pub
-rw-r--r--. 1 root root 350 Feb 2 13:52 known_hosts
[root@centos8-min8 .ssh]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.16.19 ——把公钥拷贝到其它机器
[root@centos8-min8 .ssh]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.16.20
拷贝完毕之后可在对应机器上查看到公钥信息
[root@centos8-min9 .ssh]# cat /root/.ssh/authorized_keys
[root@centos8-min8 .ssh]# ssh 192.168.16.19 ——min8可免密登录到保存了其公钥的机器
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Tue Feb 2 13:04:52 2021 from 192.168.16.1
[root@centos8-min9 ~]#
In the same way, configure the ssh password-free login of the min9 and min10 machines
4. Select master-slave mode
> Classic master-slave replication mode
Use binlog+pos to enable replication , and specify the binlog_file and log_pos of the master library in the slave library
MYSQL master and backup replication can refer to: http://t.csdn.cn/QOKci
> GTID mode
Turn on GTID, no need to find binlog and pos points, just change master to master_auto_postion=1, it will automatically find synchronization
① Overview of GTID :
1. Global transaction identifier: global transaction identified .
2. GTID transactions are globally unique, and a transaction corresponds to a GTID .
3. A GTID is executed only once on a server to avoid data confusion or master-slave inconsistency caused by repeated execution.
4、GTID用来代替classic的复制方法，不在使用binlog+pos开启复制。而是使用master_auto_postion=1的方式自动匹配GTID断点进行复制。
5、MySQL-5.6.5开始支持的，MySQL-5.6.10后开始完善。
6、在传统的slave端，binlog是不用开启的，但是在GTID中，slave端的binlog是必须开启的，目的是记录执行过的GTID（强制）。
② GTID的组成部分：
前面是server_uuid：后面是一个序列号
例如：server_uuid：sequence number
7800a22c-95ae-11e4-983d-080027de205a:10
UUID：每个mysql实例的唯一ID，由于会传递到slave，所以也可以理解为源ID。
Sequence number：在每台MySQL服务器上都是从1开始自增长的序列，一个数值对应一个事务。
③ GTID比传统复制的优势：
1、更简单的实现failover，不用以前那样在需要找log_file和log_Pos。
2、更简单的搭建主从复制。
3、比传统复制更加安全。
4、GTID是连续没有空洞的，因此主从库出现数据冲突时，可以用添加空事物的方式进行跳过。
④ GTID的工作原理：
1、master更新数据时，会在事务前产生GTID，一同记录到binlog日志中。
2、slave端的i/o 线程将变更的binlog，写入到本地的relay log中。
3、sql线程从relay log中获取GTID，然后对比slave端的binlog是否有记录。
4、如果有记录，说明该GTID的事务已经执行，slave会忽略。
5、如果没有记录，slave就会从relay log中执行该GTID的事务，并记录到binlog。
6、在解析过程中会判断是否有主键，如果没有就用二级索引，如果没有就用全部扫描。
⑤ 使用GTID搭建mysql的主从复制的主要参数：
[mysqld]
#GTID:
gtid_mode=on
enforce_gtid_consistency=on
server_id=2003306 #每个实例的server_id都要不一样
#binlog
log-bin=mysqlbin
log-slave-updates=1 #允许下端接入slave
binlog_format=row #强烈建议，其他格式可能造成数据不一致
#relay log
skip_slave_start=1
注意：建议使用mysql-5.6.5以上的最新版本。
⑥ 启动GTID的两种方法：
方法一、
1、如果是在已经跑的服务器，你需要重启一下mysql server。
2、启动之前，一定要先关闭master的写入，保证所有slave端都已经和master端数据保持同步。
3、所有slave需要加上skip_slave_start=1的配置参数，避免启动后还是使用老的复制协议。
方法二、
1、如果是新搭建的服务器，直接启动就行了。
⑦ master-slave搭建的注意事项
(1) Use GTID to mount the slave end to the master end:
1. It is best not to execute the transaction immediately after startup, but to change the master first .
2. Then execute the transaction, of course, it is not necessary.
3. Use the following sql to switch the slave to the new master .
stop slave;
change master to
master_host = 192.168.100.200,
master_port = 3306,
master_user = abobo,
master_password=123,
master_auto_position = 1;
( 2 ) If you add a new slave to the master end of the already running GTID
There are two methods:
Method 1 is applicable to the situation where the master is also newly created.
1、如果你的master所有的binlog还在。可以选择类似于上面的方法，安装slave，直接change master to到master端。
2、原理是直接获取master所有的GTID并执行。
3、优点：简单方便。
4、缺点：如果binlog太多，数据完全同步需要时间较长，并且master一开始就启用了GTUD。
方法二、适用于拥有较大数据的情况。（推荐）
1、通过master或者其他slave的备份搭建新的slave。（看第三部分）
2、原理：获取master的数据和这些数据对应的GTID范围，然后通过slave设置@@global.gtid_purged跳过备份包含的gtid。
3. Advantage: it can avoid the deficiency of the first method.
4. Disadvantages: Relatively speaking, it is a bit complicated.
5. Build a master-slave environment
① Environmental Description
This experiment builds a master and two slave environment, the version used is 5.7.28, based on GTID+row+enhanced semi-synchronous mode
GTID —— Global Transaction Identifier
Copy format - it is recommended to use the row format, there are too many pitfalls in the statement and mixed formats.
Enhanced semi-sync - rpl_semi_sync_master_wait_point = AFTER_SYNC
#5.7 is already after_sync by default
② Create master-slave replication accounts and management accounts on three MySQL servers
# Create a master-slave replication account
[mysql@centos8-min8 support-files]$ mysql -uroot -p
Enter password: 123456
mysql> create user 'gtid'@'192.168.16.%' identified by 'gtid123';
mysql> grant replication slave on *.* to 'gtid'@'192.168.16.%';
mysql> flush privileges;
# create admin account
mysql> create user 'manage'@'192.168.16.%' identified by 'manage123';
mysql> grant all privileges on *.* to 'manage'@'192.168.16.%';
mysql> flush privileges;
mysql> select Host,User from mysql.user;
③ 在主库上复制数据到所有从库，完成在某个时刻GTID的同步
[mysql@centos8-min8 mysql]$ mysqldump --single-transaction -uroot -p -A > all.sql
Enter password: 123456
[mysql@centos7-min-ha1 mysql]$ mysqldump --all-databases --triggers --routines --events -uroot -p -A > all.sql
Enter password:
[mysql@centos7-min-ha1 mysql]$
[mysql@centos7-min-ha1 mysql]$ ll
total 1132
-rw-rw-r--. 1 mysql mysql 848521 Feb 23 14:12 all.sql
[root@centos8-min8 mysql]# scp all.sql 192.168.16.19:/root/mysql/
[root@centos8-min8 mysql]# scp all.sql 192.168.16.20:/root/mysql/
④ 从各从库恢复备份并配置主从复制，开启主从同步
[mysql@centos8-min9 mysql]$ mysql -uroot -p < all.sql
[mysql@centos8-min10 mysql]$ mysql -uroot -p < all.sql
> 问题追踪
主要原因是之前的数据库里面配置了主从，再导出的时候一起导出来了,所以导入时会出现这个问题。
解决办法：在导入的服务器重置一下主数据库服务器，再重新导入sql文件即可
配置文件my.cnf修改（主从服务器配置除server-id之外其余信息一致）：
注意：每台MySQL服务器的server-id必须是不一样的！
[mysql@centos7-min-ha1 root]$ vi mysql/my.cnf
[client]
port = 3306
socket = /root/mysql/socket/mysqld-T-prod-3306.sock
default-character-set=utf8
[mysqld]
user = mysql
port = 3306
socket = /root/mysql/socket/mysqld-T-prod-3306.sock
datadir = /root/mysql/data
basedir = /root/mysql
tmpdir = /root/mysql/tmp
##server-id = 1620
##log-bin = /root/mysql/mylog/mysql-bin-T-prod-3306
##sync-binlog=1
##relay_log = /root/mysql/mylog/relay-log-T-prod-3306
###log_slave_updates=1
#GTID
gtid_mode=on
enforce_gtid_consistency=on
server-id=1621
#binlog
log-bin=/root/mysql/mylog/mysql-bin-T-prod-3306
sync-binlog=1
log_slave_updates=1
binlog_format=row
#relay log
skip_slave_start=1
[mysqld_safe]
pid-file = /root/mysql/pid/mysql-T-prod-3306.pid
log-error = /root/mysql/log/mysql-T-prod-3306.err
[root@centos7-min-ha2 ~]# vi mysql/my.cnf
#GTID
gtid_mode=on
enforce_gtid_consistency=on
server-id=1622
[mysql@centos7-min-ha3 mha]$vi /root/mysql/my.cnf
#GTID
gtid_mode=on
enforce_gtid_consistency=on
server-id=1623
Restart the master-slave mysql service and check the enabling status of gtid mode
[mysql@centos8-min8 mysql]$ support-files/mysql.server restart
The master status of the main library:
Synchronize the main library master from the library 192.168.16.22:
mysql> change master to
-> master_host='192.168.16.21',
-> master_port=3306,
-> master_user='gtid',
-> master_password='gtid123',
-> master_auto_position=1;
Synchronize the main library master from the library 192.168.16.20:
- issue tracking
1. gtid_mode is not enabled
This parameter gtid_mode is a single-item modification process, that is to say, from off to on , it needs to go through the middle OFF <-> OFF_PERMISSIVE <-> ON_PERMISSIVE <-> ON
process, and two adjacent states can be converted to each other. Otherwise an error will occur.
[mysqld]
gtid_mode=on
enforce_gtid_consistency=on
2. The slave IO process cannot connect to the communication
Open the mysql port or close the firewall
[root@centos8-min9 ~]# systemctl stop firewalld
[root@centos8-min9 ~]# systemctl disable firewalld
[root@centos8-min9 ~]# systemctl status firewalld
服务器挂掉之后MySQL主从同步失败，需要重启slave
6.安装MHA-Node节点
在所有节点上安装数据节点
安装MySQL依赖的perl环境
①安装epel源
yum install -y epel-release
②安装perl
yum install perl-DBD-MySQL
yum install perl-Config-Tiny
yum install perl-Log-Dispatch
yum install perl-Parallel-ForkManager
yum install perl-Params-Validate
③安装mha-node
tar -zxf mha4mysql-node-0.58.tar.gz
cd mha4mysql-node-0.58/
yum install perl-CPAN* -y
perl Makefile.PL
make && make install
- 问题追踪
执行yum命令失败，服务器不能访问外网
Cannot find a valid baseurl for repo: base/7/x86_64
添加DNS到网卡配置并重启网络
[root@centos8-min8 ~]# yum install perl-DBD-MySQL -y
Unzip mha4mysql-node-0.58.tar.gz and install per-cpan
[root@centos8-min8 ~]# tar -zxf mha4mysql-node-0.58.tar.gz
[root@centos8-min8 ~]# cd mha4mysql-node-0.58/
[root@centos8-min8 mha4mysql-node-0.58]# yum install perl-CPAN* -y
perl Makefile.PL
make && make install
Can't locate inc/Module/Install.pm in @INC (you may need to install the inc::Module::Install module)
Check whether the cpan software package is installed. After it is installed, the perl module can be installed through cpan
#yum install perl-CPAN*
[root@centos8-min8 mha4mysql-node-0.58]# cpan inc::Module::Install
> Issue Tracking
Failed to execute perl Makefile.PL in centos8.2, but executed successfully in centos7.8
7. Install the MHA-Manager management node
Only install the mha-manager management node on centos7-min-ha3 192.168.16.23
The media package required for the installation environment - (installed when installing MHA-Node)
①Install mha-manager
[root@centos7-min-ha3 ~]# tar -zxf mha4mysql-manager-0.58.tar.gz
[ root @ centos7 - min - ha3 ~ ] # chown - R root . root mha4 mysql - manager - 0.58
[root@centos7-min-ha3 ~]# cd mha4mysql-manager-0.58
[root@centos7-min-ha3 mha4mysql-manager-0.58]# perl Makefile.PL
[root@centos7-min-ha3 mha4mysql-manager-0.58]# make && make install
② Configure MHA
mkdir /etc/mha
mkdir -p /usr/local/mha
cd /etc/mha/
[root@centos7-min-ha3 scripts]# cat /etc/mha/mha.conf
[root@centos7-min-ha3 scripts]#
cat /usr/local/mha/scripts/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL =>'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.16.100/24'; # Virtual IP here needs to be modified according to your own environment
my $key = "1";
##my $ssh_start_vip = "/sbin/ifcfg ens33:$key $vip"; #Pay attention to network card
##my $ssh_stop_vip = "/sbin/ifcfg ens33:$key down";
my $ssh_start_vip = "sudo ip addr add $vip dev ens33:$key";
my $ssh_stop_vip = "sudo ip addr delete $vip dev ens33:$key";
my $exit_code = 0;
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
#print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
print "\n\n\n***************************************************************\n";
print "Disabling the VIP - $vip on old master: $orig_master_host\n";
print "***************************************************************\n\n\n\n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "\n\n\n***************************************************************\n";
print "Enabling the VIP - $vip on new master: $new_master_host \n";
print "***************************************************************\n\n\n\n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
"Usage: master_ip_failover –command=start|stop|stopssh|status –orig_master_host=host –orig_master_ip=ip –orig_master_port=po
rt –new_master_host=host –new_master_ip=ip –new_master_port=port\n";
}
[root@centos7-min-ha3 scripts]#
cat /usr/local/mha/scripts/master_ip_online_change
#note vip
#/bin/bash
source /root/.bash_profile
vip=`echo '192.168.16.100/24'` # Virtual IP
key=`echo '1'`
command=`echo "$1" | awk -F = '{print $2}'`
orig_master_host=`echo "$2" | awk -F = '{print $2}'`
new_master_host=`echo "$7" | awk -F = '{print $2}'`
orig_master_ssh_user=`echo "${12}" | awk -F = '{print $2}'`
new_master_ssh_user=`echo "${13}" | awk -F = '{print $2}'`
##stop_vip=`echo "ssh root@$orig_master_host /sbin/ifcfg ens33:$key down"`
##start_vip=`echo "ssh root@$new_master_host /sbin/ifcfg ens33:$key $vip"`
stop_vip=`echo "ssh root@$orig_master_host sudo ip addr delete 192.168.16.100/24 dev ens33:1"`
start_vip=`echo "ssh root@$new_master_host sudo ip addr add 192.168.16.100/24 dev ens33:1"`
if [ $command = 'stop' ]
then
echo -e "\n\n\n***************************************************************\n"
echo -e "Disabling the VIP - $vip on old master: $orig_master_host\n"
$stop_vip
if [ $? -eq 0 ]
then
echo "Disabled the VIP successfully"
else
echo "Disabled the VIP failed"
fi
echo -e "***************************************************************\n\n\n\n"
fi
if [ $command = 'start' -o $command = 'status' ]
then
echo -e "\n\n\n***************************************************************\n"
echo -e "Enabling the VIP - $vip on new master: $new_master_host \n"
$start_vip
if [ $? -eq 0 ]
then
echo "Enabled the VIP successfully"
else
echo "Enabled the VIP failed"
fi
echo -e "***************************************************************\n\n\n\n"
fi
Check connectivity of all hosts:
[root@centos7-min-ha3 scripts]#
/usr/local/bin/masterha_check_ssh --conf=/etc/mha/mha.conf
Solution:
SSH互信问题，之前是使用ssh-copy-id命令，所以在authorized_keys中只保存了除本机外的另外两台机器的公钥，需要将三台机器上的本机ssh公钥加入authorized_keys文件
重试成功
检测复制状态：
[root@centos7-min-ha3 mha]#
/usr/local/bin/masterha_check_repl --conf=/etc/mha/mha.conf
问题解决：启动三台服务器上的mysql服务
[root@centos7-min-ha3 mha]# /root/mysql/support-files/mysql.server start
问题解决：需要重启两台mysql从服务的主从同步复制功能
[root@centos7-min-ha3 mha]# su mysql
[mysql@centos7-min-ha3 mha]$ mysql -uroot -p
Enter password: 123456
mysql> start slave;
8.在主库上添加VIP
[root@centos7-min-ha1 ~]# ip addr add 192.168.16.100/24 dev ens33
删除VIP
[root@centos7-min-ha1 ~]# ip addr del 192.168.16.100/24 dev ens33
9.在管理节点启动MHA服务
启动MHA:
nohup masterha_manager --conf=/etc/mha/mha.conf > /tmp/mha_manager.log < /dev/null 2>&1 &
Check if MHA is enabled:
masterha_check_status --conf=/etc/mha/mha.conf
Notice:
3. Simulate main library failure, failover
1. MHA automatically switches the main library
MHA auto switch stuck in VIP stage via ssh drift
The main reason is that there is a problem with the configuration of the two files master_ip_failover and master_ip_online_change
[root@centos7-min-ha3 scripts]# find / -name mha.failover.complete
/usr/local/mha/mha.failover.complete
[root@centos7-min-ha3 scripts]# cd /usr/local/mha/
[root@centos7-min-ha3 mha]# rm -rf mha.failover.complete
mha.failover.complete file: After this file is generated, it will not automatically switch after allowing the main library to fail
After deleting the mha.failover.complete file on the manager machine, the VIP information can be viewed on the master machine
mha-manager is running normally, MySQL master-slave replication is normal, simulate the failure of the main library, and check that the VIP drifts from the main library 21 to the slave library 22
Stop the master MySQL service
[root@centos7-min-ha1 ~]# mysql/support-files/mysql.server stop
Shutting down MySQL........... SUCCESS!
centos7-min-ha1
centos7-min-ha2
After VIP drift, the centos7-min-ha2 slave library becomes the master library, check the master-slave status of centos7-min-ha2
View the master-slave status of the centos7-min-ha3 slave library
> Problem Tracking: Try to change the Master_Host of the slave library to VIP , the test result is that the problem cannot be solved. The reason for this problem is that the server-id configurations of min-ha3 and min-ha2 are the same.
Note: the server-id of each MySQL server must be different
2. Manually switch the main library
Premise: The state of one master and two slaves is normal, the VIP of the master library is normal, and the MHA manager is started normally
Stop the main library service, VIP switch to slave library min-ha2
[root@centos7-min-ha1 ~]# mysql/support-files/mysql.server stop
Shutting down MySQL........... SUCCESS!
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected, 1 warning (0.01 sec)
The client accesses the MySQL service through the VIP
Database master-slave status check
min-ha2 (mater): automatically without slave
min-ha3: Automatically switch the master to 192.168.16.22 synchronously
min-ha1: no slave status, no new master synchronization
subsequent recovery
① Copy the min-ha1 slave library and the min-ha2 master library
min-ha1：
② After restarting the min-ha1 MySQL service, check the Mha manager server and delete the mha.failover.complete file
After this file is generated, it will not automatically switch after allowing the main library to fail
Restart the mha-manager service
[root@centos7-min-ha3 mha]#
[root@centos7-min-ha3 mha]# nohup masterha_manager --conf=/etc/mha/mha.conf > /tmp/mha_manager.log < /dev/null 2>&1&1;
[root@centos7-min-ha3 mha]# masterha_check_status --conf=/etc/mha/mha.confmha (pid:2641) is running(0:PING_OK), master:192.168.16.22
mha.failover.complete exists, min-ha2 (master) failover is unsuccessful, VIP is still in min-ha2
Then restart the min-ha2 MySQL service, and check the master-slave status is normal
[root@centos7-min-ha2 ~]# mysql/support-files/mysql.server start
Starting MySQL. SUCCESS!
Test again to delete /usr/local/mha/mha.failover.complete, restart the mha service, and then stop the MySQL master
[root@centos7-min-ha3 mha]# rm -rf mha.failover.complete
[root@centos7-min-ha3 mha]# nohup masterha_manager --conf=/etc/mha/mha.conf > /tmp/mha_manager.log < /dev/null 2>&1&1;
[1] 3127
[root@centos7-min-ha3 mha]# masterha_check_status --conf=/etc/mha/mha.confmha (pid:3127) is running(0:PING_OK), master:192.168.16.22
[root@centos7-min-ha2 ~]# mysql/support-files/mysql.server status
SUCCESS! MySQL running (2720)
[root@centos7-min-ha2 ~]# mysql/support-files/mysql.server stop
Shutting down MySQL............ SUCCESS!
VIP successfully switched back to min-ha1 again
MySQL high availability passed, the test is complete.
Min-ha1:
Min-ha2:
Min-ha3:
Synchronize min-ha2 with min-ha1 MySQL host in the form of MySQL slave
