Mysql -MHA环境搭建

安装一台虚拟机并安装mysql

用户名密码 root/root

克隆3台出来

192.168.181.135 -master
192.168.181.134 -slave1
192.168.181.133-slave2
192.168.181.132-mha 停止mysql systemctl stop mysqld

ip	用途
192.168.181.132	MHA管理节点
192.168.181.133	slave2
192.168.181.134	slave1
192.168.181.135	master

关闭或者开放防火墙

firewall-cmd --permanent --zone=public --add-port=3306/tcp
firewall-cmd --reload

修改4台的auto.cfg文件

vim /var/lib/mysql/auto.cnf

配置 主库135-mysql的

修改配置文件 /etc/my.cnf

# endable binlog
log_bin=mysql-bin
server-id=8
# 每次提交事务都刷盘,0-不刷盘,1秒刷一次,1,2-刷log,一秒刷一次盘
sync-binlog=1

# 哪些库同步不步不设置就全步
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=performance_schema
binlog-ignore-db=sys
# 同步的库
# binlog-db-db=test
relay_log=mysql-relay-bin
log_slave_updates=1
relay_log_purge=0

• 修改server-id保证每台不一样
• sync-binlog 刷盘策略
  0-每秒提交 Redo buffer ->OS cache -> flush cache to disk，可能丢失一秒内的事务数据。由后台Master线程每隔 1秒执行一次操作。
  1-（默认值）：每次事务提交执行 Redo Buffer -> OS cache -> flush cache to disk，最安全，性能最差的方式
  2- 每次事务提交执行 Redo Buffer -> OS cache，然后由后台Master线程再每隔1秒执行OScache -> flush cache to disk 的操作
• binlog-ignore-db哪些库忽略不同步
• binlog-db-db 哪些库同步,这里不指定

重启mysql

systemctl restart mysqld

• 查看启动状态

systemctl status mysqld

修改135,134,133的密码策略

分别登录mysql
mysql -uroot -p

SHOW VARIABLES LIKE 'validate_password%';

• 修改

set global validate_password_policy=LOW;
set global validate_password_length=4;

授权

grant replication slave on *.* to 'root'@'%' identified by 'root';

grant all privileges ON *.* to 'root'@'%' identified by 'root';
# 刷新权限
flush privileges;

查看主库状态

show master status;

2台从库,134,133

修改配置文件

• 134

# endable binlog
log_bin=mysql-bin
server-id=9
sync-binlog=1
read_only=1
# 哪些库同步不步不设置就全步
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=performance_schema
binlog-ignore-db=sys
# 同步的库
# 开启relay_log
relay_log=mysql-relay-bin
log_slave_updates=1
relay_log_purge=0

• 133

# endable binlog
log_bin=mysql-bin
server-id=10
sync-binlog=1
read_only=1
# 哪些库同步不步不设置就全步
binlog-ignore-db=information_schema
binlog-ignore-db=mysql
binlog-ignore-db=performance_schema
binlog-ignore-db=sys
# 同步的库
# binlog-db-db=test
# 开启relay_log
relay_log=mysql-relay-bin
log_slave_updates=1
relay_log_purge=0

• 重启2个从库

systemctl restart mysqld;

查看主库状态

show master status;

2个从库分别执行

# 如果之前有从库要停止
stop slave;
change master to master_host='192.168.181.135',master_port=3306,master_user='root',master_password='root',master_log_file='mysql-bin.000007',master_log_pos=154;

master_host 主库ip

master_port 主库端口

master_user主库用户

master_password 主库用户对应的密码

master_log_file 主库binlog日志文件名称

master_log_pos 主库binlog日志的POS位置上面图中的position

开启从库,134,133

start slave;
show slave status \G;

• 134
  
• 133
  

主从同步测试

• 在主库上建立一个库

create database t1;

use t1;
CREATE TABLE `t1` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键自增',
  `name` varchar(50) NOT NULL COMMENT '名称',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

• 插入数据

INSERT INTO t1 (name) VALUES ('a');
INSERT INTO t1 (name) VALUES ( 'b');

从库查看

• 134
  

• 133
  
  主从搭建完成

搭建MHA

官方安装手册

安装MHA Node-4台机器

手动下载或者YUM安装
下载地址
manager
https://github.com/yoshinorim/mha4mysql-manager/releases/tag/v0.58
node
https://github.com/yoshinorim/mha4mysql-node/releases/tag/v0.58

yum install perl-DBD-MySQL -y

• 下载node节点与mannager
  
  
• 上传服务器 每个节点上都安装node节点

rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm

ll /usr/bin/{app*,filter*,purge*,save*}

MHA 132上安装manager节点

• 安装
  yum install perl-Parallel-ForkManager 出现如下信息

Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: mirrors.grandcloud.cn
 * extras: mirrors.grandcloud.cn
 * updates: mirrors.grandcloud.cn
Setting up Install Process
No package perl-Parallel-ForkManager available.
Nothing to do

 yum -y update 

更新完毕，继续执行

yum install perl-Parallel-ForkManager 

提示还是没有相关package。

• 安装EPEL
  https://fedoraproject.org/wiki/EPEL

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

• 安装依赖

 yum install perl-DBD-MySQL
yum install perl-Config-Tiny
 yum install perl-Log-Dispatch
yum install perl-Parallel-ForkManager

rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

下载配置文件

https://github.com/yoshinorim/mha4mysql-manager/tree/master/samples

.ssh互信

• 在Mha管理机器上192.168.128.132

[root@localhost samples]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:0vV/p23c7QW/HxLnaiFgvUh5Ul1qPBlxl2Gbcz9dCD4 [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|            ooo=+|
|           o.o*o+|
|          = E*.+o|
|       . B +... *|
|      . S = o..oo|
|       . . o o+o.|
|            ..o+B|
|             .o+X|
|            ...+=|
+----[SHA256]-----+
[root@localhost samples]# 

• 接着互相发送密钥 在管理机器上192.168.128.132
• 发送主节点

[root@localhost samples]# ssh-copy-id 192.168.181.135
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.135 (192.168.181.135)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.135'"
and check to make sure that only the key(s) you wanted were added.

• 2个从节点

[root@localhost samples]# ssh-copy-id 192.168.181.134
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.134 (192.168.181.134)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.134'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost samples]# ssh-copy-id 192.168.181.133
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.133 (192.168.181.133)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.133'"
and check to make sure that only the key(s) you wanted were added.

主节点 135

[root@localhost mysql]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:a/h/LiuXoN+nUE8Dr0ricS1TWia1chsMrii9Akdtl/I [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|   .   .. o      |
|  . + o. + +     |
| . . +  S X +    |
|. .. .Eo.% * .   |
| o. o =.X.+..    |
|  .. o.B.*+ o    |
|   .. ..++=B.    |
+----[SHA256]-----+
[root@localhost mysql]# ssh-copy-id 192.168.181.134
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.134 (192.168.181.134)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.134'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost mysql]# ssh-copy-id 192.168.181.133
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.133 (192.168.181.133)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.133'"
and check to make sure that only the key(s) you wanted were added.

从节点1 134

[root@localhost mysqltmp]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:KwuyNpqIclwQjwbs4MMSlIUAEti3CtTrdwQRy9Pcpgc [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|@+=. oo          |
|*=o.o.+ .        |
|*o =.=.E o       |
|o=+.o ..+        |
|.ooo  ..S.       |
|  .... ...       |
| ...o o .        |
|+.=o . o         |
|*+..  .          |
+----[SHA256]-----+
[root@localhost mysqltmp]# ssh-copy-id 192.168.181.135
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.135 (192.168.181.135)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.135'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost mysqltmp]# ssh-copy-id 192.168.181.133
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.133 (192.168.181.133)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.133'"
and check to make sure that only the key(s) you wanted were added.

从节点2 133

[root@localhost mysqltmp]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6UQ+JwH5pTvTdKwEazWjAYdx3ivw2mJkJV3olwSvR/0 [email protected]
The key's randomart image is:
+---[RSA 2048]----+
|      +=+o.      |
|      o*++*.     |
|      oo*X+=.    |
|       BB==.o.   |
|      o.S*+o  E  |
|     o =+*o      |
|      + oo       |
|     . .         |
|                 |
+----[SHA256]-----+
[root@localhost mysqltmp]# ssh-copy-id 192.168.181.135
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.135 (192.168.181.135)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.135'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost mysqltmp]# ssh-copy-id 192.168.181.134
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.181.134 (192.168.181.134)' can't be established.
ECDSA key fingerprint is SHA256:HHYxY+OyIs9Y9ocDTvq/HKdFq4T4lZoiYnivwcU6xag.
ECDSA key fingerprint is MD5:98:5b:61:f6:24:8c:a4:dc:4d:c4:b1:d9:96:19:20:de.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.181.134'"
and check to make sure that only the key(s) you wanted were added.

在管理节点132上创建工作目录

mkdir /etc/masterha
mkdir /etc/masterha/app1
cd /etc/masterha/
vim app.cnf

• 配置文件

[server default]
manager_log=/etc/masterha/mha.log
#manager工作目录
manager_workdir=/etc/masterha/

#manager的日志
manager_log=/var/log/masterha/app1/manager.log
#master服务器保存二进制日志文件的位置，二进制安装的Mysql其文件位置在mysql/data/目录下
master_binlog_dir=/var/lib/mysql
# 自动故障切换时master服务器IP地址切换脚本，如果没有则不写此项
#master_ip_online_change_script=/etc/masterha/master_ip_online_change
#手动切换时master服务器IP地址切换脚本，如果没有则不写此项（一般安装MHA时自带）
# master_ip_online_change_script=/usr/local/bin/master_ip_online_change
# 监控主库时发送ping包的时间间隔，默认为3秒，如果连续3次没有回应将自动进行故障切换
ping_interval=1
# Mysql在发生切换时二进制日志文件的保存位置
remote_workdir=/tmp

#复制用户
repl_user=root
#复制用户的密码
repl_password=root
#ssh登录用户
ssh_user=root
# 监控用户
user=root
# 监控用户的密码
password=root
 
[server1]
hostname=192.168.181.135
port=3306
 
[server2]
hostname=192.168.181.134
port=3306
# 候选master，主从切换后升为master，即使复制事件不是最新的
candidate_master=1
#默认情况下，如果一个slave的中转日志落后master二进制日志100M，那么MHA就不会选择该slave作为新的master。
# 但设置此项后MHA会忽略复制延迟，对候选master非常有用。
check_repl_delay=0
 
[server3]
hostname=192.168.181.133
port=3306

检测 ssh

[root@localhost masterha]# masterha_check_ssh --conf=/etc/masterha/app.cnf 
Mon Sep 14 14:23:40 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Sep 14 14:23:40 2020 - [info] Reading application default configuration from /etc/masterha/app.cnf..
Mon Sep 14 14:23:40 2020 - [info] Reading server configuration from /etc/masterha/app.cnf..
Mon Sep 14 14:23:40 2020 - [info] Starting SSH connection tests..
Mon Sep 14 14:23:46 2020 - [debug] 
Mon Sep 14 14:23:40 2020 - [debug]  Connecting via SSH from [email protected](192.168.181.135:22) to [email protected](192.168.181.134:22)..
Mon Sep 14 14:23:45 2020 - [debug]   ok.
Mon Sep 14 14:23:45 2020 - [debug]  Connecting via SSH from [email protected](192.168.181.135:22) to [email protected](192.168.181.133:22)..
Mon Sep 14 14:23:45 2020 - [debug]   ok.
Mon Sep 14 14:23:47 2020 - [debug] 
Mon Sep 14 14:23:41 2020 - [debug]  Connecting via SSH from [email protected](192.168.181.134:22) to [email protected](192.168.181.135:22)..
Mon Sep 14 14:23:43 2020 - [debug]   ok.
Mon Sep 14 14:23:43 2020 - [debug]  Connecting via SSH from [email protected](192.168.181.134:22) to [email protected](192.168.181.133:22)..
Mon Sep 14 14:23:46 2020 - [debug]   ok.
Mon Sep 14 14:23:47 2020 - [debug] 
Mon Sep 14 14:23:41 2020 - [debug]  Connecting via SSH from [email protected](192.168.181.133:22) to [email protected](192.168.181.135:22)..
Mon Sep 14 14:23:46 2020 - [debug]   ok.
Mon Sep 14 14:23:46 2020 - [debug]  Connecting via SSH from [email protected](192.168.181.133:22) to [email protected](192.168.181.134:22)..
Mon Sep 14 14:23:46 2020 - [debug]   ok.
Mon Sep 14 14:23:47 2020 - [info] All SSH connection tests passed successfully.

检查管理节点的状态

masterha_check_status --conf=/etc/masterha/app.cnf
app is stopped(2:NOT_RUNNING).

manager节点检测repl环境：

如果报Binlog setting check failed!错误，那么有可能是master服务器保存二进制日志文件地址错误。

masterha_check_repl --conf=/etc/masterha/app.cnf

• masterha_check_repl --conf=/etc/masterha/app.cnf

 masterha_check_repl --conf=/etc/masterha/app.cnf
Mon Sep 14 14:53:28 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Sep 14 14:53:28 2020 - [info] Reading application default configuration from /etc/masterha/app.cnf..
Mon Sep 14 14:53:28 2020 - [info] Reading server configuration from /etc/masterha/app.cnf..
Mon Sep 14 14:53:28 2020 - [info] MHA::MasterMonitor version 0.58.
Mon Sep 14 14:53:29 2020 - [info] GTID failover mode = 0
Mon Sep 14 14:53:29 2020 - [info] Dead Servers:
Mon Sep 14 14:53:29 2020 - [info] Alive Servers:
Mon Sep 14 14:53:29 2020 - [info]   192.168.181.135(192.168.181.135:3306)
Mon Sep 14 14:53:29 2020 - [info]   192.168.181.134(192.168.181.134:3306)
Mon Sep 14 14:53:29 2020 - [info]   192.168.181.133(192.168.181.133:3306)
Mon Sep 14 14:53:29 2020 - [info] Alive Slaves:
Mon Sep 14 14:53:29 2020 - [info]   192.168.181.134(192.168.181.134:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 14:53:29 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 14:53:29 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Sep 14 14:53:29 2020 - [info]   192.168.181.133(192.168.181.133:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 14:53:29 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 14:53:29 2020 - [info] Current Alive Master: 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 14:53:29 2020 - [info] Checking slave configurations..
Mon Sep 14 14:53:29 2020 - [info] Checking replication filtering settings..
Mon Sep 14 14:53:29 2020 - [info]  binlog_do_db= , binlog_ignore_db= information_schema,mysql,performance_schema,sys
Mon Sep 14 14:53:29 2020 - [info]  Replication filtering check ok.
Mon Sep 14 14:53:29 2020 - [info] GTID (with auto-pos) is not supported
Mon Sep 14 14:53:29 2020 - [info] Starting SSH connection tests..
Mon Sep 14 14:53:33 2020 - [info] All SSH connection tests passed successfully.
Mon Sep 14 14:53:33 2020 - [info] Checking MHA Node version..
Mon Sep 14 14:53:34 2020 - [info]  Version check ok.
Mon Sep 14 14:53:34 2020 - [info] Checking SSH publickey authentication settings on the current master..
Mon Sep 14 14:53:35 2020 - [info] HealthCheck: SSH to 192.168.181.135 is reachable.
Mon Sep 14 14:53:35 2020 - [info] Master MHA Node version is 0.58.
Mon Sep 14 14:53:35 2020 - [info] Checking recovery script configurations on 192.168.181.135(192.168.181.135:3306)..
Mon Sep 14 14:53:35 2020 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000008 
Mon Sep 14 14:53:35 2020 - [info]   Connecting to [email protected](192.168.181.135:22).. 
  Creating /tmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql, up to mysql-bin.000008
Mon Sep 14 14:53:36 2020 - [info] Binlog setting check done.
Mon Sep 14 14:53:36 2020 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon Sep 14 14:53:36 2020 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.181.134 --slave_ip=192.168.181.134 --slave_port=3306 --workdir=/tmp --target_version=5.7.31-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Mon Sep 14 14:53:36 2020 - [info]   Connecting to [email protected](192.168.181.134:22).. 
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to mysql-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Mon Sep 14 14:53:36 2020 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.181.133 --slave_ip=192.168.181.133 --slave_port=3306 --workdir=/tmp --target_version=5.7.31-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Mon Sep 14 14:53:36 2020 - [info]   Connecting to [email protected](192.168.181.133:22).. 
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to mysql-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Mon Sep 14 14:53:37 2020 - [info] Slaves settings check done.
Mon Sep 14 14:53:37 2020 - [info] 
192.168.181.135(192.168.181.135:3306) (current master)
 +--192.168.181.134(192.168.181.134:3306)
 +--192.168.181.133(192.168.181.133:3306)

Mon Sep 14 14:53:37 2020 - [info] Checking replication health on 192.168.181.134..
Mon Sep 14 14:53:37 2020 - [info]  ok.
Mon Sep 14 14:53:37 2020 - [info] Checking replication health on 192.168.181.133..
Mon Sep 14 14:53:37 2020 - [info]  ok.
Mon Sep 14 14:53:37 2020 - [warning] master_ip_failover_script is not defined.
Mon Sep 14 14:53:37 2020 - [warning] shutdown_script is not defined.
Mon Sep 14 14:53:37 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

测试

nohup masterha_manager --conf=/etc/masterha/app.cnf --remove_dead_master_conf --ignore_last_failover  /var/log/masterha/app1/manager.log 2>&1 &

• –remove_dead_master_conf 该参数表示当发生主从切换后，老的主库的ip将会从配置文件中移除如果故障机器修复好了，需要手工添加ip信息到配置文件中

• -manger_log 日志存放位置

• –ignore_last_failover 在缺省情况下，如果MHA检测到连续发生宕机，且两次宕机间隔不足8小时的话，则不会进行Failover，之所以这样限制是为了避免ping-pong效应。该参数代表忽略上次MHA触发切换产生的文件， 默认情况下，MHA发生切换后会在日志目录，也就是上面我设置的/data产生app1.failover.complete文件，下次再次切换的时候如 果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，这里设置为–ignore_last_failover。

• 查看状态

masterha_check_status --conf=/etc/masterha/app.cnf

app (pid:65830) is running(0:PING_OK), master:192.168.181.135

停止master节点 135

systemctl stop mysqld

管理节点查看日志

cat /var/log/masterha/app1/manager.log

• 日志

Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)
Mon Sep 14 15:08:28 2020 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin
Mon Sep 14 15:08:28 2020 - [info] HealthCheck: SSH to 192.168.181.135 is reachable.
Mon Sep 14 15:08:29 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.181.135' (111))
Mon Sep 14 15:08:29 2020 - [warning] Connection failed 2 time(s)..
Mon Sep 14 15:08:30 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.181.135' (111))
Mon Sep 14 15:08:30 2020 - [warning] Connection failed 3 time(s)..
Mon Sep 14 15:08:31 2020 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.181.135' (111))
Mon Sep 14 15:08:31 2020 - [warning] Connection failed 4 time(s)..
Mon Sep 14 15:08:31 2020 - [warning] Master is not reachable from health checker!
Mon Sep 14 15:08:31 2020 - [warning] Master 192.168.181.135(192.168.181.135:3306) is not reachable!
Mon Sep 14 15:08:31 2020 - [warning] SSH is reachable.
Mon Sep 14 15:08:31 2020 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app.cnf again, and trying to connect to all servers to check server status..
Mon Sep 14 15:08:31 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Sep 14 15:08:31 2020 - [info] Reading application default configuration from /etc/masterha/app.cnf..
Mon Sep 14 15:08:31 2020 - [info] Reading server configuration from /etc/masterha/app.cnf..
Mon Sep 14 15:08:32 2020 - [info] GTID failover mode = 0
Mon Sep 14 15:08:32 2020 - [info] Dead Servers:
Mon Sep 14 15:08:32 2020 - [info]   192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:32 2020 - [info] Alive Servers:
Mon Sep 14 15:08:32 2020 - [info]   192.168.181.134(192.168.181.134:3306)
Mon Sep 14 15:08:32 2020 - [info]   192.168.181.133(192.168.181.133:3306)
Mon Sep 14 15:08:32 2020 - [info] Alive Slaves:
Mon Sep 14 15:08:32 2020 - [info]   192.168.181.134(192.168.181.134:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:32 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:32 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Sep 14 15:08:32 2020 - [info]   192.168.181.133(192.168.181.133:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:32 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:32 2020 - [info] Checking slave configurations..
Mon Sep 14 15:08:32 2020 - [info] Checking replication filtering settings..
Mon Sep 14 15:08:32 2020 - [info]  Replication filtering check ok.
Mon Sep 14 15:08:32 2020 - [info] Master is down!
Mon Sep 14 15:08:32 2020 - [info] Terminating monitoring script.
Mon Sep 14 15:08:32 2020 - [info] Got exit code 20 (Master dead).
Mon Sep 14 15:08:32 2020 - [info] MHA::MasterFailover version 0.58.
Mon Sep 14 15:08:32 2020 - [info] Starting master failover.
Mon Sep 14 15:08:32 2020 - [info] 
Mon Sep 14 15:08:32 2020 - [info] * Phase 1: Configuration Check Phase..
Mon Sep 14 15:08:32 2020 - [info] 
Mon Sep 14 15:08:33 2020 - [info] GTID failover mode = 0
Mon Sep 14 15:08:33 2020 - [info] Dead Servers:
Mon Sep 14 15:08:33 2020 - [info]   192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:33 2020 - [info] Checking master reachability via MySQL(double check)...
Mon Sep 14 15:08:33 2020 - [info]  ok.
Mon Sep 14 15:08:33 2020 - [info] Alive Servers:
Mon Sep 14 15:08:33 2020 - [info]   192.168.181.134(192.168.181.134:3306)
Mon Sep 14 15:08:33 2020 - [info]   192.168.181.133(192.168.181.133:3306)
Mon Sep 14 15:08:33 2020 - [info] Alive Slaves:
Mon Sep 14 15:08:33 2020 - [info]   192.168.181.134(192.168.181.134:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:33 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:33 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Sep 14 15:08:33 2020 - [info]   192.168.181.133(192.168.181.133:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:33 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:33 2020 - [info] Starting Non-GTID based failover.
Mon Sep 14 15:08:33 2020 - [info] 
Mon Sep 14 15:08:33 2020 - [info] ** Phase 1: Configuration Check Phase completed.
Mon Sep 14 15:08:33 2020 - [info] 
Mon Sep 14 15:08:33 2020 - [info] * Phase 2: Dead Master Shutdown Phase..
Mon Sep 14 15:08:33 2020 - [info] 
Mon Sep 14 15:08:33 2020 - [info] Forcing shutdown so that applications never connect to the current master..
Mon Sep 14 15:08:33 2020 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
Mon Sep 14 15:08:33 2020 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Mon Sep 14 15:08:34 2020 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Mon Sep 14 15:08:34 2020 - [info] 
Mon Sep 14 15:08:34 2020 - [info] * Phase 3: Master Recovery Phase..
Mon Sep 14 15:08:34 2020 - [info] 
Mon Sep 14 15:08:34 2020 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Mon Sep 14 15:08:34 2020 - [info] 
Mon Sep 14 15:08:34 2020 - [info] The latest binary log file/position on all slaves is mysql-bin.000008:3196
Mon Sep 14 15:08:34 2020 - [info] Latest slaves (Slaves that received relay log files to the latest):
Mon Sep 14 15:08:34 2020 - [info]   192.168.181.134(192.168.181.134:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:34 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:34 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Sep 14 15:08:34 2020 - [info]   192.168.181.133(192.168.181.133:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:34 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:34 2020 - [info] The oldest binary log file/position on all slaves is mysql-bin.000008:3196
Mon Sep 14 15:08:34 2020 - [info] Oldest slaves:
Mon Sep 14 15:08:34 2020 - [info]   192.168.181.134(192.168.181.134:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:34 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:34 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Sep 14 15:08:34 2020 - [info]   192.168.181.133(192.168.181.133:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:34 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:34 2020 - [info] 
Mon Sep 14 15:08:34 2020 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Mon Sep 14 15:08:34 2020 - [info] 
Mon Sep 14 15:08:34 2020 - [info] Fetching dead master's binary logs..
Mon Sep 14 15:08:34 2020 - [info] Executing command on the dead master 192.168.181.135(192.168.181.135:3306): save_binary_logs --command=save --start_file=mysql-bin.000008  --start_pos=3196 --binlog_dir=/var/lib/mysql --output_file=/tmp/saved_master_binlog_from_192.168.181.135_3306_20200914150832.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58
  Creating /tmp if not exists..    ok.
 Concat binary/relay logs from mysql-bin.000008 pos 3196 to mysql-bin.000008 EOF into /tmp/saved_master_binlog_from_192.168.181.135_3306_20200914150832.binlog ..
 Binlog Checksum enabled
  Dumping binlog format description event, from position 0 to 154.. ok.
  No need to dump effective binlog data from /var/lib/mysql/mysql-bin.000008 (pos starts 3196, filesize 3196). Skipping.
 Binlog Checksum enabled
 /tmp/saved_master_binlog_from_192.168.181.135_3306_20200914150832.binlog has no effective data events.
Event not exists.
Mon Sep 14 15:08:35 2020 - [info] Additional events were not found from the orig master. No need to save.
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] * Phase 3.3: Determining New Master Phase..
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Mon Sep 14 15:08:35 2020 - [info] All slaves received relay logs to the same position. No need to resync each other.
Mon Sep 14 15:08:35 2020 - [info] Searching new master from slaves..
Mon Sep 14 15:08:35 2020 - [info]  Candidate masters from the configuration file:
Mon Sep 14 15:08:35 2020 - [info]   192.168.181.134(192.168.181.134:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Sep 14 15:08:35 2020 - [info]     Replicating from 192.168.181.135(192.168.181.135:3306)
Mon Sep 14 15:08:35 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Sep 14 15:08:35 2020 - [info]  Non-candidate masters:
Mon Sep 14 15:08:35 2020 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Mon Sep 14 15:08:35 2020 - [info] New master is 192.168.181.134(192.168.181.134:3306)
Mon Sep 14 15:08:35 2020 - [info] Starting master failover..
Mon Sep 14 15:08:35 2020 - [info] 
From:
192.168.181.135(192.168.181.135:3306) (current master)
 +--192.168.181.134(192.168.181.134:3306)
 +--192.168.181.133(192.168.181.133:3306)

To:
192.168.181.134(192.168.181.134:3306) (new master)
 +--192.168.181.133(192.168.181.133:3306)
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] * Phase 3.5: Master Log Apply Phase..
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Mon Sep 14 15:08:35 2020 - [info] Starting recovery on 192.168.181.134(192.168.181.134:3306)..
Mon Sep 14 15:08:35 2020 - [info]  This server has all relay logs. Waiting all logs to be applied.. 
Mon Sep 14 15:08:35 2020 - [info]   done.
Mon Sep 14 15:08:35 2020 - [info]  All relay logs were successfully applied.
Mon Sep 14 15:08:35 2020 - [info] Getting new master's binlog name and position..
Mon Sep 14 15:08:35 2020 - [info]  mysql-bin.000007:2901
Mon Sep 14 15:08:35 2020 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.181.134', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000007', MASTER_LOG_POS=2901, MASTER_USER='root', MASTER_PASSWORD='xxx';
Mon Sep 14 15:08:35 2020 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Mon Sep 14 15:08:35 2020 - [info] Setting read_only=0 on 192.168.181.134(192.168.181.134:3306)..
Mon Sep 14 15:08:35 2020 - [info]  ok.
Mon Sep 14 15:08:35 2020 - [info] ** Finished master recovery successfully.
Mon Sep 14 15:08:35 2020 - [info] * Phase 3: Master Recovery Phase completed.
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] * Phase 4: Slaves Recovery Phase..
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Mon Sep 14 15:08:35 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info] -- Slave diff file generation on host 192.168.181.133(192.168.181.133:3306) started, pid: 66075. Check tmp log /etc/masterha//192.168.181.133_3306_20200914150832.log if it takes time..
Mon Sep 14 15:08:36 2020 - [info] 
Mon Sep 14 15:08:36 2020 - [info] Log messages from 192.168.181.133 ...
Mon Sep 14 15:08:36 2020 - [info] 
Mon Sep 14 15:08:35 2020 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Mon Sep 14 15:08:36 2020 - [info] End of log messages from 192.168.181.133.
Mon Sep 14 15:08:36 2020 - [info] -- 192.168.181.133(192.168.181.133:3306) has the latest relay log events.
Mon Sep 14 15:08:36 2020 - [info] Generating relay diff files from the latest slave succeeded.
Mon Sep 14 15:08:36 2020 - [info] 
Mon Sep 14 15:08:36 2020 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Mon Sep 14 15:08:36 2020 - [info] 
Mon Sep 14 15:08:36 2020 - [info] -- Slave recovery on host 192.168.181.133(192.168.181.133:3306) started, pid: 66077. Check tmp log /etc/masterha//192.168.181.133_3306_20200914150832.log if it takes time..
Mon Sep 14 15:08:37 2020 - [info] 
Mon Sep 14 15:08:37 2020 - [info] Log messages from 192.168.181.133 ...
Mon Sep 14 15:08:37 2020 - [info] 
Mon Sep 14 15:08:36 2020 - [info] Starting recovery on 192.168.181.133(192.168.181.133:3306)..
Mon Sep 14 15:08:36 2020 - [info]  This server has all relay logs. Waiting all logs to be applied.. 
Mon Sep 14 15:08:36 2020 - [info]   done.
Mon Sep 14 15:08:36 2020 - [info]  All relay logs were successfully applied.
Mon Sep 14 15:08:36 2020 - [info]  Resetting slave 192.168.181.133(192.168.181.133:3306) and starting replication from the new master 192.168.181.134(192.168.181.134:3306)..
Mon Sep 14 15:08:36 2020 - [info]  Executed CHANGE MASTER.
Mon Sep 14 15:08:36 2020 - [info]  Slave started.
Mon Sep 14 15:08:37 2020 - [info] End of log messages from 192.168.181.133.
Mon Sep 14 15:08:37 2020 - [info] -- Slave recovery on host 192.168.181.133(192.168.181.133:3306) succeeded.
Mon Sep 14 15:08:37 2020 - [info] All new slave servers recovered successfully.
Mon Sep 14 15:08:37 2020 - [info] 
Mon Sep 14 15:08:37 2020 - [info] * Phase 5: New master cleanup phase..
Mon Sep 14 15:08:37 2020 - [info] 
Mon Sep 14 15:08:37 2020 - [info] Resetting slave info on the new master..
Mon Sep 14 15:08:37 2020 - [info]  192.168.181.134: Resetting slave info succeeded.
Mon Sep 14 15:08:37 2020 - [info] Master failover to 192.168.181.134(192.168.181.134:3306) completed successfully.
Mon Sep 14 15:08:37 2020 - [info] Deleted server1 entry from /etc/masterha/app.cnf .
Mon Sep 14 15:08:37 2020 - [info] 

----- Failover Report -----

app: MySQL Master failover 192.168.181.135(192.168.181.135:3306) to 192.168.181.134(192.168.181.134:3306) succeeded

Master 192.168.181.135(192.168.181.135:3306) is down!

Check MHA Manager logs at localhost.localdomain:/var/log/masterha/app1/manager.log for details.

Started automated(non-interactive) failover.
The latest slave 192.168.181.134(192.168.181.134:3306) has all relay logs for recovery.
Selected 192.168.181.134(192.168.181.134:3306) as a new master.
192.168.181.134(192.168.181.134:3306): OK: Applying all logs succeeded.
192.168.181.133(192.168.181.133:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.181.133(192.168.181.133:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.181.134(192.168.181.134:3306)
192.168.181.134(192.168.181.134:3306): Resetting slave info succeeded.
Master failover to 192.168.181.134(192.168.181.134:3306) completed successfully

134成为新的主节点
在134上插入数据

insert into t1(name) values('yy');

在133上查看

mysql> select * from t1;
+----+------+
| id | name |
+----+------+
|  1 | aa   |
|  2 | xt   |
|  3 | cc   |
|  4 | dd   |
|  5 | ff   |
|  6 | ee   |
|  8 | hh   |
|  9 | me   |
| 11 | ll   |
| 12 | tt   |
| 13 | pp   |
| 14 | yy   |
+----+------+

查看 发现server1没了

vim /etc/masterha/app.cnf

重启原135节点

systemctl start  mysqld

同步成功