详解MHA原理及failover切换过程

Mon Dec 31 21:52:38 2018 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Mon Dec 31 21:52:38 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/data --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.56 --binlog_prefix=mysql-bin
Mon Dec 31 21:52:39 2018 - [info] HealthCheck: SSH to 192.168.57.200 is reachable.
Mon Dec 31 21:52:39 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111)
Mon Dec 31 21:52:39 2018 - [warning] Connection failed 2 time(s)…
Mon Dec 31 21:52:40 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111)
Mon Dec 31 21:52:40 2018 - [warning] Connection failed 3 time(s)…
Mon Dec 31 21:52:41 2018 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at ‘reading initial communication packet’, system error: 111)
Mon Dec 31 21:52:41 2018 - [warning] Connection failed 4 time(s)…
Mon Dec 31 21:52:41 2018 - [warning] Master is not reachable from health checker!
Mon Dec 31 21:52:41 2018 - [warning] Master 192.168.57.200(192.168.57.200:3306) is not reachable!
Mon Dec 31 21:52:41 2018 - [warning] SSH is reachable.
Mon Dec 31 21:52:41 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha/masterha_default.conf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status…
Mon Dec 31 21:52:41 2018 - [info] Reading default configuration from /etc/masterha/masterha_default.conf…
Mon Dec 31 21:52:41 2018 - [info] Reading application default configuration from /etc/masterha/app1.conf…
Mon Dec 31 21:52:41 2018 - [info] Reading server configuration from /etc/masterha/app1.conf…
Mon Dec 31 21:52:41 2018 - [info] Setting max_ping_errors to 4, ping_interval to 1.
Mon Dec 31 21:52:43 2018 - [info] GTID failover mode = 0
Mon Dec 31 21:52:43 2018 - [info] Dead Servers:
Mon Dec 31 21:52:43 2018 - [info] 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:43 2018 - [info] Alive Servers:
Mon Dec 31 21:52:43 2018 - [info] 192.168.57.100(192.168.57.100:3306)
Mon Dec 31 21:52:43 2018 - [info] 192.168.57.121(192.168.57.121:3306)
Mon Dec 31 21:52:43 2018 - [info] Alive Slaves:
Mon Dec 31 21:52:43 2018 - [info] 192.168.57.100(192.168.57.100:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:43 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:43 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Dec 31 21:52:43 2018 - [info] 192.168.57.121(192.168.57.121:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:43 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:43 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Dec 31 21:52:43 2018 - [info] Checking slave configurations…
Mon Dec 31 21:52:43 2018 - [info] read_only=1 is not set on slave 192.168.57.100(192.168.57.100:3306).
Mon Dec 31 21:52:43 2018 - [warning] relay_log_purge=0 is not set on slave 192.168.57.100(192.168.57.100:3306).
Mon Dec 31 21:52:43 2018 - [info] read_only=1 is not set on slave 192.168.57.121(192.168.57.121:3306).
Mon Dec 31 21:52:43 2018 - [warning] relay_log_purge=0 is not set on slave 192.168.57.121(192.168.57.121:3306).
Mon Dec 31 21:52:43 2018 - [info] Checking replication filtering settings…
Mon Dec 31 21:52:43 2018 - [info] Replication filtering check ok.
Mon Dec 31 21:52:43 2018 - [info] Master is down!
Mon Dec 31 21:52:43 2018 - [info] Terminating monitoring script.
Mon Dec 31 21:52:43 2018 - [info] Got exit code 20 (Master dead).
Mon Dec 31 21:52:43 2018 - [info] MHA::MasterFailover version 0.56.
Mon Dec 31 21:52:43 2018 - [info] Starting master failover.
Mon Dec 31 21:52:43 2018 - [info]
Mon Dec 31 21:52:43 2018 - [info] * Phase 1: Configuration Check Phase…
Mon Dec 31 21:52:43 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] GTID failover mode = 0
Mon Dec 31 21:52:44 2018 - [info] Dead Servers:
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Checking master reachability via MySQL(double check)…
Mon Dec 31 21:52:44 2018 - [info] ok.
Mon Dec 31 21:52:44 2018 - [info] Alive Servers:
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.100(192.168.57.100:3306)
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.121(192.168.57.121:3306)
Mon Dec 31 21:52:44 2018 - [info] Alive Slaves:
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.100(192.168.57.100:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:44 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.121(192.168.57.121:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:44 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Dec 31 21:52:44 2018 - [info] Starting Non-GTID based failover.
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] * Phase 2: Dead Master Shutdown Phase…
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] Forcing shutdown so that applications never connect to the current master…
Mon Dec 31 21:52:44 2018 - [info] Executing master IP deactivation script:
Mon Dec 31 21:52:44 2018 - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.57.200 --orig_master_ip=192.168.57.200 --orig_master_port=3306 --command=stopssh --ssh_user=root
Mon Dec 31 21:52:44 2018 - [info] done.
Mon Dec 31 21:52:44 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Mon Dec 31 21:52:44 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] * Phase 3: Master Recovery Phase…
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase…
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000006:120
Mon Dec 31 21:52:44 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.100(192.168.57.100:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:44 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.121(192.168.57.121:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:44 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Dec 31 21:52:44 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000006:120
Mon Dec 31 21:52:44 2018 - [info] Oldest slaves:
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.100(192.168.57.100:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:44 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Dec 31 21:52:44 2018 - [info] 192.168.57.121(192.168.57.121:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:44 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:44 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] * Phase 3.2: Saving Dead Master’s Binlog Phase…
Mon Dec 31 21:52:44 2018 - [info]
Mon Dec 31 21:52:44 2018 - [info] Fetching dead master’s binary logs…
Mon Dec 31 21:52:44 2018 - [info] Executing command on the dead master 192.168.57.200(192.168.57.200:3306): save_binary_logs --command=save --start_file=mysql-bin.000006 --start_pos=120 --binlog_dir=/data/mysql/data --output_file=/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56
Creating /var/log/masterha/app1 if not exists… ok.
Concat binary/relay logs from mysql-bin.000006 pos 120 to mysql-bin.000006 EOF into /var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog …
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 120… ok.
Dumping effective binlog data from /data/mysql/data/mysql-bin.000006 position 120 to tail(143)… ok.
Binlog Checksum enabled
Concat succeeded.
Mon Dec 31 21:52:45 2018 - [info] scp from [email protected]:/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog to local:/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog succeeded.
Mon Dec 31 21:52:45 2018 - [info] HealthCheck: SSH to 192.168.57.100 is reachable.
Mon Dec 31 21:52:46 2018 - [info] HealthCheck: SSH to 192.168.57.121 is reachable.
Mon Dec 31 21:52:46 2018 - [info]
Mon Dec 31 21:52:46 2018 - [info] * Phase 3.3: Determining New Master Phase…
Mon Dec 31 21:52:46 2018 - [info]
Mon Dec 31 21:52:46 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves…
Mon Dec 31 21:52:46 2018 - [info] All slaves received relay logs to the same position. No need to resync each other.
Mon Dec 31 21:52:46 2018 - [info] Searching new master from slaves…
Mon Dec 31 21:52:46 2018 - [info] Candidate masters from the configuration file:
Mon Dec 31 21:52:46 2018 - [info] 192.168.57.100(192.168.57.100:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:46 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:46 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Dec 31 21:52:46 2018 - [info] Non-candidate masters:
Mon Dec 31 21:52:46 2018 - [info] 192.168.57.121(192.168.57.121:3306) Version=5.6.29-log (oldest major version between slaves) log-bin:enabled
Mon Dec 31 21:52:46 2018 - [info] Replicating from 192.168.57.200(192.168.57.200:3306)
Mon Dec 31 21:52:46 2018 - [info] Not candidate for the new Master (no_master is set)
Mon Dec 31 21:52:46 2018 - [info] Searching from candidate_master slaves which have received the latest relay log events…
Mon Dec 31 21:52:46 2018 - [info] New master is 192.168.57.100(192.168.57.100:3306)
Mon Dec 31 21:52:46 2018 - [info] Starting master failover…
Mon Dec 31 21:52:46 2018 - [info]
From:
192.168.57.200(192.168.57.200:3306) (current master)
±-192.168.57.100(192.168.57.100:3306)
±-192.168.57.121(192.168.57.121:3306)

To:
192.168.57.100(192.168.57.100:3306) (new master)
±-192.168.57.121(192.168.57.121:3306)
Mon Dec 31 21:52:46 2018 - [info]
Mon Dec 31 21:52:46 2018 - [info] * Phase 3.3: New Master Diff Log Generation Phase…
Mon Dec 31 21:52:46 2018 - [info]
Mon Dec 31 21:52:46 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Mon Dec 31 21:52:46 2018 - [info] Sending binlog…
Mon Dec 31 21:52:47 2018 - [info] scp from local:/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog to [email protected]:/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog succeeded.
Mon Dec 31 21:52:47 2018 - [info]
Mon Dec 31 21:52:47 2018 - [info] * Phase 3.4: Master Log Apply Phase…
Mon Dec 31 21:52:47 2018 - [info]
Mon Dec 31 21:52:47 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Mon Dec 31 21:52:47 2018 - [info] Starting recovery on 192.168.57.100(192.168.57.100:3306)…
Mon Dec 31 21:52:47 2018 - [info] Generating diffs succeeded.
Mon Dec 31 21:52:47 2018 - [info] Waiting until all relay logs are applied.
Mon Dec 31 21:52:47 2018 - [info] done.
Mon Dec 31 21:52:47 2018 - [info] Getting slave status…
Mon Dec 31 21:52:47 2018 - [info] This slave(192.168.57.100)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
Mon Dec 31 21:52:47 2018 - [info] Connecting to the target slave host 192.168.57.100, running recover script…
Mon Dec 31 21:52:47 2018 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=‘mha’ --slave_host=192.168.57.100 --slave_ip=192.168.57.100 --slave_port=3306 --apply_files=/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog --workdir=/var/log/masterha/app1 --target_version=5.6.29-log --timestamp=20181231215243 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Mon Dec 31 21:52:47 2018 - [info]
MySQL client version is 5.6.29. Using --binary-mode.
Applying differential binary/relay log files /var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog on 192.168.57.100:3306. This may take long time…
Applying log files succeeded.
Mon Dec 31 21:52:47 2018 - [info] All relay logs were successfully applied.
Mon Dec 31 21:52:47 2018 - [info] Getting new master’s binlog name and position…
Mon Dec 31 21:52:47 2018 - [info] mysql-bin.000005:120
Mon Dec 31 21:52:47 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST=‘192.168.57.100’, MASTER_PORT=3306, MASTER_LOG_FILE=‘mysql-bin.000005’, MASTER_LOG_POS=120, MASTER_USER=‘bak’, MASTER_PASSWORD=‘xxx’;
Mon Dec 31 21:52:47 2018 - [info] Executing master IP activate script:
Mon Dec 31 21:52:47 2018 - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.57.200 --orig_master_ip=192.168.57.200 --orig_master_port=3306 --new_master_host=192.168.57.100 --new_master_ip=192.168.57.100 --new_master_port=3306 --new_master_user=‘mha’ --new_master_password=‘MHAaDmIn221’
Set read_only=0 on the new master.
Mon Dec 31 21:52:48 2018 - [info] OK.
Mon Dec 31 21:52:48 2018 - [info] ** Finished master recovery successfully.
Mon Dec 31 21:52:48 2018 - [info] * Phase 3: Master Recovery Phase completed.
Mon Dec 31 21:52:48 2018 - [info]
Mon Dec 31 21:52:48 2018 - [info] * Phase 4: Slaves Recovery Phase…
Mon Dec 31 21:52:48 2018 - [info]
Mon Dec 31 21:52:48 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase…
Mon Dec 31 21:52:48 2018 - [info]
Mon Dec 31 21:52:48 2018 - [info] – Slave diff file generation on host 192.168.57.121(192.168.57.121:3306) started, pid: 4046. Check tmp log /var/log/masterha/app1/192.168.57.121_3306_20181231215243.log if it takes time…
Mon Dec 31 21:52:49 2018 - [info]
Mon Dec 31 21:52:49 2018 - [info] Log messages from 192.168.57.121 …
Mon Dec 31 21:52:49 2018 - [info]
Mon Dec 31 21:52:48 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Mon Dec 31 21:52:49 2018 - [info] End of log messages from 192.168.57.121.
Mon Dec 31 21:52:49 2018 - [info] – 192.168.57.121(192.168.57.121:3306) has the latest relay log events.
Mon Dec 31 21:52:49 2018 - [info] Generating relay diff files from the latest slave succeeded.
Mon Dec 31 21:52:49 2018 - [info]
Mon Dec 31 21:52:49 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase…
Mon Dec 31 21:52:49 2018 - [info]
Mon Dec 31 21:52:49 2018 - [info] – Slave recovery on host 192.168.57.121(192.168.57.121:3306) started, pid: 4048. Check tmp log /var/log/masterha/app1/192.168.57.121_3306_20181231215243.log if it takes time…
Mon Dec 31 21:52:51 2018 - [info]
Mon Dec 31 21:52:51 2018 - [info] Log messages from 192.168.57.121 …
Mon Dec 31 21:52:51 2018 - [info]
Mon Dec 31 21:52:49 2018 - [info] Sending binlog…
Mon Dec 31 21:52:49 2018 - [info] scp from local:/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog to [email protected]:/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog succeeded.
Mon Dec 31 21:52:49 2018 - [info] Starting recovery on 192.168.57.121(192.168.57.121:3306)…
Mon Dec 31 21:52:49 2018 - [info] Generating diffs succeeded.
Mon Dec 31 21:52:49 2018 - [info] Waiting until all relay logs are applied.
Mon Dec 31 21:52:49 2018 - [info] done.
Mon Dec 31 21:52:49 2018 - [info] Getting slave status…
Mon Dec 31 21:52:49 2018 - [info] This slave(192.168.57.121)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
Mon Dec 31 21:52:49 2018 - [info] Connecting to the target slave host 192.168.57.121, running recover script…
Mon Dec 31 21:52:49 2018 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=‘mha’ --slave_host=192.168.57.121 --slave_ip=192.168.57.121 --slave_port=3306 --apply_files=/var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog --workdir=/var/log/masterha/app1 --target_version=5.6.29-log --timestamp=20181231215243 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Mon Dec 31 21:52:50 2018 - [info]
MySQL client version is 5.6.29. Using --binary-mode.
Applying differential binary/relay log files /var/log/masterha/app1/saved_master_binlog_from_192.168.57.200_3306_20181231215243.binlog on 192.168.57.121:3306. This may take long time…
Applying log files succeeded.
Mon Dec 31 21:52:50 2018 - [info] All relay logs were successfully applied.
Mon Dec 31 21:52:50 2018 - [info] Resetting slave 192.168.57.121(192.168.57.121:3306) and starting replication from the new master 192.168.57.100(192.168.57.100:3306)…
Mon Dec 31 21:52:50 2018 - [info] Executed CHANGE MASTER.
Mon Dec 31 21:52:50 2018 - [info] Slave started.
Mon Dec 31 21:52:51 2018 - [info] End of log messages from 192.168.57.121.
Mon Dec 31 21:52:51 2018 - [info] – Slave recovery on host 192.168.57.121(192.168.57.121:3306) succeeded.
Mon Dec 31 21:52:51 2018 - [info] All new slave servers recovered successfully.
Mon Dec 31 21:52:51 2018 - [info]
Mon Dec 31 21:52:51 2018 - [info] * Phase 5: New master cleanup phase…
Mon Dec 31 21:52:51 2018 - [info]
Mon Dec 31 21:52:51 2018 - [info] Resetting slave info on the new master…
Mon Dec 31 21:52:51 2018 - [info] 192.168.57.100: Resetting slave info succeeded.
Mon Dec 31 21:52:51 2018 - [info] Master failover to 192.168.57.100(192.168.57.100:3306) completed successfully.
Mon Dec 31 21:52:51 2018 - [info]

----- Failover Report -----

app1: MySQL Master failover 192.168.57.200(192.168.57.200:3306) to 192.168.57.100(192.168.57.100:3306) succeeded

Master 192.168.57.200(192.168.57.200:3306) is down!

Check MHA Manager logs at h:/var/log/masterha/app1/app1.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.57.200(192.168.57.200:3306)
The latest slave 192.168.57.100(192.168.57.100:3306) has all relay logs for recovery.
Selected 192.168.57.100(192.168.57.100:3306) as a new master.
192.168.57.100(192.168.57.100:3306): OK: Applying all logs succeeded.
192.168.57.100(192.168.57.100:3306): OK: Activated master IP address.
192.168.57.121(192.168.57.121:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.57.121(192.168.57.121:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.57.100(192.168.57.100:3306)
192.168.57.100(192.168.57.100:3306): Resetting slave info succeeded.
Master failover to 192.168.57.100(192.168.57.100:3306) completed successfully.

当发现主库不可用时，开始failover操作
1、配置检查，从库中找到可用的mysql数据库
2、获取最新的slave
3、保存原主库挂掉时的binlog日志
4、选择可用新的主库
5、应用差异中继日志到最新的slave
6、将宕掉主库上保存的binlog日志scp到最新的slave
7、在最新的slave应用
8、All relay logs被应用完在最新的slave，后获取binlog name and position并将指向新主库主从复制搭建的语句打印出来，
9、然后Executing master IP activate script
10、在其他的slave应用差异中继日志及原主库保存的binlog
11，将其他的slave指向新的主库并开启主从复制