版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/zhangjikuan/article/details/82456129
MHA manager与node命令交互
manager上执行masterha_manager
sudo masterha_manager --conf=/u01/zjk/mha4mysql-manager-master/workdir/conf/app1.cnf --skip_check_ssh- 打印日志
Wed Aug 22 20:20:36 2018 - [info] MHA::MasterMonitor version 0.57.
Wed Aug 22 20:20:36 2018 - [debug] Connecting to servers..
Wed Aug 22 20:20:37 2018 - [debug] Connected to: 11.136.105.70(11.136.105.70:3306), user=adha
Wed Aug 22 20:20:37 2018 - [debug] Number of slave worker threads on host 11.136.105.70(11.136.105.70:3306): 8
Wed Aug 22 20:20:37 2018 - [debug] Connected to: 11.187.250.60(11.187.250.60:3306), user=adha
Wed Aug 22 20:20:37 2018 - [debug] Number of slave worker threads on host 11.187.250.60(11.187.250.60:3306): 8
Wed Aug 22 20:20:37 2018 - [debug] Connected to: 11.136.167.69(11.136.167.69:3306), user=adha
Wed Aug 22 20:20:37 2018 - [debug] Number of slave worker threads on host 11.136.167.69(11.136.167.69:3306): 8
Wed Aug 22 20:20:37 2018 - [debug] Comparing MySQL versions..
Wed Aug 22 20:20:37 2018 - [debug] Comparing MySQL versions done.
Wed Aug 22 20:20:37 2018 - [debug] Connecting to servers done.
Wed Aug 22 20:20:37 2018 - [info] Multi-master configuration is detected. Current primary(writable) master is 11.136.105.70(11.136.105.70:3306)
Wed Aug 22 20:20:37 2018 - [info] Master configurations are as below:
Master 11.187.250.60(11.187.250.60:3306), replicating from 11.136.105.70(11.136.105.70:3306), read-only
Master 11.136.105.70(11.136.105.70:3306), replicating from 11.187.250.60(11.187.250.60:3306)
Wed Aug 22 20:20:37 2018 - [info] GTID failover mode = 0
Wed Aug 22 20:20:37 2018 - [info] Dead Servers:
Wed Aug 22 20:20:37 2018 - [info] Alive Servers:
Wed Aug 22 20:20:37 2018 - [info] 11.136.105.70(11.136.105.70:3306)
Wed Aug 22 20:20:37 2018 - [info] 11.187.250.60(11.187.250.60:3306)
Wed Aug 22 20:20:37 2018 - [info] 11.136.167.69(11.136.167.69:3306)
Wed Aug 22 20:20:37 2018 - [info] Alive Slaves:
Wed Aug 22 20:20:37 2018 - [info] 11.187.250.60(11.187.250.60:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Aug 22 20:20:37 2018 - [debug] Relay log info repository: TABLE
Wed Aug 22 20:20:37 2018 - [info] Replicating from 11.136.105.70(11.136.105.70:3306)
Wed Aug 22 20:20:37 2018 - [info] 11.136.167.69(11.136.167.69:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Aug 22 20:20:37 2018 - [debug] Relay log info repository: TABLE
Wed Aug 22 20:20:37 2018 - [info] Replicating from 11.136.105.70(11.136.105.70:3306)
Wed Aug 22 20:20:37 2018 - [info] Current Alive Master: 11.136.105.70(11.136.105.70:3306)
Wed Aug 22 20:20:37 2018 - [info] Checking slave configurations..
Wed Aug 22 20:20:37 2018 - [info] Checking replication filtering settings..
Wed Aug 22 20:20:37 2018 - [info] binlog_do_db= , binlog_ignore_db=
Wed Aug 22 20:20:37 2018 - [info] Replication filtering check ok.
Wed Aug 22 20:20:37 2018 - [info] GTID (with auto-pos) is not supported
Wed Aug 22 20:20:37 2018 - [info] Checking MHA Node version..
Wed Aug 22 20:20:37 2018 - [info] Version check ok.
Wed Aug 22 20:20:37 2018 - [info] Checking SSH publickey authentication settings on the current master..
Wed Aug 22 20:20:37 2018 - [debug] SSH connection test to 11.136.105.70, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5, timeout 5
Wed Aug 22 20:20:38 2018 - [info] HealthCheck: SSH to 11.136.105.70 is reachable.
Wed Aug 22 20:20:38 2018 - [info] Master MHA Node version is 0.57.
Wed Aug 22 20:20:38 2018 - [info] Checking recovery script configurations on 11.136.105.70(11.136.105.70:3306)..
Wed Aug 22 20:20:38 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000002 --debug
Wed Aug 22 20:20:38 2018 - [info] Connecting to root@11.136.105.70(11.136.105.70:22)..
Creating /opt/TMHA2/mha4mysql-node-master/workdir if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /u01/my3306/log, up to mysql-bin.000002
Wed Aug 22 20:20:38 2018 - [info] Binlog setting check done.
Wed Aug 22 20:20:38 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Wed Aug 22 20:20:38 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='adha' --slave_host=11.187.250.60 --slave_ip=11.187.250.60 --slave_port=3306 --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.001685 --debug --slave_pass=xxx
Wed Aug 22 20:20:38 2018 - [info] Connecting to root@11.187.250.60(11.187.250.60:22)..
Checking slave recovery environment settings..
Relay log found at /u01/my3306/log, up to relaylog.001685
Temporary relay log file is /u01/my3306/log/relaylog.001685
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Wed Aug 22 20:20:38 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='adha' --slave_host=11.136.167.69 --slave_ip=11.136.167.69 --slave_port=3306 --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.001721 --debug --slave_pass=xxx
Wed Aug 22 20:20:38 2018 - [info] Connecting to root@11.136.167.69(11.136.167.69:22)..
Creating directory /opt/TMHA2/mha4mysql-node-master/workdir.. done.
Checking slave recovery environment settings..
Relay log found at /u01/my3306/log, up to relaylog.001721
Temporary relay log file is /u01/my3306/log/relaylog.001721
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Wed Aug 22 20:20:39 2018 - [info] Slaves settings check done.
Wed Aug 22 20:20:39 2018 - [info]
11.136.105.70(11.136.105.70:3306) (current master)
+--11.187.250.60(11.187.250.60:3306)
+--11.136.167.69(11.136.167.69:3306)
Wed Aug 22 20:20:39 2018 - [warning] master_ip_failover_script is not defined.
Wed Aug 22 20:20:39 2018 - [warning] shutdown_script is not defined.
Wed Aug 22 20:20:39 2018 - [debug] Disconnected from 11.136.105.70(11.136.105.70:3306)
Wed Aug 22 20:20:39 2018 - [debug] Disconnected from 11.187.250.60(11.187.250.60:3306)
Wed Aug 22 20:20:39 2018 - [debug] Disconnected from 11.136.167.69(11.136.167.69:3306)
Wed Aug 22 20:20:39 2018 - [debug] SSH check command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin --debug
Wed Aug 22 20:20:39 2018 - [info] Set master ping interval 1 seconds.
Wed Aug 22 20:20:39 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Wed Aug 22 20:20:39 2018 - [info] Starting ping health check on 11.136.105.70(11.136.105.70:3306)..
Wed Aug 22 20:20:39 2018 - [debug] Connected on master.
Wed Aug 22 20:20:39 2018 - [debug] Set short wait_timeout on master: 2 seconds
Wed Aug 22 20:20:39 2018 - [debug] Trying to get advisory lock..
Wed Aug 22 20:20:39 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..- 主库down打印日志
Wed Sep 5 20:50:42 2018 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)
Wed Sep 5 20:50:42 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin --debug
Wed Sep 5 20:50:42 2018 - [debug] SSH connection test to 11.187.250.60, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5, timeout 5
Creating /opt/TMHA2/mha4mysql-node-master/workdir if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /u01/my3306/log, up to mysql-bin.002290
Wed Sep 5 20:50:42 2018 - [info] HealthCheck: SSH to 11.187.250.60 is reachable.
Wed Sep 5 20:50:43 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '11.187.250.60' (111))
Wed Sep 5 20:50:43 2018 - [warning] Connection failed 2 time(s)..
Wed Sep 5 20:50:44 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '11.187.250.60' (111))
Wed Sep 5 20:50:44 2018 - [warning] Connection failed 3 time(s)..
Wed Sep 5 20:50:45 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '11.187.250.60' (111))
Wed Sep 5 20:50:45 2018 - [warning] Connection failed 4 time(s)..
Wed Sep 5 20:50:45 2018 - [warning] Master is not reachable from health checker!
Wed Sep 5 20:50:45 2018 - [warning] Master 11.187.250.60(11.187.250.60:3306) is not reachable!
Wed Sep 5 20:50:45 2018 - [warning] SSH is reachable.
Wed Sep 5 20:50:45 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /u01/zjk/mha4mysql-manager-master/workdir/conf/app1.cnf again, and trying to connect to all servers to check server status..
Wed Sep 5 20:50:45 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Sep 5 20:50:45 2018 - [info] Reading application default configuration from /u01/zjk/mha4mysql-manager-master/workdir/conf/app1.cnf..
Wed Sep 5 20:50:45 2018 - [info] Reading server configuration from /u01/zjk/mha4mysql-manager-master/workdir/conf/app1.cnf..
Wed Sep 5 20:50:45 2018 - [debug] Skipping connecting to dead master 11.187.250.60(11.187.250.60:3306).
Wed Sep 5 20:50:45 2018 - [debug] Connecting to servers..
Wed Sep 5 20:50:46 2018 - [debug] Connected to: 11.136.105.70(11.136.105.70:3306), user=adha
Wed Sep 5 20:50:46 2018 - [debug] Number of slave worker threads on host 11.136.105.70(11.136.105.70:3306): 8
Wed Sep 5 20:50:46 2018 - [debug] Connected to: 11.136.167.69(11.136.167.69:3306), user=adha
Wed Sep 5 20:50:46 2018 - [debug] Number of slave worker threads on host 11.136.167.69(11.136.167.69:3306): 8
Wed Sep 5 20:50:46 2018 - [debug] Comparing MySQL versions..
Wed Sep 5 20:50:46 2018 - [debug] Comparing MySQL versions done.
Wed Sep 5 20:50:46 2018 - [debug] Connecting to servers done.
Wed Sep 5 20:50:46 2018 - [info] GTID failover mode = 0
Wed Sep 5 20:50:46 2018 - [info] Dead Servers:
Wed Sep 5 20:50:46 2018 - [info] 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:46 2018 - [info] Alive Servers:
Wed Sep 5 20:50:46 2018 - [info] 11.136.105.70(11.136.105.70:3306)
Wed Sep 5 20:50:46 2018 - [info] 11.136.167.69(11.136.167.69:3306)
Wed Sep 5 20:50:46 2018 - [info] Alive Slaves:
Wed Sep 5 20:50:46 2018 - [info] 11.136.105.70(11.136.105.70:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Sep 5 20:50:46 2018 - [debug] Relay log info repository: TABLE
Wed Sep 5 20:50:46 2018 - [info] Replicating from 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:46 2018 - [info] 11.136.167.69(11.136.167.69:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Sep 5 20:50:46 2018 - [debug] Relay log info repository: TABLE
Wed Sep 5 20:50:46 2018 - [info] Replicating from 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:46 2018 - [info] Checking slave configurations..
Wed Sep 5 20:50:46 2018 - [info] Checking replication filtering settings..
Wed Sep 5 20:50:46 2018 - [info] Replication filtering check ok.
Wed Sep 5 20:50:46 2018 - [info] Master is down!
Wed Sep 5 20:50:46 2018 - [info] Terminating monitoring script.
Wed Sep 5 20:50:46 2018 - [debug] Disconnected from 11.136.105.70(11.136.105.70:3306)
Wed Sep 5 20:50:46 2018 - [debug] Disconnected from 11.136.167.69(11.136.167.69:3306)
Wed Sep 5 20:50:46 2018 - [info] Got exit code 20 (Master dead).
Wed Sep 5 20:50:46 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Sep 5 20:50:46 2018 - [info] Reading application default configuration from /u01/zjk/mha4mysql-manager-master/workdir/conf/app1.cnf..
Wed Sep 5 20:50:46 2018 - [info] Reading server configuration from /u01/zjk/mha4mysql-manager-master/workdir/conf/app1.cnf..
Wed Sep 5 20:50:46 2018 - [info] MHA::MasterFailover version 0.57.
Wed Sep 5 20:50:46 2018 - [info] Starting master failover.
Wed Sep 5 20:50:46 2018 - [info]
Wed Sep 5 20:50:46 2018 - [info] * Phase 1: Configuration Check Phase..
Wed Sep 5 20:50:46 2018 - [info]
Wed Sep 5 20:50:46 2018 - [debug] Skipping connecting to dead master 11.187.250.60.
Wed Sep 5 20:50:46 2018 - [debug] Connecting to servers..
Wed Sep 5 20:50:47 2018 - [debug] Connected to: 11.136.105.70(11.136.105.70:3306), user=adha
Wed Sep 5 20:50:47 2018 - [debug] Number of slave worker threads on host 11.136.105.70(11.136.105.70:3306): 8
Wed Sep 5 20:50:47 2018 - [debug] Connected to: 11.136.167.69(11.136.167.69:3306), user=adha
Wed Sep 5 20:50:47 2018 - [debug] Number of slave worker threads on host 11.136.167.69(11.136.167.69:3306): 8
Wed Sep 5 20:50:47 2018 - [debug] Comparing MySQL versions..
Wed Sep 5 20:50:47 2018 - [debug] Comparing MySQL versions done.
Wed Sep 5 20:50:47 2018 - [debug] Connecting to servers done.
Wed Sep 5 20:50:47 2018 - [info] GTID failover mode = 0
Wed Sep 5 20:50:47 2018 - [info] Dead Servers:
Wed Sep 5 20:50:47 2018 - [info] 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:47 2018 - [info] Checking master reachability via MySQL(double check)...
Wed Sep 5 20:50:47 2018 - [info] ok.
Wed Sep 5 20:50:47 2018 - [info] Alive Servers:
Wed Sep 5 20:50:47 2018 - [info] 11.136.105.70(11.136.105.70:3306)
Wed Sep 5 20:50:47 2018 - [info] 11.136.167.69(11.136.167.69:3306)
Wed Sep 5 20:50:47 2018 - [info] Alive Slaves:
Wed Sep 5 20:50:47 2018 - [info] 11.136.105.70(11.136.105.70:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Sep 5 20:50:47 2018 - [debug] Relay log info repository: TABLE
Wed Sep 5 20:50:47 2018 - [info] Replicating from 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:47 2018 - [info] 11.136.167.69(11.136.167.69:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Sep 5 20:50:47 2018 - [debug] Relay log info repository: TABLE
Wed Sep 5 20:50:47 2018 - [info] Replicating from 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:47 2018 - [info] Starting Non-GTID based failover.
Wed Sep 5 20:50:47 2018 - [info]
Wed Sep 5 20:50:47 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Wed Sep 5 20:50:47 2018 - [info]
Wed Sep 5 20:50:47 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
Wed Sep 5 20:50:47 2018 - [info]
Wed Sep 5 20:50:47 2018 - [debug] Stopping IO thread on 11.136.105.70(11.136.105.70:3306)..
Wed Sep 5 20:50:47 2018 - [info] Forcing shutdown so that applications never connect to the current master..
Wed Sep 5 20:50:47 2018 - [debug] Stopping IO thread on 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:47 2018 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
Wed Sep 5 20:50:47 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Wed Sep 5 20:50:47 2018 - [debug] Stop IO thread on 11.136.105.70(11.136.105.70:3306) done.
Wed Sep 5 20:50:47 2018 - [debug] Stop IO thread on 11.136.167.69(11.136.167.69:3306) done.
Wed Sep 5 20:50:48 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Wed Sep 5 20:50:48 2018 - [info]
Wed Sep 5 20:50:48 2018 - [info] * Phase 3: Master Recovery Phase..
Wed Sep 5 20:50:48 2018 - [info]
Wed Sep 5 20:50:48 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Wed Sep 5 20:50:48 2018 - [info]
Wed Sep 5 20:50:48 2018 - [debug] Fetching current slave status..
Wed Sep 5 20:50:48 2018 - [debug] Fetching current slave status done.
Wed Sep 5 20:50:48 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.002290:8646412
Wed Sep 5 20:50:48 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Wed Sep 5 20:50:48 2018 - [info] 11.136.167.69(11.136.167.69:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Sep 5 20:50:48 2018 - [debug] Relay log info repository: TABLE
Wed Sep 5 20:50:48 2018 - [info] Replicating from 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:48 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.002290:8609518
Wed Sep 5 20:50:48 2018 - [info] Oldest slaves:
Wed Sep 5 20:50:48 2018 - [info] 11.136.105.70(11.136.105.70:3306) Version=5.6.37-20180708-log (oldest major version between slaves) log-bin:enabled
Wed Sep 5 20:50:48 2018 - [debug] Relay log info repository: TABLE
Wed Sep 5 20:50:48 2018 - [info] Replicating from 11.187.250.60(11.187.250.60:3306)
Wed Sep 5 20:50:48 2018 - [info]
Wed Sep 5 20:50:48 2018 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Wed Sep 5 20:50:48 2018 - [info]
Wed Sep 5 20:50:49 2018 - [info] Fetching dead master's binary logs..
Wed Sep 5 20:50:49 2018 - [info] Executing command on the dead master 11.187.250.60(11.187.250.60:3306): save_binary_logs --command=save --start_file=mysql-bin.002290 --start_pos=8646412 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --debug
Creating /opt/TMHA2/mha4mysql-node-master/workdir if not exists.. ok.
Concat binary/relay logs from mysql-bin.002290 pos 8646412 to mysql-bin.002290 EOF into /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog ..
ok.
Concat binary/relay logs from mysql-bin.002290 pos 8646412 to mysql-bin.002290 EOF into /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog ..
parse_init_headers: file=mysql-bin.002290 event_type=15 server_id=196867798 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=mysql-bin.002290 event_type=2 server_id=196867798 length=91 nextmpos=211 prevrelay=120 cur(post)relay=211
Dumping binlog format description event, from position 0 to 120.. ok.
Dumping effective binlog data from /u01/my3306/log/mysql-bin.002290 position 8646412 to tail(8663384).. ok.
parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=15 server_id=196867798 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=2 server_id=196867798 length=77 nextmpos=8646489 prevrelay=120 cur(post)relay=197
Concat succeeded.
Wed Sep 5 20:50:49 2018 - [info] scp from root@11.187.250.60:/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog to local:/u01/zjk/mha4mysql-manager-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog succeeded.
Wed Sep 5 20:50:49 2018 - [debug] SSH connection test to 11.136.105.70, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5, timeout 5
Wed Sep 5 20:50:49 2018 - [info] HealthCheck: SSH to 11.136.105.70 is reachable.
Wed Sep 5 20:50:49 2018 - [debug] SSH connection test to 11.136.167.69, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5, timeout 5
Wed Sep 5 20:50:50 2018 - [info] HealthCheck: SSH to 11.136.167.69 is reachable.
Wed Sep 5 20:50:50 2018 - [info]
Wed Sep 5 20:50:50 2018 - [info] * Phase 3.3: Determining New Master Phase..
Wed Sep 5 20:50:50 2018 - [info]
Wed Sep 5 20:50:50 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Wed Sep 5 20:50:50 2018 - [info] Checking whether 11.136.167.69 has relay logs from the oldest position..
Wed Sep 5 20:50:50 2018 - [info] Executing command: apply_diff_relay_logs --command=find --latest_mlf=mysql-bin.002290 --latest_rmlp=8646412 --target_mlf=mysql-bin.002290 --target_rmlp=8609518 --server_id=193506223 --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --timestamp=20180905205046 --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.002564 --debug :
Relay log found at /u01/my3306/log, up to relaylog.002564
Fast relay log position search succeeded.
Target relay log file/position found. start_file:relaylog.002564, start_pos:391934.
Target relay log FOUND!
Wed Sep 5 20:50:50 2018 - [info] OK. 11.136.167.69 has all relay logs.
Wed Sep 5 20:50:50 2018 - [debug] Checking replication delay on 11.136.105.70(11.136.105.70:3306)..
Wed Sep 5 20:50:50 2018 - [debug] ok.
Wed Sep 5 20:50:50 2018 - [debug] Checking replication delay on 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:50 2018 - [debug] ok.
Wed Sep 5 20:50:50 2018 - [info] Searching new master from slaves..
Wed Sep 5 20:50:50 2018 - [info] Candidate masters from the configuration file:
Wed Sep 5 20:50:50 2018 - [info] Non-candidate masters:
Wed Sep 5 20:50:50 2018 - [info] New master is 11.136.167.69(11.136.167.69:3306)
Wed Sep 5 20:50:50 2018 - [info] Starting master failover..
Wed Sep 5 20:50:50 2018 - [info]
From:
11.187.250.60(11.187.250.60:3306) (current master)
+--11.136.105.70(11.136.105.70:3306)
+--11.136.167.69(11.136.167.69:3306)
To:
11.136.167.69(11.136.167.69:3306) (new master)
+--11.136.105.70(11.136.105.70:3306)
Wed Sep 5 20:50:50 2018 - [info]
Wed Sep 5 20:50:50 2018 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Wed Sep 5 20:50:50 2018 - [info]
Wed Sep 5 20:50:50 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Wed Sep 5 20:50:50 2018 - [info] Sending binlog..
Wed Sep 5 20:50:50 2018 - [info] scp from local:/u01/zjk/mha4mysql-manager-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog to root@11.136.167.69:/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog succeeded.
Wed Sep 5 20:50:50 2018 - [info]
Wed Sep 5 20:50:50 2018 - [info] * Phase 3.4: Master Log Apply Phase..
Wed Sep 5 20:50:50 2018 - [info]
Wed Sep 5 20:50:50 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Wed Sep 5 20:50:50 2018 - [info] Starting recovery on 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:50 2018 - [info] Generating diffs succeeded.
Wed Sep 5 20:50:50 2018 - [info] Waiting until all relay logs are applied.
Wed Sep 5 20:50:50 2018 - [info] done.
Wed Sep 5 20:50:50 2018 - [debug] Stopping SQL thread on 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:50 2018 - [debug] done.
Wed Sep 5 20:50:50 2018 - [info] Getting slave status..
Wed Sep 5 20:50:50 2018 - [info] This slave(11.136.167.69)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.002290:8646412). No need to recover from Exec_Master_Log_Pos.
Wed Sep 5 20:50:50 2018 - [debug] Current max_allowed_packet is 134217728.
Wed Sep 5 20:50:50 2018 - [debug] Tentatively setting max_allowed_packet to 1GB succeeded.
Wed Sep 5 20:50:50 2018 - [info] Connecting to the target slave host 11.136.167.69, running recover script..
Wed Sep 5 20:50:50 2018 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='adha' --slave_host=11.136.167.69 --slave_ip=11.136.167.69 --slave_port=3306 --apply_files=/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --timestamp=20180905205046 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --debug --slave_pass=xxx
Wed Sep 5 20:50:51 2018 - [info]
MySQL client version is 5.6.37. Using --binary-mode.
Applying differential binary/relay log files /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog on 11.136.167.69:3306. This may take long time...
Applying log files succeeded.
Wed Sep 5 20:50:51 2018 - [debug] Setting max_allowed_packet back to 134217728 succeeded.
Wed Sep 5 20:50:51 2018 - [info] All relay logs were successfully applied.
Wed Sep 5 20:50:51 2018 - [info] Getting new master's binlog name and position..
Wed Sep 5 20:50:51 2018 - [info] mysql-bin.003087:420069
Wed Sep 5 20:50:51 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='11.136.167.69', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.003087', MASTER_LOG_POS=420069, MASTER_USER='slave', MASTER_PASSWORD='xxx';
Wed Sep 5 20:50:51 2018 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Wed Sep 5 20:50:51 2018 - [info] Setting read_only=0 on 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:51 2018 - [info] ok.
Wed Sep 5 20:50:51 2018 - [info] ** Finished master recovery successfully.
Wed Sep 5 20:50:51 2018 - [info] * Phase 3: Master Recovery Phase completed.
Wed Sep 5 20:50:51 2018 - [info]
Wed Sep 5 20:50:51 2018 - [info] * Phase 4: Slaves Recovery Phase..
Wed Sep 5 20:50:51 2018 - [info]
Wed Sep 5 20:50:51 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Wed Sep 5 20:50:51 2018 - [info]
Wed Sep 5 20:50:51 2018 - [info] -- Slave diff file generation on host 11.136.105.70(11.136.105.70:3306) started, pid: 216831. Check tmp log /u01/zjk/mha4mysql-manager-master/workdir/11.136.105.70_3306_20180905205046.log if it takes time..
Wed Sep 5 20:50:53 2018 - [info]
Wed Sep 5 20:50:53 2018 - [info] Log messages from 11.136.105.70 ...
Wed Sep 5 20:50:53 2018 - [info]
Wed Sep 5 20:50:51 2018 - [info] Server 11.136.105.70 received relay logs up to: mysql-bin.002290:8609518
Wed Sep 5 20:50:51 2018 - [info] Need to get diffs from the latest slave(11.136.167.69) up to: mysql-bin.002290:8646412 (using the latest slave's relay logs)
Wed Sep 5 20:50:51 2018 - [info] Connecting to the latest slave host 11.136.167.69, generating diff relay log files..
Wed Sep 5 20:50:51 2018 - [info] Executing command: apply_diff_relay_logs --command=generate_and_send --scp_user=root --scp_host=11.136.105.70 --latest_mlf=mysql-bin.002290 --latest_rmlp=8646412 --target_mlf=mysql-bin.002290 --target_rmlp=8609518 --server_id=193506223 --diff_file_readtolatest=/opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --timestamp=20180905205046 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.002564 --debug
Wed Sep 5 20:50:51 2018 - [info]
Relay log found at /u01/my3306/log, up to relaylog.002564
Fast relay log position search succeeded.
Target relay log file/position found. start_file:relaylog.002564, start_pos:391934.
Concat binary/relay logs from relaylog.002564 pos 391934 to relaylog.002564 EOF into /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog ..
parse_init_headers: file=relaylog.002564 event_type=15 server_id=193506223 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=relaylog.002564 event_type=15 server_id=196867798 length=116 nextmpos=0 prevrelay=120 cur(post)relay=236
Binlog Checksum enabled
parse_init_headers: file=relaylog.002564 event_type=2 server_id=196867798 length=77 nextmpos=8217897 prevrelay=236 cur(post)relay=313
Dumping binlog format description event, from position 0 to 236.. ok.
Dumping effective binlog data from /u01/my3306/log/relaylog.002564 position 391934 to tail(428828).. ok.
parse_init_headers: file=relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog event_type=15 server_id=193506223 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog event_type=15 server_id=196867798 length=116 nextmpos=0 prevrelay=120 cur(post)relay=236
Binlog Checksum enabled
parse_init_headers: file=relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog event_type=2 server_id=196867798 length=77 nextmpos=8609595 prevrelay=236 cur(post)relay=313
Concat succeeded.
Generating diff relay log succeeded. Saved at /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog .
Permission denied (publickey,password).
ssh root@11.136.105.70 for mkdir /opt/TMHA2/mha4mysql-node-master/workdir failed with rc 255:0!
at /usr/local/bin/apply_diff_relay_logs line 450.
main::generate_and_send() called at /usr/local/bin/apply_diff_relay_logs line 497
eval {...} called at /usr/local/bin/apply_diff_relay_logs line 475
main::main() called at /usr/local/bin/apply_diff_relay_logs line 120
Wed Sep 5 20:50:51 2018 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln1089] Generating diff files failed with return code 1:0.
Wed Sep 5 20:50:51 2018 - [info] add by jikuan.zjk: may be permission denied, will scp diff_file from last_slave to manager, then scp from manager to slave
Wed Sep 5 20:50:52 2018 - [info] scp from root@11.136.167.69:/opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog to local:/u01/zjk/mha4mysql-manager-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog succeeded.
Wed Sep 5 20:50:52 2018 - [info] scp from local:/u01/zjk/mha4mysql-manager-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog to root@11.136.105.70:/opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog succeeded.
Wed Sep 5 20:50:52 2018 - [info] Generating diff files succeeded.
Wed Sep 5 20:50:53 2018 - [info] End of log messages from 11.136.105.70.
Wed Sep 5 20:50:53 2018 - [info] -- Slave diff log generation on host 11.136.105.70(11.136.105.70:3306) succeeded.
Wed Sep 5 20:50:53 2018 - [info] Generating relay diff files from the latest slave succeeded.
Wed Sep 5 20:50:53 2018 - [info]
Wed Sep 5 20:50:53 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Wed Sep 5 20:50:53 2018 - [info]
Wed Sep 5 20:50:53 2018 - [info] -- Slave recovery on host 11.136.105.70(11.136.105.70:3306) started, pid: 216853. Check tmp log /u01/zjk/mha4mysql-manager-master/workdir/11.136.105.70_3306_20180905205046.log if it takes time..
Wed Sep 5 20:50:54 2018 - [debug] Explicitly disabled relay_log_purge.
Wed Sep 5 20:50:54 2018 - [info]
Wed Sep 5 20:50:54 2018 - [info] Log messages from 11.136.105.70 ...
Wed Sep 5 20:50:54 2018 - [info]
Wed Sep 5 20:50:53 2018 - [info] Sending binlog..
Wed Sep 5 20:50:53 2018 - [info] scp from local:/u01/zjk/mha4mysql-manager-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog to root@11.136.105.70:/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog succeeded.
Wed Sep 5 20:50:53 2018 - [info] Starting recovery on 11.136.105.70(11.136.105.70:3306)..
Wed Sep 5 20:50:53 2018 - [info] Generating diffs succeeded.
Wed Sep 5 20:50:53 2018 - [info] Waiting until all relay logs are applied.
Wed Sep 5 20:50:53 2018 - [info] done.
Wed Sep 5 20:50:53 2018 - [debug] Stopping SQL thread on 11.136.105.70(11.136.105.70:3306)..
Wed Sep 5 20:50:53 2018 - [debug] done.
Wed Sep 5 20:50:53 2018 - [info] Getting slave status..
Wed Sep 5 20:50:53 2018 - [info] This slave(11.136.105.70)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.002290:8609518). No need to recover from Exec_Master_Log_Pos.
Wed Sep 5 20:50:53 2018 - [debug] Current max_allowed_packet is 134217728.
Wed Sep 5 20:50:53 2018 - [debug] Tentatively setting max_allowed_packet to 1GB succeeded.
Wed Sep 5 20:50:53 2018 - [info] Connecting to the target slave host 11.136.105.70, running recover script..
Wed Sep 5 20:50:53 2018 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='adha' --slave_host=11.136.105.70 --slave_ip=11.136.105.70 --slave_port=3306 --apply_files=/opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog,/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --timestamp=20180905205046 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --debug --slave_pass=xxx
Wed Sep 5 20:50:54 2018 - [info]
Concat all apply files to /opt/TMHA2/mha4mysql-node-master/workdir/total_binlog_for_11.136.105.70_3306.20180905205046.binlog ..
Copying the first binlog file /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog to /opt/TMHA2/mha4mysql-node-master/workdir/total_binlog_for_11.136.105.70_3306.20180905205046.binlog.. ok.
Dumping binlog head events (rotate events), skipping format description events from /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog.. parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=15 server_id=196867798 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=2 server_id=196867798 length=77 nextmpos=8646489 prevrelay=120 cur(post)relay=197
dumped up to pos 120. ok.
/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog has effective binlog events from pos 120.
Dumping effective binlog data from /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog position 120 to tail(17092).. ok.
Concat succeeded.
All apply target binary logs are concatinated at /opt/TMHA2/mha4mysql-node-master/workdir/total_binlog_for_11.136.105.70_3306.20180905205046.binlog .
MySQL client version is 5.6.37. Using --binary-mode.
Applying differential binary/relay log files /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog,/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog on 11.136.105.70:3306. This may take long time...
Applying log files succeeded.
Wed Sep 5 20:50:54 2018 - [debug] Setting max_allowed_packet back to 134217728 succeeded.
Wed Sep 5 20:50:54 2018 - [info] All relay logs were successfully applied.
Wed Sep 5 20:50:54 2018 - [info] Resetting slave 11.136.105.70(11.136.105.70:3306) and starting replication from the new master 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:54 2018 - [debug] Stopping slave IO/SQL thread on 11.136.105.70(11.136.105.70:3306)..
Wed Sep 5 20:50:54 2018 - [debug] done.
Wed Sep 5 20:50:54 2018 - [info] Executed CHANGE MASTER.
Wed Sep 5 20:50:54 2018 - [debug] Starting slave IO/SQL thread on 11.136.105.70(11.136.105.70:3306)..
Wed Sep 5 20:50:54 2018 - [debug] done.
Wed Sep 5 20:50:54 2018 - [info] Slave started.
Wed Sep 5 20:50:54 2018 - [info] End of log messages from 11.136.105.70.
Wed Sep 5 20:50:54 2018 - [info] -- Slave recovery on host 11.136.105.70(11.136.105.70:3306) succeeded.
Wed Sep 5 20:50:54 2018 - [info] All new slave servers recovered successfully.
Wed Sep 5 20:50:54 2018 - [info]
Wed Sep 5 20:50:54 2018 - [info] * Phase 5: New master cleanup phase..
Wed Sep 5 20:50:54 2018 - [info]
Wed Sep 5 20:50:54 2018 - [info] Resetting slave info on the new master..
Wed Sep 5 20:50:54 2018 - [debug] Clearing slave info..
Wed Sep 5 20:50:54 2018 - [debug] Stopping slave IO/SQL thread on 11.136.167.69(11.136.167.69:3306)..
Wed Sep 5 20:50:54 2018 - [debug] done.
Wed Sep 5 20:50:54 2018 - [debug] SHOW SLAVE STATUS shows new master does not replicate from anywhere. OK.
Wed Sep 5 20:50:54 2018 - [info] 11.136.167.69: Resetting slave info succeeded.
Wed Sep 5 20:50:54 2018 - [info] Master failover to 11.136.167.69(11.136.167.69:3306) completed successfully.
Wed Sep 5 20:50:54 2018 - [debug] Disconnected from 11.136.105.70(11.136.105.70:3306)
Wed Sep 5 20:50:54 2018 - [debug] Disconnected from 11.136.167.69(11.136.167.69:3306)
Wed Sep 5 20:50:54 2018 - [info]
----- Failover Report -----
app1: MySQL Master failover 11.187.250.60(11.187.250.60:3306) to 11.136.167.69(11.136.167.69:3306) succeeded
Master 11.187.250.60(11.187.250.60:3306) is down!
Check MHA Manager logs at dbconsole.et2:/u01/zjk/mha4mysql-manager-master/workdir/manager.log for details.
Started automated(non-interactive) failover.
The latest slave 11.136.167.69(11.136.167.69:3306) has all relay logs for recovery.
Selected 11.136.167.69(11.136.167.69:3306) as a new master.
11.136.167.69(11.136.167.69:3306): OK: Applying all logs succeeded.
11.136.105.70(11.136.105.70:3306): Generating differential relay logs up to 11.136.167.69(11.136.167.69:3306)succeeded.
Generating relay diff files from the latest slave succeeded.
11.136.105.70(11.136.105.70:3306): OK: Applying all logs succeeded. Slave started, replicating from 11.136.167.69(11.136.167.69:3306)
11.136.167.69(11.136.167.69:3306): Resetting slave info succeeded.
Master failover to 11.136.167.69(11.136.167.69:3306) completed successfully.老的master上recovery binlog
- 测试
save_binary_logs --command=test --start_pos=4 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000002 --debug
Creating /opt/TMHA2/mha4mysql-node-master/workdir if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /u01/my3306/log, up to mysql-bin.002290- monitor时ssh探测
save_binary_logs --command=test --start_pos=4 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin --debug- 真正时候的save binlog
save_binary_logs --command=save --start_file=mysql-bin.000002 --start_pos=699479232 --binlog_dir=/u01/my3306/log --output_file=/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.136.105.70_3306_20180822223644.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --debug
Creating /opt/TMHA2/mha4mysql-node-master/workdir if not exists.. ok.
Concat binary/relay logs from mysql-bin.002290 pos 8646412 to mysql-bin.002290 EOF into /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog ..
ok.
Concat binary/relay logs from mysql-bin.002290 pos 8646412 to mysql-bin.002290 EOF into /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog ..
parse_init_headers: file=mysql-bin.002290 event_type=15 server_id=196867798 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=mysql-bin.002290 event_type=2 server_id=196867798 length=91 nextmpos=211 prevrelay=120 cur(post)relay=211
Dumping binlog format description event, from position 0 to 120.. ok.
Dumping effective binlog data from /u01/my3306/log/mysql-bin.002290 position 8646412 to tail(8663384).. ok.
parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=15 server_id=196867798 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=2 server_id=196867798 length=77 nextmpos=8646489 prevrelay=120 cur(post)relay=197
Concat succeeded.slave上执行apply_diff_relay_logs
- 测试
apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=11.187.250.60 --slave_ip=11.187.250.60 --slave_port=3306 --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.001574 --debug --slave_pass=xxx
Checking slave recovery environment settings..
Relay log found at /u01/my3306/log, up to relaylog.003976
Temporary relay log file is /u01/my3306/log/relaylog.003976
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.- 查看最新的slave是否有所有的(包含最老的slave的)relaylog
apply_diff_relay_logs --command=find --latest_mlf=mysql-bin.002290 --latest_rmlp=8646412 --target_mlf=mysql-bin.002290 --target_rmlp=8609518 --server_id=193506223 --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --timestamp=20180905205046 --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.002564 --debug
Relay log found at /u01/my3306/log, up to relaylog.002564
Fast relay log position search succeeded.
Target relay log file/position found. start_file:relaylog.002564, start_pos:391934.
Target relay log FOUND!- 在新主上补全老主的日志
apply_diff_relay_logs --command=apply --slave_user='adha' --slave_host=11.136.167.69 --slave_ip=11.136.167.69 --slave_port=3306 --apply_files=/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --timestamp=20180905205046 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --debug --slave_pass=xxx
Wed Sep 5 20:50:51 2018 - [info]
MySQL client version is 5.6.37. Using --binary-mode.
Applying differential binary/relay log files /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog on 11.136.167.69:3306. This may take long time...
Applying log files succeeded.- 新主上生成其他主库的relay差异日志 （没有打通实例之间的ssh通道，所以scp会报错，修改为了先复制到manager再复制到slave）
apply_diff_relay_logs --command=generate_and_send --scp_user=root --scp_host=11.136.105.70 --latest_mlf=mysql-bin.002290 --latest_rmlp=8646412 --target_mlf=mysql-bin.002290 --target_rmlp=8609518 --server_id=193506223 --diff_file_readtolatest=/opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --timestamp=20180905205046 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --relay_dir=/u01/my3306/log --current_relay_log=relaylog.002564 --debug
Wed Sep 5 20:50:51 2018 - [info]
Relay log found at /u01/my3306/log, up to relaylog.002564
Fast relay log position search succeeded.
Target relay log file/position found. start_file:relaylog.002564, start_pos:391934.
Concat binary/relay logs from relaylog.002564 pos 391934 to relaylog.002564 EOF into /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog ..
parse_init_headers: file=relaylog.002564 event_type=15 server_id=193506223 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=relaylog.002564 event_type=15 server_id=196867798 length=116 nextmpos=0 prevrelay=120 cur(post)relay=236
Binlog Checksum enabled
parse_init_headers: file=relaylog.002564 event_type=2 server_id=196867798 length=77 nextmpos=8217897 prevrelay=236 cur(post)relay=313
Dumping binlog format description event, from position 0 to 236.. ok.
Dumping effective binlog data from /u01/my3306/log/relaylog.002564 position 391934 to tail(428828).. ok.
parse_init_headers: file=relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog event_type=15 server_id=193506223 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog event_type=15 server_id=196867798 length=116 nextmpos=0 prevrelay=120 cur(post)relay=236
Binlog Checksum enabled
parse_init_headers: file=relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog event_type=2 server_id=196867798 length=77 nextmpos=8609595 prevrelay=236 cur(post)relay=313
Concat succeeded.
Generating diff relay log succeeded. Saved at /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog .
Permission denied (publickey,password).
ssh root@11.136.105.70 for mkdir /opt/TMHA2/mha4mysql-node-master/workdir failed with rc 255:0!
at /usr/local/bin/apply_diff_relay_logs line 450.
main::generate_and_send() called at /usr/local/bin/apply_diff_relay_logs line 497
eval {...} called at /usr/local/bin/apply_diff_relay_logs line 475
main::main() called at /usr/local/bin/apply_diff_relay_logs line 120- 其他slave上补全新主的relay日志
apply_diff_relay_logs --command=apply --slave_user='adha' --slave_host=11.136.105.70 --slave_ip=11.136.105.70 --slave_port=3306 --apply_files=/opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog,/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog --workdir=/opt/TMHA2/mha4mysql-node-master/workdir --target_version=5.6.37-20180708-log --timestamp=20180905205046 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --debug --slave_pass=xxx
Wed Sep 5 20:50:54 2018 - [info]
Concat all apply files to /opt/TMHA2/mha4mysql-node-master/workdir/total_binlog_for_11.136.105.70_3306.20180905205046.binlog ..
Copying the first binlog file /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog to /opt/TMHA2/mha4mysql-node-master/workdir/total_binlog_for_11.136.105.70_3306.20180905205046.binlog.. ok.
Dumping binlog head events (rotate events), skipping format description events from /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog.. parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=15 server_id=196867798 length=116 nextmpos=120 prevrelay=4 cur(post)relay=120
Binlog Checksum enabled
parse_init_headers: file=saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog event_type=2 server_id=196867798 length=77 nextmpos=8646489 prevrelay=120 cur(post)relay=197
dumped up to pos 120. ok.
/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog has effective binlog events from pos 120.
Dumping effective binlog data from /opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog position 120 to tail(17092).. ok.
Concat succeeded.
All apply target binary logs are concatinated at /opt/TMHA2/mha4mysql-node-master/workdir/total_binlog_for_11.136.105.70_3306.20180905205046.binlog .
MySQL client version is 5.6.37. Using --binary-mode.
Applying differential binary/relay log files /opt/TMHA2/mha4mysql-node-master/workdir/relay_from_read_to_latest_11.136.105.70_3306_20180905205046.binlog,/opt/TMHA2/mha4mysql-node-master/workdir/saved_master_binlog_from_11.187.250.60_3306_20180905205046.binlog on 11.136.105.70:3306. This may take long time...
Applying log files succeeded.