table of Contents
- I. Introduction
- Two, MHA tools
- Three, MHA deployment process
- 3.1.1 Configuration
- 3.1.2 Environmental Planning
- From a multi-master configuration 3.1.3
- 3.2 MHA Configuration
- 3.2.1 master permissions granted
- 3.2.2 ssh trust
- 3.2.3 installation package mha
- 3.2.4 MHA node configuration management
- 3.2.5 MHA node detection
- 3.2.6 MHA start
- 3.2.7 MHA fault simulation
- 3.2.8 repair the damaged node joins the MHA
- MHA issue highlights
I. Introduction
MHA logic is that in order to ensure high availability MySQL, there will be a master StandBy state. During the failover mysql, MHA fault database can be done automatically switching operation in the 0 to 30 seconds, and performing failover process, the consistency of MHA can ensure the greatest degree of data in order to achieve high availability in a relative sense.
1.1MHA role
Below, the whole architecture is divided into MHA
- MHA Manager node
- MHA Node node
where the MHA Manager node is a single-deployment, MHA Node node is deployed on each MySQL Cluster nodes need to be monitored. MHA Manager will regularly detect Master node in the cluster, when the Master fails, it can automatically Standby Master or Slave upgrade to the latest data of the new Master, then the other Slave redirected to the new Master.
Two, MHA tools
Manager node:
masterha_check_ssh: MHA dependent ssh environmental monitoring tools;masterha_check_repl: MYSQL replication environment detection tools;masterga_manager: MHA Service main program;masterha_check_status: MHA operational state detection tools;masterha_master_monitor: MYSQL master node availability monitoring tools;masterha_master_swith:master: The changing tool;masterha_conf_host: Add or delete a node configuration;masterha_stop: Close MHA service tools.
Node node :( These tools are usually triggered by MHA Manager of the script, without human operation)
save_binary_logs: Copy and save the master binary log;apply_diff_relay_logs: Identify the differences relay event log and applied to other slave;purge_relay_logs: Clear relay logs (does not block SQL thread);
Custom extensions:
secondary_check_script: Detecting the availability of the master by a plurality of network routing;master_ip_failover_script: Update masterip application used;report_script: Send Report;init_conf_load_script: Load the initial configuration parameters;master_ip_online_change_script; Update master node ip address.
Three, MHA deployment process
3.1.1 Configuration
MHA have special requirements, such as each node must open the binary logs and relay logs from each node must be enabled to display its MYSQL replication environment read-onlyproperties and close the relay_log_purgefunction, where do the configuration stated in advance.
3.1.2 Environmental Planning
| Machine name | IP | Roles | Remark |
|---|---|---|---|
| manager | 172.30.200.100 | manager controller | And for managing failover |
| master | 172.30.200.101 | Primary database server | Open binlog, relay-log. Close relay_log_purge |
| slave1 | 172.30.200.102 | From the database server | Open binlog, relay-log. Close relay_log_purge |
| slave2 | 172.30.200.103 | From the database server | Open binlog, relay-log. Close relay_log_purge |
In the respective nodes / etc / hosts file configuration add the following contents:
[root@localhost ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.30.200.100 arpmgr
172.30.200.101 arpmaster
172.30.200.102 arpslave1
172.30.200.103 arpslave2Create a directory binlog
mkdir -p /data/mysqldata/binlog
chown -R mysql:mysql /data/mysqldata/binlog101 node configuration:
server-id = 200101
log-bin = /data/mysqldata/binlog/mysql-bin
binlog_format= row
max_binlog_size= 512m
relay-log = /data/mysqldata/binlog/relay-bin
expire-logs-days = 14
lower_case_table_names = 1
character-set-server = utf8
log_slave_updates = 1102 Node Configuration:
server-id = 200102
log-bin = /data/mysqldata/binlog/mysql-bin
binlog_format= row
max_binlog_size= 512m
relay-log = /data/mysqldata/binlog/relay-bin
expire-logs-days = 14
read_only = ON
relay_log_purge = 0
lower_case_table_names = 1
character-set-server = utf8
log_slave_updates = 1103 Node Configuration:
server-id = 200103
log-bin = /data/mysqldata/binlog/mysql-bin
binlog_format= row
max_binlog_size= 512m
relay-log = /data/mysqldata/binlog/relay-bin
read_only = ON
relay_log_purge = 0
expire-logs-days = 14
lower_case_table_names = 1
character-set-server = utf8
log_slave_updates = 1From a multi-master configuration 3.1.3
master node configuration:
MariaDB [(none)]>grant replication slave,replication client on *.* to 'repl'@'172.30.200.%' identified by 'repl7101';
MariaDB [(none)]> show master status;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 | 548 | | |
+------------------+----------+--------------+------------------+
1 row in set (0.000 sec)slave node configuration:
grant replication slave,replication client on *.* to 'repl'@'172.30.200.%' identified by 'repl7101';
change master to master_host='172.30.200.101',
master_user='repl',
master_password='repl7101',
master_log_file='mysql-bin.000001',
master_log_pos=548;
start slave;
show slave status\G;So far, he completed a master multi slave configuration.
3.2 MHA Configuration
3.2.1 master permissions granted
You can configure all nodes above, which has administrative privileges, currently only set permissions on the master node:
grant all on *.* to 'mhaadmin'@'172.30.%.%' identified by 'mha7101';
grant all on *.* to 'mhaadmin'@'arpmgr' identified by 'mha7101';3.2.2 ssh trust
Four nodes perform the following statement:
ssh-keygen -t rsa
ssh-copy-id -i .ssh/id_rsa.pub root@arpmgrThen arpmgrthe node above, can see authorized_keysthe information contents of the file as follows:
[root@localhost .ssh]# cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDY3yFhR5uzcFEpd+q+1Uw/cRF9ZRraygms7OZoefLFzY/ydSi6yYuCighG8WquvRep7XDNjFI71HAUagSoXiyPoCe1lqEnzpxSc+fQpIeQqEhUmLJ2bk+R83EskzwRGh+S/D4yp/swWz1vRgUGoTWevLCs33q7ZrsM8i+jB0uwZmzOV+CyQAPW9vLkRjZa4y1sx65lbR0HbdTQWQYZ4IyZauoU8XQjAIOs/CdLw2nBt8dPO53jT7NS7Ywx6eu/Wj9k/sYVVZT3jTb+pBIVs+Du5+tdUDX5aLKzxINpLlqNhorNevoC9iE0Ame1qvYonQfyWQ52Ae0y+58vFfG6PyV3 [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/ZPihYSC6ArawKRU75aQRVSFsQ5S89SrYHGWdzyluB4spj+UDUmWH1kLGYr715/HD5hh22KdLmIs7R4jviOeao1HK52fpMvklYaNtYRHuV63Zkg5sOLvLfhrHdta9wuHlW1NyWx75+wIl2LvKBRtnSddwf5ZvitJ/kChf2gpNhHAWidyjGsPoJdr0OBCNHvz1y6oON6cnMb07ExaIjptRnkbCOU0QSVjFq4+Jmh8zTTbJC2up50s15gSfWXH0+WLXmJXJGkvgHdSYqw4vJt/l25f5qAKKZsfnyfC0iyct4GyHPF6trpvQ/c2lqr/Rg4xLWgdxlyt4aBJYl5adIRK/ [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDba26wV0KwQNTb4pKuiFDCcVMNRLGMXSiJC8ucN4/KIqzoOYJ747QL8GL5F8ePnRaZ1rtOwdjnlTiC0a4Tcg4JLs+JSnJgzvepuixmGgSJfLbJ36iN1WFh6fP2GZEDdR7Qum4sBUpQyYJ20Kf9rKfQQv2wq6csK5IlFk/OoO+zTySauLnYvRxvKY2avVDXPPFJvpqimKXn59MIAoJr6YEKvncbYyqvrSgUy7klZDys9IIjYcWfO7VKjQ5bwbHrrKtNbedME+KPQld7e8ZVL66Omik4Z6ip7DQEHRKWMmuBIpL99AgOOjPLbzJFWLUPOwvy3DtmEBnZ+0NVf/1obC11 [email protected]
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwrgZtGC31EgixeY4SVl4h64m1r8LdL3hM4Be2/I+6Xw7hCzZyKKTAFgz9W/ukfx6WmZwoqp1VO/7Jp6KO1FhYOi5u0q6J1KIObFNp+3E6cB2P0q39WqmZpQ9cNPYrbs9U2Ej0L0JwUtf/xLh334PaSlv/LcNy+p1dWya2OqsBeraiXZ4MgEBzcb+0twkpfpD327VgT/mRHPmA6fPRJOOJti1u4isHeotE4i13YIqQYfBfmbfiLdXKAvgI8FuTf0i91Re/FUBOgBfBcJbqIQNR0Nh5wZ/LvNxkstDQvypZIZwiK+wN+aZZOQ7jF/+997Z9QQleC9OOoHOJR7+fisLb [email protected]Exactly shaped like a four ssh-rsaassociated key information.
The public key information described above, the remaining copy to the above four servers:
scp authorized_keys root@arpmaster:~/.ssh/
scp authorized_keys root@arpslave1:~/.ssh/
scp authorized_keys root@arpslave2:~/.ssh/Test ssh is available
[root@localhost .ssh]# ssh arpmaster
[root@localhost ~]# ssh arpslave1
[root@localhost ~]# ssh arpslave2
[root@localhost ~]# ssh arpmgr3.2.3 installation package mha
mha installation package is divided into two, one node, the other one is manager.
Four node installation:mha4mysql-node-0.57-0.el7.centos.noarch.rpm
Management node installation:mha4mysql-manager-0.57-0.el7.centos.noarch.rpm
`In the installation mha4mysql-node-0.57-0.el7.centos.noarch.rpmprocess, there are pair of perl-DBD-mysql, perl-DBIfront-dependent, steps are as follows:
yum install perl-DBD-mysql perl-DBI`In the installation mha4mysql-manager-0.57-0.el7.centos.noarch.rpmprocess, there is on the perlfront-dependent, steps are as follows:
安装yum 扩展包
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-Config-Tiny perl-Log-Dispatch-* perl-Parallel-ForkManagerThen install information, are successful, as follows:
[root@localhost ~]# rpm -ivh mha4mysql-node-0.57-0.el7.centos.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...
1:mha4mysql-node-0.58-0.el7.centos ################################# [100%]
[root@localhost ~]# rpm -ivh mha4mysql-manager-0.57-0.el7.centos.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...
1:mha4mysql-manager-0.58-0.el7.cent################################# [100%]0.58 in a super_read_only unavailable mariadb, so use the 0.57 version.
3.2.4 MHA node configuration management
[root@localhost ~]# cd /etc/mha_master/
[root@localhost mha_master]# vi /etc/mha_master/mha.cnfConfiguration file as follows:
[server default]
user=mhaadmin
password=mha7101
manager_workdir=/etc/mha_master/app1
manager_log=/etc/mha_master/manager.log
remote_workdir=/data/mha_master/app1
repl_user=repl
repl_password=repl7101
ping_interval=1
[server1]
hostname=172.30.200.101
ssh_port=22
[server2]
hostname=172.30.200.102
ssh_port=22
candidate_master=1
[server3]
hostname=172.30.200.103
ssh_port=22
no_master=13.2.5 MHA node detection
Ssh management node detects connectivity as follows:
[root@localhost ~]# masterha_check_ssh -conf=/etc/mha_master/mha.cnfThere follows a log, on behalf of Normal:
Thu Jan 9 14:43:09 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Thu Jan 9 14:43:09 2020 - [info] Reading application default configuration from /etc/mha_master/mha.cnf.. [email protected](172.30.200.103:22) to [email protected](172.30.200.101:22).. Thu Jan 9 14:43:11 2020 - [debug] ok. Thu Jan 9 14:43:11 2020 - [debug] Connecting via SSH from [email protected](172.30.200.103:22) to [email protected](172.30.200.102:22).. Thu Jan 9 14:43:11 2020 - [debug] ok. Thu Jan 9 14:43:12 2020 - [info] All SSH connection tests passed successfully.Detection MySQL replication is normal
masterha_check_repl --conf=/etc/mha_master/mha.cnfThere follows a log, indicating normal:
Thu Jan 9 14:44:54 2020 - [info] Slaves settings check done. Thu Jan 9 14:44:54 2020 - [info] 172.30.200.101(172.30.200.101:3306) (current master) +--172.30.200.102(172.30.200.102:3306) +--172.30.200.103(172.30.200.103:3306) Thu Jan 9 14:44:54 2020 - [info] Checking replication health on 172.30.200.102.. Thu Jan 9 14:44:54 2020 - [info] ok. Thu Jan 9 14:44:54 2020 - [info] Checking replication health on 172.30.200.103.. Thu Jan 9 14:44:54 2020 - [info] ok. Thu Jan 9 14:44:54 2020 - [warning] master_ip_failover_script is not defined. Thu Jan 9 14:44:54 2020 - [warning] shutdown_script is not defined. Thu Jan 9 14:44:54 2020 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK.
3.2.6 MHA start
Start mha manager:
nohup masterha_manager --conf=/etc/mha_master/mha.cnf &> /etc/mha_master/manager.log &Detecting master node status:
[root@localhost ~]# masterha_check_status --conf=/etc/mha_master/mha.cnf mha (pid:31709) is running(0:PING_OK), master:172.30.200.101Description of the main database
172.30.200.101to start properly.Close mha manager:
masterha_stop -conf=/etc/mha_master/mha.cnf
3.2.7 MHA fault simulation
masterDirectly kill mysql node[root@localhost ~]# ps -ef |grep mysql root 19864 1 0 08:51 ? 00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/data/mysqldata --pid-file=/data/mysqldata/localhost.localdomain.pid mysql 19976 19864 0 08:51 ? 00:00:13 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/data/mysqldata --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/data/mysqldata/mysqld.log --pid-file=/data/mysqldata/localhost.localdomain.pid --socket=/tmp/mysql.sock root 22166 21525 0 14:55 pts/0 00:00:00 grep --color=auto mysql [root@localhost ~]# kill -9 19864 19976MHATransfer logs.[root@localhost ~]# tail -f /etc/mha_master/manager.log From: 172.30.200.101(172.30.200.101:3306) (current master) +--172.30.200.102(172.30.200.102:3306) +--172.30.200.103(172.30.200.103:3306) To: 172.30.200.102(172.30.200.102:3306) (new master) +--172.30.200.103(172.30.200.103:3306) Master 172.30.200.101(172.30.200.101:3306) is down! Check MHA Manager logs at localhost.localdomain:/etc/mha_master/manager.log for details. Started automated(non-interactive) failover. The latest slave 172.30.200.102(172.30.200.102:3306) has all relay logs for recovery. Selected 172.30.200.102(172.30.200.102:3306) as a new master. 172.30.200.102(172.30.200.102:3306): OK: Applying all logs succeeded. 172.30.200.103(172.30.200.103:3306): This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 172.30.200.103(172.30.200.103:3306): OK: Applying all logs succeeded. Slave started, replicating from 172.30.200.102(172.30.200.102:3306) 172.30.200.102(172.30.200.102:3306): Resetting slave info succeeded. Master failover to 172.30.200.102(172.30.200.102:3306) completed successfully.From the above point of view the log,
172.30.200.102it has become the new master, and172.30.200.103stillslavethe database.
3.2.8 repair the damaged node joins the MHA
Because here is the experimental environment, you can not be everywhere mysqldump backup. If a production environment is restored, the slave can be stopped SQL thread, corresponding to the position pos in mind, and then the backup data to ensure data consistency after, synchronization data, to recover the damaged junction.
change master to master_host='172.30.200.102',
master_user='repl',
master_password='repl7101',
master_log_file='mysql-bin.000003',
master_log_pos=401;View slave status:
MariaDB [(none)]> start slave;
MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.30.200.102
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000003
Read_Master_Log_Pos: 401
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 555
Relay_Master_Log_File: mysql-bin.000003
Slave_IO_Running: Yes
Slave_SQL_Running: YesStart again, as follows:
[root@localhost ~]# nohup masterha_manager --conf=/etc/mha_master/mha.cnf &> /etc/mha_master/manager.log &
[root@localhost ~]# masterha_check_status --conf=/etc/mha_master/mha.cnf
mha (pid:31905) is running(0:PING_OK), master:172.30.200.101So far, MHA experiment is completed. Due to a production environment will use VIP, follow-up will continue writing.
MHA issue highlights
A mistake
Log error:
Thu Jan 9 11:31:36 2020 - [info] Connecting to [email protected](172.30.200.102:22)..
Can't exec "mysqlbinlog": 没有那个文件或目录 at /usr/share/perl5/vendor_perl/MHA/BinlogManager.pm line 106.
mysqlbinlog version command failed with rc 1:0, please verify PATH, LD_LIBRARY_PATH, and client optionsSolution:
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlogError two
Log error:
Checking if super_read_only is defined and turned on..DBI connect(';host=172.30.200.102;port=3306','mhaadmin',...) failed: Access denied for user 'mhaadmin'@'arpslave1' (using password: YES) at /usr/share/perl5/vendor_perl/MHA/SlaveUtil.pm line 239Solution:
manager node, execute:
grant all on *.* to 'mhaadmin'@'arpmgr' identified by 'mha7101';
grant all on *.* to 'mhaadmin'@'arpmaster' identified by 'mha7101';
grant all on *.* to 'mhaadmin'@'arpslave1' identified by 'mha7101';
grant all on *.* to 'mhaadmin'@'arpslave2' identified by 'mha7101';Error Three
Log is as follows:
Testing mysql connection and privileges..sh: mysql: 未找到命令
mysql command failed with rc 127:0!Solution:
ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql