MySQL database cluster combat --- availability MySQL MHA

A, MySQL high availability of MHA

Works 1.1 database

数据库的内部具有缓冲区，在缓冲区中的数据是待处理的数据，数据库源源不断的从缓冲区中获取数据进行处理，但是如果高并发，会造成数据缓冲区数据溢出，最终导致数据库宕机，不能响应。这只能缓解，不能根治。

1.2 High Availability

I think,High Availability is possible to reduce downtime due to routine maintenance and sudden system crashes caused.

• Database high availability need to address the following questions:

数据库的信息如何实现同步
用户如何选择连接哪个数据库
数据库宕机后如何实现高可用（尽可能不死，死后有替补）
如果数据库宕机，宕机后的数据库如何实现数据同步。

1.3 MHA

MHA (Master High Availability) currently in terms of high availability MySQL is a relatively mature solution that youshimaton by the Japanese company DeNA (Facebook now working in the company) to develop, is a good case as a failure MySQL high availability environment and the main switch from an elevated high availability software . In MySQL failover process, the MHA can be done automatically in 30 seconds from 0 to the fault switching operation of the database, and performing the handover procedure failure, the MHA to ensure consistency of the data to the maximum extent, in order to achieve real in the sense of availability .

The software consists of two parts: MHA Manager (management node) and MHA Node (node ​​data). MHA Manager can be deployed individually manage a plurality of master-slave cluster on a separate machine, it can be deployed on a slave node. MHA Node MySQL running on each server, master node MHA Manager will regularly detect the cluster, when the master fails, it can automatically slave to the latest data of the upgrade to the new master, and then all the other slave redirected to the new the master. The entire failover process is completely transparent to the application. In the MHA automatic failover process, MHA attempt to save down from the master binary log, without losing the greatest degree of assurance of data, but this is not always feasible. For example, if the primary server hardware failure or inability to access via ssh, MHA can not save the binary log, perform failover only lost the latest data. MySQL 5.5 using semi-synchronous replication, can greatly reduce the risk of data loss.MHA can be combined with semi-sync. If only one slave has received the latest binary log, MHA latest binary log can be applied to all other slave servers, so you can ensure data consistency across all nodes.Currently MHA main support infrastructure a master multi-slave, to build MHA, requested a copy cluster must have at least three database servers, a primary two act as from the library from that one acts as a master, one acts as a backup master, the other one because at least three servers, Due to cost considerations machine, Taobao also been transformed on the basis of the current Taobao TMHA already supports a master-slave.

• MHA requirements and principles:

(1) requires an odd number of nodes. When a master node hangs vote for a new master, an even number of nodes with the same number of votes situation will occur, but it will not be an odd number.
(2) coincides with the data of all the data nodes, each node has the data may be used as master
(. 3) required to install each of the master and slave node plug;
(4) When the master node linked, generally selected Last the slave node as the new master node, recent data refers to the small differences in data

Second, the configuration MHA

lab environment

CPU name	IP	function
server1	172.25.2.51	master
server2	172.25.2.52	slave（备master）
server3	172.25.2.53	slave
server4	172.25.2.54	MHA

2.1 Configuration MHA

1, the configuration server1 mysql service

1) install mysql service

yum install -y mysql-community-client-5.7.24-1.el7.x86_64.rpm mysql-community-common-5.7.24-1.el7.x86_64.rpm mysql-community-libs-5.7.24-1.el7.x86_64.rpm mysql-community-libs-compat-5.7.24-1.el7.x86_64.rpm mysql-community-server-5.7.24-1.el7.x86_64.rpm

2) change the configuration file /etc/my.cnf, after adding the following to start the database :

 29 server-id=1
 30 gtid_mode=ON
 31 enforce_gtid_consistency=ON
 32 log_bin=binlog
 33 log_slave_updates=ON

3) Check your temporary password, security initialization

4) Log database, created and used for authorized users to copy, view master status

2、On server2 and server3, Do the following:

1 | yum install -y mysql-community-client-5.7.24-1.el7.x86_64.rpm \
	mysql-community-common-5.7.24-1.el7.x86_64.rpm \
	mysql-community-libs-5.7.24-1.el7.x86_64.rpm \
	mysql-community-libs-compat-5.7.24-1.el7.x86_64.rpm \
	mysql-community-server-5.7.24-1.el7.x86_64.rpm
2 | vim /etc/my.cnf
		server-id=2（3）
 		gtid_mode=ON
 		enforce_gtid_consistency=ON
 		log_bin=binlog
 		log_slave_updates=ON
3 | systemctl start mysqld
4 | cat /var/log/mysqld.log | grep password
5 | mysql_secure_installation
6 | mysql -uroot -pWestos==123

3、On server2, Log database, modify master information, see the slave state.

4、On server3, Log database, modify master information, see the slave state.

5, at this time,再开一台虚拟机server4，安装所需软件

6、生成ssh密钥，管理节点是不需要输入密码，将生成的公钥和私钥传给数据节点

[root@server4 ~]# ssh-keygen
[root@server4 ~]# ssh-copy-id server1
[root@server4 ~]# ssh-copy-id server2
[root@server4 ~]# ssh-copy-id server3

[root@server4 ~]# scp -r .ssh server1:
[root@server4 ~]# scp -r .ssh server2:
[root@server4 ~]# scp -r .ssh server3:

6、在server1、server2和server3上，安装节点数据包

yum install -y mha4mysql-node-0.58-0.el7.centos.noarch.rpm

7、管理节点server4上创建一个目录，编辑配置文件

[root@server4 ~]# mkdir -p /etc/masterha
[root@server4 ~]# cd /etc/masterha/
[root@server4 masterha]# ls
[root@server4 masterha]# vim app1.cnf
写入：
[server default]
manager_workdir=/etc/masterha
manager_log=/var/log/masterha.log   # manager 日志文件
master_binlog_dir=/etc/masterha

password=Westos==123   #MySQL管理帐号和密码
user=root
ping_interval=1
remote_workdir=/tmp
repl_password=Westos==123
repl_user=repl   # 复制帐号和密码
ssh_user=root  # 系统ssh用户

[server1]
hostname=172.25.2.51
port=3306

[server2]
hostname=172.25.2.52
port=3306
candidate_master=1
check_repl_delay=0

[server3]
hostname=172.25.2.53
port=3306
no_master=1  ##no_master表示这个节点不能作为master

8、管理节点ssh检查管理节点ssh检查

[root@server4 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf 

• 如果报错的话，是因为server1、server2、server3之间相互不免密。

拷贝server4上的密钥给server1 2 3

[root@server4 ~]# scp -r .ssh server1:
[root@server4 ~]# scp -r .ssh server2:
[root@server4 ~]# scp -r .ssh server3:

再检测ssh，成功

9、检测复制功能

[root@server4 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf 

此时发现报错，这是因为没有给节点授权：
server1： 所有权限；

server2 和 server3： 只读


此时，再次检查复制功能:masterha_check_repl --conf=/etc/masterha/app1.cnf

2.2 手动切换maste

10、手动替换master
先关闭当前节点的manager，不关的话切不了，manager就是自动切换的工具

[root@server4 masterha]# masterha_stop --conf=/etc/masterha/app1.cnf

手动替换master节点

• 手动切换之前，需要保证主从同步正常，repl复制用户能够远程连接

masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf 
--dead_master_host=172.25.2.51 --dead_master_ip=172.25.2.51 --dead_master_port=3306
 --new_master_host=172.25.2.52 --new_master_port=3306

切换成功后，可以在server3上看到它的master已经变成了server2，而server2已经是master了

11、开启server1的mysqld服务，作为slave加入集群，将他的master改为server2

现在就完成了一次手动切换，这时会在/etc/masterha目录下生成一个app1.failover.complete文件，是来记录failover情况的，再进行failover时必须先把这个文件删除，不然不会failover
切换回172.25.2.51为master


测试:

2.3 手动切换VIP漂移

（1）首先清理app1.failover.complete

（2）server4创建一个检测进程

[root@server4 masterha]# nohup masterha_manager --conf=/etc/masterha/app1.cnf &>/dev/null &

（3）On server1 view, this process is shut down, he found opened.

（4）On server1 (the current master), close myslq service

[root@server1 ~]# systemctl stop mysqld

At this point, has been found to switch, while the manager process exits, so the need for automatic script
(5) first server1 added back to the cluster

mysql> CHANGE MASTER TO MASTER_HOST = '172.25.0.2', MASTER_USER = 'repl',
 MASTER_PASSWORD = 'Westos==123', MASTER_AUTO_POSITION = 1;

mysql> start slave;

(6) configuration scripts and vip drift , edit master_ip_failover and master_ip_online_change two scripts

• Because users access to only one entrance, so you need to configure vip

修改内容：

my $ssh_start_vip = "/sbin/ip addr add $vip dev eth0";
my $ssh_stop_vip = "/sbin/ip addr del $vip dev eth0";

cp master_ip_failover master_ip_online_change /usr/local/bin
cd /usr/local/
cd bin/
chmod +x master_ip_failover master_ip_online_change

(7) Add the following two lines in the configuration file server4

master_ip_failover_script= /usr/local/bin/master_ip_failover
master_ip_online_change_script= /usr/local/bin/master_ip_online_change

(8) was added to a vip Server2 (current master) on

• Currently server2 is the master, so give server2 add vip

（9） Test: vip drift

[root@server4 bin]# masterha_master_switch --conf=/etc/masterha/app1.cnf 
--master_state=alive --new_master_host=172.25.2.51 --new_master_port=3306 
--orig_master_is_new_slave --running_updates_limit=10000

We can see that the switch vip

***************************************************************
Disabling the VIP - 172.25.2.100/24 on old master: 172.25.2.52
***************************************************************


***************************************************************
Enabling the VIP - 172.25.2.100/24 on new master: 172.25.2.51 
***************************************************************

（10）test : Automatic switching

On server4 delete app1.failover.complete, open manager

nohup masterha_manager --conf=/etc/masterha/app1.cnf &>/dev/null &

Then close mysql, fault simulation on a master (server1), and then view the logs on server4

/var/log/masterha.log CAT , you can see the handover success, vip successfully drift