MHA high availability cluster

MHA high availability clustering

Article Directory

A, MHA Description:
Second, the deployment MHA:
The first step: mysql three main installation from the server
Step two: Modify the mysql master configuration file: /etc/my.cnf, server-id pay attention to three servers can not be the same as
the first three steps: three servers start mysql service
step Four: configure mysql master-slave synchronization (from a main two)
step 5: install MHA
step Six: start MHA

A, MHA profile:

MHA（Master High Availability）

(1 Introduction

Currently in terms of high availability MySQL is a relatively mature solution that youshimaton by the Japanese company DeNA (Facebook now working in the company) to develop, is a good switch as a failure at the primary and MySQL high availability environments from high availability software upgrade . In MySQL failover process, the MHA can be done automatically in 30 seconds from 0 to the fault switching operation of the database, and performing the handover procedure failure, the MHA to ensure consistency of the data to the maximum extent, in order to achieve real availability of meaning.

(2) The software consists of two parts:

MHA Manager (management node) and MHA Node (node ​​data). MHA Manager can be deployed individually manage a plurality of master-slave cluster on a separate machine, it can be deployed on a slave node. MHA Node MySQL running on each server, master node MHA Manager will regularly detect the cluster, when the master fails, it can automatically slave to the latest data of the upgrade to the new master, and then all the other slave redirected to the new the master. The entire failover process is completely transparent to the application.

(3) How it works:

1, the automatic failover process MHA, MHA attempt to save down from the master binary log, without losing the greatest degree of assurance of data, but this is not always feasible. For example, if the primary server hardware failure or inability to access via ssh, MHA can not save the binary log, perform failover only lost the latest data. MySQL 5.5 using semi-synchronous replication, can greatly reduce the risk of data loss. MHA can be combined with the semi-synchronous replication. If only one slave has received the latest binary log, MHA latest binary log can be applied to all other slave servers, so you can ensure data consistency across all nodes.

2, order:

① master save the binary log events from the collapse of downtime (binlog Events);
② identify slave containing the latest update;
application log difference ③ relay (relay log) to another slave;
④ Apply to save the binary log events from the master ( Events binlog);
⑤ upgrade a slave as the new master;
⑥ the other slave connected to the new master replication

Second, the deployment MHA:

Roles	You need to install Service
master(192.168.45.133)	mha4mysql-node
slave1(192.168.45.130)	mha4mysql-node
slave2(192.168.45.134)	mha4mysql-node
manager(192.168.45.135)	mha4mysql-manager、 mha4mysql-node

(1) Requirements:

This case requires automatic switching in case of failure by MHA monitoring MySQL database does not affect business.

(2) ideas:

Installation MySQL database
arranged two MySQL from a master
installation MHA software
configuration without password authentication
configuration MySQL, MHA availability
analog master failover

(3) Operating System: Cent0S7 ... 6 version, MHA version is version 0.57

First, install mysql database on three servers

(MySQL version please use 5.6.36; cmake version 2.8.6, please use), because three installation operations are consistent with the presentation here in order to install on the master.
In order to facilitate the experiment, modify the host name

#修改服务器名称，将manster服务器的主机名改为manster
hostnamectl set-hostname manster
 su
 #修改服务器名称，将slave1服务器的主机名改为slaveq
hostnamectl set-hostname slave1
 su
 #修改服务器名称，将slave2服务器的主机名改为slave2
hostnamectl set-hostname slave2
 su

#安装编译依懒环境
yum -y install gcc gcc-c++ ncurses ncurses-devel bison perl-Module-Install cmake

#挂载软件包
mount.cifs //192.168.100.3/mha /mnt
Password for root@//192.168.100.3/mha:  
#安装gmake编译软件
cd /mnt
tar zxvf cmake-2.8.6.tar.gz -C /opt
cd /opt/cmake-2.8.6/
./configure
gmake && gmake install

#安装mysql数据库
[root@localhost cmake-2.8.6]# cd /mnt
[root@localhost mnt]# tar zxvf mysql-5.6.36.tar.gz -C /opt
#编译mysql
cd /opt/mysql-5.6.36/
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DWITH_EXTRA_CHARSETS=all \
-DSYSCONFDIR=/etc
#安装
make && make install
#设置环境变量
cp support-files/my-default.cnf /etc/my.cnf
cp support-files/mysql.server /etc/rc.d/init.d/mysqld
chmod +x /etc/rc.d/init.d/mysqld
chkconfig --add mysqld
echo "PATH=$PATH:/usr/local/mysql/bin" >> /etc/profile
source /etc/profile
#创建mysql数据库，并授权
groupadd mysql
useradd -M -s /sbin/NOlogin mysql -g mysql
chown -R mysql.mysql /usr/local/mysql
mkdir -p /data/mysql
#初始化数据库
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql

Second, modify the mysql master configuration file: /etc/my.cnf, pay attention to three servers can not be the same as the server-id

---配置主服务器：
vim /etc/my.cnf
[mysql]
server-id = 1
#开启二进制日志
log_bin = master-bin
#允许从服务器进行同步
log-slave-updates = true

---配置从服务器1：
vim /etc/my.cnf
[mysql]
server-id = 2
#开启二进制日志
log_bin = master-bin
 #使用中继日志进行同步
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

---配置从服务器2：
vim /etc/my.cnf
[mysql]
server-id = 3
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

Third, the three servers start mysql service

#在三台服务器上分别创建这两个个软链接
ln -s /usr/local/mysql/bin/mysql /usr/sbin/
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/

#启动mysql
/usr/local/mysql/bin/mysqld_safe --user=mysql &
#关闭防火墙和安全功能
systemctl stop firewalld.service
setenforce 0

Fourth, the master-slave synchronization Mysql configuration (two from a master)

1, mysql master-slave configuration is relatively simple. Note that authorized
authorized user in two databases on all nodes are synchronized using a user myslave from the database manager is a user usage monitoring

mysql -u root -p             //进入数据库

mysql> grant replication slave on *.* to 'myslave'@'192.168.45.%' identified by '123';
mysql> grant all privileges on *.* to 'mha'@'192.168.45.%' identified by 'manager';
mysql> flush privileges;  //刷新数据库
#在数据库上添加下列授权（理论上不需要）以主机名授权（MHA检查时是通过主机名的形式）
mysql> grant all privileges on *.* to 'mha'@'master' identified by 'manager';
mysql> grant all privileges on *.* to 'mha'@'slave1' identified by 'manager';
mysql> grant all privileges on *.* to 'mha'@'slave2' identified by 'manager';

View binary file on the master server and synchronization points

show master status;

Set the synchronization from the server in two

#在两台从服务器上都执行下列的命令，同步主服务器的日志
mysql>  change master to master_host='192.168.45.133',master_user='myslave',master_password='123',master_log_file='master-bin.000001',master_log_pos=1292;
mysql>  start slave;    //开启slave
mysql> show slave status\G;  //查看slave

Two from the library provided only mode

mysql> set global read_only=1;
mysql> flush privileges;   //刷新数据库

Installation MHA

MHA installed on all servers in the environment in accordance with lazy, first install epel source

yum install epel-release --nogpgcheck -y

yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN

Installation node on all servers

#修改服务器名称，将manager服务器的主机名改为manager
hostnamectl set-hostname manager
 su
#挂载软件包
mount.cifs //192.168.100.3/mha /mnt
#解压安装node
cd ~
tar zxvf /mnt/mha4mysql-node-0.57.tar.gz
cd mha4mysql-node-0.57/
perl Makefile.PL
 make && make install

Installation manager on the manger server (Note: Be sure to install the components to install node manager component)

#关闭防火墙
systemctl stop firewalld.service 
setenforce 0
#解压并安装manager
cd ~
tar zxvf /mnt/mha4mysql-manager-0.57.tar.gz
mha4mysql-manager-0.57/
perl Makefile.PL 
make && make install

After the server manager installed below usr / local / bin directory will generate several tools:

• masterha_check_repl check mysql replication status
• masterha_master_monitor check whether the master is down
• masterha_check_ssh check MHA's SSH configuration
• masterha_master_switch control failover
• masterha_check_status check the current operating status MHA
• Masterha_conf_host add or remove server configuration information
• masterha_stop Close manager
• masterha_manager startup manager script

  After node installed in the / usr / local / bin will be generated following several scripts (MHA Manager typically triggered by script, without human operation)

• apply_diff_relay_logs: identifying differences relay event log events and the difference is applied to the other Slave;
• save_binary_logs: save and copy the master binary log;
• filter_mysqlbinlog: removing unnecessary ROLLBACK event (MHA no longer use this tool);
• purge_relay_logs: Clear relay log (does not block SQL thread);

  No password to access the configuration

1、在manager配置所有数据库节点的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.133
ssh-copy-id 192.168.45.130
ssh-copy-id 192.168.45.134

2、在master上配置到数据库节点slave1和slave2的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.130
ssh-copy-id 192.168.45.134

3、在slave1上配置到数据库节点master'和slave2的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.133
ssh-copy-id 192.168.45.134

4、在slave2上配置到数据库节点slave1和master的无密码认证
ssh-keygen -t rsa
ssh-copy-id 192.168.45.133
ssh-copy-id 192.168.45.130

Configuration MHA

1, copy the relevant script on the manager node to / usr / local directory

cp -ra /root/mha4mysql-manager-0.57/samples/scripts/ /usr/local/bin/
ls scripts/
 master_ip_failover：自动切换时 VIP 管理的脚本；
 master_ip_online_change：在线切换时 VIP 的管理；
 power_manager：故障发生后关闭主机的脚本；
 send_report：因故障切换后发送报警的脚本；

VIP will automatically switch management scripts copied to / usr / local / bin / directory:

cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin/

Master_ip_failover rewrite the script:

vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.45.100';
my $brdc = '192.168.45.255';
my $ifdev = 'ens33';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
my $exit_code = 0;
#my $ssh_start_vip = "/usr/sbin/ip addr add $vip/24 brd $brdc dev $ifdev label $ifdev:$key;/usr/sbin/arping -q -A -c 1 -I $ifdev $vip;iptables -F;";
#my $ssh_stop_vip = "/usr/sbin/ip addr del $vip/24 dev $ifdev label $ifdev:$key";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

4, MHA software to create a directory and copy the configuration file

mkdir /etc/masterha
 cp /root/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha/
 #编辑配置文件
vim /etc/masterha/app1.cnf

[server default]
#manager配置文件
manager_log=/var/log/masterha/app1/manager.log     
#manager日志
manager_workdir=/var/log/masterha/app1
#master保存binlog的位置，这里的路径要与master里配置的bilog的相同
master_binlog_dir=/usr/local/mysql/data
#设置自动failover时候的切换脚本。也就是上边的那个脚本
master_ip_failover_script=/usr/local/bin/master_ip_failover
#设置手动切换时候的切换脚本
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
#这个密码是前文中创建监控用户的那个密码
password=manager
remote_workdir=/tmp
#设置复制用户密码
repl_password=123
#设置复制用户的用户
repl_user=myslave
#设置发生切换后发生报警的脚本
reporl_script=/usr/local/send_report
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.45.130 -s 192.168.45.134
#设置故障发生关闭故障脚本主机
shutdown_script=""
#设置ssh的登录用户名
ssh_user=root
#设置监控用户
user=mha

[server1]
hostname=192.168.45.133
port=3306

[server2]
candidate_master=1
#设置为候选master，如果设置该参数以后，发送主从切换以后将会从此从库升级为主库
hostname=192.168.45.130
check_repl_delay=0
port=3306

[server3]
hostname=192.168.45.134
port=3306

Ssh without password authentication test, if normal final output will be successful:

masterha_check_ssh -conf=/etc/masterha/app1.cnf

Check the construction of health status

masterha_check_repl -conf=/etc/masterha/app1.cnf

Configuring a virtual ip on the master

/sbin/ifconfig ens33:1 192.168.45.100/24

Start manager

1, start mha on the server manager

 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

2, view the MHA state, you can see the current master node is mysql

masterha_check_status --conf=/etc/masterha/app1.cnf

Fault Simulation

1, start monitoring observe hunger log records

 tailf /var/log/masterha/app1/manager.log

2, turn off the master server

pkill -9 mysql

You can see from the state library, vip will switch to the library from one of them:

At this point, the client can also be through a virtual ip, connected to the database:

mysql -h 192.168.45.100 -u root -p

--------------------------- These are the MHA schema file -----------