Master-slave replication architecture evolved introduction
basic structure
(1)一主一从
(2)一主多从
(3)多级主从
(4)双主
(5)循环复制
Advanced Application Architecture Evolution
High-performance architecture
(1) 读写分离架构(读性能较高)
代码级别
MySQL proxy (Atlas,mysql router,proxySQL(percona),maxscale)、amoeba(taobao) ,xx-dbproxy等。
(2) 分布式架构(读写性能都提高):
分库分表——cobar--->TDDL(头都大了),DRDS
Mycat--->DBLE自主研发等。
NewSQL-->TiDB
High availability architecture
(1)单活:MMM架构——mysql-mmm(google)
(2)单活:MHA架构——mysql-master-ha(日本DeNa),T-MHA
(3)多活:MGR ——5.7 新特性 MySQL Group replication(5.7.17) --->Innodb Cluster
(4)多活:MariaDB Galera Cluster架构,(PXC)Percona XtraDB Cluster、MySQL Cluster(Oracle rac)架构
Availability MHA *****
Architecture Works
#主库宕机处理过程
1. 监控节点 (通过配置文件获取所有节点信息)
系统,网络,SSH连接性
主从状态,重点是主库
2. 选主
(1) 如果判断从库(position或者GTID),数据有差异,最接近于Master的slave,成为备选主
(2) 如果判断从库(position或者GTID),数据一致,按照配置文件顺序,选主.
(3) 如果设定有权重(candidate_master=1),按照权重强制指定备选主.
1. 默认情况下如果一个slave落后master 100M的relay logs的话,即使有权重,也会失效.
2. 如果check_repl_delay=0的化,即使落后很多日志,也强制选择其为备选主
3. 数据补偿
(1) 当SSH能连接,从库对比主库GTID 或者position号,立即将二进制日志保存至各个从节点并且应用(save_binary_logs )
(2) 当SSH不能连接, 对比从库之间的relaylog的差异(apply_diff_relay_logs)
4. Failover
将备选主进行身份切换,对外提供服务
其余从库和新主库确认新的主从关系
5. 应用透明(VIP)
6. 故障切换通知(send_reprt)
7. 二次数据补偿(binlog_server)
8. 自愈自治(待开发...)
# Failover
1. MHA通过脚本(masterha_master_monitor)实时(ping_interval=2)监控主库的状态(主机,数据库状态)
2. 监控到主库宕机之后,会发生自动选主(masterha_master_switch )
(1) 权重: 参数
(2) 日志量
(3) 配置文件的顺序
3. 数据补偿
(1) 当主库SSH能够连接,各个从库会立即保存(save_binary_logs)缺失部分的日志到/var/tmp,进行数据补偿.
(2) 当主库SSH无法连接,通过脚本(apply_diff_relay_logs)自动进行从库的relaylog的差异处理,进行数据补偿.
4. 构建新的主从关系,并将故障节点移除(配置文件移除)
5. MHA主进程完成工作后,自动退出.
6. 应用透明 VIP(master_ip_failover_script=/usr/local/bin/master_ip_failover)
7. 故障通知 (report_script=/usr/local/bin/send)
8. 提供额外数据补偿的手段(binlog server)
自动自愈,待开发....
Architecture Description:
1主2从,
master:db01
slave:db02 db03 ):
MHA 高可用方案软件构成
Manager软件:选择一个从节点安装
Node软件:所有节点都要安装
MHA Software Configuration
Manager 工具包主要包括以下几个工具:
masterha_manger # 启动MHA
masterha_check_ssh #检查MHA的SSH配置状况
masterha_check_repl #检查MySQL复制状况
masterha_master_monitor #检测master是否宕机
masterha_check_status #检测当前MHA运行状态
masterha_master_switch #控制故障转移(自动或者手动)
masterha_conf_host #添加或删除配置的server信息
Node工具包主要包括以下几个工具:
这些工具通常由MHA Manager的脚本触发,无需人为操作
save_binary_logs 保存和复制master的二进制日志
apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的
purge_relay_logs 清除中继日志(不会阻塞SQL线程)
MHA environment to build
planning:
主库: 51 node
从库:
52 node
53 node manager
Preparing the environment
(2 slightly from the main GTID .1)
Configuration program soft key connection
ln -s /application/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
ln -s /application/mysql/bin/mysql /usr/bin/mysql
Configure each node trust
The first time you connect manually yes to confirm,
db01:
rm -rf /root/.ssh
ssh-keygen
cd /root/.ssh
mv id_rsa.pub authorized_keys
scp -r /root/.ssh 10.0.0.52:/root
scp -r /root/.ssh 10.0.0.53:/root
各节点验证
db01:
ssh 10.0.0.51 date
ssh 10.0.0.52 date
ssh 10.0.0.53 date
db02:
ssh 10.0.0.51 date
ssh 10.0.0.52 date
ssh 10.0.0.53 date
db03:
ssh 10.0.0.51 date
ssh 10.0.0.52 date
ssh 10.0.0.53 date
install software
Download software mha
mha official website: HTTPS: //code.google.com/archive/p/mysql-master-ha/
GitHub Download: https: //github.com/yoshinorim/mha4mysql-manager/wiki/Downloads
All nodes installed Node software dependencies
yum install perl-DBD-MySQL -y
rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
Mha need to create a user in the master database db01
grant all privileges on *.* to mha@'10.0.0.%' identified by 'mha';
Manager Software Installation (db03)
yum install -y perl-Config-Tiny epel-release perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes
rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm
Configuration file preparation (db03)
Create a profile directory
mkdir -p /etc/mha
Create a log directory
mkdir -p /var/log/mha/app1
Edit mha
Profile
[root@db03 ~]# vim /etc/mha/app1.cnf
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
user=mha
password=mha
ping_interval=2
repl_password=123
repl_user=repl
ssh_user=root
[server1]
hostname=10.0.0.51
port=3306
[server2]
hostname=10.0.0.52
port=3306
[server3]
hostname=10.0.0.53
port=3306
Status Check
## 互信检查
[root@db03 ~]# masterha_check_ssh --conf=/etc/mha/app1.cnf
## 主从状态检查
[root@db03 ~]# masterha_check_repl --conf=/etc/mha/app1.cnf
MySQL Replication Health is OK.
Open MHA (db03):
[root@db03 ~]# masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
View MHA status
[root@db03 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:4719) is running(0:PING_OK), master:10.0.0.51
2.4.10 fault simulation and processing
(1) Stop the main library db01:
[root@db01 ~]# /etc/init.d/mysqld stop
观察manager 日志 tail -f /var/log/mha/app1/manager
末尾必须显示successfully,才算正常切换成功。
(2) repair the main library
[root@db01 ~]# /etc/init.d/mysqld start
(3) from the main repair
[root@db01 ~]# mysql -e "change master to master_host='10.0.0.52',master_user='repl',master_password='123' ,MASTER_AUTO_POSITION=1;"
[root@db01 ~]# mysql -e "start slave;"
[root@db01 ~]# mysql -e "show slave status \G"|grep Running:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
(4) restore configuration files
[root@db03 /etc/mha]# cat /etc/mha/app1.cnf
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
password=mha
ping_interval=2
repl_password=123
repl_user=repl
ssh_user=root
user=mha
[server1]
hostname=10.0.0.51
port=3306
[server2]
hostname=10.0.0.52
port=3306
[server3]
hostname=10.0.0.53
port=3306
(5) Start MHA
[root@db03 ~]# nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
[root@db03 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
Manager introduces additional parameters
Description: The
main library is down someone to take over?
- All logs from the node are the same, the default will be the order of the configuration file to select a new master.
- Log inconsistency from the node, automatically selects the closest to the main library from the library
- If the weight is set (candidate_master = 1), the weight for a preferred node node.
But this amount of log node 100M log behind the main library, it would not be selected. May be blended check_repl_delay = 0, close inspection of the amount of the log, the forced selection candidate nodes.
(1) ping_interval=1
# 设置监控主库,发送ping包的时间间隔,尝试三次没有回应的时候自动进行failover
(2) candidate_master=1
# 设置为候选master,如果设置该参数以后,发生主从切换以后将会将此从库提升为主库,即使这个主库不是集群中事件最新的slave
(3)check_repl_delay=0
#默认情况下如果一个slave落后master 100M的relay logs的话,MHA将不会选择该slave作为一个新的master,因为对于这个slave的恢复需要花费很长时间,通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时,这个参数对于设置了candidate_master=1的主机非常有用,因为这个候选主在切换的过程中一定是新的master
MHA function of vip
(1) preparation vip script
[root@db03 ~]# cp master_ip_failover.txt /usr/local/bin/master_ip_failover
[root@db03 ~]# cd /usr/local/bin/
[root@db03 /usr/local/bin]# yum install -y dos2unix
[root@db03 /usr/local/bin]# dos2unix master_ip_failover
[root@db03 /usr/local/bin]# chmod +x master_ip_failover
(2) to change the manager configuration file:
vi /etc/mha/app1.cnf
#添加:
master_ip_failover_script=/usr/local/bin/master_ip_failover
(3) modify the contents of the file:
[root@db03 /usr/local/bin]# vim /usr/local/bin/master_ip_failover
my $vip = '10.0.0.55/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";
(4) on a main database, a manually generated first address vip
Hand bound to the main library vip, attention must be consistent ethN and configuration files, I was eth0: (1 is the key value specified) 1
ifconfig eth0:1 10.0.0.55/24
(5) Restart mha
masterha_stop --conf=/etc/mha/app1.cnf
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
email alert
(1) add configuration options
vi /etc/mha/app1.cnf
report_script=/usr/local/bin/send
(2) a copy of the script
[root@db03 ~]# cp -a email/* /usr/local/bin
(3) Stop MHA
masterha_stop --conf=/etc/mha/app1.cnf
(4) open MHA
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
(5) close the main library, see warning messages
After the main library is closed, it will automatically switch the main library at the same time send a warning message. But the MHA will stop, we need to re-build MHA
binlog server(db03)
(1) add configuration information
vim /etc/mha/app1.cnf
[binlog1]
no_master=1
hostname=10.0.0.53
master_binlog_dir=/data/mysql/binlog
(2) create the necessary directories
mkdir -p /data/mysql/binlog
chown -R mysql.mysql /data/*
(3) pulls the master database log binlog
Note: The starting point for pulling logs, according to the current needs have been acquired from the binary log starting point library
cd /data/mysql/binlog
mysqlbinlog -R --host=10.0.0.51 --user=mha --password=mha --raw --stop-never mysql-bin.000001 &
(4) Restart MHA
masterha_stop --conf=/etc/mha/app1.cnf
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
MHA Troubleshooting
Check the status of MHA
masterha_check_status --conf=/etc/mha/app1.cnf
Check the configuration file mha
[root@db03 ~]# vim /etc/mha/app1.cnf
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=mha
ping_interval=2
repl_password=123
repl_user=repl
report_script=/usr/local/bin/send
ssh_user=root
user=mha
[server1]
hostname=10.0.0.51
port=3306
[server2]
hostname=10.0.0.52
port=3306
[server3]
hostname=10.0.0.53
port=3306
[binlog1]
no_master=1
hostname=10.0.0.53
master_binlog_dir=/data/mysql/binlog
3. Check the relationship of mutual trust
[root@db03 ~]# masterha_check_ssh --conf=/etc/mha/app1.cnf
4. Check the script
At the same time need to check the contents of the script is correct
ls -l /usr/local/bin/
-rwxr-xr-x 1 root root 2166 Aug 20 15:45 master_ip_failover
-rwxr-xr-x 1 root root 35 Aug 20 15:51 send
-rwxr-xr-x 1 root root 80213 Aug 20 15:51 sendEmail
-rwxr-xr-x 1 root root 198 Aug 20 16:01 testpl
5. Check binlogserver
ps -ef| grep mysqlbinlog
6. Make sure that the main library, the recovery binlog log
db02 [(none)]>show master status
cd /data/mysql/binlog
rm -rf *
mysqlbinlog -R --host=10.0.0.52 --user=mha --password=mha --raw --stop-never mysql-bin.000005 &
7. Check the main library VIP
ifconfig eth0:1 10.0.0.55/24
ifconfig
8. Check the status of master and slave
masterha_check_repl --conf=/etc/mha/app1.cnf
9. Start MHA
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
masterha_check_status --conf=/etc/mha/app1.cnf