Mysql semi-MHA high availability architecture design

2. Introduction to MHA
1. What is MHA
MHA (Master High Availability) is currently a relatively mature solution for MySQL high availability. It was developed by Youshimaton, a Japanese DeNA company (now working at Facebook), and is an excellent set of solutions. MySQL high availability environmentFailover and master-slave promotionHighly available software. During the MySQL failover process, MHA can automatically complete the database failover operation within 0 to 30 seconds, and during the failover process, MHA canto a greater extentEnsure data consistency to achieve true high availability.
2. Working principle of MHA:
When the master fails, the I/O thread between slaves reads the binlog position on the master, and selects the closest slave as the latest master (latest slave).
Other slaves generate differential relay logs by comparing with the latest slave and apply them.
Apply the binlog saved from the master on the latest slave, and at the same time change the latest slavePromoted to master.
Finally, apply the corresponding differential relay log on other slaves and start replicating from the new master.
3. MHA component
MHA Manager (management node) MHA Manager can be deployed separately on an independent machine to manage multiplemaster-slave cluster, can also be deployed on a slave node.
MHA Node (data node) MHA Node runs onEach MySQL serverOn the MHA Manager, the MHA Manager will regularly detect the master node in the cluster. When the master fails, it can automatically promote the data slave to the new master, and then redirect all other slaves to the new master. The entire failover process is completely transparent to the application.
4. Introduction to MHA components.
MHA Manager
runs some tools, such as masterha_manager tool implementation.Automatically monitor MySQL Masterand implementationmaster failover, other tools manually implement master failover, online master transfer, connection check, etc. One Manager can manage multiple master-slave clusters.
MHA Node
is deployed on all servers running MySQL, whether master or slave. It has three main functions:
1) Save the binary log.
If the failed master can be accessed, the master's binary log will be copied.
2) Apply the differential relay log.
Generate the differential relay log from the slave with the latest data, and then apply the differential log.
3) Clear the relay log.
Delete the relay log without stopping the SQL thread.
5. Introduction to tools in MHA
(I) Manager tool
tool descriptionmasterha_check_ssh checks MHA's SSH configurationmasterha_check_repl checks MySQL replicationmasterha_manager starts MHAmasterha_check_status detects the current MHA running status masterha_master_monitor monitors whether the master is down masterha_master_switch controls failover (automatic or manual) masterha_conf_host adds or deletes configured server information (ii) Node tool
tool
description save_binary_logs saves and copies the master's binary log apply_diff_relay_logs identifies differential relay log events And applied to other slavefilter_mysqlbinlog remove unnecessary ROLLBACK events (MHA no longer uses this tool) purge_relay_logs clear relay logs (will not block the SQL thread)
Note: Node tools are usually triggered by MHA Manager scripts and do not require manual operation..
3. MHA deployment architecture
1. Deployment planning
role IP host name server-id function remarks MHA-Manager192.168.198.50mha.itcast.cn—Management node MHA-Node (Master) 192.168.198.60master.itcast.cn10 data node write MHA -Node (Slave1) 192.168.198.70slave1.itcast.cn20 data node read MHA-Node (Slave2) 192.168.198.80slave2.itcast.cn30 data node read
2, system and software version
system version MySQL version MHA version CentOS 7.6MySQL-5.7 .31mha4mysql-manager-0.57 mha4mysql-node-0.57
3. System environment initialization
Step 1: Clone the machine, first start MASTER, and then start SLAVE1, SLAVE2 and MHA at once
Step 2: Change the host name of the computer

# hostnamectl set-hostname master.itcast.cn
# su
​
# hostnamectl set-hostname slave1.itcast.cn
# su
​
# hostnamectl set-hostname slave2.itcast.cn
# su
​
# hostnamectl set-hostname mha.itcast.cn
# su

Step 3: Change the IP address and bind the host name and IP address to the /etc/hosts file

# vim /etc/sysconfig/network-scripts/ifcfg-ens33
...
IPADDR=10.1.1.10、IPADDR=10.1.1.20、IPADDR=10.1.1.30、IPADDR=10.1.1.40
...

# systemctl stop NetworkManager
# systemctl disable NetworkManager
# systemctl restart network

Use MX to connect to 4 servers and bind the IP address and host name to the /etc/hosts file

# vim /etc/hosts
10.1.1.10 master.itcast.cn
10.1.1.20 slave1.itcast.cn
10.1.1.30 slave2.itcast.cn
10.1.1.40 mha.itcast.cn

Step 4: Turn off the firewall and SELinux

# systemctl stop firewalld
# systemctl disable firewalld
# setenforce 0
# sed -i '/SELINUX=enforcing/cSELINUX=disabled' /etc/selinux/config

Step 5: Configure the YUM source
basic source (omitted, it is recommended to use Tencent source).
After configuring the basic Tencent source, install yum epel source

[root@localhost ~]# yum -y install epel-release

Configure epel source

# wget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo
# yum clean all
# yum makecache

Configure the yum source of the local self-built warehouse
① Upload the mha-yum software package to the /soft/mha directory

# mkdir -p /soft/mha

② Configure the local yum source. When we use yum, we can automatically find the mha-yum warehouse directory

# vim /etc/yum.repos.d/local.repo
[mha]
name=mha soft
baseurl=file:///soft/mha/mha-yum
enabled=1
gpgcheck=0

Test the installation of MHA dependent software libraries

yum -y install perl-DBD-MySQL \
perl-Config-Tiny \
perl-Time-HiRes \
perl-Mail-Sender \
perl-Mail-Sendmail \
perl-MIME-Base32 \
perl-MIME-Charset \
perl-MIME-EncWords \
perl-Params-Classify \
perl-Params-Validate.x86_64 \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
net-tools

Step 6: ntpdate time synchronization (very important)

ntpdate 182.92.12.11

4. Deploy the MySQL master-slave replication environment
. Database installation location: /usr/local/mysql
Database data directory: /usr/local/mysql/data
Socket file: /tmp/mysql.sock
Port setting: 3306
Configuration file location: /usr/local/mysql/my.cnf (MySQL => installation directory => /etc directory)
☆ MASTER main server
Step 1: Upload the software to the /root directory
Step 2: Use the script to install the MySQL software

# vim mysql.sh

#!/bin/bash
yum install libaio -y
tar -xf mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz
mv mysql-5.7.31-linux-glibc2.12-x86_64 /usr/local/mysql
useradd -r -s /sbin/nologin mysql
rm -rf /etc/my.cnf
cd /usr/local/mysql
mkdir mysql-files
chown mysql:mysql mysql-files
chmod 750 mysql-files
bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql &> /root/password.txt
bin/mysql_ssl_rsa_setup --datadir=/usr/local/mysql/data
cp support-files/mysql.server /etc/init.d/mysqld
service mysqld start
echo 'export PATH=$PATH:/usr/local/mysql/bin' >> /etc/profile
source /etc/profile

source mysql.sh

Step Three: Set MySQL Password

mysql -p

mysql> set password='123';
mysql> flush privileges;

Step 4: Perform safe initialization

mysql_secure_installation

Step 5: Write the my.cnf configuration file in the MASTER host

# vim /usr/local/mysql/my.cnf

[mysqld]
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
socket=/tmp/mysql.sock
port=3306
log-error=/usr/local/mysql/data/master.err
log-bin=/usr/local/mysql/data/binlog
server-id=10
character_set_server=utf8mb4
gtid-mode=on
log-slave-updates=1
enforce-gtid-consistency

service mysqld restart

☆ SLAVE1/SLAVE2 slave server
Step 1: Upload MySQL software to the server
Step 2: Use a script to install MySQL, but no initialization is required

vim mysql.sh

#!/bin/bash
yum install libaio -y
tar -xf mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz
rm -rf /usr/local/mysql
mv mysql-5.7.31-linux-glibc2.12-x86_64 /usr/local/mysql
useradd -r -s /sbin/nologin mysql
rm -rf /etc/my.cnf
cd /usr/local/mysql
mkdir mysql-files
chown mysql:mysql mysql-files
chmod 750 mysql-files
cp support-files/mysql.server /etc/init.d/mysqld
echo 'export PATH=$PATH:/usr/local/mysql/bin' >> /etc/profile
source /etc/profile

source mysql.sh

Step 3: Use rsync to synchronize the data directory in the MASTER server to SLAVE1 and SLAVE2
MASTER:

# rm -rf /usr/local/mysql/data/auto.cnf
rsync -av /usr/local/mysql/data [email protected]:/usr/local/mysql/

rsync -av /usr/local/mysql/data [email protected]:/usr/local/mysql/

Step 4: Configure the my.cnf file
SLAVE1 for SLAVE1 and SLAVE2:

vim /usr/local/mysql/my.cnf

[mysqld]
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
socket=/tmp/mysql.sock
port=3306
log-error=/usr/local/mysql/data/slave1.err
log-bin=/usr/local/mysql/data/binlog
relay-log=/usr/local/mysql/data/relaylog
server-id=20
character_set_server=utf8mb4
gtid-mode=on
log-slave-updates=1
enforce-gtid-consistency
skip-slave-start

SLAVE2：

vim /usr/local/mysql/my.cnf

[mysqld]
basedir =/usr/local/mysql
datadir=/usr/local/mysql/data
socket=/tmp/mysql.sock
port=3306
log-error=/usr/local/mysql/data/slave2.err
log-bin=/usr/local/mysql/data/binlog
relay-log=/usr/local/mysql/data/relaylog
server-id=30
character_set_server=utf8mb4
gtid-mode=on
log-slave-updates=1
enforce-gtid-consistency
skip-slave-start

After the configuration is complete, start the MySQL software

# service mysqld restart

☆ Configure master-slave data synchronization
Step 1: Create a slave synchronization account in the MASTER server

mysql> create user 'slave'@'10.1.1.%' identified by '123456';
mysql> grant replication slave on *.* to 'slave'@'10.1.1.%';
mysql> flush privileges;

Step 2: Create an mha account (to facilitate MHA monitoring of master-slave synchronization status later)

mysql> create user 'mha'@'10.1.1.40' identified by '123456';
mysql> grant all privileges on *.* to 'mha'@'10.1.1.40';
mysql> flush privileges;

Step 3: Configure master-slave data synchronization in SLAVE1 and SLAVE2

mysql> change master to master_host='192.168.198.60',master_port=3306,master_user='slave',master_password='123456',master_auto_position=1;
mysql> start slave;
mysql> show slave status\G

At this point, the entire configuration of one master and two slaves is complete!
5. Installation of MHA software
(1) Install software on different nodes
. Note: Install on all nodes.mha-nodesoftware package, inmha managementThen install the mha-manager software package

[root@mha ~]# yum –y install mha4mysql-node-0.57-0.el7.noarch.rpm
[root@master ~]# yum –y install mha4mysql-node-0.57-0.el7.noarch.rpm
[root@slave1 ~]# yum -y install mha4mysql-node-0.57-0.el7.noarch.rpm
[root@slave2 ~]# yum –y install mha4mysql-node-0.57-0.el7.n oarch.rpm
[root@mha ~]# yum –y install mha4mysql-manager-0.57-0.el7.noarch.rpm

If the dependent software has been installed in advance, you can use rpm -ivh to install it directly.
Configuring ssh mutual trust (password-free login)
Note:
In production environments, root remote login to the server is almost prohibited, so ssh password-free login must be in the admin user Configuration is done from a security perspective.
The admin user can be any ordinary user.
This ordinary user is used by the management node of mha to remotely access all hosts in the mysql replication group and complete other tasks.
Step 1: Create an ordinary account admin on all machines with a password of 123.

# useradd admin
# echo 123|passwd --stdin admin

Step 2: Configure the mha host to the admin user mutual trust mha end of other hosts :

[root@mha ~]# su - admin
[admin@mha ~]$ ssh-keygen -P "" -f ~/.ssh/id_rsa
[admin@mha ~]$ cd .ssh/
[admin@mha .ssh]$ ls
id_rsa  id_rsa.pub
[admin@mha .ssh]$ mv id_rsa.pub authorized_keys
[admin@mha .ssh]$ for i in 10 20 30;do scp -r ../.ssh/ 10.1.1.$i:~/;done

Test password-free login:

[admin@mha .ssh]$ ssh 10.1.1.10
[admin@mha .ssh]$ ssh 10.1.1.20
[admin@mha .ssh]$ ssh 10.1.1.30

Question: Password-free SSH only requires copying the public key, but the above code copies the entire .ssh directory. Answer: Because of the mutual trust of MHA, it is required that not only MHA needs to be able to access MASTER, SLAVE1, and SLAVE2 without passwords, but MASTER must also be able to access MHA, SLAVE1, SLAVE2 without passwords... (iii) Configure the
sudo permissions of the admin user
on the MASTER host:

[root@master ~]# vim /etc/sudoers.d/admin
#User_Alias  表示具有sudo权限的用户列表; Host_Alias表示主机的列表
User_Alias MYSQL_USERS = admin
#Runas_Alias  表示用户以什么身份登录
Runas_Alias MYSQL_RUNAS = root
#Cmnd_Alias  表示允许执行命令的列表（命令需要使用完整路径）
Cmnd_Alias MYSQL_CMNDS = /sbin/ifconfig,/sbin/arping
MYSQL_USERS ALL = (MYSQL_RUNAS) NOPASSWD: MYSQL_CMNDS

Distribute this permission to SLAVE1 and SLAVE2 (when a failure occurs, the slave server can also set its own VIP)

[root@master ~]# for i in 20 30;do scp /etc/sudoers.d/admin 10.1.1.$i:/etc/sudoers.d/;done

Test whether the admin user can mount VIP (VIP can only be mounted on the MASTER machine)

# su - admin
[admin@master ~]$ sudo /sbin/ifconfig ens33:1 10.1.1.100 broadcast 10.1.1.255 netmask 255.255.255.0
[admin@master ~]$ sudo /sbin/arping -fqc 5 -w 5 -I ens33 -s 10.1.1.100 -U 10.1.1.10
[admin@master ~]$ ip a 

​Supplement
:
arping: A command used to send ARP requests to other hosts in the LAN. It can be used to test whether an IP in the LAN has been used.
​-f
: Exit after receiving the first response packet.
-q: quite mode, no output is displayed.
-c: Stop after sending the specified count ARP REQUEST packets. If the -w parameter is specified, the same number of ARP REPLY packets will be waited until timeout.
-w: Specify a timeout in seconds. Arping will exit after reaching the specified time, regardless of how many packets are sent or received during the period. In this case, arping will not stop after sending the specified count (-c) packets, but will wait until timeout or the count packets sent have been responded to before exiting.
-I: Specify the device name, the name of the network device used to send ARP REQUEST packets.
-D: Duplicate address detection mode, used to detect whether there is an IP address conflict. If there is no IP address conflict, 0 is returned.
-s: Set the IP resource address for sending ARP packets
-U: Unreasonable (forced) ARP mode to update the local information in the ARP CACHE list on other hosts, no response is required.
-h: Display help page.
(IV) Create MHA related configuration files
MHA server: Create working directory

[root@mha ~]# mkdir /etc/mha/
[root@mha ~]# mkdir -p /data/mha/masterha/app1
[root@mha ~]# chown -R admin. /data/mha

Create mha local configuration file:

[root@mha ~]# vim /etc/mha/app1.conf
[server default]

Set monitoring user and password (modification 1)

user=mha
password=123

Set the replication user and password in the replication environment (modification 2)

repl_user=slave
repl_password=123

Set the login username for ssh (modification 3)

ssh_user=admin

Set the time interval for monitoring the main database and sending ping packets. The default is 3 seconds. If there is no response after three attempts, failover will be performed automatically.

ping_interval=3

Set the working directory of mgr

manager_workdir=/data/mha/masterha/app1

Set the directory where mysql master saves binlog so that MHA can find the master's binary log (modification 4)

master_binlog_dir=/usr/local/mysql/data

Set the master's pid file (modification 5)

master_pid_file=/usr/local/mysql/data/master.itcast.cn.pid

Set the directory where mysql master saves binlog when a switch occurs (create this directory on mysql master)

remote_workdir=/data/mysql/mha

Set the mgr log file (when MHA encounters problems, mainly look at this log)

manager_log=/data/mha/masterha/app1/app1-3306.log

If there is a problem between MHA and master monitoring, MHA Manager will try to log in to the master from slave1 and slave2.

secondary_check_script=/usr/bin/masterha_secondary_check -s 192.168.198.70 -s 192.168.198.80 --user=admin --port=22 --master_host=192.168.198.60 --master_port=3306

Set the switching script when automatic failover (when a failure occurs, automatically mount VIP to SLAVE1 or SLAVE2)

master_ip_failover_script="/etc/mha/master_ip_failover.sh 192.168.198.100 1"

Set the switching script for manual switching

#master_ip_online_change_script="/etc/mha/master_ip_online_change.sh 192.168.198.100 1"

Set up a script to shut down the failed host after a failure occurs

# shutdown_script="/etc/mha/power_manager"
[server1]
hostname=192.168.198.60
port= 3306
candidate_master=1
[server2]
hostname=192.168.198.70
port= 3306
candidate_master=1
[server3]
hostname=192.168.198.80
port= 3306
candidate_master=1

(5) Upload the corresponding script to the /etc/mha directory, then change the configuration information and authorize

[root@mha ~]# ls /etc/mha/
app1.conf  master_ip_failover.sh

Note: In the script content, the network card name and connection user must be changed to admin.

my $vip  = shift;
my $interface = 'ens33';        网卡名（38行）
my $key = shift;
...
sub stop_vip() {
    my $ssh_user = "admin";     用户名（110行）
    print "=======$ssh_stop_vip==================\n";
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

​[
root@mha ~]# chmod +x /etc/mha/master_ip_*
6. Detect SSH mutual trust and MySQL master-slave status
MHA:

# su - admin

Detecting SSH mutual trust

[admin@mha ~]$ masterha_check_ssh --conf=/etc/mha/app1.conf

Check cluster status

[admin@mha ~]$ masterha_check_repl --conf=/etc/mha/app1.conf
7. Check the MHA status, and then run MHA (monitoring starts)

MHA：

[admin@mha ~]$ masterha_check_status --conf=/etc/mha/app1.conf
app1 is stopped(2:NOT_RUNNING).

​Enable
MHA Manager monitoring (key, means start MHA):

[admin@mha ~]$ nohup masterha_manager --conf=/etc/mha/app1.conf --remove_dead_master_conf --ignore_last_failover &

​If an error is reported here,

Check the monitoring status again:

[admin@mha ~]$ masterha_check_status --conf=/etc/mha/app1.conf
app1 (pid:8913) is running(0:PING_OK), master:10.1.1.10

​​Note
:

1. If normal, "PING_OK" will be displayed, otherwise "NOT_RUNNING" will be displayed, indicating that MHA monitoring is not enabled.
2. Use the admin user to start monitoring, otherwise permission denial will be reported.
3. Manually stop monitoring command: masterha_stop --conf=/etc/mha/app1.conf
   At this point, our MHA has been configured!
   4. Automatic Failover Test
   1. Install testing tools on the MASTER server

[root@master ~]# yum -y install sysbench

2. Insert test data

Create a test library test on the master server

mysql> create database test charset utf8mb4;
Query OK, 1 row affected (0.17 sec)
​
mysql> grant all on *.* to 'mha'@'localhost' identified by '123'; 
Query OK, 0 rows affected (0.14 sec)
​
mysql> flush privileges;
Query OK, 0 rows affected (0.11 sec)
​
mysql> exit
Bye
​
​
​
[root@master ~]# sysbench /usr/share/sysbench/oltp_read_only.lua \
 --mysql-host=192.168.198.60  --mysql-port=3306  --mysql-user=mha \
 --mysql-password=123456  --mysql-socket=/tmp/mysql.sock \
 --mysql-db=test  --db-driver=mysql  --tables=1 \
 --table-size=100000 --report-interval=10 --threads=128 --time=120 prepare
 
mysql> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|   100000 |
+----------+
1 row in set (0.01 sec)

3. Take snapshots of
MHA, MASTER, SLA VE1, SLAVE2
4. Simulate MASTER server failure
MHA server: monitoring log

[root@mgr ~]# tail -f /data/mha/masterha/app1/app1-3306.log

MASTER server:

service mysqld stop

The MHA log shows the result:
failover:

VIP Drift:

FAQ
1. Management node configuration file error

[root@mgr ~]# cat /etc/mha/app1.conf
[server default]

Set the monitoring user and password. This user is the database management account created on the master and has all permissions.

user=mha
password=123

Set the replication user and password in the replication environment. Note that the following permissions are required:

#REPLICATION SLAVE和REPLICATION CLIENT
repl_user=slave
repl_password=123

Set the login username for ssh

ssh_user=admin
....
[server1]
hostname=10.1.1.10
port= 3306
candidate_master=1
[server2]
hostname=10.1.1.20
port= 3306
candidate_master=1
[server3]
hostname=10.1.1.30
port= 3306
candidate_master=1

Note: Be sure to configure the correct IP and port number
2. Data read-only settings when configuring MHA

Solution: Set up slaves

The server is read-only

mysql> set @@global.read_only=1;
Query OK, 0 rows affected (0.00 sec)
​mysql
> show variables like 'read_only';
±--------------±--- ---+
| Variable_name | Value |
±--------------±------+
| read_only | ON |
±------------ --±------+
1 row in set (0.00 sec)
​3.
Wrong password for copy user permissions

Reason:
The replication user slave does not have relevant permissions, REPLICATION SLAVE andREPLICATION CLIENT
No replication user is created on the slave server
4. Other errors
The MHA cluster requires at least 2 slaves, so if there is only one slave, the check will fail!
VIP: Virtual ip
configures VIP so that after the master host in the MHA cluster goes down, the client does not need to modify the database address and can seamlessly switch through VIP.
Note that the built-in VIP drift script can only be used on the same network segment, not across network segments. Insert the code snippet here.

https://note.youdao.com/s/1GI1m8z4