Build MHA to realize MySQL high-availability cluster architecture

1. Introduction to MHA

MHA (Master High Availability) is currently a relatively mature solution in MySQL high availability. It was developed by youshimaton, a Japanese DeNA company (now working for Facebook), and is an excellent set of failover and master-slave solutions for MySQL high availability environments. Enhanced high availability software. In the process of MySQL failover, MHA can automatically complete the failover operation of the database within 0~30 seconds, and in the process of failover, MHA can ensure the consistency of data to the greatest extent, so as to achieve true High availability in a sense.

There are two roles in MHA, one is MHA Node (data node) and the other is MHA Manager (management node).

MHA Manager can be deployed on an independent machine to manage multiple master-slave clusters, or it can be deployed on a slave node. The MHA Node runs on each MySQL server, and the MHA Manager periodically detects the master node in the cluster. When the master fails, it can automatically promote the latest data slave to the new master, and then re-point all other slaves to the new master. the master. The entire failover process is completely transparent to the application.
Build MHA to realize MySQL high-availability cluster architecture
During the MHA automatic failover process, the MHA tries to save the binary log from the down primary server to ensure that the data is not lost to the greatest extent, but this is not always feasible. For example, if the primary server hardware fails or cannot be accessed via ssh, the MHA cannot keep the binary log and just fails over and loses the latest data. With MySQL 5.5's semi-synchronous replication, the risk of data loss can be greatly reduced. MHA can be combined with semi-synchronous replication. If only one slave has received the latest binary log, MHA can apply the latest binary log to all other slave servers, thus ensuring the data consistency of all nodes

Asynchronous replication

MySQL's default replication is asynchronous. The master library will immediately return the result to the client after executing the transaction submitted by the client, and does not care whether the slave library has received and processed it, so there will be a problem. The crash is gone. At this time, the transaction that has been submitted by the master may not be transmitted to the slave. If the slave is forcibly promoted to the master at this time, the data on the new master may be incomplete.

Fully synchronous replication

It means that when the main library executes a transaction, all the slave libraries execute the transaction before returning to the client. Because it needs to wait for all slave libraries to complete the transaction before returning, the performance of full synchronous replication will inevitably be seriously affected.

Semisynchronous replication

Between asynchronous replication and full synchronous replication, the master library does not return to the client immediately after executing the transaction submitted by the client, but waits for at least one slave library to receive it and write it to the relay log before returning it to the client. Compared with asynchronous replication, semi-synchronous replication improves the security of data, but it also causes a certain degree of delay, which is at least one TCP/IP round-trip time. Therefore, semi-synchronous replication is best used in low-latency networks.
Let's take a look at the schematic diagram of semi-synchronous replication:
Build MHA to realize MySQL high-availability cluster architecture

Summary: Similarities and differences between asynchronous and semi-synchronous

By default, MySQL's replication is asynchronous. After all update operations on the Master are written to Binlog, it does not ensure that all updates are replicated to the Slave. Although asynchronous operations are efficient, when there is a problem with the Master/Slave, there is a high risk of data being out of sync, and data may even be lost.
The purpose of introducing the semi-synchronous replication function in MySQL 5.5 is to ensure that when the master fails, the data of at least one slave is complete. In the case of timeout, it can also temporarily switch to asynchronous replication to ensure the normal use of services. After one salve catches up, it continues to switch to semi-synchronous mode.

working principle

Compared with other HA software, the purpose of MHA is to maintain the high availability of the Master database in MySQL Replication. Its biggest feature is that it can repair the difference logs between multiple slaves, and finally keep all slaves data consistent, and then select one of them to act as the new server. Master, and point other Slaves to it.

-从宕机崩溃的master保存二进制日志事件(binlogevents)。  
-识别含有最新更新的slave。  
-应用差异的中继日志(relay log)到其它slave。  
-应用从master保存的二进制日志事件(binlogevents)。  
-提升一个slave为新master。  
-使其它的slave连接新的master进行复制。

At present, MHA mainly supports the architecture of one master and multiple slaves. To build MHA, there must be at least three database servers in a replication cluster, one master and two slaves, that is, one acts as the master, one acts as the standby master, and the other acts as the slave database. , because at least three servers are required.
Related packages

MHA monitoring server installation: mha4mysql-manager-0.55-1.el5.noarch, mha4mysql-node-0.54-1.el5.noarch
Other master-slave cluster server installations: mha4mysql-node-0.54-1.el5.noarch

The official website address of the MHA software package: https://code.google.com/archive/p/mysql-master-ha/
The following packages are used:

mha4mysql-manager-0.55-1.el5.noarch
mha4mysql-node-0.54-1.el5.noarch

2. Build the basic environment of cluster architecture

Implementation environment:

Role	IP address	CPU name	Server Id	Types of	YOU
Manager	192.168.64.37	manager		management node	Centos7.2x86_64
master	192.168.64.7	master1	1	main mysql	Centos7.2x86_64
Candidate master	192.168.64.17	master2	2	from mysql	Centos7.2x86_64
slave	192.168.64.27	slave	3	from mysql	Centos7.2x86_64

The master provides external write services, the alternate master (actual slave, host name master2) provides read services, and the slave also provides related read services. Once the master goes down, the alternate master will be upgraded to a new master, and the slave points to The new master, manager as the management server.

1. After configuring the IP addresses of all hosts, check the selinux and firewalld settings, and close all host selinux and firewalld services to facilitate master-slave synchronization without errors in the later stage.

vim /etc/sysconfig/selinux
SELINUX=disabled   本行替换为

systemctl stop firewalld   关闭防火墙

2. Synchronize server time

vim /etc/chrony.conf
server 192.168.64.7 iburst   master2 slave配置与master1时间同步

3.
Configure epel source official website download address on all four hosts
: click to open the link wget https://mirrors.ustc.edu.cn/epel/7/x86_64/Packages/e/epel-release-7-11.noarch .rpm
4. Establish ssh non-interactive login environment on four hosts

[root@manager ~]#  ssh-keygen -t rsa
[root@manager ~]#  ssh-copy-id -i id_rsa.pub 192.168.64.37
[root@manager ~]#  scp authorized_keys id_rsa 192.168.64.17:/root/.ssh/
[root@manager ~]#  scp authorized_keys id_rsa 192.168.64.27:/root/.ssh/
[root@manager ~]#   scp authorized_keys id_rsa 192.168.64.7:/root/.ssh/

测试ssh登录面密钥
[root@master ~]# ssh [email protected]   
其它主机可以分别测试下

3. Configure MySQL master-slave replication

1. Install the node node package (master1 master2, slave) on the master and slave nodes

[root@master ~]# ls
anaconda-ks.cfg  Downloads                             original-ks.cfg  reset.sh   Videos
Desktop          mha4mysql-node-0.54-0.el6.noarch.rpm  Pictures         reset.sql
Documents        Music                                 Public           Template
[root@master ~]# l  yum install mha*

2. Modify the my.cnf file and configure master-slave synchronization.
Note: If the master MYSQL server already exists, the slave MYSQL server will only be built later. Before configuring data synchronization, the database to be synchronized on the master MYSQL server should be copied to the slave MYSQL server. On the server (for example, back up the database on the master MYSQL first, and then use the backup to restore the slave MYSQL server)
(1.) The host configuration of master1:

innodb_file_per_table
log_bin
read_only
server_id=0
skip_name_resolve=1

(2.) Host configuration of master2:

innodb_file_per_table
server_id=2
skip_name_resolve=1
read_only
relay_log_purge=0
log_bin

(3) Slave host configuration:

innodb_file_per_table
server_id=3
skip_name_resolve=1
read_only
relay_log_purge=0
log_bin

Note: Restart the mariadb service on all hosts! ! !

(3.) Create the account "mharep" for master-slave replication, which can be created on the (master1, master2) hosts, and create the MHA management account "manager" on all mysql servers.
Configuration on master1 and master2 hosts:

mysql>GRANT REPLICATION SLAVE ON *.* TO 'repluser'@'192.168.64.%' IDENTIFIED BY 'centos';
mysql> GRANT ALL ON *.* TO 'mhauser'@'192.168.64.%' IDENTIFIED BY 'centos';

Configuration on the slave host:

mysql> GRANT ALL ON *.* TO 'mhauser'@'192.168.64.%' IDENTIFIED BY 'centos';

Start creating master-slave replication:

View the node of master1:

MariaDB [(none)]> show master logs;
+--------------------+-----------+
| Log_name           | File_size |
+--------------------+-----------+
| mariadb-bin.000001 |       502 |
| mariadb-bin.000002 |       487 |
| mariadb-bin.000003 |       504 |
| mariadb-bin.000004 |       245 |
+--------------------+-----------+
4 rows in set (0.04 sec)

Configuration on master2 host:

Create master-slave replication and enable slave function

mysql>CHANGE MASTER TO MASTER_HOST='192.168.64.7',MASTER_USER='repluser' ,MASTER_PASSWORD='centos',MASTER_LOG_FILE='mariadb-bin.000004',MASTER_LOG_POS=245;
mysql> start slave;

Check the status of master2 master slave, the following two values must be yes, which means that the slave server can connect to the master server normally

Slave_IO_Running:Yes  
Slave_SQL_Running:Yes

Configuration on the slave host:

Create master-slave replication and enable the salve function

mysql>CHANGE MASTER TO MASTER_HOST='192.168.64.7',MASTER_USER='repluser' ,MASTER_PASSWORD='centos',MASTER_LOG_FILE='mariadb-bin.000004',MASTER_LOG_POS=245;
mysql> start slave;

Check the status of the slave host. The following two values must be yes, which means that the slave server can connect to the master server normally.

Slave_IO_Running:Yes  
Slave_SQL_Running:Yes

MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.64.7
                  Master_User: repluser
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mariadb-bin.000004
          Read_Master_Log_Pos: 245
               Relay_Log_File: mariadb-relay-bin.000002
                Relay_Log_Pos: 531
        Relay_Master_Log_File: mariadb-bin.000004
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 245
              Relay_Log_Space: 827
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
1 row in set (0.00 sec)

ERROR: No query specified

Notice:

The first grant command is to create an account repluser for master-slave replication, which can be created on the hosts of master1 and master2.

The second grant command is to create the MHA management account manager, which needs to be created on all mysql servers. MHA will require remote login to the database in the configuration file, so the necessary authorization is required.

Fourth, install and configure mysql-MHA

mha includes manager node and data node. The data node includes the hosts in the original MySQL replication structure, at least 3, that is, 1 master and 2 slaves. When the master fails, the master-slave structure can also be guaranteed; the master-slave replication cluster only needs to install the node Bag.

manager server: runs the monitoring script, responsible for monitoring and auto-failover; the mha manager node needs to install the node package and the manager package.
1. Two operation management nodes (mha4mysql-manager-0.55-0.el6.noarch.rpm and mha4mysql-node-0.54-0.el6.noarch.rpm) need to be installed on the manager host. On the three database hosts, only You need to install the node node of MHA.

[root@manager ~]# ls
192.168.64.17    cobbler.ks            mha4mysql-manager-0.55-0.el6.noarch.rpm
192.168.64.27    ks-post.log           mha4mysql-node-0.54-0.el6.noarch.rpm
192.168.64.7     ks-post-nochroot.log  original-ks.cfg
anaconda-ks.cfg  ks-pre.log
[root@manager ~]#  yum install mha*

The other three database nodes need to install the MHA node node (the process is omitted)! ! !

Configure MHA

Similar to the vast majority of Linux applications, proper use of MHA relies on a sensible configuration file. The configuration file of MHA is similar to the my.cnf file configuration of mysql. It is configured in the way of param=value. The configuration file is located on the management node, usually including the host name of each mysql server, mysql user name, password, working directory, etc. Wait.

(1.) Edit /etc/masterha/app1.conf with the following contents:

[root@manager ~]# vim /etc/mastermha/app1.cnf 
[server default]
user=mhauser
password=centos
manager_workdir=/data/mastermha/app1/
manager_log=/data/mastermha/app1/manager.log
remote_workdir=/data/mastermha/app1/
ssh_user=root
repl_user=repluser
repl_password=centos
ping_interval=1

[server1]
hostname=192.168.64.7
candidate_master=1
[server2]
hostname=192.168.64.17
candidate_master=1
[server3]
hostname=192.168.64.27
candidate_master=1

Explanation of configuration items:

manager_workdir=/masterha/app1 //设置manager的工作目录  
manager_log=/masterha/app1/manager.log //设置manager的日志  
user=manager//设置监控用户manager  
password=123456  //监控用户manager的密码  
ssh_user=root  //ssh连接用户  
repl_user=mharep  //主从复制用户  
repl_password=123.abc //主从复制用户密码  
ping_interval=1   //设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行railover  
master_binlog_dir=/usr/local/mysql/data   //设置master 保存binlog的位置，以便MHA可以找到master的日志，我这里的也就是mysql的数据目录  
candidate_master=1//设置为候选master，如果设置该参数以后，发生主从切换以后将会将此从库提升为主库。

SSH validity verification:

[root@manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.cnf 
Sun Apr 22 06:36:33 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
/etc/masterha/app1.cnf:No such file or directory
 at /usr/share/perl5/vendor_perl/MHA/SSHCheck.pm line 148.
[root@manager ~]# masterha_check_ssh 
masterha_check_ssh
[root@manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.cnf 
Sun Apr 22 06:37:13 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Apr 22 06:37:13 2018 - [info] Reading application default configurations from /etc/mastermha/app1.cnf..
Sun Apr 22 06:37:13 2018 - [info] Reading server configurations from /etc/mastermha/app1.cnf..
Sun Apr 22 06:37:13 2018 - [info] Starting SSH connection tests..
Sun Apr 22 06:37:18 2018 - [debug] 
Sun Apr 22 06:37:14 2018 - [debug]  Connecting via SSH from [email protected](192.168.64.27:22) to [email protected](192.168.64.7:22)..
Sun Apr 22 06:37:16 2018 - [debug]   ok.
Sun Apr 22 06:37:16 2018 - [debug]  Connecting via SSH from [email protected](192.168.64.27:22) to [email protected](192.168.64.17:22)..
Sun Apr 22 06:37:17 2018 - [debug]   ok.
Sun Apr 22 06:37:18 2018 - [debug] 
Sun Apr 22 06:37:14 2018 - [debug]  Connecting via SSH from [email protected](192.168.64.17:22) to [email protected](192.168.64.7:22)..
Sun Apr 22 06:37:16 2018 - [debug]   ok.
Sun Apr 22 06:37:16 2018 - [debug]  Connecting via SSH from [email protected](192.168.64.17:22) to [email protected](192.168.64.27:22)..
Sun Apr 22 06:37:17 2018 - [debug]   ok.
Sun Apr 22 06:37:18 2018 - [debug] 
Sun Apr 22 06:37:13 2018 - [debug]  Connecting via SSH from [email protected](192.168.64.7:22) to [email protected](192.168.64.17:22)..
Sun Apr 22 06:37:17 2018 - [debug]   ok.
Sun Apr 22 06:37:17 2018 - [debug]  Connecting via SSH from [email protected](192.168.64.7:22) to [email protected](192.168.64.27:22)..
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:080GPU/VjQmyb/Ije4lASHgZDXJv5E/QOqAcAv0wfV0.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:1
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Sun Apr 22 06:37:18 2018 - [debug]   ok.
Sun Apr 22 06:37:18 2018 - [info] All SSH connection tests passed successfully.

Validation of cluster replication:

Note: mysql database must be started

[root@manager ~]# masterha_check_repl --conf=/etc/mastermha/app1.cnf

Note: If the verification is successful, all servers and master-slave status will be automatically identified! ! !

When verifying, if you encounter this error: Can't exec "mysqlbinlog" ......

The workaround is to execute on all servers:

[css] view plain copy
ln -s /usr/local/mysql/bin/* /usr/local/bin/

Start manager:

[root@manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf 
Sun Apr 22 06:39:35 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Apr 22 06:39:35 2018 - [info] Reading application default configurations from /etc/mastermha/app1.cnf..
Sun Apr 22 06:39:35 2018 - [info] Reading server configurations from /etc/mastermha/app1.cnf..

Note: When applying Unix/Linux, we generally want a program to run in the background, so we will often use & at the end of the program to make the program run automatically. For example, we want to run mysql in the background: /usr/local/mysql/bin/mysqld_safe –user=mysql&. But there are many programs that do not want the same as mysqld, so we need the nohup command,

5. Simulate faults and detect state transitions

1. Stop the mariadb service of master1

[root@master ~]#  systemctl stop mariadb

(2.) View the MHA log
The log location specified in the configuration file above is /data/masterha/app1/manager.log

[root@manager ~]# cat/masterha/app1/manager.log

From the log information, you can see that the master failover has succeeded, and you can see the general process of failover

(3) Check the slave's replication
login to Mysql of the slave (192.168.64.27), and check the slave status

mysql> show slave status\G;

It can be seen that the IP of the master is now 192.168.64.17, which has been switched to be synchronized with 192.168.64.27. It was originally synchronized with 192.168.64.7, indicating that MHA has upgraded Candicatemaster (master2) to a new master, IO thread and SQL thread Also running correctly, MHA build successfully!!!