There is a single point of failure for a mysql host in the production environment, so we must ensure the high availability of mysql, that is, if one of the two MySQL servers is down, the other one can immediately take over.
There are generally several high-availability solutions for MySQL:
keepalived+dual master, MHA, PXC, MMM, Heartbeat+DRBD, etc. The more commonly used ones are keepalived+dual master, MHA and PXC.
This section mainly introduces the use of keepalived to achieve high availability of MySQL database. Keepalived+mysql dual master to realize MySQL-HA, we must ensure that the data of the two MySQL databases are exactly the same. The basic idea is that the two MySQL are master-slave relationship with each other. Configure the virtual IP through Keepalived to realize when one of the MySQL databases is down. After the machine, the application can automatically switch to another MySQL database to ensure the high availability of the system.
surroundings:
| mysql1 | 192.168.1.65 | mysql 、keepalived |
|---|---|---|
| mysql2 | 192.168.1.67 | mysql 、keepalived |
1. Configure two MySQL master and master synchronization
The first part of the process is that the master records the binary log. Before each transaction updates the data, the master records these changes in the binary log. MySQL writes transactions to the binary log. After the event is written to the binary log, the master notifies the storage engine to commit the transaction. The next step is for the slave to copy the master's binary log to its own relay log.
First, the slave starts a worker thread-the I/O thread. The I/O thread opens a normal connection on the master, and then starts the binlog dump process. Binlog dump process reads events from the binary log of the master. If the master has been synchronized, it will sleep and wait for the master to generate new events. The I/O thread writes these events to the relay log. The SQL slave thread handles the last step of the process. The SQL thread reads events from the relay log, replays the events in it, and updates the slave's data to make it consistent with the data in the master. As long as the thread is consistent with the I/O thread, the relay log is usually located in the OS cache, so the overhead of the relay log is small.
Master-master synchronization is the relationship between two machines as masters of each other, and writes on any one machine will be synchronized. If the mysql host has a firewall turned on, you need to turn off the firewall or create a rule.
1. Modify the MySQL configuration file. Both MySQL must enable the binlog log function. How to enable it: add the log-bin=MySQL-bin option in the [MySQL] section of the MySQL configuration file. The server-IDs of the two MySQL cannot be the same. The default In this case, the serverIDs of both MySQL servers are 1, and one of them needs to be changed to 2.
The relevant configuration in master1 is as follows:
log-bin = mysql-bin
binlog_format = mixed
server-id = 1
relay-log = relay-bin
relay-log-index = slave-relay-bin.index
auto-increment-increment = 2
auto-increment-offset = 1
!重启MySQL
The relevant configuration in master2 is as follows:
log-bin = mysql-bin
binlog_format = mixed
server-id = 2
relay-log = relay-bin
relay-log-index = slave-relay-bin.index
auto-increment-increment = 2
auto-increment-offset = 2
!重启MySQL
Note: master1 and master2 differ only in server-id and auto-increment-offset.
There is an auto-increment field in mysql, and two related configurations of auto-increment need to be set when doing the main synchronization of the database:
auto_increment_offset, auto_increment_increment. auto-increment-increment represents the increment amount of the self-increment field each time, and its default value is 1. Its value should be set to the total number of servers in the entire structure. In this case, two servers are used, so the value is set to 2. auto-increment-offset is used to set the starting point (initial value) of automatic growth in the database, because the two servers have set an automatic growth value of 2, so their starting points must be different, so as to avoid two Primary key conflict occurs during server data synchronization
Note: You can add "binlog_do_db=database name" configuration item (you can add more than one) in the my.cnf file to specify the database to be synchronized
2. Set master1 as the master server
of master2. Create an authorized account on the master1 host and allow connection on the master2 (192.168.1.67) host
mysql> grant replication slave on *.* to rep@'192.168.1.67' identified by '123.com';
Query OK, 0 rows affected, 1 warning (0.01 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
//View the current binlog status information of master1
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000002 | 609 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
//Set master1 as your own master server on master2 and enable the slave function.
mysql> change master to
-> master_host='192.168.1.65',
-> master_user='rep',
-> master_password='123.com',
-> master_log_file='mysql-bin.000002',
-> master_log_pos=609;
Query OK, 0 rows affected, 2 warnings (0.02 sec)
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)
//master1 sets up firewall communication
[root@mysqld01 ~]# firewall-cmd --add-port=3306/tcp --permanent
success
[root@mysqld01 ~]# firewall-cmd --reload
success
//View the status of the slave, the following two values must be yes, which means that the slave server can connect to the master server normally
mysql> show slave status\G
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
.....
3. Set master2 as the master server
of master1. Create an authorized account on the master2 host and allow connection on the master1 (192.168.1.65) host
mysql> grant replication slave on *.* to rep@'192.168.1.65' identified by '123.com';
Query OK, 0 rows affected, 1 warning (0.00 sec)
//View the binlog status of master2
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000002 | 452 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
//Set master2 as your own master server on master1 and turn on the slave function
mysql> change master to
-> master_host='192.168.1.67',
-> master_user='rep',
-> master_password='123.com',
-> master_log_file='mysql-bin.000002',
-> master_log_pos=452;
Query OK, 0 rows affected, 2 warnings (0.01 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
//master2 sets up firewall communication
[root@mysql02 ~]# firewall-cmd --add-port=3306/tcp --permanent
success
[root@mysql02 ~]# firewall-cmd --reload
success
//View slave status
mysql> show slave status\G
....
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
.....
4. Test So
far, the dual master configuration of mysql is
complete.//Create the zi library on master1, view and delete it on master2
mysql> create database zi;
View and delete on master2
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| mytest |
| performance_schema |
| sys |
| zi |
+--------------------+
6 rows in set (0.01 sec)
mysql> drop database zi;
Query OK, 0 rows affected (0.01 sec）
Checking the library on master1 will find that the zi library is gone
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| mytest |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
The simple understanding is that the operations on master1 will be synchronized to master2, and the operations of the same master2 will be synchronized to master1
Note: If the master MYSQL server already exists, and the slave MYSQL server will only be built later, you should copy the database to be synchronized from the master MYSQL server to the slave MYSQL server before configuring data synchronization (such as backing up the database on the master MYSQL first, Reuse the backup to restore from the MYSQL server)
Second, the installation and configuration of keepalived
Next we will complete the high availability of keepalived. Keepalived is a software solution to ensure the high availability of the cluster in cluster management. Its function is similar to heartbeat. It is used to prevent single points of failure. Keepalived is based on the VRRP protocol. The full name of VRRP is Virtual Router Redundancy Protocol, that is, virtual router redundancy protocol.
The virtual routing redundancy protocol can be considered as a protocol for achieving high availability of routers, that is, N routers that provide the same function form a router group. There is a master and multiple backups in this group. The master has a VIP that provides services to the outside world. The master will send multicast (the multicast address is 224.0.0.18). When the backup fails to receive the vrrp packet, it is considered that the master is down. At this time, it is necessary to elect a backup to be the master according to the priority of VRRP. In this way, the high availability of the router can be guaranteed.
Keepalived mainly has three modules, namely core, check and vrrp. The core module is the core of keepalived, responsible for the startup and maintenance of the main process and the loading and analysis of global configuration files. check is responsible for health checks, including various common check methods. The vrrp module is to implement the VRRP protocol.
1.
Install the keepalived software package on master1 and master2. Install the keepalived software package and service control. Before compiling and installing Keepalived, you must first install the kernel development package kernel-devel and supports libraries such as openssl-devel and popt-devel.
yum install kernel-devel openssl-devel popt-devel
yum -y install keepalived.x86_64
Pay attention to the source package to install keepalived: if you don’t know which dependencies are required for keepalived, you can go to the downloaded source decompression directory to view the contents of the INSTALL file. After executing the make install operation, the /etc/init.d/keepalived script file will be automatically generated, but You also need to manually add it as a system service, so that you can use the service and chkconfig tools to manage the keepalived service program.
Master2 host also completes the keepalived installation, the same as master1, the installation process is omitted
//Set up a firewall to prevent split brain
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload
2. Modify Keepalived's configuration file
Keepalived has only one configuration file keepalived.conf, which mainly includes the following configuration areas, namely global_defs, vrrp_instance and virtual_server.
global_defs: Mainly configure the notification object and machine ID when a fault occurs.
vrrp_instance: Used to define the VIP area and its related attributes for external services.
virtual_server: virtual server definition
The keepalived configuration file is explained as follows
! Configuration File for keepalived //!表示注释
global_defs {
router_id MYSQL-1 //表示运行keepalived服务器的一个标识
}
vrrp_instance VI_1 {
state BACKUP //指定keepalived的角色, 两台配置此处均是BACKUP,设为BACKUP将根据优先级决定主或从
interface ens33//指定HA监测网络的接口
virtual_router_id 51 //虚拟路由标识，这个标识是一个数字(取值在0-255之间,用来区分多个instance的VRRP组播)，同一个vrrp实例使用唯一的标识,确保和master2相同，同网内不同集群此项必须不同,否则发生冲突。
priority 100 //用来选举master的，要成为master，该项取值范围是1-255（在此范围之外会被识别成默认值100）,此处master2上设置为50
advert_int 1 //发VRRP包的时间间隔，即多久进行一次master选举（可以认为是健康查检时间间隔）
nopreempt //不抢占，即允许一个priority比较低的节点作为master，即使有priority更
authentication {
//认证区域，认证类型有PASS和HA（IPSEC），推荐使用PASS（密码只识别前8位）
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
//VIP区域，指定vip地址
192.168.1.100
}
}
virtual_server 192.168.1.100 3306 {
//设置虚拟服务器，需要指定虚拟IP地址和服务端口，IP与端口之间用空格隔开
delay_loop 2 //设置运行情况检查时间，单位是秒
lb_algo rr //设置后端调度算法，这里设置为rr，即轮询算法
lb_kind DR //设置LVS实现负载均衡的机制，有NAT、TUN、DR三个模式可选
persistence_timeout 60 //会话保持时间，单位是秒。这个选项对动态网页是非常有用的，为集群系统中的session共享提供了一个很好的解决方案。有了这个会话保持功能，用户的请求会被一直分发到某个服务节点，直到超过这个会话的保持时间。
protocol TCP //指定转发协议类型，有TCP和UDP两种
real_server 192.168.1.65 3306 {
//配置服务节点1，需要指定real server的真实IP地址和端口，IP与端口之间用空格隔开
weight 3 //配置服务节点的权值，权值大小用数字表示，数字越大，权值越高，设置权值大小为了区分不同性能的服务器
notify_down /etc/keepalived/bin/mysql.sh //检测到realserver的mysql服务down后执行的脚本
TCP_CHECK {
connect_timeout 3 //连接超时时间
nb_get_retry 3 //重连次数
delay_before_retry 3 //重连间隔时间
connect_port 3306 //健康检查端口
}
}
}
//master1's keepalived configuration
! Configuration File for keepalived
global_defs {
router_id mysql02
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.100
}
}
virtual_server 192.168.1.100 3306 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.67 3306 {
weight 1
notify_down /etc/keepalived/bin/mysql.sh
TCP_CHECK {
connect_port 3306
connect_timeout 3
retry 3
delay_before_retry 3
}
}
}
//master2 configuration
! Configuration File for keepalived
global_defs {
router_id mysql01
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.100
}
}
virtual_server 192.168.1.100 3306 {
delay_loop 6
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.65 3306 {
weight 1
notify_down /etc/keepalived/bin/mysql.sh
TCP_CHECK {
connect_port 3306
connect_timeout 3
retry 3
delay_before_retry 3
}
}
}
3. Create a script (master1 and master2 can be used together)
# mkdir /etc/keepalived/bin
vi /etc/keepalived/bin/mysql.sh，内容如下：
#!/bin/bash
pkill keepalived
chmod +x /etc/keepalived/bin/mysql.sh
4. Test the ip addr show dev ens33 command executed on master1 and master2 to check the control rights of master1 and master2 to VIP (cluster virtual IP).
[root@mysqld01 ~]# ip a show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:57:8d:1c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.65/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.1.100/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe57:8d1c/64 scope link
valid_lft forever preferred_lft forever
[root@mysql02 ~]# ip a show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:19:f2:8c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.67/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::65e6:f746:5344:e26a/64 scope link
valid_lft forever preferred_lft forever
It can be seen that master1 is the primary server, and master2 is the standby server. Stop the MySQL service and see if the keepalived health check program will trigger the script we wrote
Stop the mysql service of the master1 host
[root@mysqld01 ~]# systemctl stop mysqld.service
[root@mysqld01 ~]# ip a show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:57:8d:1c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.65/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe57:8d1c/64 scope link
valid_lft forever preferred_lft forever
//After stopping MySQL, you will find that VIP is not on master1, but drifted to master2. This means that stopping the MySQL service on the main service triggers the script we wrote to perform automatic failover.
Summary: Keepalived+mysql dual master Generally speaking, when small and medium-sized, adopting this architecture is the most trouble-free. After the master node fails, the keepalived high-availability mechanism is used to quickly switch to the standby node. In this scheme, there are a few things to pay attention to:
1. When using keepalived as a high-availability solution, it is best to set both nodes to BACKUP mode to avoid conflicts caused by writing the same data to the two nodes due to accidental conditions (such as split brain) due to mutual preemption;
2. Set the auto_increment_increment (auto-increment step) and auto_increment_offset (auto-increment initial value) of the two nodes to different values. The purpose is to avoid unexpected downtime of the master node. Some binlogs may not be copied to the slave in time to be applied, which will cause the self-increment of the newly written data of the slave to conflict with the original master, so use it at the beginning It is staggered, of course, if there is a suitable fault-tolerant mechanism that can solve the ID conflict between the master and the slave, it can also not be done;
3. Slave node server configuration should not be too bad, otherwise it is more likely to cause replication delays. As a slave server of a hot standby node, the hardware configuration cannot be lower than that of the master node;
4. If you are very sensitive to latency issues, you can consider using the MariaDB branch version, or directly launch the latest version of MySQL 5.7. The use of multi-threaded replication can greatly reduce replication latency;