Record the mysql master-slave architecture

Start preparation

• mysql 8.0.20 version and two centos7 servers.
• Regarding the installation of mysql on Linux, you can use Baidu by yourself, you can also use the pagoda tool to build mysql, or you can use docker to build a mysql mirror.

Master-slave theory

• The establishment of a MySQL master-slave cluster can solve MySQL's data storage and access pressure.
• Ensure data security: MySQL master-slave configuration is equivalent to one more backup data.
• Realize read-write separation: Many Internet projects are scenarios where read more and write less, so read-write separation can be performed, and the main database writes data and reads data from the database. Some middleware, such as ShardingSphere, is required to achieve read-write separation.
• Ensure high availability: When the main database goes down, a new main database can be elected from the database to ensure service availability. Achieving high availability also requires some middleware, such as MMM, MHA, MGR, etc.

Synchronization principle

• The data synchronization of the MySQL master-slave architecture is achieved through binlog log files. Open the binlog log on the master server to record each step of the database operation, and then the slave server will have an IO thread to establish a TCP connection with the master server to request the binlog log on the master server to be transmitted. The upper dump thread of the main library (the main database server, the same below) will transmit the contents of the binlog log file to the slave server through this TCP connection, and then the slave library (the slave database server, the same below) will write the read binlog log data In its own relay log file. Then another thread from the library will read the contents of the relay file to replay the operation and restore the data. As shown below
  
• MySQL binlog logs are used for master-slave data synchronization, and can also be used for cache data synchronization. For example, the cache data in redis can simulate a slave node to initiate a binlog data synchronization request to MySQL, and then write the data to redis to achieve consistency between the cache and the database.
• When building a master-slave architecture, the MySQL versions of both parties must be the same, or the master server version is lower than the slave server. There are also two nodes that need to be synchronized.

Master-slave build

Build the main database

• First open the configuration file on the master node: /etc/my.cnf. Configure to open the binlog log and specify the serverId. As shown below

[mysqld]
server-id=47
#开启binlog
log_bin=master-bin
log_bin-index=master-bin.index
skip-name-resolve
# 设置连接端口
port=3306
# 设置mysql的安装目录
basedir=/usr/local/mysql
# 设置mysql数据库的数据的存放目录
datadir=/usr/local/mysql/mysql-files
# 允许最大连接数
max_connections=200
# 允许连接失败的次数。
max_connect_errors=10
# 服务端使用的字符集默认为UTF8
character-set-server=utf8
# 创建新表时将使用的默认存储引擎
default-storage-engine=INNODB
# 默认使用“mysql_native_password”插件认证
#mysql_native_password
default_authentication_plugin=mysql_native_password

• The following configurations need attention:
  • server-id: The unique identifier of the service node. Each cluster node needs to be configured with a separate ID.
  • log_bin: Turn on binlog logging and specify the file name
  • log_bin-index: binlog log file
• The mysql service needs to be restarted after configuration: service mysqld restart
• Here you can view the synchronization status of the master node through the command:

show master status;

• file: refers to the binlog file of the current log.
• position: refers to the index in the file.
• binlog_do_db and binlog_ignore_db refer to the library that records the binlog file and the library that does not need to record the binlog file. (Blank means no configuration)

Build from database

• Also open the mysql configuration file for configuration: my.cnf

[mysqld]
#主库和从库需要不一致
server-id=48
#打开MySQL中继日志
relay-log-index=slave-relay-bin.index
relay-log=slave-relay-bin
#打开从服务二进制日志
log-bin=mysql-bin
#使得更新的数据写进二进制日志中
log-slave-updates=1
# 设置3306端口
port=3306
# 设置mysql的安装目录
basedir=/usr/local/mysql
# 设置mysql数据库的数据的存放目录
datadir=/usr/local/mysql/mysql-files
# 允许最大连接数
max_connections=200
# 允许连接失败的次数。
max_connect_errors=10
# 服务端使用的字符集默认为UTF8
character-set-server=utf8
# 创建新表时将使用的默认存储引擎
default-storage-engine=INNODB
# 默认使用“mysql_native_password”插件认证
#mysql_native_password
default_authentication_plugin=mysql_native_password

• The following configurations need attention:

  • server-id: The unique identifier of the service node.
  • relay-log: Turn on the relay log record of the slave server.
  • log-bin: Turn on binlog logging.

• Then start the MySQL slave server and set the synchronization status of his master node:

#登录从服务
mysql -u root -p;
#设置同步主节点：
CHANGE MASTER TO
MASTER_HOST='192.168.232.128',
MASTER_PORT=3306,
MASTER_USER='root',
MASTER_PASSWORD='root',
MASTER_LOG_FILE='master-bin.000004',
MASTER_LOG_POS=156,
GET_MASTER_PUBLIC_KEY=1;
#开启slave
start slave;
#查看主从同步状态
show slave status;
或者用 show slave status \G; 这样查看比较简洁

• Note: CHANGE MASTER must specify MASTER_LOG_FILE and MASTER_LOG_POS to be consistent with the main database. Subsequent checks on whether the master-slave architecture is successfully configured also need to be determined by using the two attributes of file and position to be consistent.
  
• Pay attention to the two circled attributes in the above figure. If they are consistent with the master node, it means that the master-slave synchronization is successfully built.

Master-slave synchronization test

• You can first use the show databases command to view the status of the databases in the two mysql servers. As shown below
  
• Then use the following command to add a database on the master library, and find that there is a corresponding database on the slave library. As shown below

mysql> create database syncdemo;
Query OK, 1 row affected (0.00 sec)

• Then add a table and a piece of data to the syncdemo database, and found that there are corresponding tables and data on the slave database. As shown below

mysql> use syncdemo;
Database changed
mysql> create table demoTable(id int not null);
Query OK, 0 rows affected (0.02 sec)
mysql> insert into demoTable value(1);
Query OK, 1 row affected (0.01 sec)

• So write operations on the master library will be synchronized to the slave library.
• If Slave_sql_runing=no is found when checking the status on the slave library, it means that the master-slave synchronization has failed. It may be because the write operation from the library conflicts with the synchronized sql operation, or it may be that the transaction operation is rolled back after restarting from the library.
• The data here cannot be synchronized from the slave library to the master library. If you want to synchronize from the slave library to the master library, you only need to configure the node synchronization status CHANGE MASTER TO... on the master library, then this is the dual master mode. This can also configure the mutual master cluster mode.

Master-slave synchronization expansion

• The above are all synchronization operations for the master-slave database, but in actual operation, it may only synchronize a certain database or table. That can also be configured.
• The configuration in the configuration file my.cnf of the main library is as follows:

#需要同步的二进制数据库名
binlog-do-db=masterdemo
#只保留7天的二进制日志，以防磁盘被日志占满(可选)
expire-logs-days = 7
#不备份的数据库
binlog-ignore-db=information_schema
binlog-ignore-db=performation_schema
binlog-ignore-db=sys

• The configuration file my.cnf in the slave library is configured as follows:

#如果salve库名称与master库名相同，使用本配置
replicate-do-db = masterdemo
#如果master库名[mastdemo]与salve库名[mastdemo01]不同，使用以下配置[需要做映射]
replicate-rewrite-db = masterdemo -> masterdemo01
#如果不是要全部同步[默认全部同步]，则指定需要同步的表
replicate-wild-do-table=masterdemo01.t_dict
replicate-wild-do-table=masterdemo01.t_num

• After the configuration is complete, use the shou master status command to see the effect of the two parameters binlog_do_db and binlog_lgnore_db.
• In order to ensure data consistency, data is usually written to the main library and data is read from the library. This is the separation of MySQL's read and write. If you want to achieve MySQL's read-write separation, then you need to rely on the third-party plug-in ShardingSphere.
• In order to limit the data written from the library, you can configure the read_only parameter (set global read_only=1;) so that users can be restricted from directly writing data, but it will not affect the synchronization data of the binlog log operation of reading the main library from the library. Also, read_only can only restrict the data written by ordinary operating users. If users with super authority can still write data, if you want to restrict the write data of super users, you can use the super_read_only=0 command.

GTID synchronization cluster mode

• GTID essentially uses binlog to synchronize data, but it will identify the progress of synchronization from the database based on a global transaction ID. Therefore, GTID is a global transaction ID, which is globally unique and incremented, ensuring that a unique ID is generated by the Academy of Sciences in the replication cluster for each transaction submitted on the main database.
• In the GTID-based synchronization data, the slave library will tell the master library that the GTID of the transaction has been executed by the slave library, and then the master library will send all transactions that are not executed on the slave library to the slave library for execution, and use GTID to synchronously copy the data It can be guaranteed that the same transaction will only be executed once in the specified slave library. This can avoid data inconsistencies caused by offsets.
• The configuration method of GTID is similar to the above configuration, you only need to modify the configuration in the my.cnf configuration file:

# 主库配置 
gtid_mode=on
enforce_gtid_consistency=on
log_bin=on
server_id=单独设置一个
binlog_format=row

# 从库配置
gtid_mode=on
enforce_gtid_consistency=on
log_slave_updates=1
server_id=单独设置一个

• Then start the slave library and the master library separately to start the GTID synchronous replication.

Cluster expansion

• In the above one-master one-slave cluster mode, if you need to expand into a one-master multi-slave mode, you only need to increase the binlog replication of the slave library.
• If the previous cluster has been running for a period of time, when the slave server needs to be expanded, the binlog file of the previous master library cannot be copied to the new slave server. Then add a data copy operation for the new slave server. The MySQL data backup and recovery operation is relatively simple. You can first generate the sql file of the original database, and then put it into the new slave server for execution, and then repeat the above configuration to synchronize the configuration of the slave database.

Semi-synchronous replication

• When MySQL's master-slave data is synchronized, asynchronous replication is used. The master can write to the binlog file after the user commit commits the transaction, and then returns to the client. Another thread dump of the main library will send binlog data to the slave library. The asynchrony here will cause the main library to insert data and crash after returning. At this time, the dump thread has not sent data to the slave library, resulting in data inconsistency.
• In order to solve this problem, MySQL has a semi-synchronous data replication mode to ensure data security, which is to change the asynchronous sending of binlog data to synchronous sending.
  
• When the user submits the transaction to the binlog file, it does not return immediately, but waits for the slave to receive the data and write the data into the relay-log file to respond to the master node before returning a successful response to the client. The master database is waiting for the slave database. The ack response has a timeout mechanism. The default waits for 10 seconds. If the ack from the library has not been obtained, it will be downgraded to asynchronous data replication.
• This kind of synchronously replicated data can improve data consistency compared to asynchronously replicated data. But it is not absolute consistency, because it can only guarantee that the binlog can be transmitted to at least one slave library after the transaction is committed, but it does not guarantee that the slave library transaction is executed successfully. And because of synchronous waiting, there will be a certain delay and service performance will be reduced.

Build a semi-synchronous replication mode

• This module has been included in MySQL 5.5 and above, and it is contained in the two files semisync_master.so and semisync_salve.so in the lib/plugin directory of the MySQL installation directory. You need to install the semisync_master module on the master server and the semisync_salve module on the slave server.
  
• First install the module of the main server

mysql> install plugin rpl_semi_sync_master soname 'semisync_master.so';
Query OK, 0 rows affected (0.01 sec)
mysql> show global variables like 'rpl_semi%';
+-------------------------------------------+------------+
| Variable_name | Value |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled | OFF |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_for_slave_count | 1 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
+-------------------------------------------+------------+
6 rows in set, 1 warning (0.02 sec)
mysql> set global rpl_semi_sync_master_enabled=ON;
Query OK, 0 rows affected (0.00 sec)

• The first command line: To install the semi-synchronous replication module through the extension library, you need to specify the file name of the extension library.
• The second command line: View the system global parameters, rpl_semi_sync_master_timeout is the timeout waiting time of semi-synchronous replication. Can be self-configured.
• The third command line: Turn on the semi-synchronous replication switch
• Then install the module from the server

mysql> install plugin rpl_semi_sync_slave soname 'semisync_slave.so';
Query OK, 0 rows affected (0.01 sec)
mysql> show global variables like 'rpl_semi%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| rpl_semi_sync_slave_enabled | OFF |
| rpl_semi_sync_slave_trace_level | 32 |
+---------------------------------+-------+
2 rows in set, 1 warning (0.01 sec)
mysql> set global rpl_semi_sync_slave_enabled = on;
Query OK, 0 rows affected (0.00 sec)
mysql> show global variables like 'rpl_semi%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| rpl_semi_sync_slave_enabled | ON |
| rpl_semi_sync_slave_trace_level | 32 |
+---------------------------------+-------+
2 rows in set, 1 warning (0.00 sec)
mysql> stop slave;
Query OK, 0 rows affected (0.01 sec)
mysql> start slave;
Query OK, 0 rows affected (0.01 sec)

• The installation process of the slave server is similar to that of the master server, and the slave server needs to be restarted after the installation is complete.

Master-slave architecture data delay

• In the master-slave architecture, there will be delays between data synchronization and replication and MySQL read-write separation. When the master data is inserting data, the data is being read from the database, and the delay of master-slave data replication will cause data inconsistency.
• The problem is that when the user writes data and commits the transaction, taking the Java back-end as an example, it is multi-threaded concurrent writing, and then the binlog file is pulled from the server in a single thread, and the efficiency is poor in the middle.
• MySQL supports server parallel replication after version 5.7. You can set slave_parallel_workers to a number greater than 0 on the slave server, and set the slave_parallel_type parameter to LOGLCAL_CLOCK.

mysql high-availability solution

• The MySQL clusters above are built based on their own functions and do not have high-availability functions. That is, if the master server is hung up, the slave server cannot automatically switch to the master server. If you want to use MySQL's high availability, you need some third-party tools to achieve it.
• Common ones are: MMM, MHA, MGR. Some of their common points:
  • Monitor the mater master node of the master-slave cluster architecture.
  • Automatically migrate the master node.
  • Reconfigure the slave node to synchronize the data of the new master node.
• Let me mention that MHA high-availability solutions are currently used in large Internet companies. Because MMM is almost eliminated, and MGR is a new tool that lacks authoritative certification.
• Finally, I won’t go into details about the implementation principles of three of them, and those who are interested can study them on their own.