Introduction and configuration of MySQL8.0-Replication

This article explains MySQL 8.0 replication, the source mainly refers to the official document MySQL8.0 Replication

overview

Scenarios for MySQL 8.0 replication:

Scalability: read-write separation, distributed pressure, and improved performance
Data security: the slave database can perform backup operations without polluting the data of the main database
Analysis and processing: the main library receives data, and the analysis data from the library does not affect the performance of the main library
Long-distance data distribution

For replication solutions, see Replication Solutions

MySQL 8.0 supports two ways of recording copy progress from the library :

binlog files and positions
global transaction identifiers(GTID)

MySQL 8.0 supports three replication synchronization methods:

One-way(Asynchronous) Replication
Semisynchronous Replication
Delayed Replication

MySQL 8.0 supports three replication formats Replication Formats :

Statement Based Replication (SBR)
Row Based Replication (RBR)
Mixed Based Replication (MBR)

For the configuration file parameters of replication, please refer to: Replication and Binary Logging Options and Variables

Binlog + Post

The main library writes the binlog to the relaylog of the slave library through the network, and then executes the relaylog from the library to write the binlog of the slave library . Specifies received event execution.

Each slave library will record the binlog filename+position that has been executed in the main library.

Create the primary node User

First create a user-co-slave node link for the master node to use:

mysql> CREATE USER 'repl'@'%.example.com' IDENTIFIED BY 'password';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%.example.com';
mysql> flush privileges;

In addition, the way the master node verifies the slave node was used before version 8.0 mysql_native_passwordand later caching_sha2_password . If the above code is used directly, an error will be reported:

 Last_IO_Errno: 2061
  Last_IO_Error: error connecting to master '[email protected]:3306' - retry-time: 60 retries: 1 message: Authentication plugin 'caching_sha2_password' reported error: Authentication requires secure connection.

To use a secure connection or simply create a user or change the authentication method for a user's password:

// 创建时
mysql> CREATE USER 'repl'@'%.example.com' IDENTIFIED WITH [mysql_native_password | caching_sha2_password] BY 'password';
// 更改时
mysql> ALTER USER 'repl'@'%.example.com' IDENTIFIED WITH [mysql_native_password | caching_sha2_password] BY 'password';

For details, see:

Change the master-slave configuration

server_id和server_uuid

server_idThe sum of the master and slave nodes server_uuidmust be different

server_idThe default is 1, which can be my.cnfadded in the file server_id=1, and the master and slave nodes must be different
server_uuidBy default, it will be read from datadirthe auto.cnffile, if it cannot be read, it will be created automatically:

[auto]
server-uuid=db834596-844e-11ed-b00b-000c290264b9

If the uuid is the same, the following error will be reported:

Fatal error: The slave I/O thread stops 
because master and slave have equal MySQL server UUIDs; 
these UUIDs must be different for replication to work.

These two parameters can be viewed through the following commands after logging in to mysql:

select @@server_id;
select @@server_uuid;

An example of the main library my.cnffile is as follows:

[mysqld]
server_id = 1
# logbin文件名称 如：binlog.000018
log_bin=binlog
# redolog刷盘策略
innodb_flush_log_at_trx_commit=1
# binlog刷盘策略
sync_binlog=1

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

An example from the library my.cnffile is as follows:

# 其他配置和主库类似
# 添加从库只读
read_only=on

start replication

There are two ways to record the replication progress from the node:

binlog filename + position
GTID (used by Group Replication)

Here first use the binlog filename + position method.

First log in to the master node mysql to view the latest data binlog information, run:

mysql> flush logs;
Query OK, 0 rows affected (0.01 sec)

mysql> show master status\G
*************************** 1. row ***************************
			// filename
             File: binlog.000016
         // pos
         Position: 157
     Binlog_Do_DB: 
 Binlog_Ignore_DB: 
Executed_Gtid_Set: 
1 row in set (0.00 sec)

You can datadirview the specific names of other binlogs in .

Restart the master-slave mysql to ensure that the configuration takes effect normally.

Prior to MySQL 8.0.23:CHANGE MASTER TO
after:CHANGE REPLICATION SOURCE TO

Examples are as follows:

// 先停止复制线程
mysql> stop replica;
Query OK, 0 rows affected (0.00 sec)
// 配置主节点
mysql> change replication source to
    -> SOURCE_HOST='192.168.64.133',
    -> SOURCE_USER='repl',
    -> SOURCE_PASSWORD='password',
    -> SOURCE_LOG_FILE='binlog.000015',
    -> SOURCE_LOG_POS=157;
Query OK, 0 rows affected, 2 warnings (0.00 sec)
// 配置过滤器，只监听test数据库，更多filter操作参见
// https://dev.mysql.com/doc/refman/8.0/en/change-replication-filter.html
mysql > change replication filter replicate_do_db=(test);
Query OK, 0 rows affected (0.00 sec)
// 开启复制
mysql> start replica;
Query OK, 0 rows affected (0.00 sec)
// 
mysql> show replica status\G
*************************** 1. row ***************************

 Replica_IO_State: Waiting for source to send event
                  Source_Host: 192.168.64.133
                  Source_User: repl
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: binlog.000016
          Read_Source_Log_Pos: 449
               Relay_Log_File: 192-relay-bin.000008
                Relay_Log_Pos: 659
        Relay_Source_Log_File: binlog.000016
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
			...
				// 没有报错，正常运行
               Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
			...

For startup parameters and configuration details, refer to Replication and Binary Logging Options and Variables

GTID

Information about how each transaction is committed to the master and slave can be tracked, which means that when a failover occurs, the task of the new master is greatly simplified. GTID-based replication is based on transactions, as long as the transactions submitted by the master node are also submitted by the slave nodes, then the master-slave consistency can be guaranteed

GTID format and storage

GTID is a globally unique identifier created by the master node about the submitted transaction, and it is monotonically increasing without intervals. If the previous write by binlog failed or the transaction was filtered, no GTID is currently assigned to the main library.
Format:GTID = source_id:transaction_id

The slave node keeps the same GTID as the transaction submitted by the master node. The slave node will record the GTID before the copied transaction is executed, regardless of whether the transaction is executed successfully . This record refers to writing system variables and mysql.gtid_executedthis table (or temporary storage and binlog , and then wait to write to this table), the representation is as follows:

CREATE TABLE gtid_executed (
    source_uuid CHAR(36) NOT NULL,
    interval_start BIGINT(20) NOT NULL,
    interval_end BIGINT(20) NOT NULL,
    PRIMARY KEY (source_uuid, interval_start)
)


// 压缩前
+--------------------------------------+----------------+--------------+
| source_uuid                          | interval_start | interval_end |
|--------------------------------------+----------------+--------------|
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 37             | 37           |
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 38             | 38           |
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 39             | 39           |
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 40             | 40           |
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 41             | 41           |
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 42             | 42           |
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 43             | 43           |
...

// 压缩后
+--------------------------------------+----------------+--------------+
| source_uuid                          | interval_start | interval_end |
|--------------------------------------+----------------+--------------|
| 3E11FA47-71CA-11E1-9E33-C80AA9429562 | 37             | 43           |
...

Warning
As with other MySQL system tables, do not attempt to create or modify this table yourself.

GTID life cycle

When a transaction is committed at the master, the transaction is assigned a sequence number GTIDconsisting of the master's server_uuidand the smallest non-zero transaction sequence number available. GTIDWrite to the primary node binlog, and then immediately write the transaction binlog(these two writes can be considered atomic operations). If the transaction does not write binlog(such as filtering occurs, or the transaction is read-only), then the transaction will not be assigned GTID.
When the transaction is committed, GTIDthe binlogcommit is at binlogthe beginning of the transaction ( Gtid_log_event), and after the write, the information in the file will gtidbe written into the table when the file size reaches the maximum or the host is shut down .binlogbinlogbinloggtidmysql.gtid_executed
When a transaction is assigned a GTID, the current GTID will be written to the set of GTIDs : gtid_executedthis system variable ( @@GLOBAL.gtid_executed). From 2 and 3, it can be seen mysql.gtid_executedthat the latest record may not be recorded in the table gtid(the latest one binlogmay not be extracted), but gtid_executedthe system variable will record the latest one; if and only when the binlog is closed, the gtid will be directly written mysql.gtid_executedinto the table, However, the binlog of the main library must be turned on, and the binlog can be turned off from the library.
When the master library binlogis transmitted to the slave node relaylog, the slave node reads the master library GTIDand sets the system variable gtid_nextto identify the next gtidvalue.
Before processing the incoming transaction, it will read and verify the gtid ( gtid_nextcompared with the old one) to ensure that this gtid has not been used, and only one session is currently reading it. Therefore, processing gtid from the library belongs to the process of concurrency control. gtid_ownedThe system variable records the thread id currently using GTID. If the current gtid has already been committed, then this transaction will be ignored.
If the GTID has not been used, the slave library will not create a new GTID but directly use gtid_nextthe value of the system variable as the GTID ( gtid_nextthe GTID from the master library has been assigned before, that is, the GTID of the slave library's current transaction is equal to the GTID of the master library ).
If the binlog is enabled from the slave library, then the same as the master library, write the GTID binlog( Gtid_log_event) at the beginning of the binlog written to the transaction; if the binlog is not enabled from the slave library, it will be directly persisted to mysql.gtid_executedthe table. In this case, the table record Got the latest gtid.

GTID automatic positioning

SOURCE_AUTO_POSITIONTo enable GTID replication, you need to add or . when setting the master node from the library MASTER_AUTO_POSITIONas follows:

// binlogfile + pos
mysql> change replication source to
    -> SOURCE_HOST='192.168.64.133',
    -> SOURCE_USER='repl',
    -> SOURCE_PASSWORD='password',
    // 
    -> SOURCE_LOG_FILE='binlog.000015',
    //
    -> SOURCE_LOG_POS=157;
Query OK, 0 rows affected, 2 warnings (0.00 sec)

// GTID
mysql> change replication source to
    -> SOURCE_HOST='192.168.64.133',
    -> SOURCE_USER='repl',
    -> SOURCE_PASSWORD='password',
    //
	-> SOURCE_AUTO_POSITION=1
Query OK, 0 rows affected, 2 warnings (0.00 sec)

GTID configuration

Step 1: Synchronize the servers

First make all nodes read-only for synchronizing data:

mysql> SET @@GLOBAL.read_only = ON;

Important
It is important to understand that logs containing
transactions without GTIDs cannot be used on servers where GTIDs are
enabled. Before proceeding, you must be sure that transactions without
GTIDs do not exist anywhere in the topology.

Step 2: Stop both servers

$> mysqladmin -uusername -p shutdown

Step 3: Start both servers with GTIDs enabled

Write in the configuration file my.cnf:

gtid_mode=ON
enforce-gtid-consistency=ON

Step 4: Configure the replica to use GTID-based auto-positioning

mysql> CHANGE REPLICATION SOURCE TO
     >     SOURCE_HOST = host,
     >     SOURCE_PORT = port,
     >     SOURCE_USER = user,
     >     SOURCE_PASSWORD = password,
     >     SOURCE_AUTO_POSITION = 1;

Step 5: Start the replica and disable read-only mode

mysql> START REPLICA;
mysql> SET @@GLOBAL.read_only = OFF;

Restoring GTID mode replicas

When restoring a replica in a GTID based replication setup that has encountered an error, injecting an empty transaction may not solve the problem because an event does not have a GTID.

Use mysqlbinlog to find the next transaction, which is probably the first transaction in the next log file after the event. Copy everything up to the COMMIT for that transaction, being sure to include the SET @@SESSION.gtid_next. Even if you are not using row-based replication, you can still run binary log row events in the command line client.

Stop the replica and run the transaction you copied. The mysqlbinlog output sets the delimiter to /*!*/;, so set it back:

mysql> DELIMITER ;
Restart replication from the correct position automatically:

mysql> SET GTID_NEXT=automatic;
mysql> RESET SLAVE;
mysql> START SLAVE;
Or from MySQL 8.0.22:
mysql> SET GTID_NEXT=automatic;
mysql> RESET REPLICA;
mysql> START REPLICA;

GTID restrictions

Involving non-transactional storage engines
Table updates of non-transactional storage engines (such as MyISAM) cannot be copied to tables of transactional storage engines (such as InnoDB) through GTID. The reason is: non-transactional table updates will allocate multiple GTIDs, which will cause GTID mismatches in transactional tables.
Before CREATE TABLE ... SELECT statements
MySQL 8.0.21, CREATE TABLE ... SELECTGTID cannot be used, but it can be used later.
Temporary tables
详见Restrictions on Replication with GTIDs
Preventing execution of unsupported statements
Skipping transactions
Ignoring servers
GTID mode and mysql_upgrade

GTID Configuration Parameters

Global Transaction ID System Variables