For MySQL GTID, it has been very stable after years of tempering. As an extension of the position method, it brings a lot of convenience in today's use environment. This article shares a summary of GTID technical points.
No matter which method is used for MySQL replication, it is inseparable from the binlog method. As an extension of the position method, GTID brings a lot of convenience in today's use environment. Let's review the knowledge points related to gtid step by step.
Base
First understand the basic principles of replication:
1. MySQL replication method
The master user writes data, generates events and records them in the binary log. The slave receives the binlog uploaded by the master, and then applies them in order to reproduce the operations on the master.
The traditional replication is based on (file, pos). When the master and slave are down and there is a problem when switching, the slave saves the (file, pos) on the original master, and cannot directly point to the (file, pos) on the new master.
2. The difference between position mode and GTID mode in logging
Visual comparison:
-
Master-slave replication, the default is to use the pos replication (postion) method to number each operation performed by the user (pos), and each event has a start number and an end number. GTID is a function similar to pos, which is globally common and the GTID value of events in the log file is consistent.
pos and GTID are an identifier in the log, and they are displayed in different ways in the slave. -
The generation of GTIDs is controlled by gtid_next.
On the Master, gtid_next is the default AUTOMATIC, that is, the GTID is automatically generated each time a transaction is committed. It finds an unused minimum value greater than 0 from the currently executed GTID set (ie gtid_executed) as the next transaction GTID. Also write the GTID to binlog (set gtid_next record), before actually updating the transaction record.
On the Slave, the GTID of the main library (ie, the set gtid_next record) is first read from binlog, and then the GTID is used for the transaction executed.
3. GTID Advantages
-
Simpler implementation of failover, unlike the traditional way of looking for log_file and log_Pos.
-
Easier to build master-slave replication.
-
The replication cluster has a unified way to identify the replication location, which brings convenience to cluster management.
-
Under normal circumstances, GTIDs are continuous without holes, so when data conflicts occur in the master-slave library, you can skip by adding empty things.
-
MySQL 5.7.6 version can upgrade gtid mode online
Unpopular Feature: Interface
This type of interface MySQL was initially prepared to solve the tracking of execution status and replication delay through the development of the interface. But it's basically not applicable now.
1. FUNCTION
-
GTID_SUBSET(set1,set2)
Given two sets of global transaction identifiers set1 and set2, returns true if all gtids in set1 are also in set2. Otherwise return false.
-
GTID_SUBTRACT(set1,set2)
Given two sets of global transaction identifiers set1 and set2, only return the gtid in set1 that does not belong to set2.
-
WAIT_FOR_EXECUTED_GTID_SET()
monitors all gtids applied on the server, including transactions arriving from all replication channels and user clients.
If a timeout is specified, and the timeout period elapses before all transactions in the GTID set are applied, the function stops waiting. The timeout is optional, the default timeout is 0 seconds, in which case the function always waits until all transactions in the GTID set have been applied.
-
WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS(gtid_set[, timeout][,channel])
Wait until all transactions have been applied, or until the timeout expires. 8.0.18 deprecated features, not introduced for the time being.
Reference: https://dev.mysql.com/doc/refman/5.7/en/gtid-functions.html
2. API
Tracks the gtid received from the library, return value: zero on success. Nonzero when an error occurs.
One use of the tracker mechanism is to provide a way for MySQL connectors and client applications to ensure execution. There are already mechanisms such as enhanced semi-synchronization, and there are basically no unless secondary development is performed.
-
mysql_session_track_get_first()
-
mysql_session_track_get_next()
Online analysis case:
DROP TABLE IF EXISTS test.t1;
CREATE TABLE test.t1 (i INT, f FLOAT);
--enable_session_track_info
SET @@SESSION.session_track_schema=ON;
SET @@SESSION.session_track_system_variables='*';
SET @@SESSION.session_track_state_change=ON;
USE information_schema;
SET NAMES 'utf8mb4';
SET @@SESSION.session_track_transaction_info='CHARACTERISTICS';
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SET TRANSACTION READ WRITE;
START TRANSACTION;
SELECT 1;
INSERT INTO test.t1 () VALUES();
INSERT INTO test.t1 () VALUES(1, RAND());
COMMIT;
shell> mysqltest < testscript
DROP TABLE IF EXISTS test.t1;
CREATE TABLE test.t1 (i INT, f FLOAT);
SET @@SESSION.session_track_schema=ON;
SET @@SESSION.session_track_system_variables='*';
-- Tracker : SESSION_TRACK_SYSTEM_VARIABLES
-- session_track_system_variables
-- *
SET @@SESSION.session_track_state_change=ON;
-- Tracker : SESSION_TRACK_SYSTEM_VARIABLES
-- session_track_state_change
-- ON
USE information_schema;
-- Tracker : SESSION_TRACK_SCHEMA
-- information_schema
-- Tracker : SESSION_TRACK_STATE_CHANGE
-- 1
SET NAMES 'utf8mb4';
-- Tracker : SESSION_TRACK_SYSTEM_VARIABLES
-- character_set_client
-- utf8mb4
-- character_set_connection
-- utf8mb4
-- character_set_results
-- utf8mb4
-- Tracker : SESSION_TRACK_STATE_CHANGE
-- 1
SET @@SESSION.session_track_transaction_info='CHARACTERISTICS';
-- Tracker : SESSION_TRACK_SYSTEM_VARIABLES
-- session_track_transaction_info
-- CHARACTERISTICS
-- Tracker : SESSION_TRACK_STATE_CHANGE
-- 1
-- Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS
--
-- Tracker : SESSION_TRACK_TRANSACTION_STATE
-- ________
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
-- Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS
-- SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SET TRANSACTION READ WRITE;
-- Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS
-- SET TRANSACTION ISOLATION LEVEL SERIALIZABLE; SET TRANSACTION READ WRITE;
START TRANSACTION;
-- Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS
-- SET TRANSACTION ISOLATION LEVEL SERIALIZABLE; START TRANSACTION READ WRITE;
-- Tracker : SESSION_TRACK_TRANSACTION_STATE
-- T_______
SELECT 1;
1
1
-- Tracker : SESSION_TRACK_TRANSACTION_STATE
-- T_____S_
INSERT INTO test.t1 () VALUES();
-- Tracker : SESSION_TRACK_TRANSACTION_STATE
-- T___W_S_
INSERT INTO test.t1 () VALUES(1, RAND());
-- Tracker : SESSION_TRACK_TRANSACTION_STATE
-- T___WsS_
COMMIT;
-- Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS
--
-- Tracker : SESSION_TRACK_TRANSACTION_STATE
-- ________
ok
Reference:
https://dev.mysql.com/doc/refman/5.7/en/mysql-session-track-get-first.html
https://dev.mysql.com/doc/refman/5.7/en/c -api-functions.html
Judgment method
1. How to judge whether the replication method is GTID or pos
Show slave status to view the Auto_Position field. 0 is pos mode, 1 is gtid mode.
Change gtid to pos method:
change master to master_auto_position=0;
2. Current execution gtid information
mysql> SELECT @@GLOBAL.GTID_EXECUTED;
+----------------------------------------------+
| @@GLOBAL.GTID_EXECUTED |
+----------------------------------------------+
| 39d0a7f2-702c-11ea-92a0-000c29b9a76d:1-46534 |
+----------------------------------------------+
mysql> SELECT * FROM mysql.gtid_executed;
The mysql.gtid_executed table is provided by the MySQL server for internal use. It allows replicas to use GTIDs when binary logging is disabled on the replica, and allows GTID state to be preserved in the event of binary log loss. RESET MASTER command, the gtid_executed table will be cleared.
In the event of an unexpected service stop, the gtid set in the current binary log file will not be saved in the gtid_executed table. During recovery, these gtids will be added to the table from the binary log file so that replication can continue.
3. gtid_executed
If the binary log is enabled on the MySQL server, the update of the table mysql.gtid_executed will only occur during binary rotation, because the current running GTID location can still be determined by scanning the binary log in the event of a restart.
Simply put, this table records the currently executed GTID.
The most important reason why the parameter log_slave_updates must be configured in MySQL 5.6 is that when the slave restarts, it is impossible to know the GTID location that the current slave has run to, because the variable gtid_executed is a memory value:
MySQL 5.7 persists the value of gtid_executed. The technique used is the same as how MySQL 5.6 handles the storage location of SQL threads, that is, the GTID value is persisted in an InnoDB table and submitted together with user transactions to achieve data consistency.
Triggering conditions:
-
When the binlog rotates (flush binary logs/reaches max_binlog_size) or shuts down the service, all Gtid information written to the binlog will be written to the mysql.gtid_executed table.
-
Slave library: If log_bin is not enabled or log_slave_updates is not enabled, the slave library will execute an insert mysql.gtid_executed operation for each transaction in the application relay-log.
Common commands
1. gtid settings
gtid_mode=ON #必选
enforce-gtid-consistency=true #必选
log-bin=mysql #5.6必选 5.7.5和它之后可选，为了高可用，最好设置
server-id=1 #开启log-bin的必须设置
log-slave-updates=ON # 5.6必选 5.7.5和它之后可选，为了高可用切换，最好设置ON
2. gtid skip gtid_next
stop slave;
set gtid_next='d74faa2d-5819-11e8-b248-ac853db70398:10603';
begin;commit;
set gtid_next='automatic';
start slave;
Remarks: This operation is similar to sql_slave_skip_counter, it just skips errors and cannot guarantee data consistency and requires manual intervention. It is strongly recommended that the slave enable read_only=1
3. gtid clear gtid_pureged
The actual meaning of the command: Because there is no binlog information (expire_logs_days), these gtid confirmations and rollbacks are not considered. Commonly used for backup and recovery, it is used when building a slave library.
Automatic trigger mechanism: flush, server restart
scenes to be used:
-
Disable binary logging on the replica to record GTIDs of committed replicated transactions.
-
GTIDs of transactions written to the binary log file, which is now cleared.
-
A gtid that is explicitly added to the set via the statement set @@GLOBAL.gtid_purged.
mysqldump --set-gtid-purged=off/on parameter;
whether to add GTID_PURGED' to output
4. gtid upgrade
pos upgrades the gtid method, and the conditions allow to suggest the method of rebuilding the slave library. There are risks in the following ways.
gtid_mode optional value
-
ON: Fully open GTID, if the open standby database receives a transaction without GTID, the replication is interrupted
-
ON_PERMISSIV: It can be considered as a transitional stage before turning on gtid. After the main database is set to this value, a GTID will be generated. At the same time, the standby database still tolerates transactions with and without GTID.
-
OFF_PERMISSIVE: It can be considered as a transitional stage before turning off GTID. After the main database is set to this value, it will no longer generate GTID, and the standby database can tolerate transactions with and without GTID. When the main library closes GTID, an Anonymous_Gtid event will be generated when the transaction is executed, which will be executed in the standby library: set @@session.gtid_next='anonymous'
-
OFF: Completely close GTID, if the standby database in the closed state receives a transaction with GTID, the replication will be interrupted
Switch from position mode to GTID mode:
1) Execute WARN mode at each server:
After this step is set, so that everything is allowed to violate the consistency of GTID
mysql>SET @@GLOBAL.ENFORCE_GTID_CONSISTENCY = WARN;
#这是第一个重要步骤. 您必须确保在进入下一步骤之前不会在错误日志中生成警告.
2) Execute ON mode at each server:
to ensure that all transactions cannot violate GTID consistency
mysql>SET @@GLOBAL.ENFORCE_GTID_CONSISTENCY = ON;
3) Execute OFF mode at each server:
This step indicates that the new transaction is anonymous, and the transaction allowed by colleagues to replicate is GTID or anonymous
mysql>SET @@GLOBAL.GTID_MODE = OFF_PERMISSIVE;
#需要确保这一步操作在所有的服务器上执行
4) Execute ON mode at each server:
This step indicates that the new transaction is GTID, and the transaction that colleagues allow to replicate is GTID or anonymous
mysql>SET @@GLOBAL.GTID_MODE = ON_PERMISSIVE;
#需要确保这一步操作在所有的服务器上执行
5) On each server, wait for the status variable ONGOING_ANONYMOUS_TRANSACTION_COUNT to be zero. It can be queried using:
mysql>SHOW STATUS LIKE 'ONGOING_ANONYMOUS_TRANSACTION_COUNT';
#在所有从库上查询该状态，必须为0 才能进行下一步。该状态宝石已标示为匿名的正在#进行的事务数量，如果状态值为0表示无事务等待被处理
Wait for all transactions generated to step 5 to be replicated to all servers. It is possible to do this without stopping updates: the only important thing is that all anonymous transactions are replicated.
6) GTID_MODE = ON executes on every server:
mysql>SET @@GLOBAL.GTID_MODE = ON;
7) Modify each my.cnf file:
gtid-mode=ON
ENFORCE_GTID_CONSISTENCY = ON
8) Although the GTID mode is configured for the above replication, it is still based on the Binlog method. By setting the option MASTER_AUTO_POSITION to 1, the replication can be adjusted to the replication based on GTID mode. The specific operations are as follows:
mysql>STOP SLAVE [FOR CHANNEL 'channel'];
mysql>CHANGE MASTER TO MASTER_AUTO_POSITION = 1 [FOR CHANNEL 'channel'];
mysql>START SLAVE [FOR CHANNEL 'channel'];
5. gtid 压缩 gtid_executed_compression_period
When GTID is enabled, the server periodically performs such compactions on the mysql.gtid_executed table. By setting the gtid_executed_compression_period system variable, you can control the number of transactions allowed before compressing the table, and thus the compression ratio. The default value of this variable is 1000; this means that, by default, table compaction is performed after every 1000 transactions.
Setting gtid_executed_compression_period to 0 prevents compression from being performed; however, if you do, you should prepare for a substantial increase in the amount of disk space that the gtid_executed table may require.
Use the following statement to query:
mysql> select thread_id,thread_os_id,name, processlist_command,
processlist_state
from `performance_schema`.threads where name like '%compress%';
+-----------+--------------+--------------------------------+---------------------+-------------------+
| thread_id | thread_os_id | name | processlist_command | processlist_state |
+-----------+--------------+--------------------------------+---------------------+-------------------+
| 26 | 8024 | thread/sql/compress_gtid_table | Daemon | Suspending |
+-----------+--------------+--------------------------------+---------------------+-------------------+
Note: If you find that the processlist_state value is always: "Compressing gtid_executed table", it means to compress. The memory of the record lock is requested from the operating system, so when the table gtid_executed keeps growing, it will eventually lead to MySQL OOM.
6. binlog_gtid_simple_recovery
The way MySQL iterates over binary log files during searches for GTIDs when it starts or restarts. Just to initialize gtid_executed, gtid_purged parameters, scan binlog or event related information
MySQL 5.7.7 or older binary log, you need to set binlog_gtid_simple_recovery=FALSE, if there are many non-gtid binlogs, it will greatly affect performance .
limit
It has been developed so far, but there are some scenarios that are limited.
1. create table xxx as select:
split into two parts:
create table xxxx like table;
insert into xxxx select *from table;
2. Restrictions on temporary tables
Using GTID replication mode:
1. Create temporary table and drop temporary table are not supported.
2. In the case of autocommit=1, a temporary table can be created. 3. The creation of a temporary table on the
Master side does not generate GTID information, so it will not be synchronized to the slave, but when the temporary table is deleted, a GTID will be generated, which will cause the master-slave to be interrupted.
3. Transaction operation
Involves the update of the non-transactional storage engine, and the non-transactional storage engine transactional storage engine update table cannot be executed in the same statement or the same transaction.
4.mysql_upgrade
GTID mode and mysql_upgrade. Do not enable binary logging via mysql_upgrade (--write binlog option) when running with Global Transaction Identifiers (GTIDs) enabled.
5. sql_slave_skip_counter The traditional mode of skipping postion is not supported in gtid mode.
Original link: https://www.modb.pro/db/27947