MySQL Case - abnormal synchronous parallel copy submit disorder caused

Symptom

Slave parallel copy after opening, the default will be out of order to commit the transaction, may cause synchronization interrupt;

Slave-side performance is thrown SQL thread synchronization, duplicate primary key, the modified data line is not present and the like;

GTID message similar to: 9a2a50aa-5504-11e7-9e59-246e965d93f4: 1-1371939844: 1371939846

1371939845 which is being given transaction, Intuitively, Slave end first submitted 1371939846 transactions;

Solution

MySQLversion>=5.7.5
slave_preserve_commit_order:OFF(default)->ON
注:binlog_order_commits=ON(default)

problem analysis

Consult the official WL # 6314 and WL # 7165 , here for the original language simple induction, we are interested can look at the original Level Architecture High;
WL # 6314: https://dev.mysql.com/worklog/task/ ? the above mentioned id = 6314
WL # 7165: https://dev.mysql.com/worklog/task/?id=7165
Note: commit-parent transaction in the original English text, sequence number refers to the binlog in last_commited and sequence_number; that is, simple translation of "logical time-stamped" 

WL # 6314 on slave side parallel applier

When the transaction enters prepare phase (group submitted a certain stage of the process), these transactions will receive a mark timestamp logical transaction for which mark the latest submitted;

The master terminal, the relevant process is as follows:

• In the prepare phase, the time stamp is obtained from commit_clock and stored, used to mark the latest committed transaction;
• In the commit stage (the binlog transaction have been written, but before submission engine layer), stepping operation performed on the commit_clock;

In Slave end, the relevant process is as follows:

• relaylog thread reads the coordinate event, if they have the same logical event timestamp (last_commited), then the event can be executed in parallel by the worker;

WL # 7165 parallelism on parallel optimization replication

 

Described with reference to WL # 6314, although concurrent copy has been achieved, but not the desired degree;

Example: The following diagram representative of various transaction execution order of time lines, wherein P represents a single transaction prepare phase, acquired at this stage will commit_clock timestamp, C for the write transaction binlog stage here will be commit_clock step operation;

Shown, Trx1, Trx2, Trx3 acquired the P phase are the same as FIG. Last_commited value (say 1), so that a transaction can be executed in parallel at the three Slave side; Similarly, TRX4 and not <Trx1, Trx2, Trx3> with parallel playback, since the P phase Trx4, acquired last_commited value is the value (after stepping into a 2) after the step of Trx1 been performed; 

WL # 6314 according to the logic, the Slave side that can be found seven Transaction divided into four branch, namely <Trx1, Trx2, Trx3>, <Trx4>, <Trx5, Trx6>, <Trx7>;

It should be noted that, for different branch, <Trx4> and <Trx5, Trx6> can be executed concurrently, as seen from the timeline, <Trx4> and <Trx5, Trx6> the prepare phase timeline there is overlap, which means that these two sets of affairs there is no lock conflict, it can be executed in parallel Slave;

For optimizing parallelism

Parallel copying using the improved lock to determine whether concurrent;

The basic logic is as follows:

L represents the lock phase begins, C represents the end stage lock;

A in Trx1 and Trx2 due to coincidence latch stage, there is no conflict, described Trx1 and Trx2 can be executed in parallel, but B can not work because Trx1 and Trx2 lock phase is not overlapped, can not be confirmed is not be performed in parallel ( additional not determined, not directly as parallel processing, saving performance overhead);

Judgment about lock stage, WL clear that no lock analysis, but some stages of the transaction submitted directly to the point of time as the lock and release the lock (phase commit transactions from the point of view, no problem);

• Before submitting the assumption that during the storage engine layer, all the locks have been released (end of phase lock point in time);
• Assuming that the beginning of the prepare phase, all need to have all the locks acquired (phase lock start time point);

In the MySQL binlog, L is referred tag is last_commited, C is referred to sequence_number marker;

About last_commited and sequence_number, WL # 7165 has described as follows

• Before entering the flush stage of the transaction, will be stepping transaction.sequence_number value -> is displayed as sequence_number
• Prior to the transaction layer into the engine submitted, will modify the value of global.max_committed_transaction 
  
  • = max(global.max_committed_timestamp, transaction.sequence_number) 
  • = Transaction.sequence_number (default value if binlog_order_commits ON)

Thus, in deciding whether to end the Slave SQL can execute concurrently, with reference to the following principles:

--------------------------------- ----------------------------------- --------------- ------------------------
Slave CAN transactionifthe the Execute A Smallest sequence_number 
    Among All Greater IS Executing Transactions Within last transaction.last_committed.
-------- -------------------------------------------------- ---------- ---------------------------------------

伪代码会更直观一些:
-----------------------------------------------------------------------------------------------------------
Slave logic:
    -before scheduler pushes the transaction for execution : wait until         transaction_sequence[0].sequence_number>transaction.last_committed
-----------------------------------------------------------------------------------------------------------

Therefore, using the rear lock Optimization of the degree of parallelism can indeed allow the WL # 6314 <Trx4> and <Trx5, Trx6> concurrent execution;

 

Failure scenarios reduction

Given transaction is the Slave 1371939845, follows the binlog, lack 1371939845 transaction; 

transaction on the Master sequence is as follows: 

Reference WL # 6314 format, in accordance with the transaction sequence Master drawing FIG transaction sequence, GTID, last_commited, sequence_number last two digits are used as a marker;

Since Slave is out of order submitted, these transactions are not strictly appear in the order GTID increasing in Slave in the binlog

根据WL#7165的描述, 可以得出: 在Slave上, 当Trx41执行完毕之后, Slave认为, Trx46与Trx47已经可以由coordinate进行调度, 与< Trx42, Trx43, Trx44, Trx45 > 并行执行了, 但是Trx45与Trx46, Trx47 存在业务上的先后顺序(且确实存在锁冲突), 所以先执行的Trx46删除了Trx45需要的数据, 导致同步中断;

PS: 既然Trx45和Trx46有锁冲突, 为什么Trx46会拿到84作为last_commited, 而不是88?

参考WL#7165的伪代码,
-----------------------------------------------------------------------------------------------------------
When@@global.binlog_order_commitsistrue,inprinciple we could reduce the max
    to an assignment:
     global.max_committed_transaction=transaction.sequence_number
-----------------------------------------------------------------------------------------------------------
MySQL-5.7.21的源代码:
MYSQL_BIN_LOG::ordered_commit-->
process_commit_stage_queue-->
update_max_committed
-----------------------------------------------------------------------------------------------------------

因此推测主库当时候是如下场景:<Trx35~Trx45>作为一个事务组,进入到了存储引擎的commit阶段前,会递增sequence_number,而不是一次到位的全部加上;

所以Trx46进入prepare阶段时,刚好是Trx41完成了commit阶段,所以拿到的是84,而不是88;虽然官方描述中,认为会达到最终一致的状态,但是同步过程中会存在短暂的不一致现象,这种现象被描述为"GAP";