MySQL Operation and Maintenance Case Study: Timestamp in Binlog

Abstract: This article starts with a typical case to describe the principle and practice of timestamp in Binlog. Through this article, you can understand the role and generation method of timestamp in Binlog, so that when some strange problems in this area occur, you can There are numbers, and they are well-informed. This article is selected from the background of "MySQL Operation and Maintenance Internal Reference" As we 　　all know

, in Binlog files, you often see time attributes about events, and the way they appear is as follows. #161213 10:11:35 server id 11766 end_log_pos 263690453 CRC32 0xbee3aaf5 Xid = 83631678 　　We clearly know that 161213 10:11:35 represents the time value, but what else? What else can we know about it? Case Analysis Let's 　　start with a typical case to describe the details, such as a problem encountered in Galera Cluster, you can first read a piece of Binlog content, as follows. 10:11:35 Server ID 11766 161 213 # 263 677 208 end_log_pos the CRC32 0xbfc41688 GTID [the commit = Yes] # 161 213 263 677 291 end_log_pos 10:10:44 Server ID 11766 Query thread_id the CRC32 0x02537685 = 4901481 exec_time ERROR_CODE = 0 = 0 # 161 213 10:10: 44 server id 11766 end_log_pos 263677435 CRC32 0x0e70aab6 Write_rows: table id 17852 flags: STMT_END_F











#161213 10:10:44 server id 11766  end_log_pos 263677609 CRC32 0xb58d4c61    Update_rows: table id 17853 flags: STMT_END_F
#161213 10:11:35 server id 11766  end_log_pos 263690453 CRC32 0xbee3aaf5    Xid = 83631678#161213 10:11:30 server id 11766  end_log_pos 263690501 CRC32 0x6e798470    GTID [commit=yes]
#161213 10:11:30 server id 11766  end_log_pos 263690592 CRC32 0x2b6a6d34    Query   thread_id=4900813   exec_time=5 error_code=0#161213 10:11:30 server id 11766  end_log_pos 263691291 CRC32 0xc0f9ed87    Update_rows: table id 17891 flags: STMT_END_F
# 161213 10:11:30 server id 11766 end_log_pos 263691322 CRC32 0xe40764c4 Xid = 83631679 # 161213 10:11:35 server id 11766 end_log_pos 263691370 CRC32 0xbaa4ca30 GTID [commit = yes] # 161213 10:11:35 server id 11766 end_log_pos 263691453 CRC32 = Query thread_id 4900813 exec_time 0x030f321c = 0 = 0
ERROR_CODE # 161 213 10:11:35 Server ID 11766 end_log_pos 263.69224 million 0x7584d6a1 Delete_rows the CRC32: Table 73 is the flags ID: STMT_END_F # 11766 end_log_pos 161 213 263 692 271 10:11:35 Server ID = 83.63168 million Xids the CRC32 0x03abb120
　　Listed above is the processed Binlog content, including three transactions, each transaction only lists the iconic events, such as the GTID event at the beginning of the transaction, the execution thread information and the commit event. The order of appearance is the order of Binlog content, which can be seen from the continuity of Xid.
　　In the paragraph above, focus on time information. Each event in each transaction has a time attribute. It can be seen that the first transaction was submitted at 10:11:35, and the third transaction was also submitted at this time. But the middle transaction, that is, the transaction with Xid 83631679, all event times, including the commit time, occurred at 10:11:30, 5 seconds earlier than the other two transactions!
　　Many students may have the following questions after seeing this phenomenon.　

In the Binlog file, isn't it stored in the order of submission? How can the previously submitted transactions be stored in the latter position?
Assuming the start time of transaction 83631679 is 10:11:30, then at least its commit event, the GTID event time is 10:11:35 right?
The execution time of transaction 83631679 is 5 seconds. From exec_time=5, it can be seen that this information appears, so the second question becomes more confusing.
Here you can see information such as exec_time=5, which is commendable, this is a very important information. Because it is now clearly known that transaction 83631679 started at 10:11:30. Then, this Query is executed for another 5 seconds, which can match the transaction commit time of 10:11:35. One thing to be clear now is that the transaction was submitted at 10:11:35, but what you see in the Binlog content is 10:11:30, then you need to figure out how Binlog handles the issue of recording timestamps of.
　　Timestamp is an attribute of an event, but where is the source of this attribute, that is, when was the time recorded, you can see the following code.

Log_event::Log_event(
    THD* thd_arg, uint16 flags_arg,
    enum_event_cache_type cache_type_arg,
    enum_event_logging_type logging_type_arg,
    Log_event_header *header, Log_event_footer *footer)
  : is_valid_param(false), temp_buf(0), exec_time(0),
    event_cache_type(cache_type_arg), event_logging_type(logging_type_arg),
    crc(0), common_header(header), common_footer(footer), thd(thd_arg)
{
    server_id= thd- >server_id;
    common_header->unmasked_server_id= server_id;   
    /* This is the process used to assign the time of an event when creating an event.
       It can be seen that the source of this time is the value of thd->start_time, then what we need to
       do is to figure out the source of this value*/
    common_header->when= thd->start_time;
    common_header->log_pos= 0;
    common_header ->flags= flags_arg;
}
　　As explained in the code, you need to find the source of thd->start_time. The setting of this value is easy to find, it will be done once before each statement is executed, and it is set by the function thd->set_time(). One of the very important MySQL statements is called in the entry processing function, you can take a brief look, as follows.

bool dispatch_command(THD *thd, const COM_DATA *com_data,
                     enum enum_server_command command)
{ /* local variables... */

    thd->set_command(command); /*
      Commands which always take a long time are logged into
      the slow log only if opt_log_slow_admin_statements is set.
    */
    thd->enable_slow_log= TRUE;
    thd->lex->sql_command= SQLCOM_END; /* to avoid confusing VIEW detectors */

    /* here, initialize this time to current time */
    thd-> set_time(); /* other code ... */}
　　Some students must have been clear, in fact, the timestamps in Binlog events are inherited from the statement, and a statement generates multiple events, then the timestamps of these events are all It is the same, and it is consistent with the first event (this can be verified by yourself).
　　How to explain the above question about exec_time=5? Let's look at the code again, as follows.

Log_event(
    thd_arg,
    (thd_arg->thread_specific_used ? LOG_EVENT_THREAD_SPECIFIC_F :
     0) |
    (suppress_use ? LOG_EVENT_SUPPRESS_USE_F : 0),
    using_trans ? Log_event::EVENT_TRANSACTIONAL_CACHE :
                  Log_event::EVENT_STMT_CACHE,    Log_event::EVENT_NORMAL_LOGGING,
    header(), footer()),
    data_buf(0)
{
    /* local vaiables */

    /* exec_time calculation has changed to use the same method that is used
    to fill out "thd_arg->start_time" */

    struct timeval end_time;
    ulonglong micro_end_time= my_micro_time();
    my_micro_time_to_timeval(micro_end_time, &end_time);

    /* The calculation of the time is to use the current time (the time when the execution is completed), minus
    the value of thd_arg->start_time, which has been seen above, and is the time when the statement starts to execute, that is to say, exec_time
    refers to the statement The time from start to finish, that is, the actual statement execution time*/

    exec_time= end_time.tv_sec - thd_arg->start_time.tv_sec;
    /* other code ... */
}
　　Then you can now explain why transaction 83631679 is executed 5 seconds, but all events are at 10:11:30. Just because this transaction is auto-committed, there is only one statement and it executes for 5 seconds, it's that simple, that's all.
　　Because it is automatically committed, this transaction has only one statement, and thd->set_time() will only be set once, so all events in this transaction stay at this point in time, so the above phenomenon occurs.

Divergent thinking Some students
　　may have doubts. Even if a transaction has only one statement, it is submitted. The submission time is indeed done after 5 seconds. Is there no internal handling of this problem? You can do an experiment to see that it is still one transaction and one statement, but this time it is not automatically submitted, but after a transaction starts, wait for 5 seconds, then manually submit it, and then look at the Binlog content, as follows:

mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)

mysql> insert into my1 values(13143306,'zhufeng');select sleep(5);commit;
Query OK, 1 row affected, 0 warning (0.00 sec)
+----------+
| sleep( 5) |
+----------+
| 0 |
+----------+

1 row in set (5.00 sec)

Query OK, 0 rows affected (0.01 sec)
　　for no Time error, the above three statements are executed at the same time, and there is a 5-second waiting operation in the middle. Now look at the corresponding Binlog content.

#161216 12:53:25 server id 23932 end_log_pos 449 CRC32 0x2939a45b GTID [commit=yes]
#161216 12:53:20 server id 23932 end_log_pos 522 CRC32 0x1df01360 Query thread_id=21 exec_time= error_code=0
#161216 12:53:20 server id 23932 end_log_pos 522 CRC32 0x1df01361 server id 23932 end_log_pos 614 CRC32 0xf1515ed0 Write_rows: table id 78 flags: STMT_END_F
#161216 12:53:25 server id 23932 end_log_pos 645 CRC32 0x69c2d142 Xid = 29321
　　At this point, it is obvious that the time of the two commit events of the transaction's Xid and GTID is 5 seconds later than the insertion time. This is precisely because the executed Commit statement, like the INSERT statement, is a statement, and thd->set_time() will be executed before execution, thus affecting the Binlog event generated by itself.

The sequence of events in a transaction As we have seen
　　above , in a transaction, there will be events that start transactions, events that commit transactions, and events that actually do things, such as Write_rows, etc. The order between them will be related to the timestamp. little relationship. Careful students may have found that in the example in the previous section, GTID is at the top, its time is 12:53:25, and Write_rows is in the middle, but its time is 12:53:20, and there are Does it matter?
　　In fact, this has been mentioned when I introduced the principle of MySQL 5.7 multi-thread replication. When a MySQL transaction is submitted, the operations are as follows.

Generates a GTID event based on the context after execution.
Assemble the GTID generated by the transaction.
Commit to various storage engines.
As you can see from what has been done above, the GTID information is actually generated at commit time. This will be more explicit in MySQL 5.7, because the already familiar last_committed and sequence_number are also required to construct GTIDs (see Chapter 15 for details). This also explains why the time corresponding to the GTID event is the same as Xid, because its time is inherited from the Commit statement. Of course, the same is true for Xid, because they are two different events generated by the same statement.
　　But about the order, it is related to the role of the GTID event. In MySQL Binlog, it is necessary to know the specific information of GTID in advance, so when MySQL submits and assembles the corresponding Binlog, it is put in the front, which leads to the phenomenon of time problem that we have seen so far. 　　Expansion of the problem

Let's go back and look at the case of waiting for 5 seconds at the beginning. 10:11:30 Server ID 11766 161 213 # 263 690 501 end_log_pos the CRC32 0x6e798470 GTID [the commit = Yes] # 161 213 263 690 592 10:11:30 Server ID 11766 end_log_pos the CRC32 0x2b6a6d34 Query thread_id = 4,900,813 exec_time ERROR_CODE = 0. 5 = # 161 213 10:11: 30 server id 11766 end_log_pos 263691291 CRC32 0xc0f9ed87 Update_rows: table id 17891 flags: STMT_END_F #161213 10:11:30 server id 11766 end_log_pos 263691322 CRC32 0xe40764c4 Xid = 　　5 seconds Why does an update statement execute for 836 seconds? Of course, there may be the following two reasons. The query condition corresponding to this statement is a slow query, the scan will take a long time (5 seconds), and only one row is updated (because there is only one Update_rows event), which leads to the above phenomenon. The execution itself is not slow, but when waiting for other transactions or locks to execute operations, it took 5 seconds.










Both of these reasons are possible and can be explained. The method to determine which cause is causing the problem is very simple, that is, check the slow query log file, find the slow query with thread_id of 4900813, or the slow query of the corresponding table, and must be on the instance with server_id of 11766 (this point is every time Everyone knows it, but it is sometimes ignored). If it is found, it is the first reason; if it cannot be found, it is the second reason.
　　I searched and searched, and the result was that there was no slow query during that time period.
　　No matter what the reason, it must be a slow query after 5 seconds of execution. How can it be found? Here's a little more about MySQL's slow query records, the lock waiting time is not counted here. So, if it is the second reason, then the slow query must not be found, and exec_time=5 is also very convincing for this, because the calculation of the execution time is the difference from the start time to the end time, and The calculation method of slow query is different, so this also shows that these 5 seconds are all waiting.
　　So the question is, who is it waiting for? Can you wait 5 seconds? There must be another transaction that has obtained the lock it wants, and the time to obtain the lock must be more than 5 seconds, so it needs to continue to look for it. At this point, it is known that there are two types of large transactions as follows.

A single automatically submitted statement has a long execution time (exec_time is relatively large).
The span of transaction start time and end time is long.
Looking for Binlog in ROW format, we have a good tool! Consider the following script, which will also appear in Chapter 42.

# vim summarize_binlogs.sh
#!/bin/bash
BINLOG_FILE="mysql-bin.000046"
START_TIME="2015-06-28 8:45:00"
STOP_TIME="2015-06-28 10:10:00"
mysqlbinlog --base64-output=decode-rows -vv --start-datetime="${START_TIME}"  --stop-datetime="${STOP_TIME}" ${BINLOG_FILE} | awk \
'BEGIN {xid="null";s_type=""; stm="";endtm="";intsta=0;inttal=0;s_count=0;count=0;insert_count=0;update_count=0;delete_count=0;flag=0;bf=0;period=0;} \
{
if (match($0, /^(BEGIN)/)) {bg=1;} \
if (match($0, /#.*server id/)) {if(bg==1){statm=substr($1,2,6)" "$2;cmd=sprintf("date -d \"%s\" +%%s", statm);cmd|getline intsta;close(cmd);bg=0;bf=1;}
else if(bf==1){endtm=substr($1,2,6)" "$2;cmd=sprintf("date -d \"%s\" +%%s", endtm);cmd|getline inttal;close(cmd);}} \if(match($0, /#.*Table_map:.*mapped to number/)) {printf "Timestamp :" $1 " " $2 " Table : " $(NF-4); flag=1} \else if (match($0, /#.*Xid =.*/)) {xid=$(NF)} \
else if (match($0, /(### INSERT INTO .*..*)/)) {count=count+1;insert_count=insert_count+1;s_type="INSERT"; s_count=s_count+1;}  \
else if (match($0, /(### UPDATE .*..*)/)) {count=count+1;update_count=update_count+1;s_type="UPDATE"; s_count=s_count+1;} \
else if (match($0, /(### DELETE FROM .*..*)/)) {count=count+1;delete_count=delete_count+1;s_type="DELETE"; s_count=s_count+1;}  \
else if (match($0, /^(# at) /) && flag==1 && s_count>0) {print " Query Type : "s_type " " s_count " row(s) affected" ;s_type=""; s_count=0; }  \
else if (match($0, /^(COMMIT)/)) {period=inttal-intsta;if(inttal==0){period=0};print "[Transaction total : " count " Insert(s) : " insert_count " Update(s) :" update_count " Delete(s) : " \
delete_count " Xid : "xid" period : "period" ] \n+----------------------+------------ ----------+----------------------+---------------- ------+"; \
count=0;insert_count=0;update_count=0;delete_count=0;s_type="";s_count=0;flag=0;bf=0;bg=0;} } '
　　This script can help us easily find transactions with long spans, which are represented by the period column in the result. After analyzing the data in the period column, the results are as follows.

zhufeng@zhufengmac:~$ cat sum|grep Transaction|awk '{if($19>0)print}'[Transaction total : 8 Insert(s) : 8 Update(s) : 0 Delete(s) : 0 Xid : 83631021 period : 1 ]
[Transaction total : 15 Insert(s) : 10 Update(s) : 4 Delete(s) : 1 Xid : 83631137 period : 1 ]
[Transaction total : 10 Insert(s) : 7 Update(s) : 3 Delete(s) : 0 Xid : 83631517 period : 1 ]
[Transaction total : 3 Insert(s) : 3 Update(s) : 0 Delete(s) :
[Transaction total : 16 Insert(s) : 10 Update(s) : 6 Delete(s) : 0 Xid : 83631678 period : 51 ]
[Transaction total : 12 Insert(s) : 9 Update(s) : 3 Delete(s ) : 0 Xid : 83631829 period : 1 ]
[Transaction total : 5 Insert(s) : 4 Update(s) : 1 Delete(s) : 0 Xid : 83632139 period : 1 ]
[Transaction total : 3 Insert(s) : 3 Update(s) : 0 Delete(s) : 0 Xid : 83632172 period : 1 ]
[Transaction total : 3 Insert(s) : 3 Update(s) : 0 Delete(s) : 0 Xid : 83632206 period : 1 ]
　　It was soon discovered that the transaction with Xid 83631678 was as long as 51 seconds. Then open the corresponding Binlog file to find the transaction, as follows.

#161213 10:11:35 server id 11766 end_log_pos 263677208 CRC32 0xbfc41688 GTID [commit=yes]
#161213 10:10:44 server id 11766  end_log_pos 263677291 CRC32 0x02537685    Query   thread_id=4901481   exec_time=0 error_code=0
#161213 10:10:44 server id 11766  end_log_pos 263677435 CRC32 0x0e70aab6    Write_rows: table id 17852 flags: STMT_END_F
#161213 10:10:44 server id 11766  end_log_pos 263677609 CRC32 0xb58d4c61    Update_rows: table id 17853 flags: STMT_END_F
#161213 10:11:35 server id 11766  end_log_pos 263685326 CRC32 0xc512e73c    Write_rows: table id 566 flags: STMT_END_F
#161213 10:11:35 server id 11766  end_log_pos 263685556 CRC32 0x805318e4    Write_rows: table id 566 flags: STMT_END_F
#161213 10:11:35 server id 11766  end_log_pos 263690422 CRC32 0x4de989c8    Write_rows: table id 73 flags: STMT_END_F
#161213 10:11:35 server id 11766 end_log_pos 263690453 CRC32 0xbee3aaf5 xid=83631678
　　Found something? Isn't this transaction 83631678, the first of the three shown at the beginning of this chapter? Yes, it is.
　　First of all, about the timing of GTID events and Xid events, as explained above, this is the time to submit the statement, so it is all 10:11:35, ignore it for now. And the content of the real work in the middle needs to be focused on. The first two events are Write_rows and Update_rows, their time is 10:10:44, and the time of the last three Write_rows events is 10:11:35, and the middle is as long as 51 seconds. What did you do during this time?
　　Check again that the records of the table to be modified by Update_rows in transaction 83631679 are the same records in the same table as the records to be modified by the event Update_rows that occurred at 10:10:44 in transaction 83631678. That must be waiting.
　　The database problem has been explained clearly. The only problem now is to find a business developer and ask, which table is that transaction on, and what did the DBA do in that 51-second period? The
　　DBA must continue to deal with other problems. The baton is now handed over to the development classmates, and I will hear from you tomorrow.

The problem below Time in show processlist
　　may be a problem that is easy to cause confusion in the actual operation and maintenance process. Let’s first take a look at the well-known show processlist result. Here, please focus on the Time column information in the result, as follows .

mysql> show processlist\G
*************************** 1. row ***************************
           Id: 1
         User: zhufeng.wang
         Host: localhost:51703
           db: NULL
      Command: Sleep
         Time: 5142
        State:
         Info: NULL
    Rows_sent: 0
Rows_examined: 0
*************************** 2. row ***************************
           Id: 2
         User: zhufeng.wang
         Host: localhost:62898
           db: local
      Command: Sleep
         Time: 2077
        State:
         Info: NULL
    Rows_sent: 2
Rows_examined: 2
　　In the above two lines of information, the Time values ​​are 5142 and 2077 respectively. How did they come from? For this question, all students should be relatively clear, it represents the time point when the current statement is executed, and the time difference when the show processlist command is executed. This can be proved from the following MySQL code.

/* Used to calculate the value of the Time column in Show Processlist, thd_info->start_time
   represents the start time when thread thd_info executes the last statement*/
if (thd_info->start_time)
    protocol->store_long ((longlong) (now - thd_info ->start_time));
else
    protocol->store_null();
　　Now you can do an experiment, open two sessions, connect to the same MySQL server, one session (session 1) is used for the experiment, and the other session (session 2) is used for To continuously execute the show processlist command to observe the phenomenon, the experimental results are very clear, as long as session 1 remains inactive, in the results of session 2, the time value corresponding to the connection of session 1 keeps increasing, and as long as session 1 executes any command , in the result of session 2, the Time value corresponding to the connection of session 1 will be cleared once, and then continue to increase again, and so on. After understanding the above, this phenomenon should be easy to understand now, because when executing any command (calling the function dispatch_command), thd->set_time() will be executed to clear the time to the current time, that is, the time difference is cleared.
　　But have you ever seen something like this?

mysql> show processlist\G
**************************** 1. row ******************** ******
           Id: 2
         User: zhufeng.wang
         Host: localhost:62898
           db: local
      Command: Query
         Time: 1203070
        State: User sleep
         Info: select sleep(100)
    Rows_sent: 0
Rows_examined: 0
...other Omit...
　　You can see that the Time value of session 1 is as high as 1203070 at this time, and the corresponding statement is just select sleep(100). Wondering why you only slept for 100 seconds when Time can be that high?
　　Of course, the statement here is constructed by itself, and it is also found that no matter how many times such a statement is executed, Time still keeps rising and will not be cleared. What is the reason?
　　Obviously, such a phenomenon can cause some confusion for the DBA and can cause interference when solving the problem, so it is necessary to explain it clearly here. Let's focus on the implementation of the set_time function.

inline void set_time()
  {
    /* Get the current timestamp and store it in the start_utime variable*/
    start_utime= utime_after_lock= my_micro_time();
    if (user_time.tv_sec || user_time.tv_usec)
    {
      /* It is found here that when user_time is not 0, the timestamp obtained above is directly
      ignored, but the value of user_time is used, That is to say, as long as the value of user_time
      remains unchanged, the operation of set_time will not change the latest time value of the current connection.
      We need to study clearly what user_time is! */
      start_time= user_time;
    }
    else
      /* In the normal path, if user_time is 0, update the latest time of the current connection */
      my_micro_time_to_timeval(start_utime, &start_time);
  }
　　The parameter user_time mentioned in the above code actually corresponds to MySQL The session parameter timestamp, as long as this parameter is explicitly set, user_time is not 0, then the start time of the current session is fixed at this moment.
　　Now I understand, as long as you execute a command in session 1, such as set timestamp=1490264145;, and then observe in session 2, you will find that Time is not only very large, but also when any statement is executed in session 1, in session 2 The Time value will not be cleared.
　　Therefore, if you encounter such a problem in actual operation and maintenance, you can find out whether such a statement has been executed in a connection, which creates such an illusion, because when such a problem occurs, such a statement will be misjudged as slow. query, and no such query is actually found.
　　Does this question feel familiar? Yes, such commands are often encountered in Binlog. This is what MySQL does to maintain the consistency of the master-slave execution environment, but if you do this on the main library, it is often not only not fun, but will cause confusion. the feeling of water.

Summary
　　This question seems simple, but it actually involves a lot of details about Binlog. The main purpose of talking about this is to let DBA students understand the role of timestamps in Binlog and how to generate them, so that when there are some weird problems in this area, they can know what to do with them.
　　This article is selected from "MySQL Operation and Maintenance Internal Reference: MySQL, Galera, Inception Core Principles and Best Practices", click this link to view this book on the official website of the blog post.
　　　　　　　　　　　　　　　　　　　　　　Picture description If
　　you want to get more exciting articles in time, you can search for "Blog Viewpoint" in WeChat or scan the QR code below and follow.
　　　　　　　　　　　　　　　　　　　　　　　　　Picture description


Using the Yunqi Community APP, feel comfortable~