Teach you how to solve the problem of MySQL data delay

Today, I analyzed another problem about database delay and beating, which is also typical. There are also some methods and techniques for analyzing problems in this process.

First of all, in the high-availability detection, there is a set of environmental detection intermittently. After the investigation, it is found that the database has a delay. After logging in to the slave library show slave status, you will find that the value of Seconds_behind_master is constantly beating, that is, from 0~ The frequency of 39~0~39 keeps beating, which makes people very hot.

Looking at the relevant logs of the database, I found that there is no log record that can be referred to. How to analyze this problem? Let's reproduce it first, so I captured the logs of the problem 3 times according to the rhythm, that is, continuous monitoring through show slave status. Take the output result of show slave status and save it, so that we get the offset change during the occurrence of a problem, and this change is a problem generated by SQLThread during playback.

For example, in the following output, I intercepted the relay log on the Slave side for analysis, and the corresponding field is Relay_Log_Pos

Slave_IO_State: Waiting for master to send event
         Master_Host: xxxx
         Master_User: dba_repl
         Master_Port: 4306
        Connect_Retry: 60
       Master_Log_File: mysqlbin.000044
     Read_Master_Log_Pos: 386125369
        Relay_Log_File: slave-relay-bin.000066
        Relay_Log_Pos: 386125580
    Relay_Master_Log_File: mysqlbin.000044

So quickly got the offset change: 385983806, 386062813, 386125580

Then I used mysqlbinlog to analyze the details of these logs. According to the following command, I can quickly get 3 related tables in the dumped logs.

# grep INSERT relaylog_xxxx.dump |awk '{print $3 " " $4}'|sed 's/INTO//g'|sort|uniq
 act_action_exec_info
 act_join_desc
 dic_subsidy_marketing_querylog_202008

I gradually analyzed the data operation status of each table, and the information obtained is still relatively limited. I will continue to do further analysis. For example, let's analyze the transaction volume in the entire log:

# mysqlbinlog slave-relay-bin.000066 | grep "GTID$(printf '\t')last_committed" -B 1 \
>                   | grep -E '^# at' | awk '{print $3}' \
>                   | awk 'NR==1 {tmp=$1} NR>1 {print ($1-tmp);tmp=$1}' \
>                   | sort -n -r | head -n 100
mysqlbinlog: [Warning] unknown variable 'loose-default-character-set=utf8'
5278
5268
5268
5268
5253
5253
5253
5253
5253

It can be seen that it is about 5K, which is relatively large. Where can I get these additional information? I turned on general_log in the main library, so that I can get more fine-grained operation logs.

Further analysis found that the entire business uses the display transaction method: SET autocommit=0, the entire transaction contains several large SQL, which stores a lot of operation log details, and it is called multiple times based on the Mybatis framework during the transaction operation. select count(1) from xxx operation.

After communicating with the business, the above problems were basically clarified.

The above is the detailed content of solving the problem of MySQL data delay beating.

This article address: https://www.linuxprobe.com/mysql-linux-two.html

Teach you how to solve the problem of MySQL data delay

Guess you like