Mysql binlog log analysis

1. Introduction to binlog logs

  Binlog is a binary log in the MySQL database, which is used to record all modification operations in the database, including operations such as adding, deleting, and modifying. The binlog is saved in binary format, and the operation history of the database can be viewed by parsing the binlog file. Binlog logs can be used in scenarios such as data recovery, data backup, and data synchronization. In the MySQL database, binlog has two modes: statement mode and row mode. The statement mode records the SQL statement, and the row mode records the change of each row of data. The opening and closing of the binlog log can be realized by setting the MySQL configuration file.
  Binlog is one of the very important components in the MySQL database. The full name of Binlog is Binary Log. It is a binary log file that records all modification operations in the MySQL database, including addition, deletion, and modification. These modification operations can be restored and backed up through Binlog to ensure data security and integrity.
  The role of Binlog is very important. It can be used for data recovery and backup, as well as data synchronization and replication. When performing data recovery, Binlog can be used to restore data to a certain point in time or to the state before a certain operation, thereby ensuring data integrity. When performing data backup, the Binlog file can be backed up to another server, so that when the main server has a problem, the backup server can be quickly restored to the same state as the main server.
  In addition to data recovery and backup, Binlog can also be used for data synchronization and replication. During data synchronization, the Binlog file can be transferred to other servers, thereby synchronizing the data to other servers. During data replication, the Binlog file can be transferred to the backup server, thereby keeping the data on the backup server consistent with the data on the master server.

2. Common knowledge points of binlog logs

1. Open the binlog log

  The way for mysql to enable binlog is to configure the parameter log-bin = /binlogdir/binlogname in the configuration file, where binlogdir is the storage path of binlog logs, and binlogname is the prefix of the binlog log file name. If this parameter is configured, the binlog log is enabled, and no parameters are configured. Indicates that the binlog log is disabled. The name of the generated binlog log file is usually mybinlog.000001, and the subsequent serial number increases with use. This parameter is a static parameter and needs to be restarted to take effect after modification.

2. Binlog log file size division

  Use the max_binlog_size parameter value to set the size of the binlog log file. For example, max_binlog_size = 1G, which means that the binlog log file is divided when it is larger than 1G, that is, a new binlog log file is generated after the file size is larger than 1073741824, and the serial number is increased by 1. Of course, the binlog log file division is the log record after the end of a transaction operation, so the actual size of the binlog log file is greater than or equal to 1073741824.

[bdsc@s250 logs]$ ll
total 75945060
-rw-r-----. 1 bdsc bdsc 1074332741 Jun 4 18:53 mybinlog.000001
-rw-r-----. 1 bdsc bdsc 1073822182 Jun 4 20:11 mybinlog.000002
-rw-r-----. 1 bdsc bdsc 1074277669 Jun 4 21:30 mybinlog.000003
…

3. Set the binlog log retention time

  Use the expire_logs_days parameter to set the binlog log storage time, and the binlog logs beyond the date will be automatically cleared.

mysql> show variables like ‘expire_logs_days’;
±-----------------±------+
| Variable_name | Value |
±-----------------±------+
| expire_logs_days | 7 |
±-----------------±------+
1 row in set (0.01 sec)

4. Set binlog format

  Use the binlog_format parameter setting to set the binlog log format. The row format is the most reliable, but it will generate a large number of binlog log files.

mysql> show variables like ‘binlog_format’;
±--------------±------+
| Variable_name | Value |
±--------------±------+
| binlog_format | ROW |
±--------------±------+
1 row in set (0.00 sec)

5. View binlog log content

  The binlog log file is in binary format, and it is garbled when viewed directly. We can use the mysqlbinlog command to view the log file. The log file will record the starting position of the log and the executed sql statement. When viewing, you can also use parameters to view the logs of a certain database within a specified time period. You can also use the command "mysqlbinlog --start-postion=107 --stop-position=1000 -d library name binary file" to view a database log between specified positions.

mysqlbinlog --start-datetime=‘2023-06-01 00:00:00’ --stop-datetime=‘2023-06-10 00:00:00’ -d utest mybinlog.000068

6. Clean up the binlog log according to the binlog name

  We can use the "PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 7 DAY);" command in the sql script to clean up the historical binlog logs, or we can clean up the binlog logs by name in interactive mode.

mysql> purge binary logs to ‘mybinlog.000005’;
Query OK, 0 rows affected (0.19 sec)

7. Clean up binlog logs according to time

  We can use the command purge binary logs before time to delete binlog logs before the specified time.

mysql> purge binary logs before ‘2023-06-5 10:12:00’;
Query OK, 0 rows affected (0.55 sec)

8. Export binlog logs

  Exporting the binlog log file is actually inputting the log content to the specified file through redirection when viewing.

[bdsc@s250 logs]$ mysqlbinlog --start-datetime=‘2023-06-01 00:00:00’ --stop-datetime=‘2023-06-10 00:00:00’ mybinlog.000068 > /tmp/utest.log

9. Record some binlog logs

  You can use the four parameters binlog-do-db, binlog-ignore-db, binlog-do-table, and binlog-ignore-table to customize the databases and tables that need to record binlog logs. These four parameters are often used in master-slave mode for mysql to build some tables .

3. Introduction to the three formats of binlog logs

  MySQL's binlog logs have three formats, namely Statement format, Row format, and Mixed format.

• Statement format
  Statement format is the simplest binlog format, which records the executed SQL statements, and can restore data by parsing the SQL statements. The advantage of this format is that it is simple, easy to understand and analyze, and the disadvantage is that data inconsistency may occur, because the same SQL statement may produce different results in different environments.
• Row format
  Row format is to record the changes of each row of data, including insert, delete and update operations. The advantage of this format is that data recovery is accurate and there will be no data inconsistency. The disadvantage is that the log volume is large, and a large number of logs will be generated for a large number of data changes.
• Mixed format
  Mixed format is a mixture of Statement format and Row format, and MySQL will automatically choose which format to use to record logs. For simple SQL statements, use the Statement format, and for complex SQL statements, use the Row format. The advantage of this format is that it takes into account the advantages of both simple and complex situations. The disadvantage is that data inconsistency may occur.