Little knowledge points: MySQL's redo log, undo log, binlog and Java monitoring binlog

SQL storage process

• The server establishes a connection with MySQL
• Go through the analyzer, optimizer, and executor of the Server layer in the MySQL server memory in turn
• The executor operates the InnoDB engine according to the execution plan
• InnoDB reads data from the disk data file into the buffer pool
• Before modification, the undo log will be written to save the data
• Then change the data in the buffer pool to new_data
• Write a redo log to save the modified new_data
• Write a binlog to save the modified value new_data
• The background IO thread flushes the modified value in the buffer pool to disk

1. Buffer Pool

• An important structure in InnoDB, a memory space for caching MySQL disk data
• Without Buffer Pool, disk IO operations are required for every data modification. With Buffer Pool, disk IO can be converted into memory operations, saving time and improving efficiency.
• If the power is cut off, all data will be lost, which requires the use of redo log, undo log, binlog

Two, redo log

Redo log means redo log

• Guaranteed no data loss
  • After the modification, first record the modified data to the redo log on the disk. Even if the data in the Buffer Pool is lost after a power failure, it can be restored from the redo log.
• WAL (Write-ahead logging) pre-written log
  • The mechanism of pre-writing the log first and then flushing the data
  • The redo log is sequentially written on the disk, and the data is written randomly, and the sequential write is more efficient than the random write
• redo log buffer
  • The efficiency of disk sequential writing is already very efficient, but there is still a certain gap with memory operations. Adding a memory buffer can optimize
  • MySQL runs on the operating system. If MySQL hangs up, the Buffer Pool will be lost. At this time, the cache of the OS is not lost and can be recovered. If the operating system is also hung up, the cache of the OS will also be lost.
• Order Mechanism
  • innodb_flush_log_at_trx_commit = 1: real-time write, real-time flush
    • Before each transaction is committed, the data is flushed from the redo log to disk each time
    • Lowest efficiency, low risk of data loss
  • innodb_flush_log_at_trx_commit = 0: delayed write, delayed flush
    • Every time a transaction is committed, only write data to the redo log buffer, and then let the background thread regularly flush the data
    • Highest efficiency, highest risk of data loss
  • innodb_flush_log_at_trx_commit = 2: real-time write, delayed flush
    • Before each transaction is committed, the redo log is written to the OS cache
    • The efficiency is relatively high, and the risk of data loss is relatively low. As long as the operating system is not suspended, data will not be lost. Recommended

Three, undo log

• InnoDB supports transactions, transactions can be rolled back
• The data process before the record modification is called the record undo log
• undo undo, rollback, the main function of undo log is to roll back data
• The undo log exists in the global tablespace by default. It can be simply understood that the undo log is also recorded in a MySQL table. Inserting an undo log is similar to inserting ordinary data, and you also need to write a redo log

Four, binlog

• The redo log records the modified data and provides the ability to recover from crashes
• undo log records the data before modification and provides rollback capability
• Binlog records the modified data for archiving
• binlog sync_binlog flushing strategy
  • sync_binlog = 0: Write the binlog to the OS cache before each transaction is committed, and the disk is swiped under the control of the operating system
  • sync_binlog = 1: Write binlog synchronously to disk without using OS cache
  • sync_binlog = N: After every n transaction commits, call fsync once to force the binlog in the OS cache to disk
• The difference between redo log and binlog
  • Binlog is a logical log, which records what changes have been made to which row of which table
  • Redo log is a physical log, which records what modification has been made to which record in which data page
  • Binlog is append writing, redo log is circular writing, and the log file has a fixed size, which will overwrite the previous data
  • binlog is the log of the Server layer, and redo log is the log of InnoDB

V. Summary

• Buffer Pool is a memory space managed by the MySQL process, which can reduce the number of disk IO
• redo log is a log of the InnoDB storage engine, mainly for crash recovery, innodb_flush_log_at_trx_commit controls three flushing strategies, recommended 2
• undo log is a kind of log of InnoDB storage engine, the main function is to roll back
• binlog is a kind of log in the MySQL Server layer, the main function is to archive
• There are two situations where MySQL hangs
  • The operating system is down, and the MySQL process is down
  • The operating system is not suspended, but the MySQL process is suspended

6. Java monitors MySQL binlog

6.1 Environment preparation

• Check whether MySQL has enabled binlog
  • show variables like ‘log_bin’;
  • It is not enabled by default, and ERROR 1381 - You are not using binary loggin will be reported

• open binlog
  • Modify my.cnf (some are my.ini)

• restart mysql

6.2 Java code

• Introduce dependencies

<dependency>
    <groupId>com.github.shyiko</groupId>
    <artifactId>mysql-binlog-connector-java</artifactId>
    <version>0.21.0</version>
</dependency>

• MySQL message class

public class MySQLRecord {
    
    

    private String database;
    private String table;
    private final List<UpdateRecord> records = new LinkedList<>();

    public MySQLRecord() {
    
    
    }

    public MySQLRecord(String database, String table) {
    
    
        this.database = database;
        this.table = table;
    }

    @Override
    public String toString() {
    
    
        return "{\"database\"=\"" + database + "\"," +
                "\"table\"=" + table + "\"," +
                "\"records\"=" + records + "}";
    }

    public void setRecord(UpdateRecord record) {
    
    
        records.add(record);
    }

    public void setRecords(List<UpdateRecord> records) {
    
    
        this.records.addAll(records);
    }

    public void setDatabase(String database) {
    
    
        this.database = database;
    }

    public void setTable(String table) {
    
    
        this.table = table;
    }

    public static class UpdateRecord {
    
    
        public String before;
        public String after;

        public UpdateRecord(String before, String after) {
    
    
            this.before = before;
            this.after = after;
        }

        @Override
        public String toString() {
    
    
            return "{\"before\"=\"" + before + "\"," +
                    "\"after\"=" + after + "\"}";
        }
    }

}

• Monitor Execution Class

public class mysql {
    
    
    private static final Map<Long, HashMap<String, String>> dbMap = new HashMap<>();

    public static void main(String[] args) {
    
    

        BinaryLogClient client = new BinaryLogClient("127.0.0.1", 3306, "root", "123456");
        client.setServerId(1);

        client.registerEventListener(event -> {
    
    
            EventData data = event.getData();
            if (data instanceof TableMapEventData) {
    
    
                TableMapEventData tableMapEventData = (TableMapEventData) data;
                if (!dbMap.containsKey(tableMapEventData.getTableId())) {
    
    
                    dbMap.put(tableMapEventData.getTableId(), new HashMap<String, String>(){
    
    {
    
    
                        put("database",tableMapEventData.getDatabase());
                        put("table", tableMapEventData.getTable());
                    }});
                }
            }
            if (data instanceof UpdateRowsEventData) {
    
    
                format(((UpdateRowsEventData) data).getTableId(), 0, ((UpdateRowsEventData) data).getRows());
            } else if (data instanceof WriteRowsEventData) {
    
    
                format(((WriteRowsEventData) data).getTableId(), 1, ((WriteRowsEventData) data).getRows());
            } else if (data instanceof DeleteRowsEventData) {
    
    
                format(((DeleteRowsEventData) data).getTableId(), -1, ((DeleteRowsEventData) data).getRows());
            }
        });

        try {
    
    
            client.connect();
        } catch (IOException e) {
    
    
            e.printStackTrace();
        }
    }

    // 0 update, 1 insert, -1 delete
    public static void format(Long tableId, int type,  List<?> data) {
    
    
        MySQLRecord records = new MySQLRecord(dbMap.get(tableId).get("database"), dbMap.get(tableId).get("table"));
        for (Object row : data) {
    
    
            records.setRecord(new MySQLRecord.UpdateRecord(
                    type == 0 ? Arrays.toString((Object[]) ((Map.Entry<?, ?>) row).getKey()) :
                            type == 1 ? null : Arrays.toString((Object[]) row),
                    type == 0 ? Arrays.toString((Object[]) ((Map.Entry<?, ?>) row).getValue()) :
                            type == 1 ? Arrays.toString((Object[]) row) : null));
        }
        System.out.println(records);
    }

}