redo log

redo log

effect

innoDB storage engine, you need to restart after a server failure, can accurately recover all the data has been submitted to ensure data persistence; such a transaction has been submitted (dirty pages) in memory Buffer Pool in, but the server suddenly fails, data is lost;

To solve this problem, you can modify pages flushed to disk, but because it may only modify a record, no need to waste time refreshed in real time, and modified records are not necessarily sequential, random IO slow to refresh.

Record committed transactions can be modified record, that is the value of a page of an offset amount of space in a table is updated to the number, the recorded file is called redo log. Compared to refresh the memory pages to disk, redo log flushed to disk the contents of a lot smaller, but the process is a sequence written to disk.

redo log records more than the index insert / update records and other operations, as well as to perform this action affects other actions, such as page splitting new directory entry record, modify page modify any information on the data page do more.

And binlog difference: binlog record is the official page has been operating off the disk and include all storage engines, recovery of dirty pages when Ben collapse still operating in the buffer pool the next redo log InnoDB engine for the system.

Journal

Log Format

• type: Type

  • MLOG_1BYTE: 1: 1 byte indicates an offset in the page redolog types.
  • MLOG_2BYTE: 2
  • MLOG_4BYTE: 4
  • MLOG_8BYTE: 8
  • MLOG_WRITE_STRING：30
  • MLOG_REC_INSERT: 9: indicates the use of an insert type when the redo log of the non-recorded compact row format (e.g., Redundant)
  • MLOG_COMP_REC_INSERT: 38: indicates the type of redo logs using an insert (e.g., compact / dynamic / compressed) during a non-compact format of the recording line
  • MLOG_COMP_REC_DELETE: 42: indicates a delete line using a compact format recorded in redothe log type
  • MLOG_COMP_LIST_START_DELETE and MLOG_COMP_LIST_END_DELETE: bulk delete, you can largely reduce the number of redo logs

• space id: tablespace

• page number: page number

• data: real data (for example to MLOG_COMP_REC_INSERT)

  • n_fileds: number of fields in the current record
  • n_uniques: determining the number of unique field records the column, if the primary key is the primary key number, the secondary index is a unique number + the number of primary key column index column, the same general secondary index; can be sorted based on this field is inserted
  • field1_len-fieldn_len: the size of the storage space occupied by several fields

• offset: a record before recording position on the page, easy to modify records list page, the previous record maintenance next_record property

  • end_seg_len: This field can be learned by the current record take up storage space

• len: if some type MLOG_WRITE_STRING, the number of bytes occupied by specific data

redo logs within memory operation

Mini-Transaction

Mini-Transaction (mtr) is a bottom page atomic access procedure (e.g., MAX_ROW_IDgeneration / insertion index record). A transaction can contain multiple mtr, mtr contains a number of redo logs.

Written to the log in groups

To ensure against atomic operation must be set in the form of recorded redo log to insert a record, for example, when a page split occurs, involve the Application Data Sheet, the system changes the page, change various statistics section / area , free list and other information, the new catalog records, etc. operations or complete implementation of all-or-no execution.

Log Group division: The last log after log into this series of actions to add a special log type MLOG_MULTI_REC_END：31as the end.

If an atomic operation is only one log record, then give this article a log type bit set to 1, or else produced a series of logs.

redo log write process

MySQL 512-byte pages to redo log record, this page is called block, blocka log block header, log block trailerand log block bodycomposition, header and trailer stored in the page management information.

The system will first request when the server starts a continuous redo log buffer memory as a log buffer, an example of two transactions, transaction T1 and T2 are alternately performed, it may alternatively be stored in the log buffer. They each contain two mtr, each mtr during operation, there will be a place first redo log, until the end of this mtr execution, this will all redo logs generated mtr all copied to the log buffer, after a certain period of time It will be flushed to disk.

To log bufferwrite the log is sequential, so the block will not be space debris, InnoDB provides a global variable buf_freeto indicate subsequent written redo log should be written to the block in which position, from behind the starting position is vacant .

brush disc redo logs

Brush pan ready

When the modified buffer poolwhen the page, the control block will be inserted into the flush dirty pages list, the control block stores two variables: oldest_modificationis loaded into buffer poolthe first modified mtr start lsn value corresponding to newest_modificationeach modified mtr end when the corresponding lsn value; control block in accordance with the oldest_modificationdescending order storage.

Mtr may modify a plurality of pages, the plurality of control blocks oldest_modification/ newest_modificationmay be the same.

Brush pan timing

• Insufficient log buffer space, free space is less than half
• When the transaction commits, buffer pool of dirty pages may not be the first brush plate, but the log buffer need to brush disk to prevent loss
• Timing background thread brush disc
• When a graceful server shutdown
• checkpoint when: Bulk brush from the flush list out dirty pages: If you modify the page frequently, and can not be dirty pages brush, you can not timely checkpoint, may direct user thread synchronization of the earliest changes from flush the list of dirty pages brush plate, so that the dirty pages corresponding redo log useless, a checkpoint can be

redo log files are stored

In MySQL data directory (the innodb_log_group_home_dirstorage location is determined by the name of the found file is stored in the form of a log group) named ib_logfile0... n file, the file number of the file name suffix determined by a system parameter innodb_log_files_in_groupdetermines the number of files, each the size of the file by the innodb_log_file_sizespecified.

Each ib_logfileorder cycle written log bufferin the block, the phenomenon of file will be overwritten.

Log File Format

• Before 2048 bytes (4 block): storing management information

  • log file header: redo log files stored in the current version, the beginning of the file LSN values, etc.

  • checkpoint1: mark log files can cover information

    • LOG_CHECKPOINT_NO: checkpoint frequency, incremental recording
    • LOG_CHECKPOINT_LSN: lsn maximum value redo log files can be overwritten
    • LOG_CHECKPOINT_OFFSET: lsn log file corresponding to the offset
    • LOG_CHECKPOINT_LOG_BUF_SIZE

  • Useless

  • checkpoint2

• The following byte: a storage block content, is recycled

Several global variables

• Log Sequeue Number (lsn): redo log records InnoDB global variable amount has been written includes writing a log log buffer, the initial value of 8704, the offset of the log file is 2048 bytes.

  • The first time a set of log mtr generated and added to a log buffer, lsn = 8704 + log write amount + log block header
  • The production of a set of log again, in the same Block, and can accommodate, you will only need to add the amount of the write log;
  • When turn generates a set of log, but take up a large amount, the remaining space in the current block is not received, to take up the next block, lsn = lsn + log write amount +2 * log block header + 2 * log block trailer;

• flushed_to_disk_lsn: flushed to disk redo log amount of global variables

• buf_next_to_write: marker which has log buffer in the global variable is the log of the brush disc

• innodb_flush_log_at_trx_commit =? : Indicates the need for all redo logs during the transaction generated when the brush disk to submit user threads (1) or to the background thread operation (0) or first written into a buffer (2).

checkpoint

Some redo log to determine whether the amount of disk space can be covered, namely whether it corresponds to the dirty pages have been flushed to disk

checkpoint_lsn: Represents the current total amount of redo logs may be covered by the system (the brush plate has been dirty pages), the initial value of 8074.

checkpoint steps:

• Calculation of redo logs may be covered by a corresponding lsn maximum (minimum flush list oldest_modificationbefore all log corresponding to the value lsn can be overwritten, because the list is not flush dirty pages have been stored in the brush plate), assigned to checkpoint_lsn;
• The checkpoint_lsncorresponding redo log file group offset checkpoint_offsetand the checkpointnumber (a total of how many times do checkpoint, increments) checkpoint_noto the log file header checkpoint1/2in

Ben collapse recovery

Ben collapse when the recovery occurs, flush the list has not been written to disk dirty pages change.

• Recovery of the starting point determined: Comparison of the log file checkpoint1and the checkpoint2largest checkpoint_no, and then from the corresponding checkpoint_lsn(previously covered are available, the description has been the brush disc) Start Recovery

• Recovery of end point determination: Comparison of each log file log block headerof LOG_BLOCK_HDR_DATA_LENproperties, if the note is not 512 the last fill a log file

• Recovery Methods

  • Hash tables: calculated redo logs hash(表空间id+页号), the same value in a slot and connected based on the generated time-ordered linked list, so that a repair can be a page, to avoid unnecessary random IO
  • Ben collapse skip pages have been restored: there may be flush dirty pages in the list has been brush plate, it will block in the file header dirty pages FIL_PAGE_LSN controlled according to the properties i.e. newest_modificationis greater than checkpoint_lsn, if it is then necessary to perform less than newest_modificationin FIL_PAGE_LSNthe redo logs of