What is the WAL?

HTTPS: // www.cnblogs.com/hzmark/p/wal.html 

original database and messaging middleware is the same pattern used 

are based on the sequential write performance than discrete written 

series stronger scatter

In the finish after a "Pull or Push", which originally planned to write a "storage layer design", but temporary change his mind, and would like to write an introduction about the most basic messaging middleware is the most central part: write -ahead logging (WAL).

What is the WAL

"In computer science, write-ahead logging (WAL) is a family of techniques for providing atomicity and durability (two of the ACID properties) in database systems."——维基百科

In the computer field, WAL (Write-ahead logging, Write-Ahead Logging) database system provides persistence atoms and a series of techniques.

In a system using the WAL, all the modifications are first written to the log, and then is applied to the system state. Generally comprise two portions redo and undo information.

Why do I need to use WAL, then redo and undo information it contains? For example, if a system will change to the system state directly, then reboot the system after power down the machine needs to know the operation is successful, or only partial success or failure (in order to restore state). If the WAL, then after restarting the system can be determined by comparing logs and system status is to continue to complete the operation or undo.

redo log

redo log called a redo log whenever an operation, before the operation of writing the data changes redo log, so that when the power failure occurs like the case of the system can continue to operate after the restart.

undo log

undo log called the undo log, when some changes to the implementation of half can not be completed, it can be restored to the state in accordance with changes between undo log.

Repair data (durability of transactions) with MySQL where the redo log to restart the system when Crash class, the undo log to guarantee atomicity of transactions.

Application of the message middleware WAL

WAL can be said is the foundation of messaging middleware, but also the foundation for all storage class systems.

In the message middleware, WAL MySQL is not that complicated, we just need to remember redo log. Direct messages are stored in the redo log in, just write redo log is finished, then the message is written completed:

Message is written redo log to show that the persistence
And the atomic problem does not occur, a message that is written to success, not failure that is written

Storage structure

WAL use to store data, it is necessary to organize store files, such as MySQL's binlog file. In the messaging middleware also needs a similar form to organize redo log, a file that is stored messages.

We use fixed-size file storage, so that when the index of the message, as long as know the offset, you can find the corresponding store files. The following example is an example of a file stored in a file size of 1024:

The message is continuously added to the latest storage file.

Message is stored in a file format generally described above:

Using binary message storage format
Messages are variable length

So here you will require a structure to index the message. Index into a complete message requires only two elements: the offset, size. As long as these two elements can complete data read section (a complete message) from the storage log.

The above structure is the simplest messaging middleware storage model, although from the real into practice some distance, but the core idea is the same.

Consider these questions:

Specific storage protocol message, i.e. the message storage file comprises content which need
How to optimize the index structure, supports message back, message filtering and other functions

Determine what the solution to these problems is the basic message of a middleware can use the WAL.

Crash Recovery

Said above for recovery redo log after the system make Crash recovery (in the middleware directly as data storage), then redo log specifically how we can use it for system status of it?

在消息写入的过程中，这里的状态只有一个，就是消息索引。只要消息索引构建了，那么消息就可以被消息；如果消息索引没构建，那么这条消息就是不可消费的（等价于消息没有写入）。

那么这里的恢复就是在系统Crash之后如何构建索引。

比如上图的状态是已经构建了消息0-3的索引，需要构建消息4-7的索引信息。

这个实现很简单：

消息一旦构建了索引，就记录checkpoint；checkpoint可以定期刷盘
系统恢复过程中读取checkpoint之后的消息构建索引；如果读取的消息不完整，则丢弃消息（可以采用CRC验证之类的方式来校验消息是否完成）

因为消息没有完整写入redo log的情况（写入时系统Crash了）系统并不会响应Producer消息发送成功（只有成功刷盘了才会响应成功），所以直接丢弃不完整的消息并不会对系统语义产生影响（不完整的消息没有响应Producer，也没有构建索引）。

结语

上文介绍了一下WAL，简要的描述了消息中间件存储模型和Crash Recovery，还有一些遗留的问题：

如何实现消息回溯和过滤，如何支持多tag过滤？
消息需要写文件且刷盘，如何保证写入性能？
消费时需要从文件读取消息，如何保证性能？
序列化的消息需要包含哪些属性才能完整的恢复出索引？

往期文章：

在写完上一篇《Pull or Push》之后，原本计划这一片写《存储层设计》，但是临时改变主意了，想先写一篇介绍一下消息中间件最最基础也是最核心的部分：write-ahead logging(WAL)。

什么是WAL

"In computer science, write-ahead logging (WAL) is a family of techniques for providing atomicity and durability (two of the ACID properties) in database systems."——维基百科

在计算机领域，WAL（Write-ahead logging，预写式日志）是数据库系统提供原子性和持久化的一系列技术。

在使用WAL的系统中，所有的修改都先被写入到日志中，然后再被应用到系统状态中。通常包含redo和undo两部分信息。

为什么需要使用WAL，然后包含redo和undo信息呢？举个例子，如果一个系统直接将变更应用到系统状态中，那么在机器掉电重启之后系统需要知道操作是成功了，还是只有部分成功或者是失败了（为了恢复状态）。如果使用了WAL，那么在重启之后系统可以通过比较日志和系统状态来决定是继续完成操作还是撤销操作。