Recently, I suddenly felt that many software and hardware are designed with root reason, not by desgin, but to solve the needs of that scene at that time. Once you understand it, you will feel like you are talking to designers, understand their thinking, learn their methods, and think on the same screen: live and learn.
problem thinking
1. Is the Consumer Queue Offset continuous? Why?
2. Is the Commit Log Offset continuous? Why?
3. For files written in Java, the default is big-endian or little-endian. Why?
Commit Log real distribution
While everyone is thinking, let's recall how the commit log is distributed?
In the storage root directory configured by Broker, by viewing the commit log file actually generated by Broker, you can see the distribution of data files similar to the following:
It can be seen that there are multiple real storage files, each of which uses a string of similar numbers as the file name, and has a size of 1G.
We can know from the source code that the actual abstract model is as follows:
From the figure above, we can see that:
Commit Log is a name for a class of files. In fact, there are many Commit Log files, and each of them can be called a Commit Log file. As shown in the figure, there are a total of T Commit Log files, and they are arranged according to the creation time from the past to the present.
Each Commit Log file saves messages, and saves them in the order in which they are written, and always writes the file with the largest creation time, and only one thread is writing at the same time. The first file in the figure, 1, 2, 3, 4... indicates the number of messages in this file. You can see that the 1234th message is the last message of the first Commit Log file, and the 1235th message It is the first message of the second Commit Log.
Note 1: The actual storage space occupied by all the messages in each Commit Log file is <=1G. Let's think about why this is the case.
Explanation 2: RocketMQ will lock every time the Commit Log is written, see https://github.com/apache/rocketmq/blob/7676cd9366a3297925deabcf27bb590e34648645/store/src/main/java/org/apache/rocketmq/store/ for code snippets CommitLog.java#L676-L722
We see that there are many messages in the Commit Log file, which are stored according to the established protocol. What is the specific protocol and how do you know it?
Commit Log Storage Protocol
Regarding the Commit Log storage protocol, we asked ChatGPT, and it replied me like this. Although it is wrong, the format and description of this reply are very close to the answer.
Let's look at the source code and explain it in detail: https://github.com/apache/rocketmq/blob/rocketmq-all-4.9.3/store/src/main/java/org/apache/rocketmq/store/CommitLog.java #L1547-L1587
After I sorted it out, it looks like this:
Explanation 1: The number of the message protocol I sorted out is not consistent with the code. The code only indicates the order, and the storage protocol in the real physical file will be more detailed. Explanation 2: In the "RocketMQ Distributed Message Middleware: Core Principles and Best Practices" I wrote, this figure lacks the Body content, which is added here, and other data are supplemented in more detail. Here are a few issues that need clarification:
1. The binary protocol has byte order, which is often called big endian and little endian. The big and small ends are not detailed here. Interested students can google or ask questions on ChatGPT. The answer is definitely better than what I said.
2. In java, a byte occupies 1 byte, an int occupies 4 bytes, a short occupies 2 bytes, and a long occupies 8 bytes.
3. The encoding of the Host is not simply converting the IP:Port as a string directly into a byte array, but encoding each number as a byte in turn. It will be explained in the Golang code in the next section.
4. In the encoding of the extended information, invisible characters are used as segmentation, so the extended field key-value cannot contain those two invisible characters. Which two are they, please look for them?
After we see this agreement, how can we prove that your physical file is written according to this agreement?
Use Golang to untie RocketMQ Commit Log
RocketMQ is written in java. According to the storage protocol described above, I wrote a tool in Golang that can unlock Commit Log and Cosumer Queue. Code address: https://github.com/rmq-plus-plus/rocketmq -decoder.
This tool currently supports 2 functions:
1. Specify the Commit Log site, directly parse and print the messages in the Commit Log.
2. Specify the consumption location, first parse the Consumer Queue, get the Commit Log Offset, then directly parse the Commit Log according to the Commit Log Offset, and print it.
In Golang, there is no code that depends on RocketMQ, and it relies purely on protocol decoding.
Here is an example of parsing Commit Log Offset in golang: In java, this offset is a long type, occupying 8 bytes.
In golang, the normal Commit Log Offset can be obtained by reading 8-byte data and decoding it into int64 according to the big endian sequence.
I ran a demo result, for your reference:
answer the original question
The following is my personal opinion for your reference:
1. Is the Consumer Queue Offset continuous? Why?
is continuous.
The consumer queue offset refers to the subscript of the index message in each queue, and the subscripts are of course continuous. Consumers also take advantage of this continuity to avoid submitting empty consumption sites.
Each index message occupies the same space, 20 bytes, and the structure is as follows:
The physical location here is the Commit Log Offset.
2. Is the Commit Log Offset continuous? Why?
Not consecutive.
Commit Log Offset refers to the byte offset of each message in all Commit Log files. The size of each message is uncertain, so Commit Log Offset, that is, the byte offset must be different .
And it can be known that the absolute value of the difference between every two offsets is the total length of the message bytes of the previous message.
And there is a misunderstanding in the above figure "Commit Log storage file distribution abstraction", the size of each small square is actually different.
3. For files written in Java, the default is big-endian or little-endian. Why?
Big endian. Big-endian actually has byte storage order and network transmission order. The default big-endian order in java remains the same as network transmission, which is convenient for encoding and decoding.
The first byte of the data message in each segment of the network transport layer expresses what protocol is used to transmit the subsequent data, so that when the data receiver receives the data, it first parses the protocol according to the byte order, and then decodes the subsequent data according to the protocol A sequence of bytes, in line with the way humans think and solve problems.
Discussion note : Because some versions of RocketMQ may be different, this article discusses under version 4.9.3. You can refer to this method, 解开5.0 or even other versions, and the storage protocol format of other data files.