1. Principle
1. Where does the message exist?
The place where the message is persisted is actually on the disk, in the commitlog folder in the following directory.
/root/store/commitlog
The source code is as follows:
// {@link org.apache.rocketmq.store.config.MessageStoreConfig}
// 数据存储根目录
private String storePathRootDir = System.getProperty("user.home") + File.separator + "store";
// commitlog目录
private String storePathCommitLog = System.getProperty("user.home") + File.separator + "store" + File.separator + "commitlog";
// 每个commitlog文件大小为1GB，超过1GB则创建新的commitlog文件
private int mappedFileSizeCommitLog = 1024 * 1024 * 1024;
For example, verify:
[root@iZ2ze84zygpzjw5bfcmh2hZ commitlog]# pwd
/root/store/commitlog
[root@iZ2ze84zygpzjw5bfcmh2hZ commitlog]# ll -h
total 400K
-rw-r--r-- 1 root root 1.0G Jun 30 18:21 00000000000000000000
[root@iZ2ze84zygpzjw5bfcmh2hZ commitlog]#
It can be clearly seen that the file size is 1.0G, and a new commitlog file will be created automatically if the message is written over 1.0G.
2. Explanation of key categories
2.1、MappedFile
Corresponds to the commitlog file, such as the 00000000000000000000file above .
2.2、MappedFileQueue
Is MappedFile the folder where it is, and the pair MappedFile is packaged into a file queue.
2.3、CommitLog
Targeted MappedFileQueue package use.
Two, Broker receives messages
1. Call chain
BrokerStartup.start() -》 BrokerController.start() -》 NettyRemotingServer.start() -》 NettyRemotingServer.prepareSharableHandlers() -》 new NettyServerHandler() -》 NettyRemotingAbstract.processMessageReceived()
-》 NettyRemotingAbstract.processRequestCommand() -》 SendMessageProcessor.processRequest()
2、processRequest
SendMessageProcessor.processRequest()
@Override
public RemotingCommand processRequest(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
RemotingCommand response = null;
try {
// 调用asyncProcessRequest
response = asyncProcessRequest(ctx, request).get();
} catch (InterruptedException | ExecutionException e) {
log.error("process SendMessage error, request : " + request.toString(), e);
}
return response;
}
3、asyncProcessRequest
public CompletableFuture<RemotingCommand> asyncProcessRequest(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final SendMessageContext mqtraceContext;
switch (request.getCode()) {
// 表示消费者发送的消息，发送者消费失败会重新发回队列进行消息重试
case RequestCode.CONSUMER_SEND_MSG_BACK:
return this.asyncConsumerSendMsgBack(ctx, request);
default:
// 解析header，也就是我们Producer发送过来的消息都在request里，给他解析到SendMessageRequestHeader对象里去。
SendMessageRequestHeader requestHeader = parseRequestHeader(request);
if (requestHeader == null) {
return CompletableFuture.completedFuture(null);
}
mqtraceContext = buildMsgContext(ctx, requestHeader);
// 将解析好的参数放到SendMessageContext对象里
this.executeSendMessageHookBefore(ctx, request, mqtraceContext);
if (requestHeader.isBatch()) {
// 批处理消息用
return this.asyncSendBatchMessage(ctx, request, mqtraceContext, requestHeader);
} else {
// 非批处理，我们这里介绍的核心。
return this.asyncSendMessage(ctx, request, mqtraceContext, requestHeader);
}
}
}
4、asyncSendMessage
private CompletableFuture<RemotingCommand> asyncSendMessage(ChannelHandlerContext ctx, RemotingCommand request,
SendMessageContext mqtraceContext,
SendMessageRequestHeader requestHeader) {
final byte[] body = request.getBody();
int queueIdInt = requestHeader.getQueueId();
TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
// 拼凑message对象
MessageExtBrokerInner msgInner = new MessageExtBrokerInner();
msgInner.setTopic(requestHeader.getTopic());
msgInner.setQueueId(queueIdInt);
msgInner.setBody(body);
msgInner.setFlag(requestHeader.getFlag());
MessageAccessor.setProperties(msgInner, MessageDecoder.string2messageProperties(requestHeader.getProperties()));
msgInner.setPropertiesString(requestHeader.getProperties());
msgInner.setBornTimestamp(requestHeader.getBornTimestamp());
msgInner.setBornHost(ctx.channel().remoteAddress());
msgInner.setStoreHost(this.getStoreHost());
msgInner.setReconsumeTimes(requestHeader.getReconsumeTimes() == null ? 0 : requestHeader.getReconsumeTimes());
CompletableFuture<PutMessageResult> putMessageResult = null;
Map<String, String> origProps = MessageDecoder.string2messageProperties(requestHeader.getProperties());
// 真正接收消息的方法
putMessageResult = this.brokerController.getMessageStore().asyncPutMessage(msgInner);
return handlePutMessageResultFuture(putMessageResult, response, request, msgInner, responseHeader, mqtraceContext, ctx, queueIdInt);
}
At this point, our message reception is complete, and they are all encapsulated in the MessageExtBrokerInner object.
Three, Broker message storage (persistence)
1、asyncPutMessage
Continue to look at the asyncSendMessage in the previous step
@Override
public CompletableFuture<PutMessageResult> asyncPutMessage(MessageExtBrokerInner msg) {
CompletableFuture<PutMessageResult> putResultFuture = this.commitLog.asyncPutMessage(msg);
putResultFuture.thenAccept((result) -> {
......
});
return putResultFuture;
}
2、commitLog.asyncPutMessage
public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
// 获取最后一个文件，MappedFile就是commitlog目录下的那个0000000000文件
MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
try {
// 追加数据到commitlog
result = mappedFile.appendMessage(msg, this.appendMessageCallback);
switch (result.getStatus()) {
......
}
// 将内存的数据持久化到磁盘
CompletableFuture<PutMessageStatus> flushResultFuture = submitFlushRequest(result, putMessageResult, msg);
}
}
3、appendMessagesInner
public AppendMessageResult appendMessagesInner(final MessageExt messageExt, final AppendMessageCallback cb) {
// 将消息写到内存
return cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBrokerInner) messageExt);
}
4、doAppend
@Override
public AppendMessageResult doAppend(final long fileFromOffset, final ByteBuffer byteBuffer, final int maxBlank,
final MessageExtBrokerInner msgInner) {
// Initialization of storage space
this.resetByteBuffer(msgStoreItemMemory, msgLen);
// 1 TOTALSIZE
this.msgStoreItemMemory.putInt(msgLen);
// 2 MAGICCODE
this.msgStoreItemMemory.putInt(CommitLog.MESSAGE_MAGIC_CODE);
// 3 BODYCRC
this.msgStoreItemMemory.putInt(msgInner.getBodyCRC());
// 4 QUEUEID
this.msgStoreItemMemory.putInt(msgInner.getQueueId());
// 5 FLAG
this.msgStoreItemMemory.putInt(msgInner.getFlag());
// 6 QUEUEOFFSET
this.msgStoreItemMemory.putLong(queueOffset);
// 7 PHYSICALOFFSET
this.msgStoreItemMemory.putLong(fileFromOffset + byteBuffer.position());
// 8 SYSFLAG
this.msgStoreItemMemory.putInt(msgInner.getSysFlag());
// 9 BORNTIMESTAMP
this.msgStoreItemMemory.putLong(msgInner.getBornTimestamp());
// 10 BORNHOST
this.resetByteBuffer(bornHostHolder, bornHostLength);
this.msgStoreItemMemory.put(msgInner.getBornHostBytes(bornHostHolder));
// 11 STORETIMESTAMP
this.msgStoreItemMemory.putLong(msgInner.getStoreTimestamp());
// 12 STOREHOSTADDRESS
this.resetByteBuffer(storeHostHolder, storeHostLength);
this.msgStoreItemMemory.put(msgInner.getStoreHostBytes(storeHostHolder));
// 13 RECONSUMETIMES
this.msgStoreItemMemory.putInt(msgInner.getReconsumeTimes());
// 14 Prepared Transaction Offset
this.msgStoreItemMemory.putLong(msgInner.getPreparedTransactionOffset());
// 15 BODY
this.msgStoreItemMemory.putInt(bodyLength);
if (bodyLength > 0)
this.msgStoreItemMemory.put(msgInner.getBody());
// 16 TOPIC
this.msgStoreItemMemory.put((byte) topicLength);
this.msgStoreItemMemory.put(topicData);
// 17 PROPERTIES
this.msgStoreItemMemory.putShort((short) propertiesLength);
if (propertiesLength > 0)
this.msgStoreItemMemory.put(propertiesData);
final long beginTimeMills = CommitLog.this.defaultMessageStore.now();
// Write messages to the queue buffer
byteBuffer.put(this.msgStoreItemMemory.array(), 0, msgLen);
return result;
}
This step actually saves the message in the buffer, that is, msgStoreItemMemory, which is NIO here.
private final ByteBuffer msgStoreItemMemory;
5、submitFlushRequest
Go back to the submitFlushRequest method of [2, commitLog.asyncPutMessage] again, because the previous method is to write the data into the ByteBuffer buffer, the next step is that we are going to flush the disk at this step.
public CompletableFuture<PutMessageStatus> submitFlushRequest(AppendMessageResult result, PutMessageResult putMessageResult,
MessageExt messageExt) {
// 同步刷盘
if (FlushDiskType.SYNC_FLUSH == this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;
if (messageExt.isWaitStoreMsgOK()) {
GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes(),
this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());
service.putRequest(request);
return request.future();
} else {
service.wakeup();
return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
}
}
// 异步刷盘
else {
if (!this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
flushCommitLogService.wakeup();
} else {
commitLogService.wakeup();
}
return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
}
}
6. Asynchronous brushing
class FlushRealTimeService extends FlushCommitLogService {
@Override
public void run() {
while (!this.isStopped()) {
try {
// 每隔500ms刷一次盘
if (flushCommitLogTimed) {
Thread.sleep(500);
} else {
this.waitForRunning(500);
}
// 调用mappedFileQueue的flush方法
CommitLog.this.mappedFileQueue.flush(flushPhysicQueueLeastPages);
} catch (Throwable e) {
}
}
}
}
- It can be seen that the default is to refresh the disk every 500 milliseconds
7、mappedFileQueue.flush
public boolean flush(final int flushLeastPages) {
MappedFile mappedFile = this.findMappedFileByOffset(this.flushedWhere, this.flushedWhere == 0);
if (mappedFile != null) {
// 真正的刷盘操作
int offset = mappedFile.flush(flushLeastPages);
}
}
8、mappedFile.flush
public int flush(final int flushLeastPages) {
if (this.isAbleToFlush(flushLeastPages)) {
try {
if (writeBuffer != null || this.fileChannel.position() != 0) {
// 刷盘 NIO
this.fileChannel.force(false);
} else {
// 刷盘 NIO
this.mappedByteBuffer.force();
}
} catch (Throwable e) {
log.error("Error occurred when force data to disk.", e);
}
}
return this.getFlushedPosition();
}
So far it's all over.
Four, summary
The interview was asked: How does Broker persist after receiving the message?
Respondent: There are two ways: synchronous and asynchronous. Generally choose asynchronous, synchronous efficiency is low, but more reliable. The general principle of message storage is:
The core class MappedFile corresponds to each commitlog file. MappedFileQueue is equivalent to a folder and manages all files. There is also a manager CommitLog object, which is responsible for providing some operations. Specifically, after the Broker gets the message, it first stores the message, topic, queue, etc. in the ByteBuffer, and then persists it in the commitlog file. The commitlog file size is 1G. If the size is exceeded, a new commitlog file will be created for storage, using the nio method.
5. Supplement: Synchronous/asynchronous brushing
1. Key categories
| Class name | description | Brush performance |
|---|---|---|
| CommitRealTimeService | Asynchronous flash disk && open byte buffer | highest |
| FlushRealTimeService | Asynchronous flash disk && close memory byte buffer | Higher |
| GroupCommitService | Synchronous flashing, the message will be returned after flashing the disk successfully. | lowest |
2. Graphic
3. Synchronous brushing
3.1, source code
// {@link org.apache.rocketmq.store.CommitLog#submitFlushRequest()}
// Synchronization flush
if (FlushDiskType.SYNC_FLUSH == this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
// 同步刷盘service -> GroupCommitService
final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;
if (messageExt.isWaitStoreMsgOK()) {
// 数据准备
GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes(),
this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());
// 将数据对象放到requestsWrite里
service.putRequest(request);
return request.future();
} else {
service.wakeup();
return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
}
}
putRequest
public synchronized void putRequest(final GroupCommitRequest request) {
synchronized (this.requestsWrite) {
this.requestsWrite.add(request);
}
// 这里很关键！！！，给他设置成true。然后计数器-1。下面run方法的时候才会进行交换数据且return
if (hasNotified.compareAndSet(false, true)) {
waitPoint.countDown(); // notify
}
}
run
public void run() {
while (!this.isStopped()) {
try {
// 是同步还是异步的关键方法，也就是说组不阻塞全看这里。
this.waitForRunning(10);
// 真正的刷盘逻辑
this.doCommit();
} catch (Exception e) {
CommitLog.log.warn(this.getServiceName() + " service has exception. ", e);
}
}
}
waitForRunning
protected volatile AtomicBoolean hasNotified = new AtomicBoolean(false);
// 其实就是CountDownLatch
protected final CountDownLatch2 waitPoint = new CountDownLatch2(1);
protected void waitForRunning(long interval) {
// 如果是true，且给他改成false成功的话，则onWaitEnd()且return，但是默认是false，也就是默认情况下这个if不会进。
if (hasNotified.compareAndSet(true, false)) {
this.onWaitEnd();
return;
}
//entry to wait
waitPoint.reset();
try {
// 等待，默认值是1，也就是waitPoint.countDown()一次后就会激活这里。
waitPoint.await(interval, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
log.error("Interrupted", e);
} finally {
// 给状态值设置成false
hasNotified.set(false);
this.onWaitEnd();
}
}
3.2. Summary
Summarize the main process of synchronous brushing:
The core class is GroupCommitService, and the core method is waitForRunning.
- First call the putRequest method to change hasNotified to true, and notify, that is
waitPoint.countDown(). - Followed in the run method
waitForRunning(),waitForRunning()to determine hasNotified is not true, it is true then return swap to exchange data, that is, not to await blocking direct return. - Finally, the previous step returns, and there is no blocking, then it is logical to call doCommit to perform a real refresh.
4. Asynchronous brushing
4.1, source code
The core class is: FlushRealTimeService
// {@link org.apache.rocketmq.store.CommitLog#submitFlushRequest()}
// Asynchronous flush
if (!this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
flushCommitLogService.wakeup();
} else {
commitLogService.wakeup();
}
return CompletableFuture.completedFuture(PutMessageStatus.PUT_OK);
run
// {@link org.apache.rocketmq.store.CommitLog.FlushRealTimeService#run()}
class FlushRealTimeService extends FlushCommitLogService {
@Override
public void run() {
while (!this.isStopped()) {
try {
// 每隔500ms刷一次盘
if (flushCommitLogTimed) {
Thread.sleep(500);
} else {
// 根上面同步刷盘调用的是同一个方法，区别在于这里没有将hasNotified变为true，也就是还是默认的false，那么waitForRunning方法内部的第一个判断就不会走，就不会return掉，就会进行下面的await方法阻塞，默认阻塞时间是500毫秒。也就是默认500ms刷一次盘。
this.waitForRunning(500);
}
// 调用mappedFileQueue的flush方法
CommitLog.this.mappedFileQueue.flush(flushPhysicQueueLeastPages);
} catch (Throwable e) {
}
}
}
}
Core class#method: FlushRealTimeService#run()
- Judge whether it
flushCommitLogTimedis true, the default is false, if it is true, sleep (500ms) directly and thenmappedFileQueue.flush()flash the disk. - If it is false, then enter
waitForRunning(500). Here is the key to the difference from synchronous flashing. Before the synchronous flashing, the hasNotified is changed to true, so a set of small tricks is directly: Yesreturn+doCommit, asynchronous is called directly here, and there iswaitForRunning(500)nothing right before this. The operation of hasNotified, so it will not return, but will continuewaitPoint.await(500, TimeUnit.MILLISECONDS);to block for 500 milliseconds, automatically wake up after 500 milliseconds and then flush the disk. That is to say, if the disk is refreshed asynchronously, the default is 500ms to refresh the disk once.