1. What is RocketMQ?
RocketMQ is Alibaba's open source distributed messaging middleware in 2012. It has been donated to the Apache Software Foundation and became Apache's top project on September 25, 2017. As a domestic middleware that has experienced many times of "super engineering" of Alibaba Double Eleven and has stable and outstanding performance, it has the characteristics of high performance, low latency and high reliability. Mainly used to improve performance, system decoupling, flow peaks, etc.
2. What are the characteristics of RocketMQ?
1. Flexible scalability
RocketMQ naturally supports clusters, and each of its core four components (Name Server, Broker, Producer, Consumer) can be horizontally expanded without a single point of failure.
2. Mass message accumulation capability
RocketMQ uses the zero-copy principle to achieve a super large message accumulation capability. It is said that a single machine can support billions of message accumulation, and it still maintains low write latency after accumulating so many messages.
3. Supporting sequential messages
can ensure that message consumers consume messages in the order in which they are sent. Sequential messages are divided into global order and local order. It is generally recommended to use partial order, that is, the producer realizes by sending a certain type of messages to the same queue in order.
4. Multiple message filtering methods
Message filtering is divided into server-side filtering and consumer-side filtering. The server-side filtering can be done according to the requirements of the message consumer. The advantage is that it reduces unnecessary message transmission, but the disadvantage is that it increases the burden on the message server and the implementation is relatively complicated. Consumer filtering is completely implemented by specific applications. This method is more flexible. The disadvantage is that many useless messages will be transmitted to message consumers.
5. Support transaction messages
RocketMQ supports transaction messages in addition to ordinary messages and sequential messages. This feature provides another solution for distributed transactions.
6. Retrospective consumption
Retrospective consumption refers to the news that consumers have successfully consumed. Due to business needs, RocketMQ supports retrospective consumption according to time. The time dimension is accurate to milliseconds. It can be traced forward or backward.
3. What is the difference between RocketMQ, RabbitMQ and Kafaka?
Kafka:
1. Development language: Scala development
2. Performance and throughput: The throughput is the best among all MQs, QPS 100,000, performance in milliseconds, support for cluster deployment
3. Functions: Single function
4. Disadvantages: Data loss, because The data is written to the disk buffer first, and not directly placed on the disk. Machine failure will cause data loss
5. Application scenarios: It does not matter if data is properly lost, the throughput requirements are high, and too many advanced functions are not required, such as big data scenarios.
RabbitMQ:
1. Development language: Erlang development
2. Performance and throughput: The throughput is relatively low, QPS tens of thousands, performance u second level, master-slave architecture
3. Function: Single function
4. Disadvantages: Erlang niche language development, Low throughput and troublesome cluster expansion
5. Application scenarios: Scenarios where small and medium-sized companies have low requirements for concurrency and throughput.
RocketMQ:
1. Development language: java development
2. Performance and throughput: High throughput, QPS 100,000 level, performance millisecond level, support cluster deployment
3. Function: Support various advanced functions, such as delayed message, transaction message, Message backtracking, dead letter queue, message backlog, etc.
4. Disadvantages: The relatively simple official documents may be the only disadvantage of RocketMQ at present.
5. Application scenarios: It does not matter if the data is appropriately lost, the throughput requirements are high, and too many advanced functions are not required Scenarios, such as big data scenarios.
What are the advantages and disadvantages of Kafka, ActiveMQ, RabbitMQ, RocketMQ?
| characteristic | ActiveMQ | RabbitMQ | RocketMQ | Kafka |
|---|---|---|---|---|
| Single machine throughput | Ten thousand, one order of magnitude lower than RocketMQ and Kafka | Same as ActiveMQ | 100,000 level, supporting high throughput | 100,000 level, high throughput, generally cooperate with big data systems for real-time data calculation, log collection and other scenarios |
| The impact of the number of topics on throughput | Topics can reach hundreds/thousands of levels, and throughput will drop slightly. This is a big advantage of RocketMQ. Under the same machine, it can support a large number of topics. | When the topic goes from tens to hundreds, the throughput will drop significantly. Under the same machine, Kafka tries to ensure that the number of topics is not too much. If you want to support large-scale topics, you need to add more machine resources. | ||
| Timeliness | ms level | Microsecond level, this is a major feature of RabbitMQ, with the lowest latency | ms level | Delay within ms |
| Availability | High, based on the master-slave architecture to achieve high availability | Same as ActiveMQ | Very high, distributed architecture | Very high, distributed, multiple copies of one data, a few machines down, no data loss, no unavailability |
| Message reliability | Have a low probability of losing data | Basically not lost | After parameter optimization configuration, 0 loss can be achieved | Same as RocketMQ |
| Function support | The functions in the MQ field are extremely complete | Developed based on erlang, with strong concurrency, excellent performance, and low latency | The MQ function is relatively complete, it is distributed, and the scalability is good | The function is relatively simple, mainly supporting simple MQ functions, real-time calculation and log collection in the field of big data are used on a large scale |
In summary, after various comparisons, the following are summarized:
The general business system needs to introduce MQ. At first, everyone used ActiveMQ, but now it is true that everyone does not use it much. It has not been verified by large-scale throughput scenarios and the community is not very active, so everyone should forget it, I personally don’t Recommend this;
Later, everyone started to use RabbitMQ, but it is true that the erlang language prevents a large number of Java engineers from studying and controlling it. For the company, it is almost uncontrollable, but it is indeed open source, relatively stable support, and active. high;
However, it is true that more and more companies will use RocketMQ, which is really good. After all, it is produced by Ali, but the community may be at risk of sudden pornography (Currently RocketMQ has been donated to Apache , but the activity on GitHub is actually not high ) If you have absolute confidence in your company's technical strength, I recommend RocketMQ, otherwise, go back and use RabbitMQ honestly. They have an active open source community, and they will definitely not be pornographic.
Therefore, for small and medium-sized companies with relatively average technical strength and not particularly high technical challenges, RabbitMQ is a good choice; for large companies , with strong infrastructure R&D capabilities, RocketMQ is a good choice.
If it is real-time computing, log collection and other scenarios in the field of big data , Kafka is the industry standard, absolutely no problem, the community is very active and will never be pornographic, not to mention the fact that it is almost a factual norm in this field around the world.
3. Why use MQ?
Because the project is relatively large and a distributed system is built, all remote service invocation requests are executed synchronously and often problems occur, so mq is introduced
Function description The coupling degree of decoupling system is reduced, and there is no strong dependency. Asynchronous remote calls that do not need to be executed synchronously can effectively improve the response time. After the peak-cutting request reaches the peak, the back-end service can maintain a fixed consumption rate and will not be overwhelmed.
4. What roles does RocketMQ consist of, and what are the functions and characteristics of each role?
Role role Nameserver is stateless, dynamic list; this is also one of the important differences from zookeeper. Zookeeper is stateful. Producer The message producer is responsible for sending messages to the Broker. Broker is MQ itself, responsible for sending and receiving messages, persisting messages, etc. Consumer message consumers are responsible for pulling messages from Broker for consumption, and ack after consumption.
5. What is the difference between Topic and JMS queue in RocketMQ?
Queue is the FIFO queue derived from the data structure. Topic is an abstract concept. The bottom layer of each topic corresponds to N queues, and the data actually exists on the queue.
6. Will the messages in RocketMQ Broker be deleted immediately after being consumed?
No, every message will be persisted to CommitLog. After each Consumer is connected to the Broker, it will maintain consumption progress information. When a message is consumed, only the current Consumer's consumption progress (CommitLog offset) is updated.
Follow-up: So will the news accumulate? When to clean up expired messages?
The 4.6 version will delete CommitLog files that are no longer in use after 48 hours by default
- Check the last access time of this file
- Determine whether it is greater than the expiration time
- Delete at the specified time, the default is 4 o'clock in the morning
The source code is as follows:
/**
* {@link org.apache.rocketmq.store.DefaultMessageStore.CleanCommitLogService#isTimeToDelete()}
*/
private boolean isTimeToDelete() {
// when = "04";
String when = DefaultMessageStore.this.getMessageStoreConfig().getDeleteWhen();
// 是04点，就返回true
if (UtilAll.isItTimeToDo(when)) {
return true;
}
// 不是04点，返回false
return false;
}
/**
* {@link org.apache.rocketmq.store.DefaultMessageStore.CleanCommitLogService#deleteExpiredFiles()}
*/
private void deleteExpiredFiles() {
// isTimeToDelete()这个方法是判断是不是凌晨四点，是的话就执行删除逻辑。
if (isTimeToDelete()) {
// 默认是72，但是broker配置文件默认改成了48，所以新版本都是48。
long fileReservedTime = 48 * 60 * 60 * 1000;
deleteCount = DefaultMessageStore.this.commitLog.deleteExpiredFile(72 * 60 * 60 * 1000, xx, xx, xx);
}
}
/**
* {@link org.apache.rocketmq.store.CommitLog#deleteExpiredFile()}
*/
public int deleteExpiredFile(xxx) {
// 这个方法的主逻辑就是遍历查找最后更改时间+过期时间，小于当前系统时间的话就删了（也就是小于48小时）。
return this.mappedFileQueue.deleteExpiredFileByTime(72 * 60 * 60 * 1000, xx, xx, xx);
}
7. What are the consumption modes of RocketMQ?
The consumption model is determined by Consumer, and the consumption dimension is Topic.
- Cluster consumption
1. A message will only be consumed by one Consumer in the same group.
2. When multiple groups consume a topic at the same time, each group will have one consumer to consume the data
- Broadcast consumption
The message will be consumed for each Consumer instance under a Consumer Group. That is, even if these Consumers belong to the same Consumer Group, the message will be consumed once by each Consumer in the Consumer Group.
8. Is the consumer news push or pull?
RocketMQ has no real meaning of push, which are all pulls. Although there is a push class, the actual underlying implementation uses a long polling mechanism , that is, a pull method.
The broker property longPollingEnable marks whether long polling is enabled. Default on
The source code is as follows:
// {@link org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage()}
// 看到没，这是一只披着羊皮的狼，名字叫PushConsumerImpl，实际干的确是pull的活。
// 拉取消息，结果放到pullCallback里
this.pullAPIWrapper.pullKernelImpl(pullCallback);
Follow-up: Why do you want to take the initiative to pull messages instead of using event monitoring methods?
The event-driven method is to establish a long connection and push it in real time by means of events (sending data).
If the broker actively pushes messages, it is possible that the push speed is fast and the consumption speed is slow, which will cause the message to accumulate too much on the consumer side, and at the same time, it will not be consumed by other consumers. The pull method can be pulled according to the current situation, without causing too much pressure and causing a bottleneck. So the method of pull is adopted.
9. How does the broker handle pull requests?
Consumer requests Broker for the first time
- Whether there are eligible messages in the Broker
- Yes->
- Responding to Consumer
- Waiting for the next Consumer request
- No
- DefaultMessageStore#ReputMessageService#run方法
- PullRequestHoldService connects to Hold, and it is executed every 5s to check whether there are any messages in the pullRequestTable, and push them immediately if there are any.
- Check whether there are new messages in commitLog every 1ms, and write them to pullRequestTable if there are any
- Return request when there is a new message
- Suspend the request of the consumer, that is, neither disconnect nor return data
- Use consumer's offset,
10. How does RocketMQ do load balancing?
Realize distributed storage in multiple Brokers through Topic.
producer端
The sender specifies the message queue to send the message to the corresponding broker to achieve load balancing when writing
- Improve write throughput. When multiple producers write data to a broker at the same time, performance will decrease
- Messages are distributed among multiple brokers to prepare for load consumption
The default strategy is to choose randomly:
- The producer maintains an index
- Each time the node is taken, it will increase automatically
- index takes the remainder of all brokers
- Built-in fault tolerance strategy
Other implementations:
- SelectMessageQueueByHash
- The hash is the incoming args
- SelectMessageQueueByRandom
- SelectMessageQueueByMachineRoom is not implemented
You can also customize the select method in the MessageQueueSelector interface
MessageQueue select(final List<MessageQueue> mqs, final Message msg, final Object arg);
consumer端
The average distribution algorithm is used for load balancing.
Other load balancing algorithms
Average distribution strategy (default) (AllocateMessageQueueAveragely) Circular distribution strategy (AllocateMessageQueueAveragelyByCircle) Manual configuration distribution strategy (AllocateMessageQueueByConfig) Computer room distribution strategy (AllocateMessageQueueByMachineRoom) Consistent hash distribution strategy (AllocateMessageQueueConsistentHash) Near computer room strategy (AllocateMessageQueueByMachineRoom)
Follow-up: What happens when the consumer load balancer and queue are not equal?
Consumers and queues will be allocated equally first. If the number of Consumers is less than the number of queues, there will be cases where some Consumers consume multiple queues. If the number of Consumers is equal to the number of queues, then one Consumer consumes one queue. If the number of Consumers is greater than The number of queues, then some Consumers will be spared, which is wasted.
11. Repeated consumption of messages
An important reason that affects the normal sending and consumption of messages is the uncertainty of the network.
Causes of repeated consumption
- ACK
Under normal circumstances, after the consumer actually consumes the message, it should send an ack to notify the broker that the message has been consumed normally, and remove it from the queue
When the ack cannot be sent to the broker due to network reasons, the broker will think that the entry message has not been consumed, and then the message retransmission mechanism will be activated to deliver the message to the consumer again
- Consumption pattern
In the CLUSTERING mode, the message will be guaranteed to be consumed by consumers of the same group once in the broker, but consumers of different groups will be pushed multiple times
solution
- Database Table
Before processing the message, use the message primary key to insert in the constrained field in the table
- Map
You can use map ConcurrentHashMap -> putIfAbsent guava cache in stand-alone mode
- Redis
Distributed locks are up.
12. How to make RocketMQ guarantee the sequential consumption of messages
When you use message middleware in your online business, do you need to ensure the order of messages?
If you do n’t need to ensure the order of messages, why not? If I have a scenario to ensure the order of messages, how should you guarantee it?
First, multiple queues can only guarantee the order in a single queue. The queue is a typical FIFO, natural order. Consumption of multiple queues at the same time cannot absolutely guarantee the orderliness of messages. So the summary is as follows:
The same topic, the same QUEUE, when sending a message, a thread sends a message, when consuming, a thread consumes a message in a queue.
Follow-up: How to ensure that messages are sent to the same queue?
Rocket MQ provides us with the MessageQueueSelector interface, you can rewrite the interface inside and implement your own algorithm. To give the simplest example: judge i % 2 == 0, then put it in queue1, otherwise put it in queue2.
for (int i = 0; i < 5; i++) {
Message message = new Message("orderTopic", ("hello!" + i).getBytes());
producer.send(
// 要发的那条消息
message,
// queue 选择器 ，向 topic中的哪个queue去写消息
new MessageQueueSelector() {
// 手动 选择一个queue
@Override
public MessageQueue select(
// 当前topic 里面包含的所有queue
List<MessageQueue> mqs,
// 具体要发的那条消息
Message msg,
// 对应到 send（） 里的 args，也就是2000前面的那个0
Object arg) {
// 向固定的一个queue里写消息，比如这里就是向第一个queue里写消息
if (Integer.parseInt(arg.toString()) % 2 == 0) {
return mqs.get(0);
} else {
return mqs.get(1);
}
}
},
// 自定义参数：0
// 2000代表2000毫秒超时时间
i, 2000);
}
13. How does RocketMQ ensure that messages are not lost?
First of all, there may be lost messages in the following three parts:
- Producer端
- Broker side
- Consumer端
13.1 How to ensure that the message is not lost on the Producer side
- Use send() to send messages synchronously, and the sending result is synchronously aware.
- You can retry after sending failed and set the number of retries. The default is 3 times.
producer.setRetryTimesWhenSendFailed(10);
- Cluster deployment, for example, the reason for the failure of sending may be that the current Broker is down, and it will be sent to other Brokers when retrying.
13.2 How does the Broker end ensure that the message is not lost?
- Modify the brushing strategy to synchronous brushing. By default, it is asynchronous flushing.
flushDiskType = SYNC_FLUSH
- Cluster deployment, master-slave mode, high availability.
13.3. How does the Consumer end ensure that the message is not lost?
- After the complete consumption is normal, a manual ack confirmation is performed.
14. How to deal with the message accumulation of rocketMQ
If the downstream consumer system goes down, causing millions of messages to be backlogged in the message middleware, how should you deal with it?
Have you encountered a production failure with a message backlog on the line? If you have not encountered it, consider how to deal with it. ?
The first thing is to find out what caused the accumulation of messages, whether it is caused by too many Producers, too few Consumers, or other situations. In short, locate the problem first.
Then check whether the message consumption rate is normal. If it is normal, you can temporarily solve the problem of message accumulation by launching more consumers online
Follow-up: What should I do if the Consumer and Queue are not equal, and multiple units are online but cannot consume the accumulated messages in a short time?
- Prepare a temporary topic
- The number of queues is several times the accumulation
- Queue is distributed to multiple Brokers
- Launch a Consumer as a message porter, move the messages in the original Topic to the new Topic, do not do business logic processing, just move it over
- N Consumers are online and consume the data in the temporary topic at the same time
- Fix bug
- Restore the original Consumer and continue to consume the previous Topic
Follow-up: The accumulation time is too long and the message timed out?
The message in RocketMQ will only disappear when the commitLog is deleted, and will not time out. That is to say, messages that have not been consumed will not be deleted over time.
Follow-up: Will the accumulated messages enter the dead letter queue?
No, the message will enter the retry queue (%RETRY%+ConsumerGroup) after the consumption fails, 18 times (default 18 times, all articles on the Internet say 16 times, without exception. But I don’t understand why it is 16 Second, isn’t it 18 hours?) before entering the dead letter queue (%DLQ%+ConsumerGroup).
The source code is as follows:
public class MessageStoreConfig {
// 每隔如下时间会进行重试，到最后一次时间重试失败的话就进入死信队列了。
private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
}
15. The underlying principle of RocketMQ's distributed transaction support mechanism?
Are you using RocketMQ? A great feature of RocketMQ is its support for distributed transactions. Tell me about the underlying principle of supporting this mechanism in distributed transactions?
Transactions in distributed systems can use TCC (Try, Confirm, Cancel), 2pc to solve the atomicity of messages in distributed systems
RocketMQ 4.3+ provides distributed transaction functions, through RocketMQ transaction messages, the final consistency of distributed transactions can be achieved
RocketMQ implementation method:
**Half Message: **Preprocess the message, when the broker receives such a message, it will be stored in the message consumption queue of RMQ_SYS_TRANS_HALF_TOPIC
**Check the transaction status: **Broker will start a timed task to consume messages in the RMQ_SYS_TRANS_HALF_TOPIC queue. Each time the task is executed, the transaction execution status (commit, rollback, unknown) will be confirmed to the message sender. If it is unknown, the Broker will Go to the callback regularly and recheck.
**Timeout: **If the number of checkbacks is exceeded, the message will be rolled back by default.
That is, he did not actually enter the Topic queue, but used a temporary queue to put the so-called half message, and after the transaction was submitted, the half message was actually transferred to the queue under the topic.
16. If you are asked to implement a distributed messaging middleware, how would you design and implement the overall architecture?
I personally think to answer from the following points:
- Need to consider the ability to quickly expand and naturally support clusters
- Persistent posture
- High availability
- Data 0 loss considerations
- Easy to deploy on the server side and easy to use on the client side
17. Have you read the source code of RocketMQ? If you have read it, tell me your understanding of RocketMQ source code?
If you really want me to tell me, I will complain. First of all, there is no comment. Maybe Alibaba wrote a Chinese comment before. After donating it to apache, apache felt that the Chinese comment could not be kept, and I was too lazy to write English comments, so I gave them all. deleted. The more typical design patterns are singleton, factory, strategy, and facade patterns. Singleton factories are everywhere, and strategies are impressive. For example, when sending and consuming messages, the load balancing of queue is N strategy algorithm types, including random, hash, etc. This is also one of the necessary reasons to be able to quickly expand the natural support cluster. The persistence is also relatively complete, using CommitLog to place the disk, synchronously and asynchronously.
18.How to optimize the performance of producers and consumers under high throughput?
Development
- Multi-machine deployment and parallel consumption under the same group
- A single Consumer increases the number of consumer threads
- Bulk consumption
- Message batch pull
- Business logic batch processing
Operation and maintenance
- Network card tuning
- jvm tuning
- Multithreading and cpu tuning
- Cache Page
19. Let me talk about how RocketMQ guarantees the high fault tolerance of data?
- When fault tolerance is not enabled, poll the queue for sending. If it fails, filter the failed Broker when retrying
- If the fault tolerance strategy is enabled, RocketMQ's prediction mechanism will be used to predict whether a Broker is available
- If the Broker that failed last time is available, then the Broker's queue will still be selected
- If the above situation fails, randomly select one to send
- When sending a message, it will record the time of the call and whether an error is reported, and predict the available time of the broker based on the time
In fact, it is the choice of queue when sending a message. The source code is as follows:
org.apache.rocketmq.client.latency.MQFaultStrategy#selectOneMessageQueue()
20. What should I do if any Broker goes down suddenly?
Broker master-slave architecture and multi-copy strategy. After the master receives the message, it will synchronize it to the slave, so that there is more than one message, and the master is down and the message in the slave is available, which ensures the reliability and high availability of MQ. And Rocket MQ4.5.0 has supported the Dlegder mode since the beginning, based on the raft, to achieve a real sense of HA.
21. Which NameServer does Broker register its information with?
Asking this is obviously pitting you, because Broker will register its own information on all NameServers, not one, but every one, all!