1. mqnamesrv starts 2 services on a single machine, you can specify the port number through -c xxx.properties (for example: listenPort=9877, the default port number is 9876)
2, master, slave asynchronous replication mode
The master is down, and the slave cannot automatically switch to the master. In a single master/slave mode, production messages are affected;
The slave is down and has no effect at all.
3. Producer production message polling queue strategy
Default: Random polling queue (Roundbin), maintain an auto-increment id globally, auto-increment once a message is sent, and modulo the queueSize to get the sending queue;
Custom: When sending a message, the custom messageQueueSelector class selects a specific queue according to certain rules, which can achieve sequential messages,
For example, the same order number is sent to the same queue, the code is as follows:
producer.send(msg, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Integer id = (Integer) arg;
int index = id % mqs.size();
return mqs.get(index);
}
}, orderId);
illustrate:
(1) Sequential message means that the order of consumption is consistent with the order of produced messages. Generally, it refers to the order of a type of message (such as the same order number). The producer sends a single thread in sequence and sends it to the same queue, and the consumer can send it according to the producer. sequential consumption.
(2) Disadvantages of sequential messages are: (a) queue hotspot problem, in applications with a relatively large amount of messages, individual queues are not evenly hashed, resulting in a large number of messages accumulating (b) Producer sending messages cannot take advantage of high availability, and the failure of queue sending is not possible. Will retry (c) when encountering a consumption failure message, it cannot be skipped, and the direct queue consumption is suspended
The above shortcomings, the Ali team will improve in the fourth version of Rocketmq.
4. Consumer message polling queue strategy
(1) There are two modes of RocketMQ message subscription, one is Push mode, that is, MQServer actively pushes to the consumer; the other is Pull mode, that is, the consumer actively pulls from MQServer when needed.
However, in the specific implementation, the Push and Pull modes are both actively pulled by the consumer.
(2) Multiple consumers in the same ConsumerGroup consume queue messages on average, and the allocation strategy is as follows:
If there are 5 queues and 2 consumers, then the first consumer consumes 3 queues, and the second consumer consumes 2 queues.
The average allocation strategy is adopted here, which is similar to the process of paging. All queues under TOPIC are records, and the number of consumers is equivalent to the total number of pages. So how many records per page is similar to that of a certain Consumer. Which queues to consume.
(3) At the same time, the consumer will use the RebalanceService thread to do all the queue loads under the topic once every 10 seconds, get the number of all Consumer instances under the same Consumer Group or whether the number of Topic queues has changed, and notify all Consumer instances. Redo the load balancing algorithm.
Reference: https://www.zhihu.com/question/52023315/answer/131450604
5. To query the default configuration of broker and namesrv, the commands are as follows:
sh mqbroker -p
sh mqnamesrv -p
6. Send timed messages
Rocketmq can set different levels to send messages regularly, configured in the broker, the default is: messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
level=1 means level 1 delay, level=2 means level 2 delay, and so on, the default delay level is 0, that is, no delay.
For example, when the client sends a message, it is sent with a delay of 10s. The demo code is as follows:
Message msg = new Message(topic, tags, keys, body);
msg.setDelayTimeLevel(3);
SendResult sendResult = producer().send(msg);
7. Failed to consume messages, retry the consumption mechanism regularly
(a) Send the consumer message regularly according to the delay Level of the broker. After testing, if the business processing on the consumer side fails, it returns ConsumeConcurrentlyStatus.RECONSUME_LATER.
The broker will periodically send the consumer to retry the consumption according to the set messageDelayLevel, and check the consumption progress of the broker's consumeProgress. The consumeOffset and brokerOffset have been increasing (I don't know why it is designed this way?),
End the consumption of this message until it returns ConsumeConcurrentlyStatus.CONSUME_SUCCESS; if the sent message is set with DelayTimeLevel, the broker will retry from the delay time corresponding to the messageDelayLevel after the set level;
(b) There is also a number of retries (default 16)
Both satisfy one of the messages into the dead letter queue.
8. The producer fails to send a message and retry mechanism
When an exception is thrown (RemotingException, MQClientException, MQBrokerException), the Producer will select another queue to send the message, and the following conditions must be met when retrying to send the message:
The number of retries < retryTimesWhenSendFailed+1, note: retryTimesWhenSendFailed must be less than the number of queues, and the queue will exit the retry directly after retrying (setting method: mqProducer.setRetryTimesWhenSendFailed(xxx), the default is 2 [version: 3.5.8])
When an exception is not thrown, the SendStatus status is returned normally, but sendResult.getSendStatus() != SendStatus.SEND_OK, according to retryAnotherBrokerWhenNotStoreOK=true [default is false], retry other queues before sending, and the retry logic is the same as above.
Note: Some netizens wrote that this condition needs to be met at the same time, the total time (including the time required to retry n times) < sendMsgTimeout (parameters passed in when sending a message), which is not seen in the source code. I just see sendMsgTimeout as the timeout for sending a message once, and do not participate in the calculation of the retry policy.
This is debatable, the source code has not been studied in depth, maybe I have missed something.
9. Inconsistent versions of broker and client/producer
It is found that the broker/namesrv version is 3.5.8, and the client/producer version is 3.6.2.final. At this time, the production message is normal, and the message can also be consumed, but the consumption progress can be viewed through the ConsumerProgress. The produced messages are all piled up, but the consumer has indeed been consumed;
Regardless of whether the set ConsumeFromWhere is CONSUME_FROM_LAST_OFFSET or CONSUME_FROM_FIRST_OFFSET, restarting the consumer will re-consume messages, and messages keep accumulating. After changing the version number of client/producer, it is normal.
10. Start and close broker, namesrv
#start namesrv
nohup sh mqnamesrv [-c xxx.properties] > mqnamesrv.log 2>&1 &
#Start broker[ip:port is the ip and port number of namesrv]
nohup sh mqbroker -c xxx.conf -n ip:port[;ip1:port1] > mqbroker.log 2>&1 &
Some problems with the broker were found in the test:
(1) NamesrvAddr=ip:port[;ip1:port1] can be set in xxx.conf, and -n ip:port[;ip1:port1] is not required when starting the broker
(2) Because the broker is deployed in the virtual machine and the virtual dual network card, the client cannot connect to the server normally (because the broker obtains the server's external ip when it starts),
Therefore, you need to set brokerIP1=192.168.56.101 in xxx.conf to explicitly specify the local IP; to deploy multiple brokers on the same server, you need to explicitly specify the port; listenPort=20911 (default is 10911).
Available configuration references are as follows:
namesrvAddr=192.168.56.101:9876
brokerIP1=192.168.56.101
listenPort=10911
........
Description: The default port of the broker is 10911, and the broker will use 10910 [mqadmin connection use], 10912 port [slave connection use]; so please avoid it when setting other broker port numbers.
shutdown command:
sh mqshutdown namesrv
sh mqshutdown broker
The above command will shut down all namesrv or brokers in the cluster. If only broker_slave is shut down in the test, you can use kill pid.
11. Queue number setting
It is set when the producer sends a message. Special attention: the same topic is only valid when it is created for the first time, and the subsequent modification is invalid unless the consume.json file on the broker server is modified.
The demo code is as follows: mqProducer.setDefaultTopicQueueNums(5)
Reference: http://www.mamicode.com/info-detail-327693.html
12. Setting instructions for ConsumeFromWhere
//A [new subscription group] starts to consume from the [front] position of the queue for the first time, and then starts to consume the progress of the last consumption.
ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET
//A [new subscription group] starts to consume from the [last] position of the queue for the first time, and then starts to consume the progress of the last consumption.
ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET
13. Message storage structure
The two main files are consumeQueue and common log:
(1)consume Queue
The consume queue is a logical queue of messages, which is equivalent to a directory of dictionaries. It is used to specify the location of the message on the physical file commit log, which is set through the broker configuration file.
We can specify the directory where consumequeue and commitlog are stored in the configuration. Each queue under each topic has a corresponding consumequeue file, for example: ${user.home}/store/consumequeue/${topicName}/${queueId }/${fileName}, you can configure storePathCommitLog=xxx in the broker
The storage unit in the Consume Queue is a 20-byte fixed-length binary data, which is written and read sequentially. The storage format is as follows:
| ------------ 8Byte ------------ | ------ 4Byte ---- | ----------- ---- 8Byte ------------ |
|------CommonLog Offset-------|------Size-----|------MessageTag HashCode-------|
(2)commit Log
The physical file where the message is stored, the commitlog on each broker is shared by all the queues of the local machine, without any distinction. The default location of the file is ${user.home}/store/commitlog/${fileName}, you can configure storePathConsumeQueue=xxx in the broker
The length of the message storage unit of CommitLog is not fixed, and the files are written sequentially and read randomly.
(3) Message index file
If a message contains a key value, it will use IndexFile to store the message index. The index file is mainly used to query messages based on the key. The default location of the file is as follows: ${user.home}/store/index/${fileName}, which can be found in Configured in the broker, you can configure storePathIndex=xxx in the broker
Reference: http://www.jianshu.com/p/453c6e7ff81c
14. Message queue and message storage
Rocketmq's message queues are all persistent, data structures of unlimited length (each storage unit is fixed length).
Because the message is stored in the message queue, and it has the ability to scale horizontally, and is regularly deleted according to the expiration time, the size of the message storage is related to the size of the disk. Assuming that 1K messages are counted, 96G of memory, the physical memory can cache 100 million messages.
15. Available management commands
View the consumption progress in the group: sh mqadmin consumerProgress -g xx
View the consumption instances in the group: sh mqadmin consumerConnection -g xx
View cluster status: sh mqadmin clusterList
View topic list: sh mqadmin topicList
View topic routing: sh mqadmin topicRoute -t xx
Check the status of the topic: sh mqadmin topicStatus -t xx
View cluster information for a topic: sh mqadmin topicClusterList -t xx
View the internal data structure of the consumer instance: sh mqadmin consumerStatus -g xx -i clientId
Eliminate write permissions on a broker (used to safely shut down the broker): sh mqadmin wipeWritePerm -b xx
Reset consumption offset by timestamp (for retroactive consumption): sh mqadmin resetOffsetByTime -g xx -s yy -t tt
Note: You can also query all the above data through the rocketmq-console page console.
16. Description of broker cluster deployment configuration [recommended to use multiple masters and multiple slaves, asynchronous replication mode]
The Master and Slave are paired by specifying the same brokerName parameter. The BrokerId of the Master must be 0, and the BrokerId of the Slave must be a number greater than 0.
Item 10 said that if the server has multiple network cards, it may be necessary to set the broker IP address of the local machine, and the namesrvAddr address can be set at the same time; each broker
broker_a_master.properties can be configured as follows:
brokerClusterName=DefaultCluster
brokerName=broker-a
namesrvAddr=192.168.56.101:9876;192.168.56.101:9877
brokerIP1=192.168.56.101
listenPort=10911
brokerId=0
deleteWhen=04
fileReservedTime = 48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/work/alibaba-rocketmq/store_a_master
storePathCommitLog=/work/alibaba-rocketmq/store_a_master/commitlog
storePathConsumeQueue=/work/alibaba-rocketmq/store_a_master/consumequeue
Note: Different brokers, if deployed on the same server, need to specify different storePathXXXX paths.
17. The role of Producer/Consumer Group
Producer Group：
(1) You can query how many Producer instances there are in this group through the operation and maintenance tool, command: sh mqadmin producerConnection;
(2) Transaction messages. If the Producer unexpectedly goes down, the Broker will call back any machine in the Producer Group to confirm the transaction status.
Consumer Group:
(1), You can query the consumption progress under this group through the operation and maintenance tool, how many Consumer instances, command: sh mqadmin consumerProgress/consumerConnection -g xxx
(2) In cluster mode, multiple consumers under a Consumer Group consume messages equally; in broadcast mode, group is meaningless.
18. Retrospective consumption
The consumer has successfully consumed the message. Due to business requirements, the broker needs to provide a mechanism to roll back the consumption progress according to the time dimension. Rocketmq supports the precision to milliseconds. Messages can be traced forward or backward
Use the command: sh mqadmin resetOffsetByTime -g xx -s yy -t tt
19. Message filtering
There are three ways to filter on the broker side:
(1) Filter by Message Tag, * represents all types of messages, xxx||yyy uses || to separate and consume different types of messages;
(2) Messages can be filtered through Message Header/body (official documentation, not tested);
(3) Pass the filter class defined by the client to the broker to filter the message. The detailed steps are as follows:
<a>The server where the Broker is located will start multiple FilterServer filtering processes;
<b>Pass the custom filter class to FilterServer when the Consumer starts up;
<c>Consumer pulls messages from FilterServer, FilterServer forwards the request to Broker, and after FilterServer receives the message, it filters according to the filter program uploaded by Consumer, and returns the consumption to Consumer.
The test code is as follows:
String filterCode = MixAll.file2String("/home/admin/MessageFilterImpl.java");
consumer.subscribe("TopicFilter7", "com.alibaba.rocketmq.example.filter.MessageFilterImpl", filterCode);
20. Turn off autoCreateTopicEnable in online broker configuration (official recommendation is to turn it off)
When RocketMQ sends a message, it will first obtain routing information. If it is a new message, since the corresponding Topic has not been created on MQServer,
At this time, if the above configuration is turned on, it will return the default TOPIC (RocketMQ will create a TOPIC named TBW102 on each broker) routing information.
Then the Producer will select a Broker to send the message. When the selected broker is storing the message, it will automatically create the topic when it finds that the topic of the message has not been created.
The consequence is: in the future, all the messages of the TOPIC will be sent to this broker, which cannot achieve the purpose of load balancing.
Reference: http://www.jianshu.com/p/453c6e7ff81c
21. The consumption process must ensure idempotency (de-duplication at the consumer end), and the Consumer should consume messages in batches as much as possible (set the ConsumerMessageBatchMaxSize of the Consumer [default is 1])
22、NameServer/Broker/Producer/Consumer关系
Reference: http://www.cnblogs.com/tommyli/p/5081846.html
23. 3 types of messages sent
See class CommunicationMode,
SYNC,
ASYNC,
ONEWAY
The producer.send(msg, SendCallback) method with the SendCallback class parameter is to send messages asynchronously;
producer.sendOneway(msg);
Otherwise send the message synchronously (default is to send the message synchronously).
24. Dead letter queue
Group dead letter queues according to the consumer's GroupName. If the consumer fails to consume and retries the specified number of times (default 16), it will enter the dead letter queue. Can be modified by the command updateSubGroup
Others: You can also set the number of retry queues of ConsumeGroup through updateSubGroup (default 1).
.... to be concluded later
appendix:
1. The reference website is as follows
http://www.jianshu.com/p/453c6e7ff81c
http://hiant.github.io/2016/08/26/rocketmq-0x5/
http://xingxiudong.com/2015/05/18/rocketmq-message-delay-config/
https://catslave.github.io/rocketmq/2016/08/15/RocketMQ.NameServer.html
http://www.tianshouzhi.com/api/tutorials/rocketmq
http://www.uml.org.cn/zjjs/201504011.asp
https://github.com/alibaba/RocketMQ/wiki/CLI-Admin-Tool
http://blog.csdn.net/lang_man_xing/article/details/47447797
problem solved:
http://blog.csdn.net/a417930422/article/details/50663639
http://www.mamicode.com/info-detail-327693.html
2、rocketmq-console.war
http://pan.baidu.com/s/1hs9i0lA
Final note: Don't believe the default values in the official PDF documents, different versions may be different, and the source code is the main source.