This chapter focuses on reliability, message queues to solve how to meet the needs of the business logic, while stable and reliable long-term operation.
Order news
Message sequence refers to the order message consumption and generation of the same sequence, in some business logic, we must guarantee the order. For example, producing orders, payments, shipments of these three messages have to follow the order process. Order message into a global order in which messages and partial sequence information, refer to the global order in which messages are all messages in a topic must ensure that sucking line; to ensure that every part of the order as long as the message is a message sequence consumer group can, for example, the above order, as long as guarantee of the same order of id can order three messages can consume.
Global order message
RocketMQ default does not guarantee the sequence, such as creating a topic, write queue default 8, 8 read queue. At this time, a message may be written to any of the queue; during the data read, there may be multiple consumer, every consumer may also start concurrent processing of multiple threads, so the message is consumer consumption which is of the order of consumption and sequential write message consistency is uncertain.
To ensure a global order in which messages need to first write the number of queues topic is set to 1, then the number of concurrent producers and consumers is also set to 1, (my default project is 20 concurrent threads), in simple terms, in order to ensure orderly global message of the whole topic, only the elimination of all concurrent processing, the parts are set to single-threaded process, then high concurrency, high throughput functional basic irrelevant.
Partial order message
To ensure an orderly part of the message, the sender, and consumers need to deal with. At the transmitting end sends the id to do a service to the same message with a Message Queue; during consumption to do with a message read from the message queue can not be processed concurrently, so as to achieve a partially ordered.
Which is sent to the sender queue to use MessageQueueSelector control messages, the following code: I have here is the default 0 queue all messages to send a message, of course, you can go to see your own load balancing strategy
package rocketmq.day05;
import com.alibaba.rocketmq.client.exception.MQBrokerException;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.client.producer.DefaultMQProducer;
import com.alibaba.rocketmq.client.producer.MessageQueueSelector;
import com.alibaba.rocketmq.client.producer.SendResult;
import com.alibaba.rocketmq.common.message.Message;
import com.alibaba.rocketmq.common.message.MessageQueue;
import com.alibaba.rocketmq.remoting.exception.RemotingException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
/**
* @author heian
* @create 2020-01-13-4:34 下午
* @description 消息发送到同一个消息队列
*/
public class MessageQueueSelectorDemo {
public void sendMsgDemo() throws MQClientException, RemotingException, InterruptedException, MQBrokerException {
DefaultMQProducer producer = new DefaultMQProducer("unique_producer_group__name");
producer.setInstanceName("instance1");
producer.setRetryTimesWhenSendFailed(3);//重试次数 192.168.138.47 192.168.142.133 192.168.0.102
producer.setNamesrvAddr("192.168.139.188:9876");//多个用；分割 192.168.138.47
producer.start();
for (int i = 0; i < 4; i++) {
int orderId = i;//消息发送的顺序
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat();
String format = sdf.format(date);
System.out.println("准备发送：" + format);
Message message = new Message("topicName", String.valueOf(i),format.getBytes());
SendResult sendResult= new SendResult();
// sendResult = producer.send(message);无需指定队列
sendResult = producer.send(message, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Integer id = (Integer) arg;
int index = id%mqs.size();
//return mqs.get(index);//负载均衡策略
return mqs.get(0);//指定某个队列
}
},orderId);
System.out.println("key:"+i + "消息的发送结果为：" + sendResult.toString() + "消息ID为：" + sendResult.getMsgId());
}
producer.shutdown();
}
public static void main(String[] args) throws InterruptedException, RemotingException, MQClientException, MQBrokerException {
MessageQueueSelectorDemo demo = new MessageQueueSelectorDemo();
demo.sendMsgDemo();
}
}
MessageListenerOrderly consumer side is used to solve the problem of a single message queue to be processed concurrently:
package rocketmq.day05;
import com.alibaba.rocketmq.client.consumer.DefaultMQPushConsumer;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import com.alibaba.rocketmq.client.consumer.listener.ConsumeOrderlyStatus;
import com.alibaba.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import com.alibaba.rocketmq.client.consumer.listener.MessageListenerOrderly;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.common.consumer.ConsumeFromWhere;
import com.alibaba.rocketmq.common.protocol.body.ConsumeStatus;
import com.alibaba.rocketmq.common.protocol.heartbeat.MessageModel;
import java.io.UnsupportedEncodingException;
/**
* @author heian
* @create 2019-12-09-8:12 上午
* @description
*/
public class MessageQueueSelectorConsume {
public static void main(String[] args) throws MQClientException {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("unique_consume_group_name");
consumer.setNamesrvAddr("192.168.139.188:9876");//
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.setMessageModel(MessageModel.CLUSTERING);//默认是集群模式
consumer.subscribe("topicName",null);//MessageListenerConcurrently 并发处理消息 监听类
consumer.registerMessageListener((MessageListenerOrderly) (listMsg, consumeOrderlyContext) -> {
byte[] body = listMsg.get(0).getBody();
try {
String ms = new String(body,"utf-8");
System.out.println(Thread.currentThread().getName()+"收到消息：" + ms);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return ConsumeOrderlyStatus.SUCCESS;//ConsumeConcurrentlyStatus.CONSUME_SUCCESS
});
consumer.start();
}
}
Production side load balancing (not sending the same message queue)
--发送端
key:0消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F00000000000727BA, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=0], queueOffset=55]消息ID为：C0A88BBC00002A9F00000000000727BA
key:1消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F000000000007285A, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=1], queueOffset=17]消息ID为：C0A88BBC00002A9F000000000007285A
key:2消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F00000000000728FA, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=2], queueOffset=13]消息ID为：C0A88BBC00002A9F00000000000728FA
key:3消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F000000000007299A, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=3], queueOffset=13]消息ID为：C0A88BBC00002A9F000000000007299A
--消费端
ConsumeMessageThread_13收到消息：20-1-13 下午5:47:3
ConsumeMessageThread_14收到消息：20-1-13 下午5:47:2
ConsumeMessageThread_15收到消息：20-1-13 下午5:47:1
ConsumeMessageThread_16收到消息：20-1-13 下午5:47:0Production side designated queue (transmission queue message is not the same)
--发送端
key:0消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F0000000000072A3A, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=0], queueOffset=56]消息ID为：C0A88BBC00002A9F0000000000072A3A
key:1消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F0000000000072ADA, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=0], queueOffset=57]消息ID为：C0A88BBC00002A9F0000000000072ADA
key:2消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F0000000000072B7A, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=0], queueOffset=58]消息ID为：C0A88BBC00002A9F0000000000072B7A
key:3消息的发送结果为：SendResult [sendStatus=SEND_OK, msgId=C0A88BBC00002A9F0000000000072C1A, messageQueue=MessageQueue [topic=topicName, brokerName=bogon, queueId=0], queueOffset=59]消息ID为：C0A88BBC00002A9F0000000000072C1A
--消费端
ConsumeMessageThread_2收到消息：20-1-13 下午5:57:0
ConsumeMessageThread_2收到消息：20-1-13 下午5:57:1
ConsumeMessageThread_2收到消息：20-1-13 下午5:57:2
ConsumeMessageThread_2收到消息：20-1-13 下午5:57:3Consumer end the MessageListenerOrderly when the following four consumer settings can still use: two parameters to set the number of threads consumption thread before setConsumeThreadMin setConsumeThreadMax setPullBatchSize setConsumeMessageBatchMaxSize, PullBatchSize refers to the maximum number of messages from a queue get a message broker, the default is 32, ConsumeMessageBatchMaxSize refers to the consumer in the Executor (i.e. where call processing MessageListener) a number of inbound (List <MessageExt> msgs maximum length of the list) message, the default value is 1.
Note that MessageListenOrderly refers not simply prohibit concurrent processing, achieving MessageListenOrderly in, added a lock for each consumer queue, each message before consumption, we need to get the message corresponding lock, thus ensuring the same time consumption is not the same message queue concurrent consumption, but different consumer queue can be complicated by the consumer, which is why no queue and there is no reason for the order of consumption.
Consumer duplication
For distributed message queue, it must at the same time do not repeat delivery and delivery is difficult, in the case of fish and can not have both, rocketmq choose a certain delivery, to ensure that messages are not lost, but may result in duplication consumption. For example, you state the consumer does not return success, that this provision of consumer information how the second time until successful. Resolution message repeated consumption in two ways: The first is to ensure that the consumer logic of power and so on, the second is a consumer safeguard their own record: You have to consume a row of data in a table, you can add a consumption status code 00 consumer consumption 02 to 01 completed consumer, before the consumer go to the next query whether this news is consumed before, then see the message or not to discard. Of course, these are the users themselves to achieve.
Dynamic changes in the machine
A Message Queue cluster composed of multiple machines, continuous and stable provision of services, as business needs or hardware failure, often need to increase or decrease the role of the machine, the machine dynamic changes in how the case under the following does not affect the server.
Dynamic changes in NameServer
NameServer coordinator of rocketmq into the group, the individual components of the cluster is to obtain various properties and address information through NameServer, the main function is divided into two parts: the individual broker regularly report their status information to NameServer; the other is the individual client, This is what we project producers, consumers, and a command-line tool for the latest status information NameServer. So they have to tell NameServer address before starting the broker, producers and consumers, in order to improve reliability, it is recommended to start multiple NameServer, because much of its resources occupied, and the broker can be deployed on the same machine. Once you have multiple NameServer, a decrease NameServer no effect on other components.
There are four ways to set NameServer address below in descending order of priority mode introduced in turn:
- By setting the code, such as producers by producer.setNamesrvAddr ( "ip1: port1; ip2: potr2"); in mqadmin tool by -n ip1: port1; ip2: port2 parameters are set, if the custom command tools, also by defaultMQAdminExt.setNamesrvAddr ( "ip1: port1; ip2: port2") set
- Using java startup parameters, corresponding to the option is rocketmq.namesrv.addr
- Linux environment variable parameters through this, set the variable before starting: NAMESRV_ADDR
- Set by the http service, when the above methods are not used, the program will want to send a request to http address to get NameServer address, the default URL is http: //jmenv.tbsite.net.8080/rocketmq/nsaddr (Taobao test address) , covered by rocketmq.namesrv.domain jmenv.tbsite.net parameter; parameter sadder covered by rocketmq.namesrv.domain.subgroup
The fourth way may seem tedious, but it only supports dynamic increase NameServer, without having to restart the other components. Each request for two minutes using this method other components will be a URL, obtain the latest NameServer address.
Dynamic changes in broker
As the business grew, the time required for the expansion of the cluster can dynamically increase the role of the broker machine. An increase of only broker will not affect the original topic, originally created good reading and writing of data in the topic is still carried out on those who broker the original.
After the cluster expansion, the one that can put new topic assigned to the new broker machine, balanced use of resources, so that is one way to update an existing topic by updateTopic configuration command to create a new queue on the new broker. For example TestTopic is a topic of current theme, because of the need to increase the amount of data expansion, new additions broker machine address: 192.168.0.1: 10911, this time to execute the following command: result in new broker machine to create eight new TestTopic read and write queue
sh ./bin/mqadmin updateTopic -b 192.168.0:10911 -t TestTopic -n 192.168.0.100:9876Because the business changes or replacement of the machine need to reduce the broker, this time how does it work? Reduce the broker to see whether the producers have continued to run, when a topic is only one master broker, the broker to stop off, messages sent will be affected, need to stop this before the broker, stop sending messages.
When a topic there are multiple master broker, to stop one of them, whether this time will be lost news? There are ways used to send messages and producers answer relationship, if the use of synchronous send (MSG), has automatic retry logic within DefaultMQProducer, wherein a broker stopped, it will automatically send a message to another broker, not the phenomenon of lost messages to occur. If the send message send (msg, callback), or a sendOneWay embodiment, will be lost in the handover process messages asynchronously. Since in the asynchronous and sendOneWay two ways, Producer.setRentryTimesWhenSendFailed setting does not work, does not retry transmission failure. DefaultMQProducer NameServer default 30 seconds to request the latest routing message, obtaining queue information producer if not stopped at the Broker, longer follow-up automatically sends a message to the queue.
If the program producer can be suspended, it can be switched smoothly in a case where the master and slave. After the producer can close shut master broker, this time to read all the messages will be slave to the machine, the message consumption is not affected (can not write messages to the backup machine), after the completion of the replacement master broker, start the master data based on the original broker, then send the message in the normal start producers.
Kill pid command with Linux on it to close properly broker, there is a shutdown function under BrokerController, this function is added to the ShutdownHook years, when using the kill command of Linux (can not kill -9), shutdown function will first be performed either mqadmin tools provided by RocketMQ to shut down broker, the principle is the same.
The impact of various fault messages
We hope that the message queue has been stable cluster can run, but sometimes the fault is always inevitable, here is the fault conditions may occur, see how treatment
1, broker normally closed, and then restart
This situation belongs controllable software problems, the data in memory will not be lost, automatically standby switching. If consumers continue to run, master broker failure to restart the process, the end consumer will be automatically connected to the corresponding slave machine, the message is lost and there will be no deviation. When the master broker restart the machine, the consumer side and back to reconnect to the master broker. But note that: the master broker starts when the consumer correct information from the slave broker consumption, not to forcibly stop the consumer end consumer, if the consumer end stop spending, start this time after the master broker in consumer start, at which point it will read the offset value on the master machine has lagged behind, then before consumption over the news and back again over consumption caused by repeated consumption again.
If the first case there are favorite producers continuously running a fault Broker master producer synchronous mode if the transmission message, a retry would be the other master machine sends a message to the topic, if not asynchronously retry will be lost during the handover message, as it name server every 30 seconds to obtain routing information, if the information acquired in the master broker queue has not stopped, subsequent to this queue does not send messages.
2, broker Crash abnormal, then start; os crash, reboot; power off the machine, but it can immediately restore power;
In a software fault, memory data may be lost, so different brush plate strategies, the impact is different. If the master, slave are configured to be identical to the first SYNC_FLUSH same, then the message will be lost asynchronous brush disc appears.
3, damage to the disk; CPU, motherboards, memory and other critical damage
Is a hardware failure, the original machine disk data may be lost, if the master and the slave machine configured the way synchronous replication occurs above a machine malfunction, messages are not lost, if it is asynchronous replication, messaging naturally can not sync, so at this time when the copy of the message will be lost.
Overall, the master-slave master in order to ensure high availability from the best way to synchronous replication, synchronization is the best way to message producers to write, brush disk brush can also take synchronous disk (less efficient) can be eliminated single point of failure, even if a device does not appear extreme situations message is lost!
Message priority
Applications need to handle several types of messages in some scenarios, different messages of different priority levels. rocketmq is a FIFO queue, the message does not support the priority level or topic. Simple business priority needs to be addressed through indirect means.
The first relatively simple case, when the topic if there are many similar types of messages, such as AB, AC, AA, when the great AB, AC's message, but the message processing is slow when there will be a lot of queue AB, AC type of message waiting, if at this time a small amount of AA added to the queue, it will at the back AB, AC type message, to wait a long time before being treated (FIFO fifo). If traffic needs to AA types of messages in time, can be similar types of messages are split into two in the topic, such as AA type in the message in topicaA, AB, AC topicbB placed, and then from the consumer side two two were subscribed topic, not because AB, AC type too many messages to be delayed.
The second case and a similar, but without creating a lot of topic, give an actual application scenario: an order system, accepting requests 100 Express stores over, these written request to mq, order processing by the producer read by the consumer from the message queue and process the message, a single process up to 10,000 per day, if a certain store 100 express several booms, such as a store to an order request issued by 20000, so that other 99 stores and so it may be forced to deal with a store did not finish, that is, two days after the order to be processed, which is obviously unfair. This time can create a topic, topic of the set number of message queue 100, 100 stores this correspondence, the producer writes the message queue corresponding to the specified, the default is to use the consumer side DefaultMQConsumer cyclic manner to read a topic All the queues, while the sharp increase in the traffic message store process becomes not ye, but does not affect normal processing of the other stores. DefaultMQConsumer default PullBatchSize 32, which is read each time from 32 up to read the message, to this parameter may be fairer to 1consume.setPullBatchSize (1), but the throughput is low.
The third case is a strong demand priority, the priority of the above cases the message is not high, like a mechanism to ensure fair handling, processing certain messages to avoid due to the larger amount of blocking other messages. Existing simultaneously process a, b in an application program, c three types of messages, a first priority in a, b, c for the second and third priority. For this the user is required to achieve their encoding the priority control. If the above three types of messages on a topic which may be used to pull to the pull mode, autonomous control MessageQueue queue traversal message read message; if the message inside the three topic, it is necessary to start this three consumption, achieve control three consumer end consumer priority.
