Foreword
This article would like to introduce some time ago in writing enode a distributed message, the way to achieve the queue equeue. This idea I think the message queue is not out, but by Ali after learning of rocketmq, their use c # to achieve a simple version of a lightweight. On the one hand can make yourself more in-depth grasp of some of the frequently asked questions by writing the message queue of the queue; it can also be used and enode integration, support for messaging enode in command and the domain event. Currently in the .net platform, relatively easy to use message queues, the most common is Microsoft's MSMQ it, there are also client-side like rabbitmq of .net. The message queues are very strong and mature. But when I learned kafka and Ali rocketmq (an earlier version called metaq, since renamed rocketmq metaq 3.0), that rocketmq design ideas attracted me, because I not only understand their thoughts, as well as its complete source code can be learned. But rocketmq is written in java, and there is no client-side .net, it can not be used directly (for their friends who are interested can write client-side of a .net), so learning the rocketmq design documentation, and most of the code after determine their own with c # to write an out.
The open source address: https://github.com/tangxuehua/equeue , project queue contains all the source code and examples of how to use. Enode project can also be seen in how to use.
Glossary equeue message queue
Topic
A topic that is a theme. A system, we can be divided into a number of news topic, so that we can by topic, send messages to a different queue.
Queue
The next topic, we can set a plurality of queue, each queue is what we usually call message queue; because queue is entirely dependent on a specific topic, so when we want to send a message, the message always specify What topic belongs Yes. Then equeue will be able to know how many queue of this topic. But in the end sent to which queue it? For example, there is a topic the next four queue, the message that for under this topic, send, in the end of the queue to which to send it? There must be a process that message is routed. Equeue current practice is to send a message, the user needs to specify the topic and a message corresponding to a type of object parameters for the route. The topic equeue get all the queue, and then a modulo hash code based on the number of queue object parameter finally obtained number of queue to be transmitted to the transmission queue to know which. This process of routing messages is to send a message in this party do, that is to say the following producer. The reason is not on the news server is done because it allows the user to decide how to route messages with greater flexibility.
Producer
Is the message queue of producers. We know that the essence of the message queue is to achieve a publish-subscribe model, that producer - consumer model. Producers to produce news, consumer spending news. Producer So here is used to produce and send a message.
Consumer
Is the message queue of the consumer, a message can have multiple consumers.
Consumer Group
Consumer groups, which may be for all of us is a new concept. To come up with a reason why consumer groups, in order to achieve the following crowdsumption say. A consumer group contains some consumers, if consumers are to be clustered consumption, and that these consumers will average in the packet of information consumption.
Broker
equeue the message broker in charge transfer, i.e., receiving a message sent from the producer, and persistent messages to disk, and then receives a request pull message sent from the consumer, and then pull the corresponding message to the consumer on request. Therefore, as will be appreciated Broker message queue servers, provide a message reception, storage, pull service. Visible, broker for equeue is the core, it must not hang, hang up once, that producer, consumer can not achieve a publish-subscribe.
Crowdsumption
Cluster consumption means that a consumer under consumer group, the average queue under the topic of consumption. Specifically, how the average can look at the architecture diagram below, where the first briefly describe in words. If there is a next topic queue 4, and there is currently a Group consumer, the consumer has four of the packet, that each consumer is assigned to a queue in this topic, so that to achieve the queue at the average consumption topic purpose. If the consumer group at only two consumer, that each consumer will consume 2 queue. If there are three consumer, the consumer first two queue, two behind each consumer a queue, so as to achieve as far as possible the average consumption. So, we can see that we should try to make the number of queue number and topic of the consumer under the same consumer group or a multiple relationship. In this way the number of each consumer consumption queue is always the same, so the pressure will be about the same for each consumer server. Provided that the current number of each queue at this topic in the news almost always more. This is something we can route messages to hash key according to a user-defined to ensure.
Broadcast Consumer
Broadcast Consumer means a consumer can subscribe to a certain topic of the news that it will receive all the queue at the topic in the news, regardless of what the consumer's group Yes. So for the consumer for broadcast, consumer group no practical significance. consumer may at instantiation, we can specify a cluster or broadcast consumer spending.
Consumer progress (offset)
Consumer progress means that when a message in a queue in the consumption of a consumer group in the consumer, equeue is to know where the current consumption by recording the consumption location (offset). So that the consumer start spending continued after the restart from this position. For example, a topic there are four queue, a consumer group has four consumer, then each consumer assigned to a queue, and then separately for each consumer consumption own queue in the news. equeue will be recorded separately for each consumer consumption schedule of its queue, thus ensuring the restart after each consumer to know where to begin next continue to spend. In fact, perhaps after the next restart is not consumed by the consumer of the queue, but rather consumed by the consumer group in the other, so it does not matter, because we have recorded the location of the queue of the consumer. So we can see, consumption and consumer location has nothing to do, the consumer position is entirely a property of the queue, where consumption is currently used to record to. In addition it is important that one topic may be more consumer group in the consumer subscription. Different consumer group in the consumer consumption even with a queue under the same topic, the progress that the consumer is stored separately. In other words, different consumer consumption within the consumer group completely isolated from each other is not affected. Another point is that, for the consumer and broadcast cluster consumption of consumer durable progress of the place is different, consumer spending is on the progress of the cluster broker, that is, while consumer spending broadcast schedule is stored on the consumer's Message Queue Server on the local disk. The reason for this is because the design for the cluster consumption due to a queue of consumers may be replaced, because the consumer The number of consumer in the group may increase or decrease, then will be re-calculated for each consumer what the consumer is queue, can understand this to? So, when there is a queue of consumer changes, new consumer how to know where to start spending this queue it? If the queue is stored in the forward progress of the consumer on a consumer server, it is difficult to get the consumer progress, because it is possible that the server has hung up, or the shelf, are likely. And because the broker for all consumer is always in the service, so, in the case of a cluster of consumption, consumption locations queue is subscribed topic is stored on the broker, when stored do isolation according to different consumer group, to ensure that the consumer under a different consumer group consumer progress complementary impact. Then, for radio consumption, due to the situation will change in a queue of the consumer does not occur, so we do not need to let broker to save the consumption location, it is stored on the consumer's own server.
What equeue that?
By the map, we can intuitively understand equeue. This figure is brought from rocketmq design documentation, huh, huh. Because equeue design ideas and rocketmq completely consistent, so I took over use. Each time the producer can be a topic message, transmitted according to a certain routing policy (producer customizable) messages sent to a particular queue. Then the consumer can consume queue at a specific topic in the news. The figure above, there are two TOPIC_A consumer, the consumer is two years in a group, so the queue at the average consumer should TOPIC_A but because there are three queue, so the first consumer assigned to two queue, the second assigned to a consumer. For TOPIC_B, since there is only a consumer, all queue at that TOPIC_B by its consumption. All topic information, queue information, as well as the message itself, it is stored on the server broker. This is not reflected in the figure above. The figure focuses on the relationship between producer, consumer, topic, queue these four things, not concerned about the physical server deployment architecture.
Reflections on key issues
How to communicate between 1.producer, broker, consumer three
Because it is implemented in c #, and because typically deployed in the LAN, in order to achieve high performance communication, we can use the socket for asynchronous communication. .net itself provides good support asynchronous socket communication; we can also use zeromq to achieve high-performance socket communication. Originally intended to be used directly zeromq to implement communication modules like, but then you learn a bit .net built-in socket communication-related knowledge, not difficult to find, so on their own to achieve one, huh, huh. Benefits own realization is that I can define your own message protocol, which is currently part of the code that implements the ecommon base class library, is an independent business scenarios can be taken with the independent foundation class library. Interested can go down and look at the code download. After some of his own performance tests, we found that the performance of the communication module is good. A broker, four producer at the same time send a message to the broker, 4W messages that can be sent per second without problems, more producer have not tested.
2. What message persistence,
Persistent messages aspects main consideration is the performance issue, there is news how fast read.
1. First, the message on one broker does not require the broker has been saved on the server, because these messages will always be consumed. According to Ali rocketmq design, the default will be deleted once one day has been consumed over the news. So, we can understand the message on the broker should not be unlimited growth, as it will be periodically removed. It is not necessary to consider the issue on a broker message does not fit.
2. How fast persistent messages? In general, I think there are two ways: 1) sequential write disk files; 2) with ready-made key, value of nosql products to store; rocketmq currently using to write the file mode, the difficulty of this approach is to write file is more complex, because all messages are sequentially append to the end of the file, although the performance is very high, but also a high degree of complexity; all such messages can not all be written in a file, a file after reaching a certain size need to split, split once points will have a lot of questions, huh, huh. How to read is more complex issues after the split. Because it is written to the file as well as the order, then we also need a message every starting position and length in the file needs to be recorded, so consumer in the consumer message, to get the message from a file based on offset. In short a lot of issues to be considered. If you are using nosql to persist messages that can save all kinds of problems we encounter when writing papers, we only need to care about how the key message and the message in the queue corresponding offset up to. Another point doubt, queue in the information to be persistent it? Must first think clearly put the queue is something. When the broker receives a message, of course, is first persisted, after the completion of the need to put the message in the queue. However, because the memory is limited, we can not put this information directly into the queue, we actually want to only when the message is key to put in nosql where required, or if the file is to persist, it is the place message offset in the file offset, i.e., that position is stored in the file (such as which line number). So, in fact, queue index is just a message. Persistent queue that is necessary to do? Be persistent, so the broker after the restart when the queue recovery time can be shortened. It requires persistence and message synchronization persistence it? Obviously do not need, we can asynchronous timing of each persistent queue, and then restore the queue, you can start with the persistence of partial recovery, then put the rest by persistent message queue supplement to achieve due to the asynchronous persistence the slow part can be tied. So, after the above analysis, the message itself is placed in nosql, queue all in memory.
How that message persistence of it? I think the best way is to have each message has a global sequence number, once the message is written nosql, the global sequence number of the message on the set, and then when we update the information queue corresponding to the message global sequence number passed to queue, this queue will be able to queue their own global sequence number of the local message and the message sequence number mapping relationship. Related code is as follows:
public MessageStoreResult StoreMessage(Message message, int queueId)
{ var queues = GetQueues(message.Topic); var queueCount = queues.Count; if (queueId >= queueCount || queueId < 0)
{ throw new InvalidQueueIdException(message.Topic, queueCount, queueId);
} var queue = queues[queueId]; var queueOffset = queue.IncrementCurrentOffset(); var storeResult = _messageStore.StoreMessage(message, queue.QueueId, queueOffset);
queue.SetMessageOffset(queueOffset, storeResult.MessageOffset); return storeResult;
}Nothing more telling than the code, huh, huh. Code based on the idea that a message object and a receiving queueId, queueId represents the current message queue to put the first few years. Then the internal logic is to first of all get the message queue of the topic, because the queue and topic are in memory, so there is no performance problems. Then check the current passed in queueId is legitimate. If valid, then locate the queue, then IncrementCurrentOffset method, the internal number by 1 and returns the queue, then the message persistent, lasting time and the queueId queueOffset with persistence, it returns a completion message global sequence number. Since the content of the message will messageStore internal, queueId, queueOffset, and a global sequence number of the message stored together as a whole to the nosql, key message is global sequence number, value is in front of said whole (serialized as binary). Then, in SetMessageOffset method call queue, the message of the global queueOffset and offset mapping relationship can be established. Finally returns a result. MessageStore.StoreMessage memory to achieve approximately as follows:
public MessageStoreResult StoreMessage(Message message, int queueId, long queueOffset)
{ var offset = GetNextOffset();
_queueCurrentOffsetDict[offset] = new QueueMessage(message.Topic, message.Body, offset, queueId, queueOffset, DateTime.Now); return new MessageStoreResult(offset, queueId, queueOffset);
}GetNextOffset is to get the next global message sequence number, QueueMessage is the above mentioned "whole", because it is the realization of memory, so I used a ConcurrentDictionary to save it queueMessage object. If implemented with nosql messageStore, where the need to write nosql, key message is global sequence number, value is queueMessage binary sequence data. Through the above analysis we can see that the global serial number + queueId + queueOffset overall record as a persistent message will be together with us. This has two very good characteristics: 1) persistence of the implement position atomic transaction message in the message queue and persistence in; and 2) we can always restore the queue according to the information of all of these persistent queueMessage because queueMessage message contains the location information and a message in a queue;
Based on this message store, when a consumer to consume a location message, we can first find the queue by queueId, then queueOffset (passed over by the consumer) offset by global news get the message, then according to the Global offset as the key to get the message from the nosql. In fact now equeue bulk pull message, which is a socket request is not pulling a message, but to pull a group, the default is 32 messages. Such consumer can use less network requests to get more messages, the message can accelerate the rate of consumption.
Message routing when a message transmission details 3.producer
producer when sending messages, how do you know how many queue it under the current topic? Every time a message is sent to the broker to check on it? Obviously not, so the performance of sending the message would not increase. then what should we do? It is asynchronous, huh. producer can send a request to the timing broker, acquires the number of queue under topic, and then saved. So that each producer when sending messages, as long as you can get from the local cache. Because the number of topic on the broker queue generally does not change, so this makes sense cache. That there is a problem, the current producer for the first time to send a message to a topic, where queue to do? Because the timing of threads do not want to know the number of queue in which topic broker to take, because the producer end there is not a topic, because the message have not been sent. That is the need to determine what, if the current topic does not count information queue, the queue to obtain count information directly from the broker. Then cached, sending the current message. Then when the second transmission, because the cache has been the message, so we do not have the broker took, and regular follow-thread count queue will automatically update to the topic of the next. Well, the queue count producer with topic, that user when sending a message, the framework will be able to put this topic of queueCount delivered to the user, then the user can according to their own needs first of several routes the message to the queue.
How to achieve load balancing 4.consumer
consumer load balancing means that, in the case of consumer spending cluster, how to make the same consumer group in the average consumption of consumer queue at the same topic. So this is essentially a load balancing queue will be assigned to process the average consumer's. So how to achieve it? Load balancing by the above definition, as long as we, do load balancing, it is necessary to determine the consumer group and topic; and then get all the consumer at the consumer group, and all the queue under the topic; then for the current consumer, can be calculated the current consumer which should be assigned to the queue. We can get the current consumer using the following function should be assigned to which several queue.
public class AverageAllocateMessageQueueStrategy : IAllocateMessageQueueStrategy
{ public IEnumerable<MessageQueue> Allocate(string currentConsumerId, IList<MessageQueue> totalMessageQueues, IList<string> totalConsumerIds)
{ var result = new List<MessageQueue>(); if (!totalConsumerIds.Contains(currentConsumerId))
{ return result;
} var index = totalConsumerIds.IndexOf(currentConsumerId); var totalMessageQueueCount = totalMessageQueues.Count; var totalConsumerCount = totalConsumerIds.Count; var mod = totalMessageQueues.Count() % totalConsumerCount; var size = mod > 0 && index < mod ? totalMessageQueueCount / totalConsumerCount + 1 : totalMessageQueueCount / totalConsumerCount; var averageSize = totalMessageQueueCount <= totalConsumerCount ? 1 : size; var startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod; var range = Math.Min(averageSize, totalMessageQueueCount - startIndex); for (var i = 0; i < range; i++)
{
result.Add(totalMessageQueues[(startIndex + i) % totalMessageQueueCount]);
} return result;
}
}Function in the realization of not much analyzed. The purpose of this function is given input, returns the consumer assigned to the current queue. The principle of allocation is evenly distributed. Well, with this function, we can easily achieve a load balancing. We can open a regular job on the inside of each consumer is running the job from time to time to conduct a load balancing, which is performed once the above functions, the latest queue to get the current consumer binding. Because each consumer has a groupName property that represents the current consumer belongs to which group. So, when we will be able to get all the load balancing broker under the current consumer group; on the other hand, since each consumer knows what its own subscription topic, so with topic information, you can get all the queue under the topic information, and with these two information, each consumer will be able to do your own load balancing a. Look at the following code:
_scheduleService.ScheduleTask(Rebalance, Setting.RebalanceInterval, Setting.RebalanceInterval);
_scheduleService.ScheduleTask(UpdateAllTopicQueues, Setting.UpdateTopicQueueCountInterval, Setting.UpdateTopicQueueCountInterval);
_scheduleService.ScheduleTask(SendHeartbeat, Setting.HeartbeatBrokerInterval, Setting.HeartbeatBrokerInterval);Each consumer will start the task three internal timing, the timing of the first task pledged to do a load balancing; the second task all said to be regularly updated information on the current topic of queueCount consumer subscriptions, and the latest information is stored queueCount locally; the third task that the current consumer periodically sends a heartbeat to the broker, this broker will be able to know whether a heartbeat consumer is still alive, maintaining all the consumer information on the broker. If there is no new discovery in a timely manner or send heartbeat over the consumer, it will be considered new or dead consumer. Because the maintenance of all the consumer information on the broker, so he will be able to provide access to services, such as group queries consumer in the group according to a consumer.
Through these three regular tasks, the consumer will be able to complete the load balancing. Look at Rebalance method:
private void Rebalance()
{ foreach (var subscriptionTopic in _subscriptionTopics)
{ try {
RebalanceClustering(subscriptionTopic);
} catch (Exception ex)
{
_logger.Error(string.Format("[{0}]: rebalanceClustering for topic [{1}] has exception", Id, subscriptionTopic), ex);
}
}
}Code is very simple, is to do load balancing process for each subscribed topic. Look again at RebalanceClustering method:
View Code
The code above much of the analysis is to first get all the consumer according to consumer group and topic, then do the sorting process for the consumer. The reason to do the sorting process is to ensure that does not change as much as possible to the existing distribution of load balancing. The next step is to get all queue at the topic from a local, also in accordance queueId do some sort. Then it is to call the above allocation algorithm to calculate the current consumer should be assigned to which queue. The last call UpdatePullRequestDict method used to add or delete a queue for processing. For the new queue, you want to create a separate worker thread, start pulling messages from broker; deleted for queue, corresponding to stop work, stop pulling messages.
Introduction and analysis above, we know how to achieve consumer equeue load balancing. We can see that, because the update queue under each topic is asynchronous timing, and load balancing itself is timed and maintenance of consumer information on broker is not true, because each consumer to the broker is not sending heartbeat transmitted in real time, but such is sent once every 5s. Because all of these are asynchronous design, it may result in the load balancing process, with a queue may be simultaneously two consumer spending. This is called, a message we can do is consume at least once, but can not do a equeue level message will only be consumed once. In fact rocketmq like this idea, too, to give a message will only be achieved once the consumer (as too costly and too complicated, in fact, for a distributed environment, are less likely to do a message will be consumed once), instead of using ensure that a message will be consumed at least once (ie at least once). so use equeue, the application process for each party to do so on their own power message.
5. How to achieve real-time push messaging
Real-time push message, there are two approaches: a push mode (push) and a pull mode (pull). The push mode refers broker active push news for all consumers subscribed to the topic; pull refers to the way consumers take the initiative to broker pull message; for push mode, the biggest advantage is real-time, because there is a new message , will be immediately pushed to the consumer. But there is a downside is that if consumers consume too late, it will push the message to consumers, which would lead to the consumer end of the message is blocked. And by pulling the way, there are two implementations: 1) a way to pull in rotation, such as whether there are new messages every 5s rotation, the disadvantage of this approach is Xiaoxibushi, but the progress of consumption by the consumer himself entirely to control the ; 2) takes longer to pull connected, is not in rotation, between the consumer and the broker channel remains connected, then a new message broker will use this channel to send messages to consumers.
Way pull message via the long connections equeue currently employed. Long length to achieve through the socket connection. But although called long connection, nor is constantly open, but will also design a timeout restrictions, such as no more than a maximum length of connection 15s, over 15s, the broker send a reply to the consumer, tell the consumer is currently no news; then consumer after receiving the reply to this, you know you want to continue a long connection to initiate the next pull. Then, if the inside, a new message on the broker in this 15s, then the broker can immediately take the initiative of the consumers connected to notify the appropriate length, the message to the consumer. Therefore, it can be seen, the broker when processing the request message pulling consumers, if no new messages, this will hold live socket connection, hold up to 15s, over 15s, return information is sent to tell the consumer current no message, and then send consumers pull message request over again. With this pull model is based on a long connection, we can achieve two benefits: 1) real-time push messaging; 2) news consumption by the consumer schedule control;
In addition, equeue also achieved in the consumer's own automatic current limiting function. If the broker is currently a lot of news, namely speed greater than the speed of news producers to produce consumer spending news, there will be news on that broker accumulation. That at this time when consumers pull message, there will always be news to pull, but the consumer has no time to deal with so many messages. Therefore equeue design a framework built restrictor (flow control, flow control), that is, the upper limit may be formulated to allow for accumulation of a message consumer end, such as 3000, exceeding this number (configurable), then the consumer will equeue those with little slower frequency pull message. How such a delay in milliseconds (configurable time delay) and then pull. This simple to achieve the purpose of flow control.
6. How to deal with cases of failure messages consumption
As a message queue, consumers could always throw an exception when consuming news, the situation in equeue in this case is the failure of news consumption. By introducing the above consumption progress, we know each queue for a particular consumer group, has a unique consumer progress. Indeed, to the consumer to take the local message after pulled, may be consumed in two ways, one is parallel consumer a linear consumption.
Parallel consumption means that if the disposable pull up current message 32, it will pass to start equeue task (i.e., multi-threaded opening) parallel manner for each message consumer;
Linear consumption means that the message is a sequentially separate single thread consumption, consumption, and the order of the same order of pulling over.
For linear consumption, if consumption of time before a message fails, an exception is thrown, and then how to do it? Possible approach is to think of a retry three times, but after a retry or if the failure? Not because the message behind the message can not lead to the consumer, right? Ha ha! In this case, first talk about rocketmq in the handling of it: it's practice, when faced with consumption of failure, not immediately retry, but directly to send this message to a retry queue on the broker, after sent successfully, it can consume down to the next message. Because once sent to a retry queue, it means that the message is always the last to be consumed, because the message is not lost. But if sent to a retry queue broker is not successful? This one? ! In fact, this situation should not arise, if there is, it is basically broker hung up, huh, huh.
rocketmq in this case, that failure will put this message into the local memory queue, and slowly consume it. Then continue to the next consumer behind the news. Now you must be very concerned about how to queue the offset is updated? Here involves the concept of a sliding door. When a batch of messages pulled from the broker to the local consumer, the consumer is not immediately, but first into a local SortedDictionary, key messages in the queue is the location where, value is the message itself. Because it is a sort of dictionary, so the key message means that the smallest is the foremost message, the biggest news is the rearmost news. Then either parallel or linear consumer spending, as long as a message is consumed, it is removed from the SortedDictionary out there. Each time a message is removed, always returns the current SortedDictionary in the smallest key, and then we can determine whether this key and whether forward than the last, and if so, this latest update of the offset queue . Because every time a message is removed, always returns the smallest key SortedDictionary in the current, so if the current offset is 3, 4 and offset to the news consumer has failed, so it will not be removed, but the offset is Although all of these messages 5,6,7,8 consumer success, but as long as the offset for this message 4 is not removed, it is the smallest key will not move forward. This is called a sliding door concept. Is like running on the tracks in a motor car, offset the forward movement is like a motor car continues to move forward. Because we want to offset always we will continue to move forward, so I do not want a failure message in front of the consumer so that the sliding door stops moving (that is, we always want the smallest key can increase in size), so we will try to get consumer failure message can not hinder the forward movement of the sliding door. So only consume failure message into the retry queue.
Another point to note: not always successful completion of a news consumption, it will immediately tell the broker updated offset, such as that performance is certainly low, broker will be too busy, a better way is to simply update queue in local memory the offset, then the timing of such a 5s, the latest update to offset broker. Therefore, since the existence of this asynchronous, will also lead to the possibility of a message is repeated consumption, because the broker is certainly offset the slower progress than the actual consumption, there is a time difference 5s. So, once again stressed that the party must properly handle the processing of the application message idempotent! For example enode frame, for each command message, the inner frame are made of command processing idempotent. So using the application enode framework itself need not be considered for command of processing power and so on.
The above mentioned parallel linear consumption and consumption, in fact, to update the offset is the same as nothing more than a multi-threaded parallel consumption while removing messages from successful consumer SortedDictionary, whereas only a single thread to remove the single-threaded in SortedDictionary messages. So we have to pass the lock mechanism to ensure the operation of SortedDictionary is thread-safe. Currently with the line layer security ReaderWriterLockSlim to achieve the method call. Interested friends can take a look at the code.
Finally, focus, huh, huh. equeue not yet achieved the failed message back to the retry queue broker. This feature will be considered later added to the list.
7. broker how to solve the problem of single point
This problem is rather complicated, does not support the current equeue broker of master-salve or master-master, but a single point. I think a mature message queue, in order to ensure that a broker's time to hang up, to try to ensure there are other broker can take over it, so as to make the reliability of the message queue server. But the issue is too complicated. Just the way master-slave rocketmq currently implemented. As long as the primary master is hung, that producer will not send a message to a broker, because the broker slave is read-only, can not directly accept a new message, the slave is allowed to be only broker consumer pull message.
The problem, to be discussed clearly, need a lot of knowledge distributed area. Due to space reasons, we discussed here do not, in fact, I myself do not know how in the end the design. Daniel who want a lot of guidance, how to achieve high availability broker, ha!
Reproduced in: https: //my.oschina.net/Raymond/blog/226063