RabbitMQ and Kafka contrast

# Preface the open source community has a lot of good middleware queue, such as RabbitMQ and Kafka, each queue looks like its characteristics, during the project selection, often dazzled, overwhelmed. For RabbitMQ and Kafka, in the end you should choose? # RabbitMQ RabbitMQ architecture ## concept is a distributed system ** broker **: service program runs each node, add or delete function to maintain the queue of the node and forwards the request queue operations. ** master queue **: Each queue is divided into a main queue and a plurality of image queues. ** mirror queue **: mirroring the queue, as a backup master queue. After the master node where the queue hang up, the system is upgraded to queue Mirror master queue, the operation queue is responsible for processing client requests. Note, mirror queue mirror only, not designed to bear the pressure reader client. ! [File] (https://img2018.cnblogs.com/blog/1203146/201910/1203146-20191003142502668-692418737.jpg) shown above, there are two nodes in the cluster, each node has a broker, each broker maintains a queue of the machine, and can communicate with each other between borker. There are two cluster queues A and B, each queue is divided into master queue and mirror queue (backup). So how production and consumption on the queue to achieve it? ## queue consumption! [File] (https://img2018.cnblogs.com/blog/1203146/201910/1203146-20191003142502892-764881573. JPG) shown above there are two consumer consumption queues A, both attached to a consumer on different machines in a cluster. RabbitMQ any node in the cluster have all queues meta information on the cluster, so any node connected to the cluster can be, the main difference is that some consumer even in the node where the master queue, and some even in the non-master queue node on. Because the mirror queue and master queue to be consistent, they need synchronization mechanism, because of the consistency of the limit, causing all read and write operations must operate on the master queue (think about it, why read from the master queue should be read in ? and separate read and write database is not the same.), and then by the master node to node synchronization mirror queue is located. Even if the consumer is connected to the non-master queue node, the consumer operations will be routed to the node where the master queue, in order to consume. ## production queue! [File] (https://img2018.cnblogs.com/blog/1203146/201910/1203146-20191003142503113-427343022. jpg) principle and consumption, as if connected to a non-master queue node, the route in the past. ** ** insufficient due master queue single node, resulting in a performance bottleneck, throughput is limited. While To improve performance, the internal use of the Erlang language, but ultimately could not escape the fatal flaw in the architecture design. # Kafka honest, Kafka I think it is a modified version of RabbitMQ saw this flaw was devised to improve the point is this: the queue of a single master to become more master, that is, a machine could not carry qps, then I to use multiple machines carry qps, put a traffic queue is uniformly dispersed on multiple machines can not it what? Note that the data is no intersection between a plurality of master, i.e. a message sent to either the master Queue, or sent to another master queue. This is called the inside of each Partition in Kafka master queue, i.e. a slice. Queue has a plurality of master slices, each slice have several main sub-fragments do backup synchronization mechanism similar RabbitMQ. ! [File] (https://img2018.cnblogs.com/blog/1203146/201910/1203146-20191003142503323-2045676015.jpg) above, we have omitted a different queue, only one queue (Kafka on the assumption that the cluster called Topic ). Each producer random to send messages to the primary slice, slice resynchronization after the primary to the secondary slice. ! [File] (https://img2018.cnblogs.com/blog/1203146/201910/1203146-20191003142503580-1215564775. JPG) when the virtual queue is read out of the concept of a Group, a Topic internal message, a consumer will only be routed to the same Group, the same message consumer consumption of a Group is not the same; shared among Group a Topic, multiple copies looks is a queue. Therefore, in order to reach more than a Topic Group share data, Kafka does not like that news consumption RabbitMQ is completed immediately deleted, but you must configure save the date in the background, that is, save only the most recent message, a message will be over this time deleted from the disk, thus ensuring a period of time, Topic Group data for all visible (this feature makes Kafka is very suitable for a company's data bus). Queue read also read the main fragment, and correspondence between the slices one by one in order to optimize performance, and the main consumer, the consumer if the number is greater than the number of fragments, some consumers message is not present. Thus, Kafka is definitely designed for high throughput, such as setting the number of fragments is 100, then there are 100 machines to carry a flow Topic, of course, better than the stand-alone performance of RabbitMQ. # This article summarizes only the contrast Kafka and RabbitMQ, but open-source queue is more than just these two, ZeroMQ, RocketMQ, JMQ and so on, it did not look limited time, so the comparison is not within the scope of this article. So, do not be confused by these sorts of queue, and identify the key differences from the architecture, combined with their actual needs (such as this will only examine only throughput from a demand) to get easy selection. Finally summarized as follows: lower throughput: Kafka and RabbitMQ can be. High throughput: Kafka. RabbitMQ and reference this article from KafKa official document, so you really want to get to know a middleware principle best to see an official document, the document which detailed design, we can compare the design of their own, and to find themselves in line with the actual middleware situation. Transfer: https: //www.cnblogs.com/haolujun/p/9632835.html