Comparison of common message middleware

1. Comparison of Kafka, RabbitMQ and Redis message middleware

In distributed systems, message middleware is often used for data exchange between systems.
According to the actual business demand scenarios and operation and maintenance costs, you can choose the product that suits you.

2. Introduction of related concepts

• Kafka
  1. Based on Pull mode to process message consumption
  2. Pursue high throughput
  3. The initial purpose is to log collection and transmission
  4.0.8 version began to support replication, does not support transactions, the message duplication, loss, errors are not strict Data collection business that requires and is suitable for Internet services that generate large amounts of data.

• RabbitMQ
  RabbitMQ is an open-source message queuing system developed using Erlang language and implemented based on the AMQP protocol.
  The main features of AMQP
  1. Message-oriented, queueing, routing (including point-to-point and publish / subscribe), reliability, and security
  2. The AMQP protocol is more used in enterprise systems, and the scenarios that require high data consistency, stability, and reliability, and the performance and throughput requirements are second.
  Mainly used in: dubbo framework (zookeeper is used for registration center), spring cloud, etc.

• Redis
  Redis is a NoSQL database based on Key-Value pairs, and is very active in development and maintenance.
  Although it is a Key-value database storage system, it itself supports MQ functions, so it can be used as a lightweight queue service.

3. Features comparison

1. In terms of application scenarios,
RabbitMQ complies with the AMQP protocol and is developed by the erlang language with high internal concurrency. It is used for real-time messaging with high reliability requirements.

Kafka is Linkedin's open-source message subscription system in December 2010. It is mainly used to process live streaming data and large amounts of data.

2. In the architectural model,
RabbitMQ follows the AMQP protocol. The broker of RabbitMQ consists of Exchange, Binding, and queue, where exchange and binding form the routing key of the message; the client Producer communicates by connecting the channel and the server, and the Consumer obtains the message from the queue. Consumption (long connection, queue message will be pushed to the consumer side, the consumer reads data from the input stream in a loop). rabbitMQ is centered on the broker; there is a message confirmation mechanism.

Kafka complies with the general MQ structure, producer, broker, consumer, consumer as the center, the consumer information of the message consumer information is saved, the consumer pulls the bulk data from the broker according to the point of consumption; no message confirmation mechanism.

3. In terms
of throughput, Kafka has high throughput, internal batch processing of messages, zero-copy mechanism, data storage and retrieval are local disk sequential batch operations, with O (1) complexity, efficiency of message processing Very high.
RabbitMQ is slightly inferior to kafka in terms of throughput. Their starting point is different. rabbitMQ supports reliable delivery of messages, supports transactions, and does not support batch operations; based on the reliability of storage, storage can use memory or hard disk.

4. In terms of availability,
RabbitMQ supports the queue of the mirror, the main queue is invalid, and the mirror queue takes over
the broker of kafka to support the active and standby mode

5. In terms of cluster load balancing,
kafka uses zookeeper to manage the brokers and consumers in the cluster, and can register topics to zookeeper; through the coordination mechanism of zookeeper, the producer saves the broker information of the corresponding topic, which can be randomly or polled and sent to the broker ; And the producer can specify shards based on semantics, and the message is sent to a shard in the broker.
RabbitMQ's load balancing requires a separate loadbalancer to support it.

Four, application scenarios

Rabbitmq is more reliable than kafka.
Kafka is more suitable for IO high-throughput processing. For example, ELK log collection Kafka and RabbitMq are general purpose message brokers. They are all for distributed deployment. But their assumptions about the definition of the message semantic model are very different.

• In the following scenarios, you are more suitable to use Kafka. You have a large number of events (more than 100,000 per second), you need to partition, sequential, at least one successful delivery to consumers who mix online and packaged consumption, you want to be able to reread the message, you can accept the current limit Node level is highly available or you do n’t mind getting support from software that is still in toddlers through forum / IRC tools.

• In the following scenarios, you are more suitable to use RabbitMQ. You have fewer events (over 20,000 / sec) and need to find consumers through complex routing logic, you want message delivery to be reliable, you do n’t care about the order of message delivery, you need to support cluster-nodes now High level of availability or you need 7 * 24 hours of paid support (of course, you can also use the forum / IRC tool)

• Redis message push (based on distributed pub / sub) is mostly used for real-time message push and does not guarantee reliability

Redis message push (based on distributed pub / sub) is mostly used for real-time message push, and does not guarantee reliability. Other mq and kafka are guaranteed to be reliable but have some delay (non-real-time systems do not guarantee delay). Redis-pub / sub will be emptied when power is turned off, and the use of redis-list as a message push has persistence, but it is not completely reliable and will not be lost.

• Redis is an in-memory database! Redis's father did a disque, do you want to try it. mq generally uses a subscription-publish model. If you consider performance, the main focus is on whether the consumption model is pull or push. The most influential should be the storage structure. Kafka's performance can only be effective when the number of topics is less than 64. determined by partition. Message loss in extreme cases, for example: after the master writes the message, the host machine goes down and the hard disk is damaged