[Turn] to ensure high availability message queue

Original http://prchen.com/2019/06/23/%E4%BF%9D%E8%AF%81%E6%B6%88%E6%81%AF%E9%98%9F%E5%88% 97% E7% 9A% 84% E9% AB% 98% E5% 8F% AF% E7% 94% A8% E6% 80% A7 /

In actual production, the introduction of the message queue art will certainly bring benefits, such as reducing the coupling system, the asynchronous response speed, flattened peak pressure. However, any technology is a double-edged sword, simultaneously introduced the message queue must take some risks: the more dependent on external systems introduced, the more error-prone. Once the message middleware is down, it will cause the entire system unusable. Therefore, to ensure high availability of the message queue is very important.

MQ how to ensure high availability?

RabbitMQ high availability mode

RabbitMQ ensure high availability has three modes: stand-alone mode, normal mode cluster, the cluster mirroring mode

• Stand-alone mode

  For developers to use the test environment, NA production environment

• Ordinary cluster model

  MQ plurality deployed in a clustered environment, create message queue (Queue) would be placed in a concentrated MQ example , every other instance of the message queue of the synchronization metadata (profile can be understood as, but not the message itself). When the consumer needs to pull message, if the message queue is not connected to the host instance, then the instance to be on a message queue where the pull message instances .

  

  Ordinary cluster model of drawbacks:

  1. May occur within the MQ cluster large data transfers , larger overhead
  2. Availability low. Once the queue instance where the hang up, the message is lost, consumers can not normal consumption

• Mirroring cluster model

  In this mode, the message queue is created , whether or metadata message in a message queue will exist in multiple instances , then each time the producer sends a message to the MQ instance, will automatically send messages to multiple instances synchronization message for message queue.

  

  How do I turn the mirror cluster model?

  在上一篇文章中提到，RabbitMQ有很好的管理控制台，只需要在后台新增一个镜像集群模式的策略，命令数据同步到所有或指定数量的MQ实例中，当生产者再次创建消息队列时应用这个策略，即可自动同步数据到其他实例中。

  镜像集群模式的优势：

  高可用。MQ集群中任何一台机器挂了都没事，有备用机器。

  镜像集群模式的弊端：

  1. 由于RabbitMQ不支持分布式集群，消息队列同步到所有MQ实例会增大网络带宽压力，系统性能开销也相应提高
  2. 扩展性不好。无法线性扩展消息队列，一旦在集群中增加机器，就要同步所有消息队列中的数据。

Kafka的高可用模式

Kafka集群由多个Kafka实例构成，每个实例被称为broker，每个broker是一个节点；当生产者创建一个topic时，这个topic可以划分为多个partition（区），每个partition可以存在于不同的broker上，每个partition就放一部分数据。

上述模式是天然的分布式消息队列，即一个 topic 的数据，分布在不同的MQ上，每台MQ只放一部分数据。

实际上 RabbmitMQ 不是分布式消息队列，它只是提供了一些集群、HA(High Availability, 高可用性) 机制的传统消息队列。

Kafka 0.8 以前，是没有 HA 机制的，任何一个 broker 宕机了会导致 broker 上的 partition 就挂掉，无法读写，没有什么高可用性可言。

Kafka 0.8 以后，提供了 HA 机制，即 replica（复制品） 副本机制。每个 partition 的数据都会同步到其它机器上，形成自己的多个 replica 副本。所有 replica 会选举一个 leader 出来，那么生产和消费都跟这个 leader 打交道，然后其他 replica 被称为 follower。写的时候，leader 会负责把数据同步到所有 follower 上去，读的时候就直接读 leader 上的数据即可。只能读写 leader？很简单，要是你可以随意读写每个 follower，那么就要解决数据一致性的问题，系统复杂度太高，很容易出问题。Kafka 会均匀地将一个 partition 的所有 replica 分布在不同的机器上，保证了高可用。

如此一来，就保证了消息队列的高可用。每个 broker 中的 partition 在其他机器上都有副本，因此不用担心某个 broker 宕机。如果宕机的 broker 中有某个 partition 的 leader，那么此时会从 follower 中重新选举一个新的 leader 出来，大家继续读写新的 leader 即可。这就是Kafka实现高可用的模式。

When writing a message producer , the producer wrote leader, then leader writes data to local disk, then the other follower on their own initiative to pull data from the leader. Once all the data is good follower synchronization, it will be sent to the leader ack, ack after receipt of all follower of the leader, will return to write messages to the successful producer. (Of course, this is only one mode, this behavior may be appropriately adjusted)

When consumer spending messages , read only from the leader, but only when a message has been successfully synchronized all follower have returned ack when the news will be read by consumers.

to sum up

This paper describes the RabbitMQ and Kafka ensure high availability mode.

RabbitMQ ensure high availability has three modes: Normal mode, normal mode and mirror cluster cluster mode.

Wherein the common cluster message queue exists only on a single machine, so that when necessary to pull message, a large amount of data transmission, a large overhead .

Compared to ordinary cluster model, mirroring the cluster model put the message queue stored on multiple servers, while ensuring the high availability cluster but because simply, not a distributed system, increasing the system overhead, and poor scalability .

Kafka support distributed , high availability is no mechanism before 0.8, provides after 0.8 Replica-copy good way to ensure high availability of the message queue. Partition on each machine has its copies on other machines, and each partition has a leader, other machines for the follower, producers and consumers will only read and write operations on the leader , when the leader hangs up, will re-election from the follower in a new leader, it does not affect read and write.