The difference between Kafka and partition copy queue RocketMQ

Kafka recent study, found that the core concept RocketMQ still there are some differences, following me down Kafka partition and RocketMQ

The difference between the queues.

RocketMQ queue

Each subject will have RocketMQ plurality of queues, distributed on each broker cluster, the following distribution:

In broker queue will be abstracted into a consumer queue, in a clustered mode, each queue for each consumer group can only be one consumer subscription spending, but a consumer can consume multiple queues, which also ensures that the cluster mode under the message will not be repeated consumption, as shown below: Zhengzhou infertility hospital rankings: http: //wapyyk.39.net/zz3/zonghe/1d427.html

In RocketMQ the open source version, when you create a theme, created by the cluster model, the number of queues specified topic in the cluster, such as cluster has two broker, select when creating a theme we queue number is 4, it will be in every broker in create a theme for the four queues, then the topic in the cluster will have 4 * 2 queue number, here is a good place is not precisely control the number of queues, but not the problem.

RocketMQ redundant message implemented by master-slave mode, in a production environment, will be taken to build a multi-mode Master Slave multiple clusters, queue data synchronization between master and slave has two kinds of synchronous and asynchronous replication.

Thus, RocketMQ rely consumer queue, and the queue data synchronization redundant message from the main. Zhengzhou infertility check: http: //wapyyk.39.net/zz3/zonghe/1d427.html

Kafka and partition copy

Partition concept Kafka is one of its core concepts, zoning mechanism makes Kafka has the ability to expand horizontally, on its partitions, Kafka can also set the copy partition, greatly improving the reliability Kafka message.

在 Kafka 中,一个主题在集群中会拥有一个以上分区,每个分区在每个消费集群中只能有一个消费者进行订阅消费,,但是一个消费者可以消费多个队列,与 RocketMQ 队列一样:

我们可以通过调整主题的分区数量提高消息的吞吐量,还可以为分区设置副本因子,即该分区在集群中拥有多少个副本(replica),副本分为 leader replica 与 follower replica,它们之间通过 ISR(in-sync replica)与 leader replica 保持数据同步。

在创建主题topic-demo时,可以指定主题在集群中的分区数量,以及副本因子大小:

--partitions 4 --replication-factor 2

以上参数为该主题创建了 4 个分区,副本因子为 2,我现在有个集群,有 3 个 broker:

nodel brokerid=O 
node2 brokerid=l 
node3 brokerid=2

根据 Kafka 的默认分配:

node1: topic-demo-0、topic-demo-1
node2: topic-demo-1、topic-demo-2、topic-demo-3
node3: topic-demo-0、topic-demo-2、topic-demo-3

有没有发现,每个分区都分配了一个副本,而且分区的分布尽量均衡,分区副本尽量不在同一个节点上,如果我们设置副本因子为 3,原理一样。

不同于 RocketMQ 队列,Kafka 的分区可以在集群中精确设置多少个,然后随机均衡地分布在集群上,还可以自由定义副本的多少,而 RocketMQ 的 Master-Slave 模式看起来仅有一份副本,当然为了节省存储空间以及提高性能,一般副本因子设置 2 也就够了。

相对比 RocketMQ 的队列与主从同步机制,Kafka 的分区与副本机制显得更加灵活,而且也更加合理。


Guess you like

Origin blog.51cto.com/14510269/2441933
Recommended