RocketMQ, KafKa contrast and RabitMQ

Message Queue MQ effect

Decoupling

Decoupling is the message queue to solve the most essential issues.

The final consistency

Eventual consistency refers to the state of the two systems consistent, either succeed or fail. Some message queue can not guarantee eventual consistency.

broadcast

Message Queuing essential functions which may be broadcast. With the message queue, we only need to be concerned about whether the message delivery queue, as to who would like to subscribe, downstream of things, no doubt greatly reduce the workload of the development and the FBI.

Peak shifting and flow control

A typical usage scenario is the spike clipping service for traffic scene.

Due to limited space, this article focuses on relatively message queue, please refer to the detailed scenarios clipping funnel flow model: "What is the traffic peak clipping? How to solve the spike scene clipping service "

RocketMQ, KafKa contrast and RabitMQ

characteristic	ActiveMQ	RabbitMQ	RocketMQ	KafKa
Development language	java	erlang	java	scala
Stand-alone Throughput	Ten thousand	Ten thousand	100 000	100 000
Timeliness	ms level	Class us	ms level	ms level within
Cluster management			name server	zookeeper
High Availability	High (master-slave architecture)	High (master-slave architecture) mirroring mode implemented, there will be a performance bottleneck when large volumes of data	Very high (distributed architecture)	Very high (distributed architecture)
From the switching master		Automatic switching, the first to join the cluster will become the slave master, because the data before the newly added slave does not synchronize master, so there will be some data may be lost.	Does not support automatic switching, can not send a message to the master after master failure, consumer probably 30s (default) perceived this event, after which the slave consumption; if the master can not be restored, there may be some of the information is lost when asynchronous replication	Automatic switching, N copies, allows the N-1 failure; failure after master slave select automatically from a main
Message write performance		RAM RocketMQ of about 1/2, DISK 1/3 RAM about performance properties.	Well, 10 bytes per test, stand-alone single broker about 7w / s, single 3broker about 12w / s	Very good, each 10 bytes tests: one million / s
Features	Mature product, the company has been in many applications, there are more documents; better support a variety of protocols	Based on erlang development, so the concurrent competence, excellent performance, low latency, rich management interface.	MQ features a more complete, better scalability	MQ supports only major features like backtracking message queries and other functions not provided, use a wide range of big data scene.

为啥RabbitMQ能做到us级，其他的都是ms级呢？

主要有以下几个原因。

erlang语言天然支持高并发。
基于内存镜像模式实现，RabbitMQ的us级主要指基于RAM模式。

RocketMQ、KafKa和RabitMQ选择

Kafka

Kafka主要特点是基于Pull的模式来处理消息消费，追求高吞吐量，一开始的目的就是用于日志收集和传输，适合产生大量数据的互联网服务的数据收集业务。

大型公司建议可以选用，如果有日志采集功能，肯定是首选kafka了。

RocketMQ

天生为金融互联网领域而生，对于可靠性要求很高的场景，尤其是电商里面的订单扣款，以及业务削峰，在大量交易涌入时，后端可能无法及时处理的情况。

RoketMQ在稳定性上可能更值得信赖，这些业务场景在阿里双11已经经历了多次考验，如果你的业务有上述并发场景，建议可以选择RocketMQ。

RabbitMQ

RabbitMQ，结合erlang语言本身的并发优势，性能较好，社区活跃度也比较高，但是不利于做二次开发和维护。

如何保证消息的一致性和如何进行消息的重试机制？

如何保证消息的一致性

保证消息的一致性有三种方式。

定时补偿+幂等消费(TCC)
推拉结合
分布式事务

如何进行消息的重试机制

下面只分析RocketMQ的消息重试。

RocketMQ只有当消费模式为集群模式时，Broker才会自动进行重试，对于广播消息是不会重试的。消费者从Broker拉取消息失败时，RocketMQ会通过消息重试机制重新投递消息，这肯定不会一直投递。当投递到最大重试次数之后，就会加入到死信队列，我们只要对死信队列的数据做人工补偿。

认定为消费失败规则

RocketMQ规定，以下三种情况统一按照消费失败处理并会发起重试。

业务消费方返回ConsumeConcurrentlyStatus.RECONSUME_LATER
业务消费方返回null
业务消费方主动/被动抛出异常

注意对于抛出异常的情况，只要我们在业务逻辑中显式抛出异常或者非显式抛出异常，broker 也会重新投递消息，如果业务对异常做了捕获，那么该消息将不会发起重试。因此对于需要重试的业务，消费方在捕获异常的时候要注意返回 ConsumeConcurrentlyStatus.RECONSUME_LATER 或 null 并输出异常日志，打印当前重试次数。（推荐返回ConsumeConcurrentlyStatus.RECONSUME_LATER）

个人不建议捕获异常，让MQ认定为失败。

RocketMQ重试时间窗口

messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h

死信的业务处理方式

默认的处理机制中，如果我们只对消息做重复消费，达到最大重试次数之后消息就进入死信队列了。

我们也可以根据业务的需要，定义消费的最大重试次数，每次消费的时候判断当前消费次数是否等于最大重试次数的阈值。

如：重试三次就认为当前业务存在异常，继续重试下去也没有意义了，那么我们就可以将当前的这条消息进行提交，返回 broker 状态ConsumeConcurrentlyStatus.CONSUME_SUCCES，让消息不再重发，同时将该消息存入我们业务自定义的死信消息表，将业务参数入库，相关的运营通过查询死信表来进行对应的业务补偿操作。

RocketMQ 的处理方式为将达到最大重试次数（16 次）的消息标记为死信消息，将该死信消息投递到 DLQ 死信队列中，业务需要进行人工干预。实现的逻辑在 SendMessageProcessor 的 consumerSendMsgBack 方法中，大致思路为首先判断重试次数是否超过 16 或者消息发送延时级别是否小于 0，如果已经超过 16 或者发送延时级别小于 0，则将消息设置为新的死信。死信 topic 为：%DLQ%+consumerGroup。

发送失败如何重试

设置生产者在 3s 内没有发送成功则重试 3 次的代码如下。

   /**同步发送消息，如果 3 秒内没有发送成功，则重试 3 次*/
    DefaultMQProducer producer = new DefaultMQProducer("DefaultProducerGroup");
    producer.setRetryTimesWhenSendFailed(3);
    producer.send(msg, 3000L);

参考

https://www.jianshu.com/p/fec054f3e496
https://gitbook.cn/books/5d340810c43fe20aeadc88db/index.html