RabbitMQ solutions to common problems

Previous blog about RabbitMQ record of service to build and simple entry, but the light may not be enough.
To RabbitMQ will be used in the production, it is necessary to consider and solve many problems.

table of Contents

• Message converter

• How to ensure that producers message is not lost?

• How consumers to prevent message loss?

• Message prefetch

• Dead Letter Switch

---

Message converter

RabbitMQ native only send a byte array, the integration and SpringBoot, Spring allows developers to send an object, because the Spring of sending and receiving messages carried out some processing.

Default message to the converter: SimpleMessageConverterconversion logic roughly as follows:

• If the request is based on the contentType textstarts, the message is converted to String. Before forwarding is determined whether a given character set, if not given in a default character set UTF-8conversion.
• If contentType equal application/x-java-serialized-objectthen the message transmission sequence of Java.
• If the above condition is not satisfied, no conversion, as transmission.

Custom message converter

If necessary, you can also choose to implement their own message converter.
Create a class that implements MessageConverterthe interface, the producers realize toMessagemethod, consumers realize fromMessagemethod.

public class MyMessageConverter implements MessageConverter {

	//生产者发送转换
	@Override
	public Message toMessage(Object o, MessageProperties messageProperties) throws MessageConversionException {
		//使用FastJson
		Message message = new Message(JSONObject.toJSONBytes(o), messageProperties);
		return message;
	}

	//消费者接收转换
	@Override
	public Object fromMessage(Message message) throws MessageConversionException {
		return JSONObject.parse(message.getBody());
	}
}

template set custom converter

//设置自定义消息转换器
template.setMessageConverter(new MyMessageConverter());

How to ensure that producers message is not lost?

Message sent successfully in two ways:

• Broker message arrives
• Message is successfully routed to the Queue

General From a business point of view, only the message is routed to the Queue, has truly sent successfully.

For both cases, RabbitMQ provides two callbacks: ConfirmCallback and ReturnCallback.

ConfirmCallback

ConfirmCallback for a "message reaches Broker", as to whether the message is routed to it regardless of the Queue.

When sending a message to the Broker, RabbitMQ will issue a response, he told the producers that they have received the message.

Code

Open sender confirmation

//开启 发送者确认
connectionFactory.setPublisherConfirms(true);

Send a message written confirmation callback class

/**
 * @Author: pch
 * @Date: 2020/1/11 11:06
 * @Description: email消息发送方确认回调
 */
public class EmailConfirmCallback implements RabbitTemplate.ConfirmCallback {

	@Override
	public void confirm(CorrelationData correlationData, boolean ack, String cause) {
		//发送消息时没传递correlationData，这里就为null
		System.err.println("消息附带标识：" + correlationData);
		//消息是否到达broker
		System.err.println("消息是否到达broker：" + ack);
		//消息发送失败的原因(RabbitMQ服务宕机等)
		System.err.println("失败原因："+cause);
		
		if (!ack) {
			//消息发送失败,写入到数据库，等待后续处理
		}
	}
}

template transmittal confirmation callback

//开启 发送者确认
template.setConfirmCallback(new EmailConfirmCallback());

User registration interface, after registering successfully send a message service code

@PostMapping("register")
public Object register(String name, String email) {
	//保存注册信息逻辑...
	System.out.println(name + "-注册成功,开始发送email消息...");

	//发送消息到队列
	Map<String, Object> map = new HashMap<>(2);
	map.put("name", name);
	map.put("email", email);

	//发送消息时，可以携带一个CorrelationData保存业务主键
	CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString().toUpperCase());
	rabbitTemplate.convertAndSend("emailExchange", "email_error", JSONObject.toJSONString(map),correlationData);

	return name + "-注册成功";
}

Request, the console as shown below:

The message has arrived Broker, but whether the message is routed to the Queue, here you can not judge a need to use ReturnCallback.

ReturnCallback

ReturnCallback callback for success it is whether the message is routed to a queue, usually used in conjunction with ConfirmCallback.

If a message arrives Broker but not be routed to the Queue, it will trigger ReturnCallback. For example: Exchange is not bound Queue.
On the contrary, it will not trigger ReturnCallback.

mandatory

When receiving the message routing RabbitMQ, will operate according to different mandatory, mandatory default is false.

If mandatory is false, when a message can not be routed, RabbitMQ directly discards the message.
When the mandatory true, the message can not be routed, RabbitMQ will call Basic.Return command message is returned to the producer.

Code

Open sender confirmation

//开启 发送者确认
connectionFactory.setPublisherConfirms(true);

Composing messages failed callback class

/**
 * @Author: pch
 * @Date: 2020/1/11 12:46
 * @Description: email消息失败回调
 */
public class EmailReturnCallback implements RabbitTemplate.ReturnCallback {
	@Override
	public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
		System.err.println("消息内容:" + new String(message.getBody()));
		System.err.println("响应状态码:" + replyCode);
		System.err.println("响应内容:" + replyText);
		System.err.println("exchange:" + exchange);
		System.err.println("rou![在这里插入图片描述](https://img-blog.csdnimg.cn/20200111130632936.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyMDk5ODMz,size_16,color_FFFFFF,t_70)tingKey:" + routingKey);
		
		//消息没有被路由到Queue，写入数据库，待后续处理...
	}
}

Mandatory mode is turned on, and set a callback example

//开启mandatory模式（开启失败回调）
template.setMandatory(true);
template.setReturnCallback(new EmailReturnCallback());

When sending a message, the wrong set routingKey, the console displays the following:

Set the correct routingKey, will not trigger callback.

Backup switch

When the switch statement, a property may be provided: alternate-exchangeto help set a backup switch.
When a message can not be routed to the Queue, RabbitMQ message will switch to alternate routes, standby switch is typically declared as FANOUTtype to ensure that the message will be routed to the Queue.

Note that: even if the message did not succeed in routing the main switch, as long as the routing success in standby switch, it will not trigger ReturnCallback callback.
ReturnCallback will trigger a callback when the standby switch statement, only when the main switch and the standby switch can not be successfully routed.

to sum up

ConfirmCallback针对消息没有到达Broker的回调处理，ReturnCallback针对消息到达Broker但是没路由到Queue的回调处理。
一般将两者配合使用，可以保证消息发送100%不丢失。

RabbitMQ也支持事务，也可以做到消息发送不丢失，但是开启事务后性能严重下降，不建议使用。

消费者如何防止消息丢失？

除了生产者要保证消息100%发送外，消费者也必须确保消息不丢失，这样才能最终确保消息不丢失。

RabbitMQ的消息确认模式：

• AcknowledgeMode.NONE
• AcknowledgeMode.MANUAL
• AcknowledgeMode.AUTO

默认为自动确认，即消息被消费者取走后Queue就会将其删除，不管其是否消费成功。

如果对数据的要求不高，如：日志记录，哪怕丢失一部分日志也无所谓，则可以使用自动确认，这样可以保证最好的性能。

但是如果对数据要求很高，则必须改为：手动确认。

消息手动确认

在手动确认模式下，消息被消费者取走之后，Queue不会将其删除，而是将消息的状态改为Unacked待确认，消费者获取到消息进行消费时，可能成功也可能失败。

消费成功时，则通知RabbitMQ，将消息从Queue中删除。

消费失败时，有两种选择，一是通知RabbitMQ将消息放回队列，交给其他消费者消费、二是认为消息是废数据，让RabbitMQ直接丢弃即可。

不管是否消费成功，都必须进行消息确认，否则消息会一直处于Unacked状态，堆积在Queue中。

消费者获取消息后宕机，消息会丢失吗？

不会，手动确认模式下，即使消费者获取到消息，Queue也不会将其删除，只是将消息的状态改为Unacked待确认。
消费者宕机后，连接就会断开，RabbitMQ检测到连接断开后，会将消息状态改为Ready分发给其他消费者。

虽然不会丢失消息，但是会带来另外一个问题：消息重复消费。

例如：消费者获取消息成功消费后，在消息确认之前服务突然宕机，RabbitMQ会认为消息没有被成功消费，会分发给其他消费者，导致消息被重复消费，可以通过“消息幂等性”解决，后面会介绍。

开启消息手动确认，也可通过yml方式

@Bean("simpleContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(ConnectionFactory connectionFactory){
	SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
	factory.setConnectionFactory(connectionFactory);
	//消息手动确认
	factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
	return factory;
}

消费者：消息的确认和退回

@Component
public class EmailConsumer {

	//监听的队列名称
	@RabbitListener(queues = "emailQueue", containerFactory = "simpleContainerFactory")
	public void consumer(Map<String, String> map, Message message, Channel channel) throws Exception {
		long deliveryTag = message.getMessageProperties().getDeliveryTag();
		System.out.println("消费者,消息ID:" + deliveryTag);

		if (sendEmail(map.get("name"), map.get("email"))) {
			//邮件发送成功，确认消息	消息ID	是否批量确认
			channel.basicAck(deliveryTag, false);

		}else {
			//消息退回 		消息ID      是否批量确认    是否重回队列
			channel.basicNack(deliveryTag, false, true);

			//消息退回，只能单条 建议用basicNack
			//channel.basicReject(deliveryTag, true);
		}
	}

	//发送邮件
	private boolean sendEmail(String name, String email){
		System.out.println("发送邮件:" + email);
		//编写业务逻辑....

		return true;
	}
}

消息预取

聊消息预取之前，先说一说RabbitMQ的消息分发机制。

默认情况下，RabbitMQ会以最快的速度，将消息以轮询的方式全部分发给消费者，尽管消费者还来不及处理。
这样可以保证RabbitMQ本身不会因为消息堆积而影响性能，但是对消费者而言却不太友好。

举个例子：Queue中有100个消息，同时启动两个消费者，RabbitMQ会立即给每个消费者分发50个消息，假设消费者A性能很强，1S就能消费完50个消息，而消费者B性能弱，需要10S才能消费完。这时消费者A就会很空闲，消费者B就会很忙碌，没有充分利用A的性能。

而消息预取则可以根据消费者的实际情况来进行设置，开启消息预取后，RabbitMQ不会直接分发所有的消息，而是根据给定的预取数量来分发，当消费者全部处理完毕后，RabbitMQ才会进行下一轮的消息分发。

例如：Queue中有100个消息，消费者消费1个消息耗时1S，消息预取设为1，则RabbitMQ每秒分发一个消息，如下图所示：

开启消息预取后，还可以对消息进行批量确认，从而进一步提升性能。

设置消息预取的数量

@Bean("simpleContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(ConnectionFactory connectionFactory){
	SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
	factory.setConnectionFactory(connectionFactory);
	//消息手动确认
	factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
	//消息预取数量
	factory.setPrefetchCount(100);
	return factory;
}

消息批量确认

//deliveryTag当前消息的标识，RabbitMQ会自动将一组消息确认
channel.basicAck(deliveryTag, true);

消息预取的数量需要设置一个合适的值，太小性能差但是数据可靠性高，太大性能高但是数据可靠性差。

消息重复消费

开启消息预取后，可能存在“消息重复消费”的问题。
批量获取100条消息，成功消费了99条，在消费最后一条时系统宕机，RabbitMQ会将100条消息重新分发给其他消费者。

RabbitMQ not provide a solution, we need to solve their own business, a common solution:

• Plus database table fields marked distinction consumption.
• Use Redis to save the consumer to identify successful.

Dead Letter Switch

When a statement Queue, which can be set to a "dead letter Switch" ( dead-letter-exchange), a case when the message, the message will be redirected to RabbitMQ on the dead letter rerouting switch.

• Message is returned (channel.basicNack)
• Message Expiration (TTL)
• Message Number (x-max-length) exceeds the maximum limit and the queue is removed
• The total size of the message (x-max-length-bytes) exceeds the maximum limit and the queue is removed

When a message appears above situation, if the message is discarded RabbitMQ not want, you can save the data by providing a dead letter switch.

Badmail effect switch that is: when the message can not be correctly consumer spending, routed to another queue, waiting to retry or manual intervention.

Statement queue, dead letter switch, the message sent to the expiration deadExchange rerouted to the queue.

@Bean
public Queue queue(){
	Map<String, Object> map = new HashMap<>();
	//设置过期时间
	map.put("x-message-ttl", 1000);
	//设置死信交换机
	map.put("x-dead-letter-exchange", "deadExchange");
	//死信交换机路由时新的routingKey
	map.put("x-dead-letter-routing-key", "dead.key");
	return new Queue("emailQueue", true, false, false, map);
}

Message returned, as well.

//消息退回 如果设置了死信交换机，则会被发送到死信交换机重新路由
channel.basicNack(deliveryTag,false,false);

Avoid endless loop

Require special attention when using RabbitMQ avoid an infinite loop, the message is always the consumer can not be correct, but RabbitMQ continue to send a message to consumers, resulting in an infinite loop, for example:

• Consumers only one time, a message rollback messages back and let Queue
• Dead Letter Queue is set to switch the original switch