Ali mns large message transmission

Very large message transfer

Update time: 2016-11-23 17:16:08

problem background

The maximum message size limit of the Alibaba Cloud Message Service MNS queue is 64K, which basically meets the requirements of messages as a control flow information exchange channel under normal circumstances. However, in some special scenarios, when the message data is relatively large, only message fragmentation can be used.

So how can we deliver messages larger than 64K based on MNS without message slicing? There is a solution.

solution

1. Before the producer sends a message to MNS, if it finds that the message body is larger than 64K, it will upload the message body data to OSS first;

2. Then, the producer sends the Objcet information corresponding to the data to MNS as a message;

3. The consumer reads the message from the MNS queue and determines whether the message content is the object information of OSS;

4. If the message content is the object information of the OSS, download the corresponding object content from the OSS and return it to the upper-layer program as the message body;

5. For messages whose size is less than 64K, go directly to MNS.

The specific process is shown in the following figure:

Program implementation

The large message sample code provides a Java language version of the above scheme. The main functions are encapsulated into classes: BigMessageSizeQueue

The public methods provided by BigMessageSizeQueue are as follows:

1. publicBigMessageSizeQueue(CloudQueue cq,OSSClient ossClient,String ossBucketName)
2. //构造函数，cq为普通的mnsqueue对象，ossClient和ossBucketName包含了大消息中转的oss region和bucket
3. publicMessage putMessage(Message message)// 发送消息
4. publicMessage popMessage(int waitSeconds)// 接收消息
5. publicvoid deleteMessage(String receiptHandle)//删除消息
6. publicvoid setBigMessageSize(long bigMessageSize)//设置大消息的阈值（大于这个值的消息会走OSS），默认64K；
7. publicvoid setNeedDeleteMessageObjectOnOSSFlag(boolean flag)// 设置是否需要删除OSS上的消息，默认yes；

Please refer to the code in Demo.java in the attachment for the specific example code.

Precautions

1. The main message network bandwidth for large messages is the network bandwidth of producers and consumers when using this scheme to send large-size messages.

2. The network transmission time of large messages is longer, and the probability of being affected by network fluctuations is greater. It is recommended to do necessary retry at the upper layer.