Prospect Review

Bulk message

For the sending of messages, sometimes it may be necessary to send more than one at a time, such as log messages.

Bulk operations can improve performance.

In this section, the old horse will add a little batch feature with you.

batch

Bulk sending of messages

Producer implementation

Interface definition

/**
 * 同步发送消息-批量
 * @param mqMessageList 消息类型
 * @return 结果
 * @since 0.1.3
 */
SendBatchResult sendBatch(final List<MqMessage> mqMessageList);

/**
 * 单向发送消息-批量
 * @param mqMessageList 消息类型
 * @return 结果
 * @since 0.1.3
 */
SendBatchResult sendOneWayBatch(final List<MqMessage> mqMessageList);
复制代码

Supports sending multiple messages at a time.

interface implementation

The producer is implemented as follows.

@Override
public SendBatchResult sendBatch(List<MqMessage> mqMessageList) {
    final List<String> messageIdList = this.fillMessageList(mqMessageList);
    final MqMessageBatchReq batchReq = new MqMessageBatchReq();
    batchReq.setMqMessageList(mqMessageList);
    String traceId = IdHelper.uuid32();
    batchReq.setTraceId(traceId);
    batchReq.setMethodType(MethodType.P_SEND_MSG_BATCH);
    return Retryer.<SendBatchResult>newInstance()
            .maxAttempt(maxAttempt)
            .callable(new Callable<SendBatchResult>() {
                @Override
                public SendBatchResult call() throws Exception {
                    return doSendBatch(messageIdList, batchReq, false);
                }
            }).retryCall();
}

@Override
public SendBatchResult sendOneWayBatch(List<MqMessage> mqMessageList) {
    List<String> messageIdList = this.fillMessageList(mqMessageList);
    MqMessageBatchReq batchReq = new MqMessageBatchReq();
    batchReq.setMqMessageList(mqMessageList);
    String traceId = IdHelper.uuid32();
    batchReq.setTraceId(traceId);
    batchReq.setMethodType(MethodType.P_SEND_MSG_ONE_WAY_BATCH);
    return doSendBatch(messageIdList, batchReq, true);
}


private SendBatchResult doSendBatch(List<String> messageIdList,
                               MqMessageBatchReq batchReq,
                               boolean oneWay) {
    log.info("[Producer] 批量发送消息 messageIdList: {}, batchReq: {}, oneWay: {}",
            messageIdList, JSON.toJSON(batchReq), oneWay);
    // 以第一个 sharding-key 为准。
    // 后续的会被忽略
    MqMessage mqMessage = batchReq.getMqMessageList().get(0);
    Channel channel = getChannel(mqMessage.getShardingKey());
    //one-way
    if(oneWay) {
        log.warn("[Producer] ONE-WAY send, ignore result");
        return SendBatchResult.of(messageIdList, SendStatus.SUCCESS);
    }
    MqCommonResp resp = callServer(channel, batchReq, MqCommonResp.class);
    if(MqCommonRespCode.SUCCESS.getCode().equals(resp.getRespCode())) {
        return SendBatchResult.of(messageIdList, SendStatus.SUCCESS);
    }
    throw new MqException(ProducerRespCode.MSG_SEND_FAILED);
}
复制代码

ps: There is a difference between this and a single send, and that is the choice of channel. Because only one can be selected, the sharding-key of each message cannot be taken into account.

Handling of Brokers

message distribution

// 生产者消息发送-批量
if(MethodType.P_SEND_MSG_BATCH.equals(methodType)) {
    return handleProducerSendMsgBatch(channelId, json);
}

// 生产者消息发送-ONE WAY-批量
if(MethodType.P_SEND_MSG_ONE_WAY_BATCH.equals(methodType)) {
    handleProducerSendMsgBatch(channelId, json);
    return null;
}
复制代码

Implementation

/**
 * 处理生产者发送的消息
 *
 * @param channelId 通道标识
 * @param json 消息体
 * @since 0.1.3
 */
private MqCommonResp handleProducerSendMsgBatch(String channelId, String json) {
    MqMessageBatchReq batchReq = JSON.parseObject(json, MqMessageBatchReq.class);
    final ServiceEntry serviceEntry = registerProducerService.getServiceEntry(channelId);
    List<MqMessagePersistPut> putList = buildPersistPutList(batchReq, serviceEntry);

    MqCommonResp commonResp = mqBrokerPersist.putBatch(putList);

    // 遍历异步推送
    for(MqMessagePersistPut persistPut : putList) {
        this.asyncHandleMessage(persistPut);
    }
    return commonResp;
}
复制代码

Here, the message list is persisted.

The demonstrated persistence strategy is as follows:

@Override
public MqCommonResp putBatch(List<MqMessagePersistPut> putList) {
    // 构建列表
    for(MqMessagePersistPut put : putList) {
        this.doPut(put);
    }

    MqCommonResp commonResp = new MqCommonResp();
    commonResp.setRespCode(MqCommonRespCode.SUCCESS.getCode());
    commonResp.setRespMessage(MqCommonRespCode.SUCCESS.getMsg());
    return commonResp;
}
复制代码

Bulk ACK for messages

illustrate

The previous implementation is to perform an ACK after each message consumption is completed.

For the message consumption of the pull strategy, we can wait for the end of the current batch and issue an ACK receipt uniformly.

Consumption realization

The adjustment is implemented as follows:

for(MqTopicTagDto tagDto : subscribeList) {
    final String topicName = tagDto.getTopicName();
    final String tagRegex = tagDto.getTagRegex();
    MqConsumerPullResp resp = consumerBrokerService.pull(topicName, tagRegex, size);

    if(MqCommonRespCode.SUCCESS.getCode().equals(resp.getRespCode())) {
        List<MqMessage> mqMessageList = resp.getList();
        if(CollectionUtil.isNotEmpty(mqMessageList)) {
            List<MqConsumerUpdateStatusDto> statusDtoList = new ArrayList<>(mqMessageList.size());
            for(MqMessage mqMessage : mqMessageList) {
                IMqConsumerListenerContext context = new MqConsumerListenerContext();
                final String messageId = mqMessage.getTraceId();
                ConsumerStatus consumerStatus = mqListenerService.consumer(mqMessage, context);
                log.info("消息：{} 消费结果 {}", messageId, consumerStatus);

                // 状态同步更新
                if(!ackBatchFlag) {
                    MqCommonResp ackResp = consumerBrokerService.consumerStatusAck(messageId, consumerStatus);
                    log.info("消息：{} 状态回执结果 {}", messageId, JSON.toJSON(ackResp));
                } else {
                    // 批量
                    MqConsumerUpdateStatusDto statusDto = new MqConsumerUpdateStatusDto();
                    statusDto.setMessageId(messageId);
                    statusDto.setMessageStatus(consumerStatus.getCode());
                    statusDto.setConsumerGroupName(groupName);
                    statusDtoList.add(statusDto);
                }
            }

            // 批量执行
            if(ackBatchFlag) {
                MqCommonResp ackResp = consumerBrokerService.consumerStatusAckBatch(statusDtoList);
                log.info("消息：{} 状态批量回执结果 {}", statusDtoList, JSON.toJSON(ackResp));
                statusDtoList = null;
            }
        }
    } else {
        log.error("拉取消息失败: {}", JSON.toJSON(resp));
    }
}
复制代码

If ackBatchFlag = false, the processing logic is the same as before.

If ackBatchFlag = true, put the message in the list first, and execute it uniformly after the end.

broker implementation

message distribution

//消费者消费状态 ACK-批量
if(MethodType.C_CONSUMER_STATUS_BATCH.equals(methodType)) {
    MqConsumerUpdateStatusBatchReq req = JSON.parseObject(json, MqConsumerUpdateStatusBatchReq.class);
    final List<MqConsumerUpdateStatusDto> statusDtoList = req.getStatusList();
    return mqBrokerPersist.updateStatusBatch(statusDtoList);
}
复制代码

accomplish

The default persistence implementation, updated as follows:

@Override
public MqCommonResp updateStatusBatch(List<MqConsumerUpdateStatusDto> statusDtoList) {
    for(MqConsumerUpdateStatusDto statusDto : statusDtoList) {
        this.doUpdateStatus(statusDto.getMessageId(), statusDto.getConsumerGroupName(),
                statusDto.getMessageStatus());
    }

    MqCommonResp commonResp = new MqCommonResp();
    commonResp.setRespCode(MqCommonRespCode.SUCCESS.getCode());
    commonResp.setRespMessage(MqCommonRespCode.SUCCESS.getMsg());
    return commonResp;
}
复制代码

Traverse each element and update the state.

summary

Asynchronous and batch are the two most common ways to improve performance.

Batch implementation is the easiest and most effective.

I hope this article is helpful to you. If you like it, please like, collect and forward it.

I'm an old horse, and I look forward to seeing you again next time.

open source address

The message queue in java.(java simple version mq implementation) github.com/houbb/mq

Extended reading

rpc - Implementing rpc from scratch github.com/houbb/rpc

[mq] Realize batch sending and receipt of mq-12-messages from scratch

Prospect Review

Bulk message

Bulk sending of messages

Producer implementation

Interface definition

interface implementation

Handling of Brokers

message distribution

Implementation

Bulk ACK for messages

illustrate

Consumption realization

broker implementation

message distribution

accomplish

summary

open source address

Extended reading

Guess you like