Table of contents
2. Simulate message queue based on List structure
4.1 Stream-based single-consumer mode
4.2 Stream-based message queue-consumer group
4.3 The Stream structure is used as a message queue to realize asynchronous order placement
1. What is a message queue?
Literally means a queue for storing messages . The simplest message queue model includes 3 roles:
Message queue : store and manage messages, also known as message broker (Message Broker )
Producer : send message to message queue
Consumers : Get messages from message queues and process them
Redis provides three different ways to implement message queues:
(1) List structure : Simulate message queue based on List structure
(2) PubSub : Basic peer-to-peer messaging model
(3) Stream : A relatively complete message queue model
2. Simulate message queue based on List structure
The list-based queue is very simple. It uses the combination of LPUSH and RPOP to achieve the effect of the queue. I have already learned it when I learned the List type structure before.
However, it should be noted that when there is no message in the queue, the RPOP or LPOP operation will return null, and it will not block and wait for the message like the blocking queue of the JVM. Therefore, BRPOP or BLPOP should be used here to achieve the blocking effect.
This implementation is relatively simple and has its own advantages and disadvantages.
advantage:
Using Redis storage, not limited by the upper limit of JVM memory
Based on Redis persistence mechanism, data security is guaranteed
can satisfy message order
shortcoming:
Unable to avoid message loss
Only supports single consumer
3. PubSub-based message queue
PubSub (publish-subscribe) is a messaging model introduced in Redis 2.0 . As the name implies, consumers can subscribe to one or more channels (channels), and after producers send messages to the corresponding channels, all subscribers can receive relevant messages.
Its characteristic is that different consumers can subscribe to different channels and support multiple consumers to consume.
grammar:
SUBSCRIBE channel [channel] : Subscribe to one or more channels
PUBLISH channel msg : send a message to a channel
PSUBSCRIBE pattern[pattern] : subscribe to all channels matching the pattern pattern
As shown in the figure, the top is the producer and the bottom is the consumer:
The advantage of this method is that it can achieve multi-consumption , but it also has many defects .
Including: does not support data persistence, cannot avoid message loss (only the latest message can be read), there is an upper limit for message accumulation, and data loss when exceeded (because it is stored in memory)
4. Stream-based message queue
Stream is a new data type introduced by Redis 5.0, which can implement a very complete message queue.
4.1 Stream-based single-consumer mode
There are two main commands here:
1. Command to send a message (XAdd):
For example:
users is the queue name, * represents the automatically generated queue ID, name jack age 21 is the message
2. One of the ways to read messages: XREAD
Focus on the ID part here, 0 means read from the first one, and $ means read the latest news, which is the last one.
For example, to read the first message using XREAD :
In business development, we can call the XREAD blocking method cyclically to query the latest news, so as to achieve the effect of continuously monitoring the queue . The pseudo code is as follows:
Note here: When we specify the start ID as $ , it means to read the latest message . If more than one message arrives in the queue during the process of processing a message , we can only get it next time. The latest one , there will be a problem of missing messages.
Features of the XREAD command based on the STREAM type message queue:
Messages can be traced back
A message can be read by multiple consumers
can block reads
There is a risk of missing messages
4.2 Stream-based message queue-consumer group
Consumer Group: Divide multiple consumers into a group and listen to the same queue. Has the following characteristics:
In layman's terms, multiple consumers are in a competitive relationship in a queue, and multiple consumers process queue messages to speed up message processing. And the consumer group will add a logo to the message to record the latest read message. If the message is processed but not submitted, the message will enter the pending state and enter the pending-list, which will not cause data loss .
grammar:
Create a consumer group
key: queue name
groupName: consumer group name
ID: start ID mark, $ represents the last message in the queue, 0 represents the first message in the queue
MKSTREAM: automatically create a queue when the queue does not exist
Delete the specified consumer group (ket is the queue, groupname is the group name)
XGROUP DESTORY key groupNameAdd a consumer to the specified consumer group (consumername is the consumer name)
XGROUP CREATECONSUMER key groupname consumernameDelete the specified consumer in the consumer group
XGROUP DELCONSUMER key groupname consumernameRead messages from consumer group
XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...]group: consumer group name
consumer: consumer name, if the consumer does not exist, a consumer will be created automatically
count: the maximum number of this query
BLOCK milliseconds: the longest waiting time when there is no message
NOACK: no manual ACK, get it Automatically confirm
STREAMS key after the message: specify the queue name
ID: get the start ID of the message
Note here: ID is not only 0 and $ symbols.
">": start from the next unconsumed message
If the consumer has acquired the message but has not submitted the message : Get the consumed but unconfirmed message from the pending-list according to the specified id, such as 0, starting from the first message in the pending-list.
View the messages in the Pending-List:
Get unread messages in Pending-List and submit.
The basic idea of consumers listening to messages (pseudocode):
Finally, a small comparison:
4.3 The Stream structure is used as a message queue to realize asynchronous order placement
need:
-
Create a message queue of type Stream named stream.orders
-
Modify the previous Lua script for placing an order in seconds, and add a message directly to stream.orders after confirming that you are eligible for snap-up, and the content includes voucherId, userId, and orderId
-
When the project starts, start a thread task, try to get the message in stream.orders, and complete the order\
1. Modify the Lua script and send a message to the stream queue after judging that the order condition is met
---
--- Generated by EmmyLua(https://github.com/EmmyLua)
--- Created by 华子.
--- DateTime: 2022/12/11 19:12
---
--需要用到的参数
-- 1.优惠券id
local voucherId = ARGV[1];
-- 2.用户id
local userId = ARGV[2];
-- 3. 订单Id
local orderId = ARGV[3];
--数据key
--1.库存key
local stockKey = "seckill:stock"..voucherId;
--2.订单key
local orderKey = "seckill:order".. voucherId
--业务脚本
--1.判断库存是否充足
if(tonumber(redis.call('get',stockKey) <=0)) then
--库存不足，返回1
return 1
end
--2.判断用户是否下单
if (redis.call('sismember',orderKey,userId)) then
--用户已经存在，不可重复下单
return 2
end
-- 可以正常执行逻辑，扣库存 incrby stockKey -1
redis.call('incrby',stockKey,-1);
-- 下单(保存用户)
redis.call('sadd',orderKey,userId);
--发送消息到队列中 XADD stream.orders * k1 v1 k2 v2
redis.call('xadd','stream.orders','*','userId',userId,'voucherId',voucherId,'id',orderId)
return 0;2. Java code, abandon the code that originally read the blocking queue message, start a thread to read the thread message, and perform asynchronous message processing to place an order.
Focus on executing the asynchronous order code
package com.hmdp.service.impl;
import cn.hutool.core.bean.BeanUtil;
import com.hmdp.dto.Result;
import com.hmdp.entity.VoucherOrder;
import com.hmdp.mapper.VoucherOrderMapper;
import com.hmdp.service.ISeckillVoucherService;
import com.hmdp.service.IVoucherOrderService;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.hmdp.utils.RedisIdWorker;
import com.hmdp.utils.UserHolder;
import lombok.extern.slf4j.Slf4j;
import org.redisson.api.RLock;
import org.redisson.api.RedissonClient;
import org.springframework.aop.framework.AopContext;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.redis.connection.stream.*;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import java.time.Duration;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* <p>
* 服务实现类
* </p>
*
* @author 虎哥
* @since 2021-12-22
*/
@Service
@Slf4j
public class VoucherOrderServiceImpl extends ServiceImpl<VoucherOrderMapper, VoucherOrder> implements IVoucherOrderService {
@Resource
private ISeckillVoucherService iSeckillVoucherService;
@Resource
private RedisIdWorker redisIdWorker;
@Resource
private StringRedisTemplate stringRedisTemplate;
@Resource
private RedissonClient redissonClient;
//获取lua脚本
private static final DefaultRedisScript<Long> SECKILL_SCRIPT;
static {
SECKILL_SCRIPT = new DefaultRedisScript<>();
SECKILL_SCRIPT.setLocation(new ClassPathResource("seckill.lua"));
SECKILL_SCRIPT.setResultType(Long.class);
}
//获取线程池
private static final ExecutorService SECKILL_ORDER_EXECUTOR = Executors.newSingleThreadExecutor();
//在类初始化的是否就执行异步下单的任务
@PostConstruct
public void init(){
SECKILL_ORDER_EXECUTOR.submit(new VoucherOrderHandle());
}
//执行异步下单的任务
private class VoucherOrderHandle implements Runnable{
String queName = "stream.orders";
@Override
public void run() {
while (true){
try {
//1.获取消息队列中的订单信息XGROUP GROUP g1 c1 COUNT 1 Block 2000 STREAMS stream.orders >
List<MapRecord<String, Object, Object>> list = stringRedisTemplate.opsForStream().read(
Consumer.from("g1", "c1"),
StreamReadOptions.empty().count(1).block(Duration.ofSeconds(2)),
StreamOffset.create(queName, ReadOffset.lastConsumed())// lastConsumed 代表>号
);
//2.判断消息获取是否成功
if (list == null || list.isEmpty()){
//2.1如果失败，继续下次循环
continue;
}
//3.解析获取到的消息list
MapRecord<String, Object, Object> records = list.get(0);//获取消息中的第一个结果
Map<Object, Object> value = records.getValue();//getValue()方法获取结果中的键值对，也就是我们的key，value
//3.1转成VoucherOrder对象
VoucherOrder voucherOrder = BeanUtil.fillBeanWithMap(value, new VoucherOrder(), true);
//4.创建订单
handlerVoucherOrder(voucherOrder);
//5.ACK提交
stringRedisTemplate.opsForStream().acknowledge(queName,"g1",records.getId());
} catch (Exception e) {
handlePendingList();
log.error("获取订单信息异常{}",e);
}
}
}
private void handlePendingList() {
while (true){
try {
//1.获取pending-list队列中的订单信息XGROUP GROUP g1 c1 COUNT 1 STREAMS stream.orders 0
List<MapRecord<String, Object, Object>> list = stringRedisTemplate.opsForStream().read(
Consumer.from("g1", "c1"),
StreamReadOptions.empty().count(1),
StreamOffset.create(queName, ReadOffset.from("0"))// lastConsumed 代表>号
);
//2.判断消息获取是否成功
if (list == null || list.isEmpty()){
//2.1如果失败，说明pending-list队列中没有消息，结束循环
break;
}
//3.解析获取到的消息list
MapRecord<String, Object, Object> records = list.get(0);//获取消息中的第一个结果
Map<Object, Object> value = records.getValue();//getValue()方法获取结果中的键值对，也就是我们的key，value
//3.1转成VoucherOrder对象
VoucherOrder voucherOrder = BeanUtil.fillBeanWithMap(value, new VoucherOrder(), true);
//4.创建订单
handlerVoucherOrder(voucherOrder);
//5.ACK提交
stringRedisTemplate.opsForStream().acknowledge(queName,"g1",records.getId());
} catch (Exception e) {
log.error("获取订单信息异常{}",e);
try {
Thread.sleep(20);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
}
}
//获取锁
private void handlerVoucherOrder(VoucherOrder voucherOrder) {
//获取用户id
Long userId = voucherOrder.getUserId();
//1.创建锁对象
//SimpleRedisLock lock = new SimpleRedisLock(stringRedisTemplate, "order:" + userId);
RLock lock = redissonClient.getLock("order:" + userId);
//2.尝试获取锁
boolean isLock = lock.tryLock();
if (!isLock){
//获取锁失败
log.error("获取锁失败");
return;
}
try {
proxy.createVoucherOrder(voucherOrder);
} finally {
//释放锁
lock.unlock();
}
}
private IVoucherOrderService proxy;
@Override
public Result seckillVoucher(Long voucherId) {
Long userId = UserHolder.getUser().getId();
long orderId = redisIdWorker.nextId("order");
//1.执行lua脚本
Long result = stringRedisTemplate.execute(
SECKILL_SCRIPT,
Collections.emptyList(),
voucherId.toString(), userId.toString(),String.valueOf(orderId)
);
//2.判断返回结果是否为0
int r = result.intValue();
if (r != 0) {
//3.如果不为0，代表没有下单资格
Result.fail(r==1?"库存不足!":"不可重复下单!");
}
//5.获取代理对象
proxy = (IVoucherOrderService) AopContext.currentProxy();
//6.返回一个订单id
return Result.ok(orderId);
}
//创建订单
@Transactional
public void createVoucherOrder(VoucherOrder voucherOrder) {
Long userId = voucherOrder.getUserId();
//6.根据优惠券id和用户id判断订单是否已经存在
//如果存在，则返回错误信息
int count = query().eq("user_id", userId).eq("voucher_id", voucherOrder.getVoucherId()).count();
if (count > 0) {
log.error("用户已经购买!");
return;
}
boolean success = iSeckillVoucherService.update()
.setSql("stock=stock-1")
.eq("voucher_id", voucherOrder.getVoucherId()).gt("stock", 0)
.update();
if (!success){
//扣减失败
log.error("扣减失败");
return;
}
save(voucherOrder);
}
}