Article Directory
- Redis optimization spike, Redis message queue to achieve asynchronous spike
- 1. Second kill optimization
- Two, Redis message queue
- 3. Realize asynchronous seckill based on Stream message queue
Redis optimization spike, Redis message queue to achieve asynchronous spike
Undertake Redis - Coupon spike, oversold inventory, distributed locks, Redisson articles
1. Second kill optimization
1.1 Review of the "one person, one order" seckill business code
@Service
public class VoucherOrderServiceImpl extends ServiceImpl<VoucherOrderMapper, VoucherOrder> implements IVoucherOrderService {
@Resource
private ISeckillVoucherService seckillVoucherService;
@Resource
private RedisIdWorker redisIdWorker;
@Resource
private StringRedisTemplate stringRedisTemplate;
@Resource
private RedissonClient redissonClient;
@Override
public Result seckillVoucher(Long voucherId) {
// 1.查询优惠券
SeckillVoucher voucher = seckillVoucherService.getById(voucherId);
// 2.判断秒杀是否开始
if (voucher.getBeginTime().isAfter(LocalDateTime.now())) {
return Result.fail("秒杀活动尚未开始");
}
// 3.判断秒杀是否结束
if (voucher.getEndTime().isBefore(LocalDateTime.now())) {
return Result.fail("秒杀活动已经结束");
}
// 4.判断库存是否充足
if (voucher.getStock() < 1) {
return Result.fail("库存不足");
}
UserDTO user = UserHolder.getUser();
// 创建锁对象
// 锁定的范围与之前一样。切记不能把order锁住，范围太大了，以后有关order的都被锁住了
// 之前的方式：SimpleRedisLock lock = new SimpleRedisLock("order:" + user.getId(), stringRedisTemplate);
// TODO 使用Redisson客户端获取锁对象
RLock lock = redissonClient.getLock("lock:order:" + user.getId());
// 获取锁
// 订单大概是500ms，我们这里可以设定为秒
boolean isLock = lock.tryLock();
// 判断是否获取锁成功
if (!isLock) {
// 不成功
// 我们要避免一个用户重复下单,既然获取锁失败，说明在并发执行，我们要避免并发执行
return Result.fail("不允许重复下单");
}
// 成功
// createVoucherOrder方法执行过程中可能会有异常，我们放到try...catch中
try {
// 获取当前对象的代理对象 强转
VoucherOrderServiceImpl proxy = (VoucherOrderServiceImpl) AopContext.currentProxy();
return proxy.createVoucherOrder(voucherId);
}finally {
// 出现异常做锁的释放
lock.unlock();
}
}
// 如果在方法上添加synchronized,说明同步锁是this，当前对象
// 不建议 把synchronized放在方法上，锁住此对象后，不管任何一个用户来了，都是这把锁，也就意味着整个方法被串行化了
// 所谓“一人一单”，只需要对同一个用户加锁即可，如果不是同一个用户，无需加锁
@Transactional
public Result createVoucherOrder(Long voucherId) {
// 新增一人一单的判断
UserDTO user = UserHolder.getUser();
// user.getId().toString()转换成字符串也无法保证线程安全，因为每次的String都不一样
// 我们可以加一个intern，是一个字符串对象规范表示，回去字符串常量池中找一找和此字符串的值一样的字符串地址并返回
// 查询订单
int count = query().eq("user_id", user.getId())
.eq("voucher_id", voucherId).count();
// 判断是否存在
if (count > 0) {
// 用户至少下过一单，不能再下了
return Result.fail("一人一单，不可重复下单");
}
// 说明没买过，继续执行代码
// 5.扣减库存
boolean success = seckillVoucherService.update()
.setSql("stock = stock-1") //set stock = stock-1
.eq("voucher_id", voucherId) //where voucher_id= voucherId
.gt("stock", 0)//where stock>0
.update();
if (!success) {
return Result.fail("扣减失败，可能库存不足");
}
// 6.创建订单
// 我们只管订单id，代金券id，下单用户id
VoucherOrder voucherOrder = new VoucherOrder();
// 6.1 订单id
// 使用自定义id生成器生成id
long orderID = redisIdWorker.nextId("order");
voucherOrder.setId(orderID);
// 6.2 用户id
// 我们之前编写的登录拦截器
voucherOrder.setUserId(user.getId());
// 6.3 代金券id
voucherOrder.setVoucherId(voucherId);
save(voucherOrder);
// 7.返回订单id
return Result.ok(orderID);
}
}
There are a large number of database operations in the code, and the overall business performance is not very good
The average time spent reached 497 milliseconds
1.2 Idea of asynchronous spike
First review the process of the previous seckill business
-
The front end initiates a request to reach our Nginx, and then Nginx will load balance our request to our tomcat
-
However, to execute various logics in tomcat, query coupons, judge flash inventory, query orders, verify one order per person, reduce inventory, and create orders. The time-consuming of the entire business is the sum of the time-consuming of various logics
Among them, query coupons, query orders, reduce inventory, and create orders to access the database in four steps. The concurrency capability of the database itself is relatively poor, not to mention that inventory reduction and order creation are write operations to the database. In order to avoid security issues, distributed locks are added. , which also greatly reduces business performance
How to improve it?
Divide the business logic into two different threads. The function of the main thread is to judge the purchase qualification of the user. If qualified, an independent thread is opened to handle the time-consuming operation of reducing inventory and creating orders. In this way, the efficiency Greatly improve
The performance of Redis is better than that of MYSQL. We can completely put the operation of judging the spike inventory and checking the operation of one person one order into Redis.
After the main thread comes in, it first finds Redis to complete the judgment on the eligibility for seckill. If it is qualified, it will perform subsequent operations of reducing inventory and placing orders.
How do we judge the flash sale inventory and check one person one order in Redis?
We need to cache the coupon inventory information and related order information in redis. What kind of data structure should we choose?
-
Inventory is relatively simple , you only need a common String structure, the key is the id of the coupon, and the value is the value of the inventory. When making inventory judgments in the future, check to see if the value is greater than 0
When it is judged that there is stock, the value should be reduced by 1, and this should be reduced in advance
-
One order per person , we need to record in Redis which users have purchased the current coupon, and when another user comes in the future, we only need to determine whether this user has purchased
For this, we use the set collection, which can ensure the uniqueness of the elements, and can store multiple values in a key. After a user places an order successfully in the future, we will record the user's id, and then use more users, we will record in turn That's fine. If it is found that the set already exists, then the purchase is definitely not allowed
The final flow chart in Redis
There are many judgments on Redis. The business process is relatively long. We must ensure atomicity during execution. At this time, we need to use Lua scripts.
This part does not need to write java code, Lua script is required
What do we need to do to enter Tomcat?
Core write operations such as placing an order and reducing inventory that take a long time do not appear in the current process
When do you place an order to reduce inventory?
We open an independent thread to read the user information and coupon information saved in advance, and then we can complete the write operation of the asynchronous database. When we return the order id to the user, the user can pay, and the seckill business is over. up
So when do we write coupon information and user information into the database, and complete the operation of placing an order and reducing inventory, it is actually not that important (the requirements for effectiveness are not high), and the data is written according to the frequency that the MySQL database can bear into the database
1.3 Judgment of eligibility for seckill based on Redis
Case : Improve seckill business and improve concurrency performance
Requirements :
- Save the coupon information to Redis while adding the flash coupon
- Based on Lua script, judge the flash sale inventory, one order per person, and determine whether the user snapped up successfully
- If the snap-up is successful, encapsulate the coupon id and user id and store them in the blocking queue
- Start the thread task, continuously obtain information from the blocking queue, and realize the function of placing an order asynchronously
1.3.1 Modify VoucherServiceImpl
Save the coupon information to Redis while adding the flash coupon
Coupon inventory information is stored permanently without expiration time
@Override
@Transactional
public void addSeckillVoucher(Voucher voucher) {
// 保存优惠券
save(voucher);
// 保存秒杀信息
SeckillVoucher seckillVoucher = new SeckillVoucher();
seckillVoucher.setVoucherId(voucher.getId());
seckillVoucher.setStock(voucher.getStock());
seckillVoucher.setBeginTime(voucher.getBeginTime());
seckillVoucher.setEndTime(voucher.getEndTime());
seckillVoucherService.save(seckillVoucher);
// TODO 优惠券库存信息保存到Redis中
stringRedisTemplate.opsForValue().set("seckill:stock:"+voucher.getId(),voucher.getStock().toString());
}
have a test
{
"shopId": 1,
"title": "100元代金券",
"subTitle": "周一至周五均可使用",
"rules": "全场通用\n无需预约\n可无限叠加\n不兑现、不找零\n仅限堂食",
"payValue": 8000,
"actualValue": 10000,
"type": 1,
"stock": 10,
"beginTime": "2022-01-25T10:09:17",
"endTime": "2025-01-26T12:09:04"
}
The first requirement is completed
1.3.2 Lua script writing
Based on Lua script, judge the flash sale inventory, one order per person, and determine whether the user snapped up successfully
Writing Lua scripts
To use messy script to complete the following content
-- 1. 参数列表
-- 1.1 优惠券id
-- 需要去redis中读取库存数量，判断是否充足，其中key的前缀seckill:stock是固定的，后面的id是需要传入的
local voucherId = ARGV[1]
-- 1.2 用户id
-- 需要知道用户id，才能判断用户之前是否下过单
local userId = ARGV[2]
-- 2.数据相关key
-- 2.1 库存key, lua中是用 .. 拼接字符串
local stockKey = 'seckill:stock:' .. voucherId
-- 2.2 订单key，值是一个set集合，集合名称就是下面。内容是购买订单的用户的id，这样可以记录谁购买了谁没有购买
local orderKey = 'seckill:order:' .. voucherId
-- 3.lua脚本业务
-- 3.1 判断库存是否充足
-- redis.call('get',stockKey)得到的结果是字符串，是无法和数字比较的
if (tonumber(redis.call('get',stockKey))<=0) then
-- 3.2库存不足
return 1
end
-- 3.2判断用户是否下单
-- 借助命令SISMEMBER命令，判断一个给定的值是不是当前set集合中的一个成员，如果存在返回1，不存在返回0
if (redis.call('SISMEMBER',orderKey,userId) ==1) then
-- 3.3redis中存在，说明是重复下单
return 2
end
-- 3.4 扣库存 incrby stockKey -1
redis.call('incrby',stockKey,-1)
-- 3.5 下单，保存用户 sadd orderKey userId
redis.call('sadd',orderKey,userId)
-- 成功返回0
return 0
1.3.3 Redis+lua judges whether the user snapped up successfully
If the snap-up is successful, encapsulate the coupon id and user id and store them in the blocking queue, and complete the following operations
Transform the VoucherOrderServiceImpl class
No blocking queues have been written yet
// TODO 获取脚本
// RedisScript是一个接口，我们使用一个实现类，泛型就是返回值的类型
private static final DefaultRedisScript<Long> SECKILL_SCRIPT;
// TODO 静态代码块做初始化
static {
SECKILL_SCRIPT = new DefaultRedisScript<>();
// TODO 借助Spring提供的方法区resource下找
SECKILL_SCRIPT.setLocation(new ClassPathResource("seckill.lua"));
// TODO 配置一下返回值类型
SECKILL_SCRIPT.setResultType(Long.class);
}
@Override
public Result seckillVoucher(Long voucherId) {
Long userId = UserHolder.getUser().getId();
// TODO 1.执行lua脚本
// 我们没有传入key，所以传入一个空集合即可
Long result = stringRedisTemplate.execute(
SECKILL_SCRIPT,
Collections.emptyList(),
voucherId.toString(), userId.toString()
);
// TODO 2.判断结果是否为0（0有购买资格，不为0没有购买资格）
int r = result.intValue();
if (r!=0){
// TODO 2.1 不为0没有购买资格
return Result.fail(r==1 ? "库存不足":"不能重复下单");
}
// TODO 2.2 为0有购买资格,把下单的信息保存到堵塞队列,后续可以异步完成下单业务
// 生成订单id
long orderId = redisIdWorker.nextId("order");
// 组合队列有点麻烦，先写到这里
// TODO 3. 返回订单id
return Result.ok(orderId);
}
1.3.4 Asynchronous order placement based on blocking queue
The code we just wrote did not write a blocking queue, let's write it
Complete the following functions
- If the snap-up is successful, encapsulate the coupon id and user id and store them in the blocking queue
- Start the thread task, continuously obtain information from the blocking queue, and realize the function of placing an order asynchronously
This place is to encapsulate the user id, coupon id, and related order id that successfully grabbed the order and put them in a queue.
The queue contains information about all orders waiting to be placed
Then we will start a thread task to execute related order operations asynchronously, so that placing an order later will not affect the time-consuming of our entire business
@Service
@Slf4j
public class VoucherOrderServiceImpl extends ServiceImpl<VoucherOrderMapper, VoucherOrder> implements IVoucherOrderService {
@Resource
private ISeckillVoucherService seckillVoucherService;
@Resource
private RedisIdWorker redisIdWorker;
@Resource
private StringRedisTemplate stringRedisTemplate;
@Resource
private RedissonClient redissonClient;
//TODO 获取脚本
// RedisScript是一个接口，我们使用一个实现类，泛型就是返回值的类型
private static final DefaultRedisScript<Long> SECKILL_SCRIPT;
//静态代码块做初始化
static {
SECKILL_SCRIPT = new DefaultRedisScript<>();
// 借助Spring提供的方法区resource下找
SECKILL_SCRIPT.setLocation(new ClassPathResource("seckill.lua"));
// 配置一下返回值类型
SECKILL_SCRIPT.setResultType(Long.class);
}
// TODO 创建堵塞队列
// 有一个特点：当一个线程尝试从这个队列里获取元素时，如果没有元素，这个线程就会堵塞，知道队列中有元素他才会被唤醒
// 1024*1024表示队列的长度
private BlockingQueue<VoucherOrder> orderTasks = new ArrayBlockingQueue<>(1024 * 1024);
// TODO 创建一个线程池
private static final ExecutorService SECKILL_ORDER_EXECUTOR = Executors.newSingleThreadExecutor();
@PostConstruct//当前类初始化完毕后执行这个方法
private void init() {
SECKILL_ORDER_EXECUTOR.submit(new VoucherOrderHandler());
}
// TODO 创建一个线程任务
// 什么时候执行这个任务呢？用户秒杀抢购之前，因为一旦用户开始秒杀，阻塞队列中就会有新的订单，这个任务就应该去取出订单相关信息
private class VoucherOrderHandler implements Runnable {
@Override
public void run() {
// 不断的从堵塞队列orderTasks取出信息，然后执行
while (true) {
// take就是一个阻塞方法，获取队列中的头部,如果没有就等到有
try {
// TODO 1.获取队列中的订单信息
VoucherOrder voucherOrder = orderTasks.take();
// TODO 2.创建订单
handleVoucherOrder(voucherOrder);
} catch (Exception e) {
// e.printStackTrace();
log.error("处理订单异常", e);
}
}
}
}
/**
* 订单处理方法
* 我们在Redisson脚本中加了一个次锁了，在这里为什么还要再加一次锁呢？
* 做一个兜底方案，万一Redis出现问题，有个保证
*
* @param voucherOrder
*/
private void handleVoucherOrder(VoucherOrder voucherOrder) {
// TODO 1.获取用户
Long userId = voucherOrder.getUserId();
// TODO 2.创建锁对象
RLock lock = redissonClient.getLock("lock:order:" + userId);
// TODO 3.获取锁
boolean isLock = lock.tryLock();
// TODO 4.判断是否获取锁成功
if (!isLock) {
// 获取锁失败
log.error("不允许重复下单");
return;
}
// TODO
try {
// 之前这种获取代理对象的方式是获取不到的,因为不在同一个线程下面了
// VoucherOrderServiceImpl proxy = (VoucherOrderServiceImpl) AopContext.currentProxy();
// return proxy.createVoucherOrder(voucherId);
// TODO 在主线程获取，这个地方使用
proxy.createVoucherOrder(voucherOrder);
} finally {
// 出现异常做锁的释放
lock.unlock();
}
}
private VoucherOrderServiceImpl proxy;
@Override
public Result seckillVoucher(Long voucherId) {
Long userId = UserHolder.getUser().getId();
// 1.执行lua脚本
// 我们没有传入key，所以传入一个空集合即可
Long result = stringRedisTemplate.execute(
SECKILL_SCRIPT,
Collections.emptyList(),
voucherId.toString(), userId.toString()
);
// 2.判断结果是否为0（0有购买资格，不为0没有购买资格）
int r = result.intValue();
if (r != 0) {
// 2.1 不为0没有购买资格
return Result.fail(r == 1 ? "库存不足" : "不能重复下单");
}
// TODO 2.2 为0有购买资格,把下单的信息保存到堵塞队列,后续可以异步完成下单业务
// TODO 阻塞队列，将用户id，订单id，优惠券id，保存到阻塞队列中
VoucherOrder voucherOrder = new VoucherOrder();
// TODO 2.3 生成订单id
long orderId = redisIdWorker.nextId("order");
voucherOrder.setId(orderId);
// TODO 2.4 用户id
voucherOrder.setUserId(userId);
// TODO 2.5 代金券id
voucherOrder.setVoucherId(voucherId);
// TODO 2.6 放入阻塞队列
orderTasks.add(voucherOrder);
// TODO 3.获取代理对象
proxy = (VoucherOrderServiceImpl) AopContext.currentProxy();
// 3. 返回订单id
return Result.ok(orderId);
}
// 如果在方法上添加synchronized,说明同步锁是this，当前对象
// 不建议 把synchronized放在方法上，锁住此对象后，不管任何一个用户来了，都是这把锁，也就意味着整个方法被串行化了
// 所谓“一人一单”，只需要对同一个用户加锁即可，如果不是同一个用户，无需加锁
@Transactional
public void createVoucherOrder(VoucherOrder voucherOrder) {
// 新增一人一单的判断
Long userId = voucherOrder.getUserId();
// user.getId().toString()转换成字符串也无法保证线程安全，因为每次的String都不一样
// 我们可以加一个intern，是一个字符串对象规范表示，回去字符串常量池中找一找和此字符串的值一样的字符串地址并返回
// 查询订单
int count = query().eq("user_id", userId)
.eq("voucher_id", voucherOrder.getVoucherId()).count();
// 判断是否存在
if (count > 0) {
// 用户至少下过一单，不能再下了
log.error("用户至少下过一单，不能再下了");
return;
}
// 说明没买过，继续执行代码
// 5.扣减库存
boolean success = seckillVoucherService.update()
.setSql("stock = stock-1") //set stock = stock-1
.eq("voucher_id", voucherOrder.getVoucherId()) //where voucher_id= voucherId
.gt("stock", 0)//where stock>0
.update();
if (!success) {
log.error("扣减失败，可能库存不足");
return;
}
// TODO 创建订单
save(voucherOrder);
}
}
}
1.3.4 Summary of Preliminary Optimization of Seckill
-
What is the optimization idea of seckill business?
Change synchronous order to asynchronous order
Simultaneous ordering: After the request comes, judge whether you are qualified, and if you are qualified, you can place an order yourself. Due to various reasons, it takes a long time
Asynchronous order placement: the business is divided into two parts, one part judges the qualifications for snapping up, if qualified, it will end immediately and return the order number; the other part executes the business that takes a long time
- First use Redis to complete the inventory balance, one-person-one-order judgment, and complete the order grabbing business
- Then put the order business into the blocking queue, and use an independent thread to place the order asynchronously
-
What are the problems of asynchronous seckill based on blocking queue?
- memory limit problem
We are using the blocking queue in JDK. If the JVM memory used is not limited, in the case of high concurrency, there may be countless order objects that need to be created and placed in the blocking queue, which will cause memory overflow (so set the length of a queue)
If the queue is full, it cannot be stored
- Data Security Issues
We now store these order information based on the memory. If the service suddenly goes down, all the order information in the memory will be lost. The user has paid but there is no relevant order data in the background
Two, Redis message queue
Message Queue (Message Queue), a queue for storing messages
The simplest message queue model includes the following roles
- Message queue : store and manage messages, also known as message broker (Message Broker)
- Producer : send message to message queue
- Consumers : Get messages from message queues and process them
Benefits of message queues :
Decoupling for greater efficiency
Imagine as a courier, a takeaway cabinet, and a consumer
Suppose a courier comes to deliver the courier (there is no take-out cabinet), and the delivery is delivered without a courier cabinet, but the consumer is working in the company. At this time, the user tells the courier to wait for me. If the courier waits, it will be a waste The courier's time, if not, the user may not be able to find the takeaway
With the express cabinet, it is different. The delivery staff can pick up the items and put them in the delivery cabinet, and then the user can pick them up whenever they have time. At this time, the courier and the consumer are decoupled.
The same is true for the seckill operation. When someone rushes to buy a product, we don’t rush to actually place an order. We first judge whether we are eligible. If we are eligible to buy, we don’t write to the database at this time, but store the information in the message queue.
At this time, we start an independent thread as a consumer, which continuously obtains messages from the queue, and actually completes the function of placing an order
The function is similar to the reader queue, but there are two differences
- The message queue is an independent service independent of the JVM service and is not limited by the JVM memory
- The message queue is not only used for queue storage, but also to ensure the security of the data. It must be persisted. No matter whether the service is down or restarted, the data will not be lost. After the message is delivered to the consumer, the consumer is required to confirm the message. If the consumer Without confirmation, the message will still exist in the queue, and it will be delivered to the consumer next time, allowing it to continue processing until it succeeds (ensure that the message is consumed at least once)
Redis provides three different ways to implement message queues
- List structure: Simulate message queue based on List structure
- PubSub: the basic peer-to-peer messaging model
- Stream: a relatively complete message queue model
I have also learned two message queues before
2.1 Implement message queue based on List
The List data structure of Redis is a two-way linked list, which can easily simulate the queue effect
When queuing, the entry and exit are not at the same time, so we can use LPUSH combined with RPOP or RPUSH combined with LPOP to achieve
List-related commands: Redis command - general command, String type, Key hierarchy, Hash type, List type, Set type, SortedSet type_redis command to view the hierarchy
but! ! When there is no message in the queue, the RPOP or LPOP operation will return null, unlike the blocking queue of the JVM that will block and code the message
Therefore, BROPO or BLPOP should be used here to achieve the blocking effect
What are the benefits of this approach compared to blocking queues in the JDK ?
-
Independent storage independent of JVM, independent of JVM memory
-
Data security, Redis supports data persistence, after storing in the queue, the data will not be lost
-
It can satisfy the order of messages (queue characteristics: first in first out)
What are the disadvantages ?
-
Unable to avoid message loss
If we take out the message from Redis and hang up before the execution is finished, then this task will be gone in Redis and cannot be executed again
-
Only supports single consumer
Once a task is taken by a consumer, it is removed from the queue and cannot be picked up by other consumers. It is impossible to realize the requirement that a message is consumed by many people
2.2 Implement message queue based on PubSub
PubSub (publish-subscribe) is a messaging model introduced in Redis version 2.0. As the name implies, consumers can subscribe to one or more channels, and after producers send messages to corresponding channels, all subscribers can receive relevant information
Related commands
- SUBSCRIBE channel [channel] : Subscribe to one or more channels
- PUBLISH channel msg : send a message to a channel
- PSUBSCRIBE pattern [pattern] : Subscribe to all channels matching the pattern pattern. pattern is a wildcard
? : represents a character
* : represents 0 or more characters
[ae]: Indicates that it can be a character or e character
Producer : There is basically no change in the party sending the message, except that the channel name must be included when sending the message
Consumer : Multiple consumers can be allowed to subscribe, and you can specify your own channel name when subscribing
Let's demonstrate
Consumer 1 subscribes and finds that it is inherently blocking
Consumer 2 subscribes
The producer publishes the message, and the two consumers can receive the message
What are the characteristics of PubSub-based message queues?
advantage
- Adopt publish-subscribe model to support multi-production and multi-consumption
shortcoming
-
Does not support data persistence
The essence of the List structure is not a message queue, but a linked list, but we use it as a message queue
PubSub itself is designed to send messages. If no one subscribes to the message we publish, the message will be lost.
-
Unable to avoid message loss
If no one receives it after posting, it will be lost
-
There is an upper limit to the accumulation of messages, and data will be lost when it exceeds
2.3 Stream-based message queue
Stream is a kind introduced by Redis 5.0 新数据类型, which can implement a very complete message queue.
Official Website : Commands | Redis
2.3.1 Stream single consumption mode
2.3.1.1 Sending messages
Send message command xadd
- key : queue name
- NOMKSTREAM : If the queue does not exist, whether to create the queue automatically, the default is to create automatically, if this value is given, it will not be created
- MAXLEN | MINID [= | ~] threshold [LIMIT count] : Set the maximum number of messages for the message queue. For example, we set the maximum number of messages in the message queue to 1000. If it exceeds 1000, some old messages will be removed. No upper limit is set if no value is given
- *|ID : Specify the unique id of the message, * means it is automatically generated by Redis, the format is "timestamp-incremental number"
- field value : represents the field and value, and the message body stored in the message is called Entry. The format is multiple key-value key-value pairs, and a key-value pair is an Entry
For example:
2.3.1.2 Read messages
One of the ways to read messages: xread
-
Count count : The maximum number of messages read each time, you can read multiple messages at a time, or you can read one message
-
BLOCK milliseconds : When there is no message, whether to block and the length of the block.
BLOCK If no parameter is given, it will not block, if there is a message, it will return directly, if there is no message, it will return empty; if the parameter is given, it will be blocked, if there is a message, it will return, and if there is no message, it will be blocked;
milliseconds is the waiting time, if it is 0, it means waiting forever
-
STREAMS KEY [KEY ...] : Which queue to read messages from, key is the queue name. A pair of queues can be specified at the same time
-
ID : The initial message id, only returns messages greater than this ID.
0: means start from the first message
$: Represents starting from the latest news
have a test
We found that the same message can be read twice.
It will not be deleted after reading, it will exist permanently
During development, we can call the XREAD blocking method cyclically to query the latest news, so as to achieve the effect of continuously monitoring the queue
while(true){
//尝试读取队列中的消息，最多堵塞2秒
Object msg = redis.execute("XREAD COUNT 1 BLOCK 2000 STREAMS users $");
if(mes == null){
continue;
}
// 处理消息
handleMessage(msg);
}
Notice! When we specify the actual ID as $, it means to read the latest message. If more than one message is sent to the large queue during the process of processing a message, only the latest one can be obtained next time, and a message will appear The problem of missed messages
2.3.1.3 XREAN command characteristics of STREAN type message queue
XREAN command characteristics of STREAN type message queue
- Messages can be traced back
- A message can be read by multiple consumers
- can block reads
- There is a risk of missing messages
2.3.2 Stream consumer group mode
Consumer Group : Divide multiple consumers into a group and listen to the same queue. Have the following characteristics:
-
news diversion
Messages in the queue will be distributed to different consumers in the group instead of repeated consumption, thereby speeding up message processing. To a certain extent, the problem of message accumulation can be avoided
-
message sign
The consumer group will maintain a mark to record the last message processed, even if the consumer is down and restarted, it will still read the message after the mark. Make sure every message is consumed
-
message confirmation
After the consumer gets the message, the message is in the pending state and stored in a pending-list. After processing, the message needs to be confirmed by XACK, and the message is marked as processed before being removed from the pending-list. This perfectly solves the problem of message loss
2.3.2.1 Create a consumer group
Create a consumer group
XGROUP create key groupName ID [MKSTREAM]
-
key:
queue name
-
groupName：
The name of the consumer group
-
ID：
When this group listens to messages, where does it start to listen?
0: means start from the first message
$: Represents starting from the latest news
-
MKSTREAM : Automatically create a queue if it does not exist
Other common commands :
#删除指定的消费者组
XGROUP DESTORY key groupName
# 给指定的消费者组添加消费者
XGROUP CREATECONSUMER key groupname consumername
#删除消费者组中的指定消费者
XGROUP DELCONSUMER key groupnameconsumername
Under normal circumstances, there is no need to add consumers by yourself. When we specify a consumer from the group and listen to messages, if we find that the consumer does not exist, it will automatically create it for us.
2.3.2.2 Read messages
How to monitor news?
XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...]
-
group：
consumer group name
-
consumer：
Consumer name, if the consumer does not exist, a consumer will be created automatically
-
count：
The maximum number of this query
-
BLOCK milliseconds：
When there is no message, whether it is blocked and the length of the blockage.
BLOCK If no parameter is given, it will not block, if there is a message, it will return directly, if there is no message, it will return empty; if the parameter is given, it will be blocked, if there is a message, it will return, and if there is no message, it will be blocked;
milliseconds is the waiting time, if it is 0, it means waiting forever
-
NOACK：
The parameter means that no consumer confirmation is required. If this parameter is not given, it means that consumer confirmation is required, so generally we do not give this parameter
-
STREAMS KEY [KEY …]:
Which queue to read messages from, the key is the queue name. Two queues can be specified at the same time
-
ID：
start message id
">": start from the next unconsumed message
Others: Obtain the consumed but unconfirmed messages from the pending-list according to the specified id, such as 0, starting from the first message in the pending-list
Under normal circumstances, we should give the greater than number to read the unconsumed messages. If there is an abnormal situation, go to the pending-list to read the consumed but unprocessed messages
have a test
We read several messages above, but never confirmed
Be sure to have a message, confirm a message
2.3.2.3 Acknowledgment message
How to confirm the news?
XACK key group ID [ID ...]
-
key
queue name
-
group
group name
-
ID
To confirm which message, turn the pending message into a correctly processed (processed) message, so that it will be removed from the pending-List
test
For example, let's confirm the messages corresponding to the last five ids in the s1 queue and g1 group
2.3.2.4 View pending-list
If we read a message, but there is an exception and no confirmation, the message will be placed in the pending-list, so how do we check the pending-list
Official website: XPENDING | Redis
XPENDING key group [[IDLE min-idle-time] start end count [consumer]]
-
key
queue name
-
group
group name
-
IDLE min-idle-time
free time. After getting the message, confirm the previous period of time.
For example, if we give a parameter of 5000, the messages whose idle time exceeds 5000ms will enter the pending-list queue
-
**start end **
starting range. There are a lot of messages in the pending-list, and these two parameters tell it what the minimum id and maximum id we want to get
-: The minus sign represents the smallest
+: The plus sign represents the largest
-
count
The amount we want to get
-
consumer
Which consumer's pending-list to get
Read the first message in the pending-list
just read normally
XREADGROUP GROUP g1 C1 COUNT 1 BLOCK 200 STREAMS S1 0
2.3.2.5 Consumer group - the basic idea of consumers listening to messages
Use the while loop to get the messages in the consumer group all the time
2.3.2.6 XREANGROUP command features of STREAN type message queue
First of all, it has all the advantages of XREAD single consumption mode, and makes up for some shortcomings
XREAN command characteristics of STREAN type message queue
- Messages can be traced back
- A message can be read by multiple consumers
- can block reads
XREANGROUP command characteristics of STREAN type message queue
-
Messages can be traced back
After the message is consumed, it will not be deleted from the queue, and other people can also come and get it (this is for different groups)
-
Multiple consumers can compete for news and speed up consumption
-
can block read
-
No risk of missing messages
The consumer group will go to the mark, where the last consumption was, and the next time it comes back, it can be read from the last mark
-
There is a message confirmation mechanism to ensure that the message is consumed at least once
Independent of JVM, not limited by JVM
Persistence can be done in Redis, security is guaranteed
2.4 Summary
3. Realize asynchronous seckill based on Stream message queue
Requirements :
① Create a message queue of type Stream named stream.orders
MKSTREAM : Automatically create a queue if it does not exist
XGROUP CREATE stream.orders g1 0 MKSTREAM
② Modify the previous Lua script for placing an order in a flash sale, and add a message directly to stream.orders after confirming that you are eligible for snap-up, the content includes voucherld, userld, orderld
For the parameter *|ID in XADD : specify the unique id of the message, * means that it is automatically generated by Redis, and the format is "time stamp-incremental number"
Changed two lines of code
-- 1. 参数列表
-- 1.1 优惠券id
-- 需要去redis中读取库存数量，判断是否充足，其中key的前缀seckill:stock是固定的，后面的id是需要传入的
local voucherId = ARGV[1]
-- 1.2 用户id
-- 需要知道用户id，才能判断用户之前是否下过单
local userId = ARGV[2]
-- TODO 1.3 订单id
local orderId = ARGV[3]
-- 2.数据相关key
-- 2.1 库存key, lua中是用 .. 拼接字符串
local stockKey = 'seckill:stock:' .. voucherId
-- 2.2 订单key，值是一个set集合，集合名称就是下面。内容是购买订单的用户的id，这样可以记录谁购买了谁没有购买
local orderKey = 'seckill:order:' .. voucherId
-- 3.lua脚本业务
-- 3.1 判断库存是否充足
-- redis.call('get',stockKey)得到的结果是字符串，是无法和数字比较的
if (tonumber(redis.call('get',stockKey))<=0) then
-- 3.2库存不足
return 1
end
-- 3.2判断用户是否下单
-- 借助命令SISMEMBER命令，判断一个给定的值是不是当前set集合中的一个成员，如果存在返回1，不存在返回0
if (redis.call('SISMEMBER',orderKey,userId) == 1) then
-- 3.3redis中存在，说明是重复下单
return 2
end
-- 3.4 扣库存 incrby stockKey -1
redis.call('incrby',stockKey,-1)
-- 3.5 下单，保存用户 sadd orderKey userId
redis.call('sadd',orderKey,userId)
-- TODO 3.6 发送消息到队列当中，XADD stream.orders * k1 v1 k2 v2.....
redis.call("xadd","stream.orders","*","userId",userId,"voucherId",voucherId,"id",orderId)
return 0
Transform seckill business logic
The java code is indeed cut down a lot
//TODO 获取脚本
// RedisScript是一个接口，我们使用一个实现类，泛型就是返回值的类型
private static final DefaultRedisScript<Long> SECKILL_SCRIPT;
//静态代码块做初始化
static {
SECKILL_SCRIPT = new DefaultRedisScript<>();
// 借助Spring提供的方法区resource下找
SECKILL_SCRIPT.setLocation(new ClassPathResource("seckill.lua"));
// 配置一下返回值类型
SECKILL_SCRIPT.setResultType(Long.class);
}
private VoucherOrderServiceImpl proxy;
@Override
public Result seckillVoucher(Long voucherId) {
// 获取用户id
Long userId = UserHolder.getUser().getId();
// TODO 获取订单id，这不操作移到上面
long orderId = redisIdWorker.nextId("order");
// 1.执行lua脚本
// 我们没有传入key，所以传入一个空集合即可
Long result = stringRedisTemplate.execute(
SECKILL_SCRIPT,
Collections.emptyList(),
voucherId.toString(), userId.toString(),String.valueOf(orderId)
);
// 2.判断结果是否为0（0有购买资格，不为0没有购买资格）
int r = result.intValue();
if (r != 0) {
// 2.1 不为0没有购买资格
return Result.fail(r == 1 ? "库存不足" : "不能重复下单");
}
// 3.获取代理对象
proxy = (VoucherOrderServiceImpl) AopContext.currentProxy();
// 4. 返回订单id
return Result.ok(orderId);
}
③ When the project starts, start a thread task, try to get the message in stream.orders, and complete the order
This step directly looks at the complete code
//创建一个线程池
private static final ExecutorService SECKILL_ORDER_EXECUTOR = Executors.newSingleThreadExecutor();
@PostConstruct//当前类初始化完毕后执行这个方法
private void init() {
SECKILL_ORDER_EXECUTOR.submit(new VoucherOrderHandler());
}
private class VoucherOrderHandler implements Runnable {
String queueName = "stream.orders";
@Override
public void run() {
while (true) {
try {
// TODO 1.获取消息队列中的订单信息,XREADGROUP GROUP g1 c1 COUNT 1 BLOCK 2000 STREAMS stream.orders >
// 为什么返回一个List？因为count不是1的话，可能会有多条消息
List<MapRecord<String, Object, Object>> list = stringRedisTemplate.opsForStream().read(
// 组是属于消费者的一部分信息，在这里统一叫做Consumer(一定是Spring包下的)
Consumer.from("g1", "c1"),
// StreamReadOptions.empty()创建一个空的，count(1)每次读取一条消息
// block(Duration.ofMillis(2000))，只能传入一个Duration参数
StreamReadOptions.empty().count(1).block(Duration.ofMillis(2000)),
// queueName消息队列名字,ReadOffset.lastConsumed()最近一个未消费消息
StreamOffset.create(queueName, ReadOffset.lastConsumed())
);
// TODO 2.判断消息获取是否成功
if(list ==null || list.isEmpty() ){
// TODO 2.1 如果获取失败，说明没有消息，继续下一次循环
// 如果获取失败，说明没有消息，继续下一次循环
continue;
}
// TODO 3.解析消息中的订单信息
// 明确知道count是1，所以直接获取0就行
MapRecord<String, Object, Object> record = list.get(0);
// 消息id
RecordId id = record.getId();
// 将Map转成VoucherOrder
Map<Object, Object> values = record.getValue();
// true表示出现错误忽略
VoucherOrder voucherOrder = BeanUtil.fillBeanWithMap(values, new VoucherOrder(), true);
// TODO 3.如果获取成功，可以下单
handleVoucherOrder(voucherOrder);
// TODO 4.ACK确认,XACK stream.orders g1 id
stringRedisTemplate.opsForStream().acknowledge(queueName,"g1",id);
} catch (Exception e) {
log.error("处理订单异常", e);
try {
Thread.sleep(20);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
// TODO 处理时消息时报异常了，消息没有被确认，会进入pending-list，我们在这里需要去pendingList取出来再去做处理
handlePendingList();
}
}
}
private void handlePendingList() {
while (true) {
try {
// TODO 1.获取pending-list中的订单信息,XREADGROUP GROUP g1 c1 COUNT 1 STREAMS stream.orders 0
// 为什么返回一个List？因为count不是1的话，可能会有多条消息
List<MapRecord<String, Object, Object>> list = stringRedisTemplate.opsForStream().read(
// 组是属于消费者的一部分信息，在这里统一叫做Consumer(一定是Spring包下的)
Consumer.from("c1", "g1"),
// StreamReadOptions.empty()创建一个空的，count(1)每次读取一条消息
StreamReadOptions.empty().count(1),
// queueName消息队列名字,ReadOffset.from("0"),读取pending-list第一条消息
StreamOffset.create(queueName, ReadOffset.from("0"))
);
// TODO 2.判断消息获取是否成功
if(list ==null || list.isEmpty() ){
// TODO 2.1 如果获取失败，说明pending-list没有异常消息，结束循环
// 如果获取失败，说明没有消息，继续下一次循环
continue;
}
// TODO 3.解析消息中的订单信息
// 明确知道count是1，所以直接获取0就行
MapRecord<String, Object, Object> record = list.get(0);
// 消息id
RecordId id = record.getId();
// 将Map转成VoucherOrder
Map<Object, Object> values = record.getValue();
// true表示出现错误忽略
VoucherOrder voucherOrder = BeanUtil.fillBeanWithMap(values, new VoucherOrder(), true);
// TODO 3.如果获取成功，可以下单
handleVoucherOrder(voucherOrder);
// TODO 4.ACK确认,XACK stream.orders g1 id
stringRedisTemplate.opsForStream().acknowledge(queueName,"g1",id);
} catch (Exception e) {
log.error("处理订单异常", e);
// 这个地方不用递归，直接下一次循环处理就行
}
}
}
}
/**
* 订单处理方法
* 我们在Redisson脚本中加了一个次锁了，在这里为什么还要再加一次锁呢？
* 做一个兜底方案，万一Redis出现问题，有个保证
*
* @param voucherOrder
*/
private void handleVoucherOrder(VoucherOrder voucherOrder) {
// 1.获取用户
Long userId = voucherOrder.getUserId();
// 2.创建锁对象
RLock lock = redissonClient.getLock("lock:order:" + userId);
// 3.获取锁
boolean isLock = lock.tryLock();
// 4.判断是否获取锁成功
if (!isLock) {
// 获取锁失败
log.error("不允许重复下单");
return;
}
try {
// 之前这种获取代理对象的方式是获取不到的,因为不在同一个线程下面了
// VoucherOrderServiceImpl proxy = (VoucherOrderServiceImpl) AopContext.currentProxy();
// return proxy.createVoucherOrder(voucherId);
// 在主线程获取，这个地方使用
proxy.createVoucherOrder(voucherOrder);
} finally {
// 出现异常做锁的释放
lock.unlock();
}
}
private VoucherOrderServiceImpl proxy;
@Override
public Result seckillVoucher(Long voucherId) {
// 获取用户id
Long userId = UserHolder.getUser().getId();
// 获取订单id，这不操作移到上面
long orderId = redisIdWorker.nextId("order");
// 1.执行lua脚本
// 我们没有传入key，所以传入一个空集合即可
Long result = stringRedisTemplate.execute(
SECKILL_SCRIPT,
Collections.emptyList(),
voucherId.toString(), userId.toString(), String.valueOf(orderId)
);
// 2.判断结果是否为0（0有购买资格，不为0没有购买资格）
int r = result.intValue();
if (r != 0) {
// 2.1 不为0没有购买资格
return Result.fail(r == 1 ? "库存不足" : "不能重复下单");
}
// 3.获取代理对象
proxy = (VoucherOrderServiceImpl) AopContext.currentProxy();
// 4. 返回订单id
return Result.ok(orderId);
}
// 如果在方法上添加synchronized,说明同步锁是this，当前对象
// 不建议 把synchronized放在方法上，锁住此对象后，不管任何一个用户来了，都是这把锁，也就意味着整个方法被串行化了
// 所谓“一人一单”，只需要对同一个用户加锁即可，如果不是同一个用户，无需加锁
@Transactional
public void createVoucherOrder(VoucherOrder voucherOrder) {
// 新增一人一单的判断
Long userId = voucherOrder.getUserId();
// user.getId().toString()转换成字符串也无法保证线程安全，因为每次的String都不一样
// 我们可以加一个intern，是一个字符串对象规范表示，回去字符串常量池中找一找和此字符串的值一样的字符串地址并返回
// 查询订单
int count = query().eq("user_id", userId)
.eq("voucher_id", voucherOrder.getVoucherId()).count();
// 判断是否存在
if (count > 0) {
// 用户至少下过一单，不能再下了
log.error("用户至少下过一单，不能再下了");
return;
}
// 说明没买过，继续执行代码
// 5.扣减库存
boolean success = seckillVoucherService.update()
.setSql("stock = stock-1") //set stock = stock-1
.eq("voucher_id", voucherOrder.getVoucherId()) //where voucher_id= voucherId
.gt("stock", 0)//where stock>0
.update();
if (!success) {
log.error("扣减失败，可能库存不足");
return;
}
// 创建订单
save(voucherOrder);
}
}