redis数据库队列（list），集合（set）元素设置类似过期（expire）功能

转自 https://blog.csdn.net/leean950806/article/details/78669070/

**（2018-07-23更新：
本案例使用的解决方案以及代码，在较大数据量的操作情境下存在严重redis性能问题——大集合的一次性删除操作可能导致redis阻塞，正常业务将无法访问、操作redis。如果需要使用相关功能，请使用资料中的1方法或自行优化redis删除队列操作的执行时间）**

问题：

项目需要为每个用户维护一个列表，存放一些数据。列表中的值有过期时间，过期的值查询可以找到也可以找不到，还会有一个验证，所以无所谓。但是redis队列只有一个整体的过期功能，没有每个元素的单独过期功能，所以如果用户一直不停向队列塞东西，队列就会变的越来越大。这显然不合理。

资料：

https://stackoverflow.com/questions/16545321/how-to-expire-the-hset-child-key-in-redis

https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis/

查了一下资料，目前给队列、集合元素单独设置过期不可能做到（redis4.0.2）。但是有其他方法可以做到类似功能。

参考的一篇文章提出两种方法：

1.使用SortedSet，使用score参数代表unix时间，程序定期使用ZRANGEBYSCORE清除过期项

2.将集合拆分成多个按时间排序、自动过期的小集合

1方法显然更方便、高效，但是项目必须跑程序为每个集合定期维护，可能产生很多不必要的麻烦，所以我选择2解决方法。

解决方案：

（重要的话说三遍：这个解决方案中过期的值有可能被返回，过期的值有可能被返回，过期的值有可能被返回。如果需要返回确定不过期的值，请在value中加unix时间作验证）

使用一个工具类封装redis操作，自动进行redis集合的拆分和查询。直接上代码：

必须获得redis读、写、设置过期、查询key是否存在函数

expire表示过期时间（秒）； blockSize表示分块大小（秒），不能大于expire

package com.study.javaweb.test1.utils;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * User: longjuanfeng Date: 2017-11-29
 */
public class RedisAutoExpireUtils<ValueType, SetResponse> {
    private Logger logger = LoggerFactory.getLogger(RedisAutoExpireUtils.class);

    private RedisSetter<ValueType, SetResponse> redisSetter;

    private RedisGetter<ValueType> redisGetter;

    private RedisExpire redisExpire;

    private RedisExists redisExists;

    private Integer expire;

    //default size is expire
    private Integer blockSize;

    public RedisAutoExpireUtils(RedisSetter<ValueType, SetResponse> redisSetter, RedisGetter<ValueType> redisGetter, RedisExpire redisExpire, RedisExists redisExists, Integer expire) throws Exception {
        this(redisSetter, redisGetter, redisExpire, redisExists, expire, expire);
    }

    public RedisAutoExpireUtils(RedisSetter<ValueType, SetResponse> redisSetter, RedisGetter<ValueType> redisGetter, RedisExpire redisExpire, RedisExists redisExists, Integer expire, Integer blockSize) throws Exception {
        if (blockSize > expire) {
            throw new Exception("blockSize should not larger than expire");
        }
        this.redisSetter = redisSetter;
        this.redisGetter = redisGetter;
        this.redisExpire = redisExpire;
        this.redisExists = redisExists;
        this.expire = expire;
        this.blockSize = blockSize;
    }

    public SetResponse setRedisValue(String key, ValueType value) {
        Integer nowTime = new Long(System.currentTimeMillis() / 1000).intValue();
        Integer blockTail = nowTime % blockSize;
        Integer stampPos = nowTime - blockTail;
        String timeStamp = String.valueOf(stampPos);
        logger.info("timeStamp of {} is {}", nowTime, timeStamp);
        String keyWithStamp = key + ":" + timeStamp;
        Boolean exists = redisExists.exists(keyWithStamp) == 1;
        SetResponse result = redisSetter.addValue(keyWithStamp, value);
        if (!exists) {
            Integer expireTime = expire + blockSize - blockTail;
            redisExpire.setExpire(keyWithStamp, expireTime);
            logger.info("set expire of {} is {}", nowTime, expireTime);
        }
        return result;
    }

    public ValueType getRedisValue(String key) {
        Integer nowTime = new Long(System.currentTimeMillis() / 1000).intValue();
        Integer checkBlockNum = expire / blockSize + 1;
        Integer blockTail = nowTime % blockSize;
        Integer stampPos = nowTime - blockTail;
        for (int i = 0; i < checkBlockNum; i++) {
            String timeStamp = String.valueOf(stampPos);
            logger.info("check timeStamp of {} is {}", nowTime, timeStamp);
            String keyWithStamp = key + ":" + timeStamp;
            ValueType value = redisGetter.getValue(keyWithStamp);
            if (value != null) {
                logger.info("find value at timeStamp of {}", timeStamp);
                return value;
            }
            stampPos = stampPos - blockSize;
        }
        return null;
    }

    @FunctionalInterface
    public interface RedisSetter<ValueType, SetResponse> {
        SetResponse addValue(String key, ValueType value);
    }

    @FunctionalInterface
    public interface RedisGetter<ValueType> {
        ValueType getValue(String key);
    }

    @FunctionalInterface
    public interface RedisExpire {
        void setExpire(String key, Integer expire);
    }

    @FunctionalInterface
    public interface RedisExists {
        Integer exists(String key);
    }
}

使用方法
redisTestRepository是一个读写redis功能类

package com.study.javaweb.test1.repository;

import com.study.javaweb.test1.BaseTest;
import com.study.javaweb.test1.utils.RedisAutoExpireUtils;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;

/**
 * User: longjuanfeng Date: 2017-11-28
 */
public class RedisTest extends BaseTest {
    private Logger logger = LoggerFactory.getLogger(getClass());

    @Autowired
    private RedisTestRepository redisTestRepository;

    @Test
    public void redisAutoExpireUtilsTest1() throws Exception {
        Integer fixedExpireTime = 20;
        String testKey = "redisUtilTest1";
        RedisAutoExpireUtils<String, Long> redisAutoExpireUtils = new RedisAutoExpireUtils<>(redisTestRepository::setAdd, redisTestRepository::setGet, redisTestRepository::setExpire, key -> redisTestRepository.checkExists(key) ? 1 : 0, fixedExpireTime, fixedExpireTime / 3);

        redisAutoExpireUtils.setRedisValue(testKey, "12233");
        redisAutoExpireUtils.setRedisValue(testKey, "12234");

        Thread.sleep(10000);
        redisAutoExpireUtils.setRedisValue(testKey, "12235");
        redisAutoExpireUtils.setRedisValue(testKey, "12236");

        String result = redisAutoExpireUtils.getRedisValue(testKey);
        logger.info("get pop data of {} : {}", testKey, result);


        Thread.sleep(5000);
        result = redisAutoExpireUtils.getRedisValue(testKey);
        logger.info("get pop data of {} : {}", testKey, result);
        result = redisAutoExpireUtils.getRedisValue(testKey);
        logger.info("get pop data of {} : {}", testKey, result);

        Thread.sleep(7000);
        result = redisAutoExpireUtils.getRedisValue(testKey);
        logger.info("get pop data of {} : {}", testKey, result);
    }
}

运行结果

[18:24:10:294] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.setRedisValue(RedisAutoExpireUtils.java:70) - timeStamp of 1511951050 is 1511951046
[18:24:10:883] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.setRedisValue(RedisAutoExpireUtils.java:77) - set expire of 1511951050 is 22
[18:24:10:884] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.setRedisValue(RedisAutoExpireUtils.java:70) - timeStamp of 1511951050 is 1511951046
[18:24:20:886] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.setRedisValue(RedisAutoExpireUtils.java:70) - timeStamp of 1511951060 is 1511951058
[18:24:20:887] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.setRedisValue(RedisAutoExpireUtils.java:77) - set expire of 1511951060 is 24
[18:24:20:889] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.setRedisValue(RedisAutoExpireUtils.java:70) - timeStamp of 1511951060 is 1511951058
[18:24:20:890] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951060 is 1511951058
[18:24:20:894] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:93) - find value at timeStamp of 1511951058
[18:24:20:896] [INFO] - com.study.javaweb.test1.repository.RedisTest.redisAutoExpireUtilsTest1(RedisTest.java:36) - get pop data of redisUtilTest1 : 12236
[18:24:25:897] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951065 is 1511951064
[18:24:25:899] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951065 is 1511951058
[18:24:25:901] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:93) - find value at timeStamp of 1511951058
[18:24:25:903] [INFO] - com.study.javaweb.test1.repository.RedisTest.redisAutoExpireUtilsTest1(RedisTest.java:44) - get pop data of redisUtilTest1 : 12235
[18:24:25:904] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951065 is 1511951064
[18:24:25:907] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951065 is 1511951058
[18:24:25:909] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951065 is 1511951052
[18:24:25:910] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951065 is 1511951046
[18:24:25:911] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:93) - find value at timeStamp of 1511951046
[18:24:25:912] [INFO] - com.study.javaweb.test1.repository.RedisTest.redisAutoExpireUtilsTest1(RedisTest.java:46) - get pop data of redisUtilTest1 : 12234
[18:24:32:913] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951072 is 1511951070
[18:24:32:914] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951072 is 1511951064
[18:24:32:915] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951072 is 1511951058
[18:24:32:916] [INFO] - com.study.javaweb.test1.utils.RedisAutoExpireUtils.getRedisValue(RedisAutoExpireUtils.java:89) - check timeStamp of 1511951072 is 1511951052
[18:24:32:917] [INFO] - com.study.javaweb.test1.repository.RedisTest.redisAutoExpireUtilsTest1(RedisTest.java:54) - get pop data of redisUtilTest1 : null

与预期一致

设计思路：
将整个时间线切分成block，用每个block头的time stamp作为redis的key，单独设置过期。查询时，查多个block。
redis util分块

待解决的问题：
blockSize过大，会导致应过期数据堆积，当blockSize = expire时，redis最多需要额外存储100%的数据。如果redis空间紧张，应该适当减小blockSize
blockSize过小，会导致redis读取次数增多，redis读取平均增加 expire/（blockSize*2）次，如果redis访问太慢应该适当增加blockSize

blockSize很小时可以考虑用multi加快访问速度

Hash存取同理

--------------------- 本文来自尤里安龙卷风的CSDN 博客，全文地址请点击：https://blog.csdn.net/leean950806/article/details/78669070?utm_source=copy

redis数据库队列（list），集合（set）元素设置类似过期（expire）功能

猜你喜欢