How to optimize Redis memory usage

1. Introduction

1.1 Introduction

Redis is a fast non-relational data storage system, which is characterized by supporting multiple data structures and high performance. Redis can be used in various application scenarios such as caching, queues, publish/subscribe, and leaderboards.

1.2 Means of memory optimization

Redis stores all data in memory, so judicious use of memory is crucial. Reasonable optimization of memory usage can improve the performance of Redis, increase the amount of data Redis can handle, and reduce the possibility of Redis downtime.

Two, memory usage analysis

2.1 Redis memory usage mechanism

Redis memory usage mainly includes the following two aspects:

1. Basic use of server memory: After the Redis service is started, Redis will apply for a part of memory from the operating system for server operation and maintenance connections, client information, etc.
2. Memory allocation of database key-value pairs: When adding key-value pairs to Redis, Redis will determine the type of memory to use and the way to allocate memory according to the size of the value. When the size of a single key-value pair exceeds a certain threshold, Redis will save the key-value pair to a temporary file on disk, thereby saving memory space.

2.2 Redis memory usage bottleneck analysis

The bottleneck of Redis memory usage mainly includes the following three aspects:

1. Operating system limitation: Under 32-bit operating system, the maximum memory that Redis can use is about 3GB, but under 64-bit operating system, there is no such limitation.
2. Redis memory configuration: The maxmemory parameter in the Redis configuration is used to control the maximum amount of memory that Redis can use. If this parameter is configured too small, resulting in Redis not having enough memory to store key-value pairs, an Out Of Memory error will be generated.
3. Redis data structures: Using different data structures also affects the bottleneck of Redis memory usage. For example, in Redis, using a hash table is more memory efficient than using a list. Therefore, in the actual application scenario, the data structure suitable for the current business scenario should be selected first, so as to maximize the use of memory resources.

public class RedisOptimization {
    
    

    private static Jedis jedis;

    public static void main(String[] args) {
    
    
        jedis = new Jedis("localhost", 6379);
        
        // 内存优化
        optimizeMemoryUsage();
    }

    /**
     * Redis 内存使用优化
     */
    public static void optimizeMemoryUsage() {
    
    
        // 设置最大内存
        jedis.configSet("maxmemory", "4gb");

        // 查看内存使用情况
        String memoryUsage = jedis.info("memory");
        System.out.println(memoryUsage);
    }
}

3. Redis memory usage optimization

3.1 Data Compression

3.1.1 Compression rules

Redis provides a variety of data types, and each type needs to follow different rules when performing data compression. According to the characteristics of the data type, the Redis compression rules are as follows:

• String type: data of string type cannot be compressed.
• List type: If all elements in the list are integer types, they will be stored in memory as compact integer arrays, which take up less space than ordinary list types.
• Collection type: If all elements in the collection are of integer type, they will be stored in memory as a compact integer array, which takes up less space than ordinary collection types.
• Hash type: If all elements in the hash are short string types, they will be stored in Quicklist, which can reduce memory usage.
• Ordered collection type: If the member elements and scores in the ordered collection are of integer type, they will be stored in a compact integer array, which takes up less space than the ordinary ordered collection type.

3.1.2 Compression strategy

For data that needs to be compressed, multiple compression strategies are supported in Redis, as follows:

• Quicklist compression: For long linked list structure, it can be converted into Quicklist structure and then compressed.
• RDB file compression: memory snapshot RDB files can be compressed using Gzip, LZF, Snappy and other algorithms to reduce storage space.
• LRU memory elimination: You can choose to only clear the least recently used data, while preventing cache penetration caused by a large amount of data being cleared.

3.2 Memory recovery

3.2.1 Memory elimination strategy

Redis supports multiple memory elimination strategies to reclaim memory, as follows:

• noeviction: It is not allowed to reclaim memory. When the memory is insufficient, the operation will report an error.
• allkeys-lru: Select the least used one from all keys (Redis data structure), which is equivalent to the "Least Recently Used" algorithm.
• allkeys-random: Randomly recycle all keys.
• volatile-lru: From the key-value pairs whose expiration time has been set, select the least recently used one to recycle.
• volatile-random: Randomly delete a key from the key-value pair whose expiration time is set.
• volatile-ttl: According to the TLL of the key-value pair, recycle the key-value pair with the latest time.

3.2.2 Periodically clear expired data

Redis has a built-in mechanism for periodically clearing expired data. It can be set through the maxmemory-policyand parameters in the configuration file .maxmemory-samples

3.3 Performance optimization

3.3.1 Reasonably set the cache expiration time

Reasonably setting the cache expiration time can avoid the memory waste caused by the cache being occupied for a long time. Different invalidation strategies can be selected according to business characteristics, such as first-in-first-out and least-used strategies.

3.3.2 Using memory optimization tools

Redis has a number of built-in memory optimization tools, which can be redis-cliused through commands, as follows:

• info memory: View Redis memory usage.
• slowlog get: View slow query records.
• monitor: View Redis operation logs in real time.

4. Countermeasures

4.1 Memory upgrade

If the current Redis memory usage is too high, you can consider upgrading the memory of the Redis machine, but it should be noted that a single Redis process supports a maximum of 512MB of memory. If this limit is exceeded, data fragmentation is required to reduce memory usage.

4.2 Data Fragmentation

Data sharding is a commonly used Redis memory optimization strategy, which divides large-scale data into multiple small blocks for processing. Different data blocks can be stored in different Redis instances, thereby reducing the memory usage of a single Redis. However, data sharding needs to pay attention to data consistency and read-write synchronization.

Five, Redis advanced optimization

5.1 Preheating technology

Preheating technology refers to loading some hot data into memory in advance after Redis starts, thereby reducing the response time when users access. The specific implementation can query the data and write it into Redis through scheduled tasks or manual triggering, leaving enough time for Redis to load the data into memory.

5.2 Persistence technology

There are two persistence technologies in Redis, RDB and AOF.

5.2.1 RDB

RDB is Redis Database, which is in the form of a snapshot. All data is written to the disk to generate a .Snapshot file for persistence; at the same time, it supports compression of the snapshot file to reduce the file size. Compared with the AOF method, the RDB method can perform better cache recovery, but some recent write operations may be lost.

5.2.2 AOF

AOF is Append-Only File, which appends all write operations performed by Redis to the file (default name is appendonly.aof) for persistence. These write operations include commands such as SET and DEL. Compared with the RDB method, the AOF method can better ensure data security, but when the amount of data is relatively large, the size of the AOF file will also become large, and it will also have a certain impact on the recovery efficiency of the cache.

public class RedisService {
    
    
    // Redis连接池配置信息
    private static final JedisPoolConfig POOL_CONFIG = new JedisPoolConfig();
    // Redis服务器IP地址
    private static String ADDR = "localhost";
    // Redis的端口号
    private static int PORT = 6379;
    // 访问密码
    private static String AUTH = null;
    // 可用连接实例的最大数目，默认值为8
    private static int MAX_ACTIVE = 8;
    // 控制一个pool最多有多少个状态为idle(空闲的)的jedis实例，默认值也是8
    private static int MAX_IDLE = 8;
    // 等待可用连接的最大时间，单位毫秒，默认值为-1，表示永不超时
    private static int MAX_WAIT = -1;
    // 连接超时的时间
    private static int TIMEOUT = 3000;
    // 在borrow一个jedis实例时，是否需要验证，若为true则所有jedis实例均是可用的
    private static boolean TEST_ON_BORROW = true;

    // 初始化Redis连接池
    private static JedisPool jedisPool = null;

    static {
    
    
        try {
    
    
            POOL_CONFIG.setMaxTotal(MAX_ACTIVE);
            POOL_CONFIG.setMaxIdle(MAX_IDLE);
            POOL_CONFIG.setMaxWaitMillis(MAX_WAIT);
            POOL_CONFIG.setTestOnBorrow(TEST_ON_BORROW);

            jedisPool = new JedisPool(POOL_CONFIG, ADDR, PORT, TIMEOUT, AUTH);
        } catch (Exception e) {
    
    
            e.printStackTrace();
        }
    }

    /**
     * 获取jedis实例
     * 
     * @return
     */
    public synchronized static Jedis getJedis() {
    
    
        try {
    
    
            if (jedisPool != null) {
    
    
                Jedis resource = jedisPool.getResource();
                return resource;
            } else {
    
    
                return null;
            }
        } catch (Exception e) {
    
    
            e.printStackTrace();
            return null;
        }
    }

    /**
     * 释放资源
     * 
     * @param jedis
     */
    public static void returnResource(final Jedis jedis) {
    
    
        if (jedis != null) {
    
    
            jedisPool.returnResource(jedis);
        }
    }
}