Redis - 13. Development specification

This article mainly introduces the development specifications when using Redis, and explains it from the following aspects.

• key-value design

• command use

• client use

• related tools

Through the introduction of this article, the problems caused by using the Redis process can be reduced.

1. Key-value design

1.1. Key name design

1.1.1. Readability and manageability

Prefix the business name (or database name) (to prevent key conflicts), separated by colons, such as business name: table name: id

ugc:video:1

1.1.2. Simplicity

On the premise of ensuring semantics, control the length of the key. When there are many keys, the memory usage cannot be ignored, for example:

user:{uid}:friends:messages:{mid}简化为u:{uid}:fr:m:{mid}。

1.1.3, do not contain special characters

Negative example: contains spaces, newlines, single and double quotes, and other escape characters

1.2, value design

1.2.1. Reject bigkey

To prevent network card traffic and slow query, the string type should be controlled within 10KB, and the number of hash, list, set, and zset elements should not exceed 5000.

Counterexample: A list with 2 million elements.

For non-string bigkeys, do not use del to delete, use hscan, sscan, zscan to delete gradually, and at the same time, pay attention to prevent automatic deletion of bigkey expiration time (for example, if a 2 million zset is set to expire in 1 hour, the del operation will be triggered, causing Blocking, and the operation will not appear in the slow query (latency can be checked)), search method and delete method

1.2.2. Select the appropriate data type

For example: entity type (reasonable control and use of data structure memory encoding optimization configuration, such as ziplist, but also pay attention to the balance between saving memory and performance)

Counter example:

set user:1:name tom
set user:1:age 19
set user:1:favor football

Positive example:

hmset user:1 name tom age 19 favor football

1.2.3. Control the life cycle of the key

Redis is not a trash can. It is recommended to use expire to set the expiration time (the expiration time can be broken up if conditions permit to prevent centralized expiration), and the non-expired data should focus on idletime.

2. Command usage

2.1, O(N) commands focus on the number of N

For example, hgetall, lrange, smembers, zrange, sinter, etc. are not impossible to use, but the value of N needs to be specified. If there is a need for traversal, hscan, sscan, and zscan can be used instead.

2.2. Disable command

It is forbidden to use keys, flushall, flushdb, etc. online, and ban commands through the rename mechanism of redis, or use scan to process progressively.

2.3. Reasonable use of select

Redis's multi-database is weak, and numbers are used to distinguish it. Many clients support it poorly. At the same time, multi-databases for multi-service are actually processed by single thread, which will interfere.

2.4. Use batch operations to improve efficiency

• Native commands: eg mget, mset.

• Non-native commands: You can use pipeline to improve efficiency.

But pay attention to controlling the number of elements in a batch operation  (for example, within 500, it is actually related to the number of element bytes).

Note the difference between the two:

• Native is an atomic operation, and pipeline is a non-atomic operation.

• The pipeline can package different commands, which cannot be done natively

• The pipeline needs to be supported by both the client and the server.

2.5. It is not recommended to use the Redis transaction function too much

The transaction function of Redis is weak (rollback is not supported), and the cluster version (self-developed and official) requires that the key of a transaction operation must be on a slot (can be solved by using the hashtag function)

2.6. The Redis cluster version has special requirements for using Lua

1. All keys should be passed by the KEYS array, the redis command called in redis.call/pcall, the position of the key must be the KEYS array, otherwise the error will be returned directly, "-ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS arrayrn"

2. All keys must be in one slot, otherwise it will directly return error, "-ERR eval/evalsha command keys must in same slotrn"

2.7, monitor command

When using the monitor command if necessary, be careful not to use it for a long time.

3. Client use

3.1. Avoid using one Redis instance for multiple applications

Irrelevant business split, public data as a service.

3.2, use connection pool

It can effectively control the connection and improve efficiency at the same time. The standard usage method:

执行命令如下：
Jedis jedis = null;
try {
    jedis = jedisPool.getResource();
    //具体的命令
    jedis.executeCommand()
} catch (Exception e) {
    logger.error("op key {} error: " + e.getMessage(), key, e);
} finally {
    //注意这里不是关闭连接，在JedisPool模式下，Jedis会被归还给资源池。
    if (jedis != null)
        jedis.close();
}

3.3. Fusing function

Under high concurrency, it is recommended that the client add a fuse function (such as netflix hystrix)

3.4. Reasonable encryption

Set a reasonable password, and use SSL encrypted access if necessary (supported by Alibaba Cloud Redis)

3.5. Elimination strategy

According to your own business type, select the maxmemory-policy (maximum memory elimination policy), and set the expiration time.

The default strategy is volatile-lru, that is, after the maximum memory is exceeded, the lru algorithm is used to remove keys from expired keys to ensure that non-expired data will not be deleted, but OOM problems may occur.

Other strategies are as follows:

• allkeys-lru: Delete keys according to the LRU algorithm, regardless of whether the data has a timeout attribute, until enough space is made available.

• allkeys-random: Randomly delete all keys until enough space is made.

• volatile-random: Randomly delete expired keys until enough space is made.

• volatile-ttl: According to the ttl attribute of the key-value object, delete the data that will expire recently. If not, fall back to noeviction policy.

• noeviction: No data will be removed, all write operations will be rejected and the client error message "(error) OOM command not allowed when used memory" will be returned. At this time, Redis only responds to read operations.

4. Related tools

4.1. Data synchronization

Data synchronization between redis can be used: redis-port

4.2, big key search

redis big key search tool

4.3. Hot key search

The internal implementation uses monitor, so it is recommended to use facebook's redis-faina for a short time. Aliyun Redis has solved the hot key problem at the kernel level

5. Delete bigkey

• The following operations can be accelerated using the pipeline.

• Redis 4.0 already supports asynchronous deletion of keys, welcome to use.

5.1, Hash delete: hscan + hdel

public void delBigHash(String host, int port, String password, String bigHashKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
        List<Entry<String, String>> entryList = scanResult.getResult();
        if (entryList != null && !entryList.isEmpty()) {
            for (Entry<String, String> entry : entryList) {
                jedis.hdel(bigHashKey, entry.getKey());
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    //删除bigkey
    jedis.del(bigHashKey);
}

5.2, List delete: ltrim

public void delBigList(String host, int port, String password, String bigListKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    long llen = jedis.llen(bigListKey);
    int counter = 0;
    int left = 100;
    while (counter < llen) {
        //每次从左侧截掉100个
        jedis.ltrim(bigListKey, left, llen);
        counter += left;
    }
    //最终删除key
    jedis.del(bigListKey);
}

5.3, Set delete: sscan + srem

public void delBigSet(String host, int port, String password, String bigSetKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<String> scanResult = jedis.sscan(bigSetKey, cursor, scanParams);
        List<String> memberList = scanResult.getResult();
        if (memberList != null && !memberList.isEmpty()) {
            for (String member : memberList) {
                jedis.srem(bigSetKey, member);
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    //删除bigkey
    jedis.del(bigSetKey);
}

5.4, ​​SortedSet delete: zscan + zrem

public void delBigZset(String host, int port, String password, String bigZsetKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Tuple> scanResult = jedis.zscan(bigZsetKey, cursor, scanParams);
        List<Tuple> tupleList = scanResult.getResult();
        if (tupleList != null && !tupleList.isEmpty()) {
            for (Tuple tuple : tupleList) {
                jedis.zrem(bigZsetKey, tuple.getElement());
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    //删除bigkey
    jedis.del(bigZsetKey);
}