1. Key-value design
1. Key name design
- (1) [Recommendation]: Readability and manageability
Prefix with business name (or database name) (to prevent key conflict), separated by colons, such as business name: table name: id
ugc:video:1- (2) [Suggestion]: Conciseness
On the premise of ensuring semantics, control the length of the key. When there are many keys, the memory usage cannot be ignored, for example:
user:{uid}:friends:messages:{mid}简化为u:{uid}:fr:m:{mid}。- (3) [Mandatory]: Do not include special characters
Counter example: contains spaces, newlines, single and double quotes, and other escape characters
2. value design
- (1) [Mandatory]: reject bigkey (to prevent network card traffic, slow query)
The string type is controlled within 10KB, and the number of hash, list, set, and zset elements should not exceed 5000.
Counter example: a list with 2 million elements.
For non-string bigkeys, do not use del to delete them. Use hscan, sscan, and zscan to delete them gradually. At the same time, pay attention to prevent the problem of automatic deletion of bigkey expiration time (for example, a 2 million zset set to expire in 1 hour will trigger the del operation, causing Blocking, and the operation will not appear in the slow query (latency can be checked)), find method and delete method
- (2) [Recommended]: Select the appropriate data type.
For example: entity type (to reasonably control and use data structure memory encoding to optimize configuration, such as ziplist, but also pay attention to the balance between saving memory and performance)
Counterexample:
set user:1:name tom
set user:1:age 19
set user:1:favor footballNormal example:
hmset user:1 name tom age 19 favor football3. [Recommendation]: Control the life cycle of the key, redis is not a trash can.
It is recommended to use expire to set the expiration time (if the conditions allow, the expiration time can be broken up to prevent centralized expiration), and the data that does not expire should focus on idletime.
Second, the use of commands
1. [Recommended] O(N) command pays attention to the number of N
For example, hgetall, lrange, smembers, zrange, sinter, etc. are not impossible to use, but the value of N needs to be specified. If there is a need for traversal, hscan, sscan, and zscan can be used instead.
2. [Recommended]: Disable command
It is forbidden to use keys, flushall, flushdb, etc. online, disable commands through the rename mechanism of redis, or use the scan method for progressive processing.
3. [Recommendation] Use select reasonably
The multi-database of redis is weak, and it uses numbers to distinguish it. Many clients have poor support. At the same time, multi-database for multi-service is actually single-threaded processing, which will cause interference.
4. [Recommended] Use batch operations to improve efficiency
原生命令：例如mget、mset。
非原生命令：可以使用pipeline提高效率。But pay attention to controlling the number of elements in a batch operation (for example, within 500, which is actually related to the number of element bytes).
Note that the two are different:
1. 原生是原子操作，pipeline是非原子操作。
2. pipeline可以打包不同的命令，原生做不到
3. pipeline需要客户端和服务端同时支持。5. [Suggestion] Redis transaction function is weak, it is not recommended to use too much
The transaction function of Redis is weak (does not support rollback), and the cluster version (self-developed and official) requires that the key of a transaction operation must be in a slot (can be solved by using the hashtag function)
6. [Recommendation] The Redis cluster version has special requirements for using Lua:
- 1. All keys should be passed by the KEYS array. The redis command called in redis.call/pcall, the key position must be the KEYS array, otherwise an error will be returned directly, "-ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS array"
- 2. All keys must be in 1 slot, otherwise return error, "-ERR eval/evalsha command keys must in same slot"
7. [Recommendation] When using the monitor command if necessary, be careful not to use it for a long time.
3. Client use
1. [Recommended]
Avoid using one Redis instance for multiple applications
Positive example: unrelated business split, public data service.
2. [Recommended]
Using a database with a connection pool can effectively control connections and improve efficiency at the same time. Standard usage:
执行命令如下：
Jedis jedis = null;
try {
jedis = jedisPool.getResource();
//具体的命令
jedis.executeCommand()
} catch (Exception e) {
logger.error("op key {} error: " + e.getMessage(), key, e);
} finally {
//注意这里不是关闭连接，在JedisPool模式下，Jedis会被归还给资源池。
if (jedis != null)
jedis.close();
}Here is an article on JedisPool optimization methods:
3.【Suggestion】
Under high concurrency, it is recommended that the client add a circuit breaker function (such as netflix hystrix)
4. 【Recommendation】
Set a reasonable password, and use SSL encryption to access if necessary (supported by Alibaba Cloud Redis)
5.【Suggestion】
According to your own business type, select maxmemory-policy (maximum memory elimination policy), and set the expiration time.
The default strategy is volatile-lru, that is, after the maximum memory is exceeded, the lru algorithm is used to remove keys in expired keys to ensure that non-expired data will not be deleted, but OOM problems may occur.
Other strategies are as follows:
- allkeys-lru: Delete keys according to the LRU algorithm, regardless of whether the data has a timeout attribute set, until enough space is vacated.
- allkeys-random: Randomly delete all keys until enough space is made.
- volatile-random: Randomly delete expired keys until enough space is made.
- volatile-ttl: Delete data that will expire recently according to the ttl attribute of the key-value object. If not, fallback to the noeviction strategy.
- noeviction: No data will be removed, all write operations will be rejected, and the client error message "(error) OOM command not allowed when used memory" will be returned. At this time, Redis only responds to read operations.
4. Related tools
1. [Recommended]: Data synchronization
Data synchronization between redis can be used: redis-port
2. [Recommended]: big key search
3. [Recommendation]: Hotspot key search (monitor is used for internal implementation, so it is recommended to use it for a short time)
阿里云Redis已经在内核层面解决热点key问题，欢迎使用。
Five appendix: delete bigkey
1. 下面操作可以使用pipeline加速。
2. redis 4.0已经支持key的异步删除，欢迎使用。1. Hash delete: hscan + hdel
public void delBigHash(String host, int port, String password, String bigHashKey) {
Jedis jedis = new Jedis(host, port);
if (password != null && !"".equals(password)) {
jedis.auth(password);
}
ScanParams scanParams = new ScanParams().count(100);
String cursor = "0";
do {
ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
List<Entry<String, String>> entryList = scanResult.getResult();
if (entryList != null && !entryList.isEmpty()) {
for (Entry<String, String> entry : entryList) {
jedis.hdel(bigHashKey, entry.getKey());
}
}
cursor = scanResult.getStringCursor();
} while (!"0".equals(cursor));
//删除bigkey
jedis.del(bigHashKey);
}2. List delete: ltrim
public void delBigList(String host, int port, String password, String bigListKey) {
Jedis jedis = new Jedis(host, port);
if (password != null && !"".equals(password)) {
jedis.auth(password);
}
long llen = jedis.llen(bigListKey);
int counter = 0;
int left = 100;
while (counter < llen) {
//每次从左侧截掉100个
jedis.ltrim(bigListKey, left, llen);
counter += left;
}
//最终删除key
jedis.del(bigListKey);
}3. Set delete: sscan + srem
public void delBigSet(String host, int port, String password, String bigSetKey) {
Jedis jedis = new Jedis(host, port);
if (password != null && !"".equals(password)) {
jedis.auth(password);
}
ScanParams scanParams = new ScanParams().count(100);
String cursor = "0";
do {
ScanResult<String> scanResult = jedis.sscan(bigSetKey, cursor, scanParams);
List<String> memberList = scanResult.getResult();
if (memberList != null && !memberList.isEmpty()) {
for (String member : memberList) {
jedis.srem(bigSetKey, member);
}
}
cursor = scanResult.getStringCursor();
} while (!"0".equals(cursor));
//删除bigkey
jedis.del(bigSetKey);
}4. SortedSet delete: zscan + zrem
public void delBigZset(String host, int port, String password, String bigZsetKey) {
Jedis jedis = new Jedis(host, port);
if (password != null && !"".equals(password)) {
jedis.auth(password);
}
ScanParams scanParams = new ScanParams().count(100);
String cursor = "0";
do {
ScanResult<Tuple> scanResult = jedis.zscan(bigZsetKey, cursor, scanParams);
List<Tuple> tupleList = scanResult.getResult();
if (tupleList != null && !tupleList.isEmpty()) {
for (Tuple tuple : tupleList) {
jedis.zrem(bigZsetKey, tuple.getElement());
}
}
cursor = scanResult.getStringCursor();
} while (!"0".equals(cursor));
//删除bigkey
jedis.del(bigZsetKey);
}