Regarding the Redis big key (Key), we define the big key in terms of [space complexity] and [time complexity] of accessing it.
The former mainly represents the memory size of the redis key; the latter represents the number of elements contained in the Redis set data type (set/hash/list/sorted set) key. The following two examples:
1 String key with a size of 200MB (Maximum 512MB for String Object); large memory space is occupied
1 Hash key containing 100000000 (1kw) fields, the corresponding access mode (such as hgetall) has high time complexity
Because the memory space complexity processing time is very small, testing the del 200MB String key takes about 1 millisecond,
and deleting a Hash key with 1kw fields will block the Redis process for tens of seconds. Therefore, this article only analyzes large collection keys from the time complexity. The risks of deleting such a large key, and how to delete it gracefully.
In a Redis cluster, applications should try to avoid using large keys; the direct impact can easily lead to a "sloping problem" in the capacity and requests of the cluster. For specific analysis, see the article: redis-cluster-imbalance . However, in the actual production process, there will always be unreasonable business use, and such large keys will appear; when the DBA finds out, it promotes business optimization and transformation, and then deletes this large key; if it is deleted directly, the DEL command may block the Redis process for tens of seconds , causing serious impact on application and Redis cluster availability.
The risk of directly deleting large keys
When the DEL command deletes a key of a single collection type, the time complexity of the command is O(M), where M is the number of elements contained in the collection type Key.
DEL key
Time complexity: O(N) where N is the number of keys that will be removed. When a key to remove holds a value other than a string, the individual complexity for this key is O(M) where M is the number of elements in the list, set, sorted set or hash. Removing a single key that holds a string value is O(1).
In the production environment, there have been many failures that caused Redis to block, failover and application avalanches due to the deletion of large keys in the business.
Testing to delete a large key of a collection type is time-consuming. Generally, 100w to hundreds of w elements can be cleared per second; if a large key with thousands of w elements is used, Redis will be blocked for 10 seconds,
which may cause the cluster to determine that Redis has failed and a fault has occurred. switching; or an avalanche of applications.
Description: Redis is single-threaded.
A single command that takes too much time will block other commands, which may easily cause application avalanches or failover of the Redis cluster.
So avoid using time-consuming commands in a production environment.
Time-consuming for Redis to delete large set keys, test estimation, please refer to; it is related to factors such as hardware environment, Redis version and load
The number of Key type Items takes time| Hash | ~1 million | ~1000ms |
| List | ~1 million | ~1000ms |
| Set | ~1 million | ~1000ms |
| Sorted Set | ~1 million | ~1000ms |
When we find that there is a large key in the cluster and want to delete it, how to delete the large key gracefully?
How to gracefully delete all kinds of large keys
从Redis2.8版本开始支持SCAN命令，通过m次时间复杂度为O(1)的方式，遍历包含n个元素的大key.
这样避免单个O(n)的大命令，导致Redis阻塞。 这里删除大key操作的思想也是如此。
Delete Large Hash Key
通过hscan命令，每次获取500个字段，再用hdel命令，每次删除1个字段。
Python代码：
|
1
2
3
4
5
6
7
8
|
def del_large_hash():
r = redis.StrictRedis(host=
'redis-host1', port=
6379)
large_hash_key =
"xxx"
#要删除的大hash键名
cursor =
'0'
while cursor !=
0:
cursor, data = r.hscan(large_hash_key, cursor=cursor, count=
500)
for item
in data.items():
r.hdel(large_hash_key, item[
0])
|
Delete Large Set Key
删除大set键，使用sscan命令，每次扫描集合中500个元素，再用srem命令每次删除一个键
python代码：
|
1
2
3
4
5
6
7
8
|
def del_large_set():
r = redis.StrictRedis(host=
'redis-host1', port=
6379)
large_set_key =
'xxx'
# 要删除的大set的键名
cursor =
'0'
while cursor !=
0:
cursor, data = r.sscan(large_set_key, cursor=cursor, count=
500)
for item
in data:
r.srem(large_size_key, item)
|
Delete Large List Key
删除大的List键，未使用scan命令； 通过ltrim命令每次删除少量元素。
Python代码：
|
1
2
3
4
5
|
def del_large_list():
r = redis.StrictRedis(host=
'redis-host1', port=
6379)
large_list_key =
'xxx'
#要删除的大list的键名
while r.llen(large_list_key)>
0:
r.ltrim(large_list_key,
0,
-101)
#每次只删除最右100个元素
|
Delete Large Sorted set key
删除大的有序集合键，和List类似，使用sortedset自带的zremrangebyrank命令,每次删除top 100个元素。
Python代码：
|
1
2
3
4
5
|
def del_large_sortedset():
r = redis.StrictRedis(host=
'large_sortedset_key', port=
6379)
large_sortedset_key=
'xxx'
while r.zcard(large_sortedset_key)>
0:
r.zremrangebyrank(large_sortedset_key,
0
,
99
) #The
time complexity is lower, each deletion is O(log(N)+100)
|
Redis Lazy Free
Starting from version 3.4, Redis will support the lazy delete free method, and the process of deleting large keys will not block normal requests.
参考:
lazy redis
issue 1748
Delete Large Hash
http://blog.csdn.net/wsliangjian/article/details/52329320