Pit record -Redis use scan instead of keys

[Advance notice]
article by author: Zhang Yaofeng in conjunction with their own experience in the production of finishing, forming easy to understand article
writing is not easy, reproduced please specify, thank you!
The Spark Code Cases Address: https://github.com/Mydreamandreality/ sparkResearch

---

Online issues

• Timing of tasks corresponding to the keys * wildcard match key
• Other services during this time (need to use Redis) alarm can not be normal service
• Check the operation and maintenance platform log: alarm service requests within this time all the throws redis connection timeout, set redis service timeout is 200ms

identify the problem

• First, determine cause of command keys
• And Redis amount of data to be particularly large, otherwise it is not trigger this bug, before a small amount of data when there is no problem
• Secondly keyscommand itself (million and more) will be particularly slow in particularly large amount of data redis case
• The most deadly is that this command will block redis multiplexed io the main thread, if the thread is blocked, and other services to request redis will all be blocked during this period, leading to a series of reactions in response Caton, connection timeout and many more

problem solved

• The online environment, which can block the operation of the main thread, and the algorithm time complexity is O (n) should prohibit the use of

alternative plan

• redis.scan

How to scan to solve the problem online

• scan and keys are O (n) complexity
• scan also supports wildcard characters fuzzy search
• scan does not block the main thread
• scan support cursors return data in batches iteration
  Note: scan the returned data is likely to be repeated, the client needs to take the initiative to re

scan Note

• Cursor scan command from the start from 0 0 End
• scan command each time you return data will be required to carry the values ​​for the next cursor, access time according to this value then, if the returned data is empty, is not necessarily the data needs to be confirmed by the cursor

scan command Use Cases

scan 0 match my*key count 10000

JavaRedisApi-scan command Use Cases

    public Set<String> scan(String key, long count) {
        Set<String> keys = redisTemplate.execute(
                (RedisCallback<Set<String>>) connection -> {
                    Set<String> keyTmp = new HashSet<>();
                    Cursor<byte[]> cursor = connection.scan(new ScanOptions.ScanOptionsBuilder().match(key + "*").count(count).build());
                    while (cursor.hasNext()) {
                        keyTmp.add(new String(cursor.next()));
                    }
                    return keyTmp;
                });
        return keys;
    }

Regression testing results

• Normal service execution