In redis, the use of wildcards is supported, such as '? 'or'', so we can use commands like `keys` to get all the data in a db in redis. But there is a problem that if you do this, the efficiency is very unsatisfactory in the case of a large amount of data. In general, the keys xxx type of operation will always be indispensable in the slowlog of redis (if someone performs this operation on it )
The scan command can help us solve the situation of using the keys command to traverse a large amount of data and cause the server to block. It only traverses a small part of the data each time, and the time complexity corresponding to each operation is O(1).
The scan command is a cursor-based iterator. After the scan command is called, it will return a new cursor to the user. The user needs to use this new cursor as the cursor parameter of the scan command in the next iteration, continuing the previous one. Iteration process. When the cursor parameter of the scan command is set to 0, the server will start a new iteration. When the cursor value returned by the server to the user is 0, the iteration has ended.
But for RedisCluster, it is not possible to perform scan operations on all keys, but for other data types, such as hash, zset, a series of hscan, zscan operations can be performed. In fact, it can be seen from the implementation of jedisCluster that if you want to scan all keys, you need to implement MultiKeyCommands, but RedisCluster does not support this type of operation, and manages operations such as pipeline, mget, and mset.
But in redisCluster, you can get the connection of each node through getClusterNodes, traverse and query in turn:
val jedisCluster = new JedisCluster(hostPorts)
val clusterNodes = jedisCluster.getClusterNodes.values()
for (clusterNode: JedisPool <- clusterNodes) {
var scanCursor = "0"
val resource: Jedis = clusterNode.getResource
Execute the scan command on each node. The end condition of the scan command is not only to return the next index=0, but also to obtain an empty list. At this time, it is also announced that there is no new key.
At this point, the scan operation can be performed through a do/while loop:
for (clusterNode: JedisPool <- clusterNodes) {
var scanCursor = "0"
var scanResult: util.List[String] = new util.ArrayList[String]()
do {
val resource: Jedis = clusterNode.getResource
val scan: ScanResult[String] = resource.scan(scanCursor, new ScanParams().`match`(“xxx*"))
scanCursor = scan.getStringCursor
scanResult = scan.getResult
for (key: String <- scanResult) {
//xxx
}
resource.close()
} while (!scanCursor.equals("0") && !scanResult.isEmpty)
}
When scanning, you can set count (number) and match (matching pattern) to facilitate filtering according to conditions.
However, it is necessary to count whether the total result of the key value executed by the scan method is the same as the execution result of keys (under the same operating conditions, redis cannot add keys at will).
After our verification, in cluster mode, putting all keys under set (considering that master/slave keys are duplicated, keys in cluster mode will ignore duplicate keys, but each redis connection will not scan) is the same :
val keys = new mutable.HashSet[String]()
for (clusterNode: JedisPool <- clusterNodes) {
var scanCursor = "0"
var scanResult: util.List[String] = new util.ArrayList[String]()
do {
val resource: Jedis = clusterNode.getResource
val scan: ScanResult[String] = resource.scan(scanCursor, new ScanParams().`match`(PROD_SKU_PRICE + "*"))
scanCursor = scan.getStringCursor
scanResult = scan.getResult
for (key: String <- scanResult) {
keys.add(key)
}
resource.close()
} while (!scanCursor.equals("0") && !scanResult.isEmpty)
}
print (keys.size)
After verification, the total number is the same. Through this test, it is also found that the count of the scan command will not really return the number of key values each time (although it still exists in the future), but will only return the key value smaller than the count value. quantity.
However, considering that the scan command cannot be applied in RedisCluster mode, we can make certain modifications to the command so that it can support the operation according to the hash tag. Since keys with the same hash tag will be stored on the same redis node, only a single redis connection operation in RedisCluster can be performed at this time, which is somewhat similar to the read-write separation scheme mentioned earlier: http://brandnewuser.iteye.com /blog/2315636 , directly in the previous scheme, add the scan method (in the inherited JedisCluster class), since there is no corresponding key in this operation, it is necessary to additionally pass the hashTag to specify which node to use for processing:
public ScanResult<String> scan(final String hashTag, final String cursor, final ScanParams params) {
ZhenQueryContextHolder.getInstance().setQueryContext(new ZhenQueryContext(OperationType.READ));
return new ZhenJedisClusterCommand<ScanResult<String>>(connectionHandler, maxRedirections) {
@Override
public ScanResult<String> execute(Jedis connection) {
return connection.scan(cursor, params);
}
}.run(hashTag);
}
refer to: