Redis Pipeline Technology & Partitioning

Redis pipeline technology
Redis is a TCP service based on client-server model and request/response protocol. This means that usually a request will follow the following steps:
the client sends a query request to the server, and listens for the Socket to return, usually in blocking mode, waiting for the server to respond.
The server processes the command and returns the result to the client.
Redis pipeline technology
Redis pipeline technology can continue to send requests to the server when the server does not respond, and finally read the responses of all servers at one time.
Example To
view the redis pipeline, just start the redis instance and enter the following command:
$(echo -en "PING\r\n SET w3ckey redis\r\nGET w3ckey\r\nINCR visitor\r\nINCR visitor\r\nINCR visitor\ r\n"; sleep 10) | nc localhost 6379

+PONG
+OK
redis
:1
:2
:3
In the above examples, we use the PING command to check whether the redis service is available, then we set the value of w3ckey to redis, and then we Get the value of w3ckey and make the visitor increment 3 times.
In the returned results, we can see that these commands are submitted to the redis service at one time, and finally the responses of all servers are read at one
time . Advantages of
Pipeline Technology The most significant advantage of pipeline technology is to improve the performance of redis services.
some test data
In the following test, we will use the Ruby client of Redis, which supports the pipeline technology feature, and test the speed improvement effect of the pipeline technology.
require 'rubygems'
require 'redis'
def bench(descr)
start = Time.now
yield
puts "#{descr} #{Time.now-start} seconds"
end
def without_pipelining
r = Redis.new
10000.times {
r.ping
}
end
def with_pipelining
r = Redis.new
r.pipelined {
10000.times {
r.ping
}
}
end
bench("without pipelining") {
without_pipelining
}
bench("with pipelining") {
with_pipelining
}
From Mac OS in LAN The data from executing the above simple script on X systems shows that the round-trip latency has been improved to a fairly low level with pipelining enabled.
without pipelining 1.185238 seconds
with pipelining 0.250783 seconds
As you can see, we are 5 times faster and more efficient when the pipe is turned on. Redis Partitioning Partitioning is the process of splitting data into multiple Redis instances, so each instance holds only a subset of keys. Advantages of Partitioning Allows us to construct larger databases by of the sum of the memory of multiple computers. Multiple cores and multiple computers allow us to expand computing power; multiple computers and network adapters allow us to expand network bandwidth. Disadvantages of Partitioning Some features of Redis are not very good at partitioning: operations involving multiple keys are generally not supported. For example, when two sets are mapped to different redis instances, you cannot perform the intersection operation on the two sets. Redis transactions involving multiple keys cannot be used. When using partitions, data processing is more complex, for example you need to process multiple rdb/aof files and backup persistent files from multiple instances and hosts. Adding or removing capacity is also more complicated. Most redis clusters support the ability to add and delete nodes at runtime with transparent data balancing, but other systems like client partitions, proxies, etc. do not support this feature. However, a technique called presharding can help. Partition Types Redis has two types of partitions. Suppose there are 4 Redis instances R0, R1, R2, R3, and multiple keys representing users such as user:1, user:2, there are different ways to select which instance the key is stored in for a given key . That is, there are different systems for mapping a key to a Redis service.


















Range partitioning
The simplest partitioning method is range partitioning, which maps a range of objects to a specific Redis instance.
For example, users with IDs from 0 to 10000 are saved to instance R0, users with IDs from 10001 to 20000 are saved to R1, and so on.
This method is feasible, and used in practice, the disadvantage is that there must be a mapping table from the range to the instance. This table needs to be managed, and also needs a mapping table of various objects, which is usually not a good method for Redis.
Hash Partitioning
Another partitioning method is hash partitioning. This works for any key, not necessarily object_name: This form is as simple as described below:
convert the key to a number with a hash function, such as the crc32 hash function. Executing crc32(foobar) on key foobar will output an integer like 93024922.
Modulo this integer, convert it to a number between 0-3, and you can map this integer to one of four Redis instances. 93024922 % 4 = 2, which means the key foobar should be stored in the R2 instance. Note: The modulo operation is the remainder of the division, usually implemented with the % operator in many programming languages.