Redis series articles:

Article Directory

- Redis pipeline
- - - Request/response protocol and RTT
    - Redis pipeline (Pipelining)
Redis publish and subscribe

Redis pipeline

Request/response protocol and RTT

Redis is a TCP service based on a client-server model and request/response protocol.

This means that usually a request will follow the following steps:

The client sends a query request to the server and listens to the Socket to return, usually in blocking mode, waiting for the server to respond.
The server processes the commands and returns the results to the client.

Therefore, for example, the following is the execution of 4 command sequences:

Client: INCR X
Server: 1
Client: INCR X
Server: 2
Client: INCR X
Server: 3
Client: INCR X
Server: 4

The client and server are connected through the network. This connection can be fast (loopback interface) or slow (a network connection with multiple jumps is established). Regardless of the network delay, data packets can always reach the server from the client, and return data from the server to the client.

This time is called RTT (Round Trip Time). When a client needs to perform multiple requests in a batch, it is easy to see how this affects performance (for example, adding many elements to the same list, Or fill the database with a lot of Keys). For example, if the RTT time is 250 milliseconds (under a very slow connection), even if the server can handle 100k requests per second, we can only handle 4 requests per second at most.

If the loopback interface is used, the RTT is much shorter (for example, it only takes 44 milliseconds for my host to ping 127.0.0.1), but it is still a lot of overhead in a batch write operation.

Fortunately, there is a way to improve this situation.

Redis pipeline (Pipelining)

The one-time request/response server can handle new requests even if the old requests have not been responded. In this way, multiple commands can be sent to the server without waiting for a reply, and finally the reply is read in one step.

This is pipelining, a technique that has been widely used for decades. For example, many POP3 protocols have been implemented to support this function, which greatly speeds up the process of downloading new mail from the server.

Redis has long supported pipelining technology, so no matter what version you are running, you can use pipelining to operate Redis.

Here is an example of usage:

1. Open the redis server

service redis_6379 start

2. Use nc to establish a connection with the server and send multiple commands

echo -e "set k1 99\n incr k1\n get k1" | nc localhost 6379

Carriage return output:

Insert picture description here

The first command set k1 99outputs ok
The second command incr k1k1 value +1 outputs 100
The third command get k1queries the value of k1 and outputs 100

$3Means that the width of 100 is 3.

\n It is the OK symbol in linux, the OK symbol in window is:\r\n

Important note : When using pipes to send commands, the server will be forced to reply to a queue reply, which takes up a lot of memory. So, if you need to send a large number of commands, it is best to process them in batches according to a reasonable number, such as 10K commands, read the reply, and then send another 10k command, and so on. This speed is almost the same, but in reply to the 10k command queue, a very large amount of memory is needed to organize the returned data content.

Redis publish and subscribe

Redis publish and subscribe (pub/sub) is a message communication mode: the sender (pub) sends the message, and the subscriber (sub) receives the message.

Redis clients can subscribe to any number of channels.

The following figure shows the relationship between channel channel1 and the three clients that subscribe to this channel-client2, client5 and client1:

Insert picture description here

When a new message is sent to channel channel1 through the PUBLISH command, the message will be sent to the three clients that subscribe to it:

Insert picture description here

Instance

The following example demonstrates how publish and subscribe works. Three redis-cli clients need to be opened.

1. Open two redis-cli and subscribe to the bo channel

Insert picture description here

2. Open another client to publish messages: apple

Insert picture description here

3. You can see that both clients have received a message published by another client: apple

Insert picture description here

Application scenario

WeChat chat history:

Insert picture description here

Real-time data is achieved through publish/subscribe
The chat records within three days are placed in the sorted_set ordered list of redis, and the time is used as the score to sort
The longer data must be put in the mysql database

Let’s think about the above example. When I 发布/订阅receive a message from a client, we need to adjust sorted_set to store, and then adjust the database to store. Then I don’t want to call it now. Is there a better way? Of course there are:

Insert picture description here

We split the sorted_set into a redis, and realize the notification by subscribing to real-time redis, and the database microservices also need to subscribe to the current redis to achieve the notification effect.

Redis things

MULTI , EXEC , DISCARD and WATCH are commands related to Redis transactions. A transaction can execute multiple commands at once, with the following two important guarantees:

The transaction is a separate isolated operation: all commands in the transaction are serialized and executed sequentially. During the execution of the transaction, it will not be interrupted by command requests sent by other clients.
A transaction is an atomic operation: either all commands in the transaction are executed, or none of them are executed.

The EXEC command is responsible for triggering and executing all commands in the transaction:

If the client uses MULTI to start a transaction, but fails to execute EXEC because of the disconnection , all commands in the transaction will not be executed.
On the other hand, if the client successfully executes EXEC after opening the transaction , all commands in the transaction will be executed.

usage

The MULTI command is used to start a transaction and it always returns OK. After MULTI is executed, the client can continue to send any number of commands to the server. These commands will not be executed immediately, but will be placed in a queue. When the EXEC command is called, all the commands in the queue will be executed.

On the other hand, by calling DISCARD , the client can clear the transaction queue and give up executing the transaction.

1. Open client1, set the value bo of k1, and turn on things, and then enter the query command get k1

127.0.0.1:6379> set k1 bo
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> get k1
QUEUED
127.0.0.1:6379>

2. Open client2, open things, delete k1, and execute things

127.0.0.1:6379> multi
OK
127.0.0.1:6379> del k1
QUEUED
127.0.0.1:6379> exec
1) (integer) 1
127.0.0.1:6379>

K1 is successfully deleted at this time

3. Execute the things in client1

127.0.0.1:6379> exec
1) (nil)
127.0.0.1:6379>

Found that key1 has been deleted at this time!

Note: Redis is single-process, single-thread, and single-instance, so whoever exec comes first will execute whose thing first!

Error in things

The following two errors may be encountered when using transactions:

Before the transaction executes EXEC , the command enqueued may be wrong. For example, the command may produce syntax errors (wrong number of parameters, parameter name error, etc.), or other more serious errors, such as insufficient memory (if the server uses maxmemoryto set the maximum memory limit of words).
The command may fail after EXEC is called. For example, the command in the transaction may process the wrong type of key, such as using the list command on the string key, and so on.

For errors that occur before the execution of EXEC , the client's previous approach is to check the return value of the command enqueuing: if the command returns when enqueuing QUEUED, then enqueue is successful; otherwise, enqueue failed. If a command fails while enqueuing, most clients will stop and cancel the transaction. However, starting from Redis 2.6.5, the server will record the failure of the command to enter the queue, and when the client calls the EXEC command, it will refuse to execute and automatically abandon the transaction.

As for the errors that occur after the execution of the EXEC command, no special treatment is given to them: even if an error occurs during the execution of certain commands in the transaction, other commands in the transaction will continue to be executed.

Why Redis things don’t support roll back

If you have experience with relational databases, the practice of "Redis does not roll back when the transaction fails, but continues to execute the remaining commands" may make you feel a little strange.

The following are the advantages of this approach:

Redis commands will only fail due to incorrect syntax (and these problems cannot be found when enqueuing), or the command is used on the wrong type of key: that is, from a practical point of view, the failed command Is caused by programming errors, and these errors should be found in the development process, and should not appear in the production environment.
Because there is no need to support rollbacks, the internals of Redis can be kept simple and fast.

There is a view that Redis's handling of transactions will cause bugs, but it should be noted that under normal circumstances, rollbacks cannot solve the problems caused by programming errors. For example, if you originally wanted to add 1 to the value of the key through the INCR command, but accidentally added 2, or executed INCR on the wrong type of key , there is no way to handle these situations with rollback.

Give up things

When the DISCARD command is executed , the transaction will be abandoned, the transaction queue will be emptied, and the client will exit from the transaction state:

127.0.0.1:6379> multi
OK
127.0.0.1:6379> discard
OK
127.0.0.1:6379>

Use check-and-set operations to achieve optimistic locking

The WATCH command can provide check-and-set (CAS) behavior for Redis transactions.

The WATCH keys will be monitored and will find out whether these keys have been changed. If at least one monitored key is modified before EXEC is executed, the entire transaction will be cancelled, and EXEC returns nil-reply to indicate that the transaction has failed.

1. Client1 sets k1 to 1, then turns on things, k1 value +1

127.0.0.1:6379> set k1 1
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> incr k1
QUEUED

2. Client2 watches k1 and then turns on things to get the value of k1

127.0.0.1:6379> watch k1
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> get k1
QUEUED

3. Client1 executes things and queries the value of k1

127.0.0.1:6379> set k1 1
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> incr k1
QUEUED
127.0.0.1:6379> exec
1) (integer) 2
127.0.0.1:6379> get k1
"2"

4. Client2 performs things

127.0.0.1:6379> watch k1
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> get k1
QUEUED
127.0.0.1:6379> exec
(nil)

It can be seen that when client2 executes a thing, since the watched k1 has been added by client1, it will return (nil) when client2 gets the value of k1 when executing the thing, which proves that the thing will not be executed.

Redis Key expiration time

Usually, the expiration time is not set when Redis keys is created. They will always exist unless removed using the displayed command, for example, using the DEL command.

EXPIREOne type of command can be associated with a key with additional memory overhead. When keys perform expiration operations, Redis will ensure that they are deleted in accordance with the prescribed time.

key expiration time and permanent effectiveness can EXPIREand PERSIST update or delete the expiration time command (or other commands) to carry out.

The expiration time of redis will not be extended with access.
The expiration time will be eliminated when writing occurs.

redis set expiration time

set key value ex seconds
expire key seconds

redis clear expiration time

There are two ways to expire Redis keys: passive and active.

When some clients try to access it, the key will be discovered and actively expired.

Of course, this is not enough, because some expired keys will never be accessed. In any case, these keys should expire, so set the expiration time of the keys regularly and randomly. All these expired keys will be deleted from the key space.

Specifically, what Redis does 10 times per second:

Test 20 random keys for related expiration detection.
Delete all expired keys.
If more than 25% of the keys expire, repeat step 1.

This is a trivial probability algorithm. The basic assumption is that our sample is this key control, and we keep repeating expiration detection until the percentage of expired keys is less than 25%, which means that in any given At a given time, up to 1/4 of the expired keys will be cleared.

The purpose is to slightly sacrifice some memory, but to ensure the high performance of redis!

See Through Redis Series (3): Redis pipeline, publish/subscribe, things, expiration time detailed introduction

Article Directory

Redis pipeline

Request/response protocol and RTT

Redis pipeline (Pipelining)

Redis publish and subscribe

Instance

Application scenario

Redis things

usage

Error in things

Why Redis things don’t support roll back

Give up things

Use check-and-set operations to achieve optimistic locking

Redis Key expiration time

redis set expiration time

redis clear expiration time

Guess you like