Redis is very easy to use. Have you ever understood what protocol it uses?

Redis

A very important reason why Redis is widely used is its high performance. Therefore, we must pay attention to all the factors, mechanisms and countermeasures that may affect Redis performance.

A small partner came back from the interview and said that the interviewer asked him some Redis questions, but he didn't seem to answer.

I said, don't you use Redis very sluggishly? What problem is holding you up.

He said that the thing is like this. At the beginning, some basic questions were asked, such as several basic data types and usage scenarios of Redis, as well as some problems with master-slave replication and clustering, which were all fine.

Then I asked about the two persistence methods of Redis. I said that with the two methods of RDB and AOF, the RDB data file is small and the recovery speed is fast, but it has an impact on performance and is not suitable for real-time storage. AOF is now the most commonly used persistence method. One of its advantages is real-time and has the least impact on the performance of Redis half-length.

Then the interview asked again, do you know the format of the file after AOF persistence?

Answer: It seems to be a text file, right?

Okay, text file, do you know what rules it has? In other words, does it have anything to do with the Redis protocol?

Answer: Ah, this, um, I’m not sure.

Now let's take a look at the relationship between AOF and RESP agreement:

1. Speaking of two persistence methods.
2. What is the RESP protocol
3. Hands-on implementation of a simple protocol analysis command line tool

 

Let's start with persistence. Although when you mention Redis, the first thing that comes to mind is caching, but Redis is not just as simple as caching. It is positioned as an in-memory database, which can store many types of data structures and can also be used as simple messages. Queue usage. Since it is a database, the persistence function is essential.

Two persistence methods of Redis

Redis provides two persistence methods, one is the RDB method, and the other is the AOF method. AOF is currently a more popular persistence solution.

RDB way

RDB persistence uses snapshots to write snapshots of data sets in memory to disk within a specified time interval. It comes in the form of a compact compressed binary file. You can copy the snapshot to another server to create a server copy of the same data, or restore the data after restarting the server. RDB is the default persistence method of Redis and a necessary solution for earlier versions.

RDB is controlled by the following parameters.

# 设置 dump 的文件名
dbfilename dump.rdb

# 持久化文件的存储目录
dir ./

# 900秒内，如果至少有1个key发生变化，就会自动触发bgsave命令创建快照
save 900 1

# 300秒内，如果至少有10个key发生变化，就会自动触发bgsave命令创建快照
save 300 10

# 60秒内，如果至少有10000个key发生变化，就会自动触发bgsave命令创建快照
save 60 10000

Persistence process

The above mentioned several trigger persistence mechanisms in the configuration file, such as 900 seconds, 300 seconds, and 60 seconds. Of course, you can also manually execute the command save or bgsave to trigger. bgsave is a non-blocking version. It generates snapshots by fork out of the child process, while save blocks the main process and is not recommended.

1. First, the bgsave command is triggered;

2. The parent process forks a child process. This step is a relatively heavyweight operation, and it is also an important reason why the performance of RDB is not as good as AOF;

3. After the parent process forks the child process, it can respond normally to other commands sent by the client;

4. The child process starts the persistence work, and completes the snapshot storage of the existing data;

5. After the child process completes the operation, notify the parent process;

 

Advantages of RDB:

• RDB is a compact and compressed binary file that represents a snapshot of Redis data at a certain point in time. Very suitable for scenarios such as backup and full copy. For example, perform bgsave backup every 6 hours and copy the RDB file to a remote machine or file system (such as hdfs) for disaster recovery.

• Redis loads RDB to restore data much faster than AOF.

Disadvantages of RDB:

• There is no way to achieve real-time persistence/second-level persistence for RDB data. Because bgsave has to perform a fork operation to create a child process every time it runs, it is a heavyweight operation and the cost of frequent execution is too high.

• RDB files are saved in a specific binary format. There are multiple RDB versions in the Redis version evolution process, and there is a problem that the old version of the Redis service cannot be compatible with the new version of the RDB format.

AOF method

AOF is controlled by the following parameters.

# appendonly参数开启AOF持久化
appendonly yes

# AOF持久化的文件名，默认是appendonly.aof
appendfilename"appendonly.aof"

# AOF文件的保存位置和RDB文件的位置相同，都是通过dir参数设置的
dir ./

# 同步策略
# appendfsync always
appendfsync everysec
# appendfsync no

# aof重写期间是否同步
no-appendfsync-on-rewrite no

# 重写触发配置
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# 加载aof出错如何处理
aof-load-truncated yes

# 文件重写策略
aof-rewrite-incremental-fsync yes

To solve the problem that RDB is not suitable for real-time persistence, Redis provides AOF persistence method to solve it. AOF is also the most popular persistence method at present.

AOF (append only file), records each write command in an independent log, and re-executes the commands in the AOF file when restarting to achieve the purpose of recovering data.

1. All write commands will be appended to aof_buf (buffer);

2. The AOF buffer is synchronized to the hard disk according to the corresponding strategy;

3. As AOF files become larger and larger, AOF files need to be rewritten regularly to achieve the purpose of compression;

4. When the Redis server restarts, you can load the AOF file for data recovery;

 

What is stored in the AOF file

I randomly swiped a few commands in the local test redis environment, and then opened the appendonly.aof file to view it, and found that the content inside was like the following.

 

RESP protocol

The Redis client communicates with the server using the RESP protocol. This protocol is a communication protocol designed specifically for Redis, but it can also be used in other client-server communication scenarios.

The RESP protocol has the following characteristics:

• Simple to implement;
• Quick analysis
• Can read

The client sends a command to the server, the server parses the command after receiving the command, then executes the corresponding logic, and then returns to the client. Of course, this send and reply are all in the format of the RESP protocol.

Under normal circumstances, we will use redis-cli or some client tools to connect to the Redis server.

./redis-cli

Then the command sending and return results of the entire interactive process are as follows, the green part is the sent command, and the red part is the returned result.

 

This is the part that we are all familiar with. However, this does not reveal the true face of the RESP agreement.

Try telnet

RESP is implemented based on the TCP protocol, so in addition to using various client tools and the redis-cli tool provided by Redis, you can also use telnet to view it. With telnet, you can see the original data format returned by RESP.

My local Redis uses the default port 6379, and no requirepass is set, let's try to connect with telnet.

telnet 127.0.0.1 6379

Then execute the same commands as before, and the results sent and returned are as follows, the green part is the sent command, and the red part is the returned result.

 

Well, some commands return okay, but like get str:hello, in addition to the world value itself, there is an extra line of $5 on the returned result, is it a bit confused?

Agreement rules

Request command

The rules for a command sent from the client to the server are as follows:

*<参数数量> CR LF
$<参数 1 的字节数量> CR LF
<参数 1 的数据> CR LF
...
$<参数 N 的字节数量> CR LF
<参数 N 的数据> CR LF

RESP uses \r\n as the delimiter to indicate the number of specific parameters of this command. From the point of view of the command, spaces separated by spaces represent one parameter. For example, set str:hello world command has 3 parameters. Indicate the number of characters and specific content of each parameter.

Using this command as an example, corresponding to RESP protocol rules will become as follows:

*3\r\n$3\r\nset\r\n$9str:hello\r\n$5world\r\n

 

Server reply

Redis commands return many different types of replies.

By checking the first byte of the data sent back by the server, you can determine what type of response the response is:

1. The first byte of status reply is "+"

For example, the reply of ping command, +PONG\r\n

2. The first byte of error reply is "-"

For example, input a command that does not exist in redis, or set wrong parameters for some commands, such as input auth. If there is a password parameter after the auth command, it will return an error response type if you do not input it.

-ERR wrong number of arguments for 'auth' command\r\n

3. The first byte of integer reply (integer reply) is ":"

For example, INCR, DECR auto-increment and auto-decrement commands, the returned result is this: 2\r\n

4. The first byte of bulk reply is "$"

For example, perform a get operation on the string type, $5\r\nworld\r\n, the number 5 after $ means that the returned result has 5 characters, followed by the actual content of the returned result.

5. The first byte of multi bulk reply is "*"

For example, commands that return multiple results such as LRANGE key start stop or hgetall, such as the results returned by the lrange command:

*2\r\n$6\r\nnews-2\r\n$6\r\nnews-1\r\n

The multiple batch replies are consistent with the format of the command sent by the client as mentioned earlier.

Implement a simple Redis interactive tool

Knowing the Redis protocol rules, we can write a simple client by ourselves. Of course, through the official website, we can see that there are already various languages, and each language has more than one client tool.

 

For example, there are so many kinds of Java language clients, among which Jedis should be the most used. Since there are already such useful wheels, of course, there is no need to re-create the wheels, mainly to deepen the impression.

 

The RESP protocol is based on the TCP protocol and can be connected in socket mode.

publicSocketcreateSocket()throwsIOException{
Socket socket =null;
try{
socket =newSocket();
socket.setReuseAddress(true);
socket.setKeepAlive(true);
socket.setTcpNoDelay(true);
socket.setSoLinger(true,0);

socket.connect(newInetSocketAddress(host, port), DEFAULT_TIMEOUT);
socket.setSoTimeout(DEFAULT_TIMEOUT);
outputStream = socket.getOutputStream();
inputStream = socket.getInputStream();
returnsocket;
}catch(Exception ex) {
if(socket !=null) {
socket.close();
}
throwex;
}
}

Then all that is left is to parse the returned results. The tools I made have reached this point. The following is the return output of some simple commands.

 

The code has been put on github, if you are interested, you can clone it and take a look.

>https://github.com/huzhicheng/medis

I hope it can be helpful to everyone's study, and the friends I like can help forward + follow, thank you everyone~