Two ways of Redis data persistence and Redis to achieve subscription publishing

Redis data persistence method

Redis is an in-memory database. If the database state in memory is not saved to disk, once the server process exits, the database state in the server will disappear, and the data inside will be lost at the same time. Therefore, Redis provides a persistence function.

In master-slave replication, RDB is used as a backup (on the slave).

Redis has two data persistence methods, RDB and AOF respectively. Let's introduce these two methods respectively.

1. RDB method (Redis DataBase)

1. What is RDB?

Save the database state of Redis in memory to disk. The RDB file is a compressed binary file. The database state when the RDB file is generated can be restored through this file (by default, it is persisted to the dump.rdb file and stored in redis). After restarting, it will automatically read the files in it. It is reported that the time to synchronize to the memory is 20-30 seconds for a string type key of 10 million and a 1GB snapshot file. In layman's terms, the data set in the memory is written to the disk as a snapshot within a specified time interval, which is called a Snapshot in the jargon. When the data is restored, the snapshot file is directly read into the memory. .

2. Introduction to the RDB persistence process

Redis first creates (fork) a child process for persistence. The child process writes all data to a temporary file. This temporary file is a temporary RDB file. This file needs to be persisted and waited until it is persisted. After the conversion process is over, the RDB temporary file is replaced with the last persisted file, and an official RDB file is also generated. During the entire process, the main thread does not perform any IO operations, which ensures extremely high performance .

Application:
If large-scale data recovery is required and the integrity of data recovery is not very sensitive, then the RDB method is more efficient than the AOF method. But the disadvantage of RDB is that the last persistence is down, and data may be lost. The default persistence method of Redis is RDB , and we generally don't need to modify this configuration.

3. The configuration of RDB in the configuration file

The file name saved by rdb is dump.rdb , which are configured in the snapshot option in the configuration file.

In the configuration file, the default settings for the persistence frequency of snapshots are as follows:

#设置Redis进行快照镜像的频率
save 900 1         #如果900秒（15分钟）内，至少有一个key进行了修改，我们就进行持久化操作（写入到数据库的持久化文件中）
save 300 10       #如果300秒内（6分钟），有10个key进行了修改，就进行持久化操作
save 60 10000   #如果60秒内（1分钟），有10000个key进行了修改，就进行持久化操作（多线程）

4. RDB trigger mechanism

• When the save rule in the configuration file is met, the rdb rule will be automatically triggered to generate an rdb file;
• When the fluxhall command is executed, the rdb rule will also be triggered to generate the rdb file;
• When exiting Redis, an rdb file will also be generated.

The backup will automatically generate a dump.rdb file.

5. How to restore rdb files?

• Just put the dump.rdb file in our redis startup directory , redis will automatically check dump.rdb to restore the data when it starts.
• Use the config get dircommand to view the startup directory, as long as the dump.rdb file is in this directory, the data in it will be automatically restored after startup.
  

6. Introduction of RDB manual execution commands SAVE and BGSAVE

• The SAVE command will block the Redis server process until the RDB file is created. During the server process is blocked, the server cannot process any command requests;
• The BGSAVE command forks a child process, and then the child process is responsible for creating the RDB file, and the server process (parent process) continues to process the command request. (Generally use this method)
• Before the end of the creation of the RDB file, that is, before the data persistence is completed, the client executes the SAVE or BGSAVE command again, it will be rejected by the server.

7. Application scenarios and disadvantages

• Suitable for large-scale data recovery;
• For occasions where data integrity is not high.

In the production environment, we will back up the dump.rdb file.

Disadvantages:

• It takes a certain time interval to operate.
• If Redis goes down unexpectedly, the last modified data is gone.
• When forking a child process, it will take up a certain amount of memory space.

2. AOF method (Append Only File)

Record all our write commands, history, and execute this file once when restoring.

1. The working principle of the AOF method

It records each write operation in the form of a log, and records all the instructions that Redis has executed (read operations are not recorded). Only files can be appended but files cannot be rewritten. Redis will read the files at the beginning of startup. Build data , in other words, after Redis restarts, it will execute the write command from front to back according to the content of the log file to complete the data recovery work.

The default AOF method is unlimited appending of files, which will cause the files to become larger and larger.

2. AOF persistence process

First, the parent process goes back to fork a child process, and then the child process rewrites the data to an AOF temporary file according to the data snapshot in the memory, and then the parent process caches the write command and writes the write command To the original AOF file, when the child process is finished writing, it will notify the parent process. After receiving the notification, the parent process writes the write command in the cache to the AOF temporary file, and then the parent process replaces the temporary file with the original The AOF file is renamed to complete the persistence operation, and finally if a write command appears later, it will be appended to the new AOF file.

3. The configuration of AOF in the configuration file

The file name saved by AOF is: appendonly.aof , this method is not enabled by default. If you need to use AOF, you need to manually enable it in the configuration file. The configuration details are as follows:

#开启aof模式的配置，默认是不开启，因为Redis默认是使用rdb方式持久化的，在大部分的情况下，rdb完全够用了。
appendonly no

#aof持久化的文件名字
appendfilename "appendonly.aof"

#配置持久化同步sync的选项（持久化策略），默认是everysec每秒同步一次
# appendfsync always     #  每次修改都会同步sync ，这种方式比较耗性能
appendfsync everysec    #  每一秒执行一次同步sync，可能会丢失这1s的数据。
# appendfsync no           #  不执行同步sync，这个时候操作系统自己同步数据，速度最快。
#是否进行重写选项。在 aof 重写或者写入 rdb 文件的时候，会执行大量 IO，此时对于 everysec 和 always 的 aof 模式来说，执行同步(fsync) 会造成阻塞过长时间，no-appendfsync-on-rewrite 字段设置为默认设置为 no。如果对延迟要求很高的应用，这个字段可以设置为 yes，否则还是设置为 no，这样对持久化特性来说这是更安全的选择。设置为 yes 表示 rewrite 期间对新写操作不同步fsync,暂时存在内存中,等 rewrite 完成后再写入。
#默认为 no，建议 no，可以保证数据的安全性。Linux 的默认 fsync 策略是 30 秒。可能丢失 30 秒数据。
no-appendfsync-on-rewrite no

#aof文件的自动重写配置。当目前的aof文件大小超过上一次重写的aof文件大小的百分之多少的时候进行重写，当aof文件增长到一定大小的时候，Redis能够调用bgrewriteaof对日志进行重写。默认设置为100，设为100意思就是当前aof文件大小是上次日志重写的aof文件大小的两倍时，自动启动新的日志重写过程。
auto-aof-rewrite-percentage 100

#允许重写最小aof文件的大小设置，如果超过了64MB就会去写一些其它的文件。
auto-aof-rewrite-min-size 64mb

#指redis在恢复时，会忽略最后一条可能存在问题的指令。默认值yes。即在aof写入时，可能存在指令写错的问题(突然断电，写了一半)，这种情况下，设置为yes会log并继续，设置为no会直接恢复失败
aof-load-truncated yes

#加载redis时，可以识别AOF文件以“redis”开头。
#字符串并加载带前缀的RDB文件，然后继续加载AOF尾巴
aof-use-rdb-preamble yes

The AOF mode is not enabled by default. We need to configure it manually. Just change the appendonly nomiddle noto yesenable the AOF mode, and then restart Redis to take effect.

4. What if there is a problem with the appendonly.aof file, is damaged or damaged?

When the appendonly.aof file is damaged, we will report the following error (connection refused) when we connect to Redis:

This is because there is a problem with the appendonly.aof file, and Redis cannot be started, so we need to repair this aof file. Redis provides us with a redis-check-aof tool, which can repair the appendonly.aof file. The file is in the bin directory.
Command: After redis-check-aof --fix appendonly.aof

executing this command, we can see that the AOF file is repaired successfully, just connect again.

5. Advantages and disadvantages

(1) Advantages:

• Every modification is synchronized, the integrity of the file will be better;
• Synchronize once every second, you may lose one second of data;
• Never synchronize, the highest efficiency;

(2) Disadvantages:

• Compared with data files, AOF is much larger than RDB, and the repair speed is also slower than RDB;
• AOF runs slower than RDB, so the default configuration of Redis is RDB.

Expand

1. The RDB persistence method can perform snapshot storage of your data within a specified time interval;

2. The AOF persistence method records each write operation to the server. When the server restarts, these commands will be re-executed to restore the original data . The AOF command uses the Redis protocol to append and save each write operation to the end of the file. Redis can also Rewrite the AOF file in the background so that the volume of the AOF file is not too large;

3. Only caching. If you only want your data to exist while the server is running, you don't need to use any persistence.

4. Open two persistence methods at the same time

• In this case, when Redis restarts, it will load the AOF file first to restore the original data , because under normal circumstances the data set saved by the AOF file is more complete than the data set saved by the RDB file.

• The RDB data is not real-time, and the server will only find the AOF file when the server restarts when both are used at the same time. Should I only use AOF? It is not recommended here, because RDB is more suitable for backing up the database (AOF is not good for backup when it is constantly changing), fast restart, and there will be no potential AOF BUG, ​​keep RDB as a precautionary preparation.

5. Performance recommendations

• Because the RDB file is only used for backup purposes, it is recommended to only persist the RDB file on the Slave (slave), and only a 15-minute backup is enough. Only save 900 1 is the rule.

• If AOF is enabled, the advantage is that in the worst case, only data of no more than two seconds will be lost. The startup script is simpler. Just load your own AOF file. The cost is that it brings continuous IO and the other is At the end of AOF rewrite, it is inevitable to write the new data generated during the rewrite process to the new file. The blocking opportunity is inevitable. As long as the hard disk permits, the frequency of AOF rewrite should be reduced as much as possible. The default value of 64M for AOF rewrite is too small. It can be set to above 5G, and the default value exceeds 100% of the meta size and can be rewritten to an appropriate value.

• If you don’t Enable AOF, you can achieve high availability by only relying on Master-Slave Repllcation, which can save a lot of IO and reduce the system fluctuations caused by redeite. The cost is if the Master/Slave are dropped at the same time (power failure) , It will lose more than ten minutes of data. The startup script also compares which of the RDB files in the two Master/Slave is updated, and then loads the newer one. Weibo is this architecture.

Redis publish and subscribe

Redis publish and subscribe (pub/sub) is a message communication mode . The sender (pub) sends a message, and the subscriber (sub) receives the message. Redis clients can subscribe to any number of channels.

Such as WeChat, Weibo, follow system, etc.

Subscribe/publish model

The message publisher first publishes the message to the channel (Redis server), and then the channel will publish the message to the subscribers. As long as the user who subscribes to the channel (message subscriber) will receive the message sent from the channel, if not You will not receive messages if you subscribe to this channel.

For example : For example, if you push an article on WeChat's official account, when the author writes an article, he publishes the article to the official account, and then users who follow the official account will receive the information sent by the official account .

In the whole process, the author is equivalent to the news publisher in the figure, and then the official account is equivalent to the channel, and the users who follow the official account (channel) are the message subscribers.

Subscription relationship introduction

1. The following figure shows the channel channel1 and the relationship between the three clients that subscribe to this channel-client2, client5 and client1.

2. When a new message is PUBLISHsent to the channel channel1 through a command, the message will be sent to the three clients that subscribe to it:

3. Introduction to related commands

(1) PSUBSCRIBE pattern [pattern .....]: Subscribe to one or more channels that conform to a given mode.

(2) PUBSUB subcommand [argument [argument ....]]: View the status of the subscription and publishing system.

(3) PUBLISH channel message: Send the information message to the specified channel channel.

(4) PUNSUBSCRIBE [pattern [pattern ....]]: Unsubscribe all channels of a given mode.

(5) SUBSCRIBE channel [channel ....]: Subscribe to the information of a given one or more channels.

(6) UNSUBSCRIBE [channel [channel ....]]: Refers to unsubscribe from the given channel

The above commands are widely used to build instant messaging applications, such as: online chat rooms (chatroom) and real-time broadcasting, real-time reminders, etc.

4. Test:

First prepare two clients, one for the role of message publisher (pub) and one for the role of message subscriber (sub).

(1) Subscribers subscribe to java channels:

(2) Publishers publish messages in java channels:

(3) Subscribers receive messages from Java channels:

5. Principle:

• Redis is implemented in C. By analyzing the pubsub.c file in the Redis source code, we can understand the underlying implementation of the publish and subscribe mechanism.

• Redis implements publish and subscribe functions through commands such as publish, subscribe, and psubsctibe.

• After subscribing to a channel through the sunsctibe command, a dictionary is maintained in the redis-server. The key of the dictionary is a channel, and the value of the dictionary is a linked list. The linked list stores all the clients that subscribe to this channel. The key to the subscribe command is to add the client to the subscription list of a given channel.

• Send a message to subscribers through the publish command, redis-server will use the given channel as a key, look up the linked list that records all clients subscribing to this channel in the channel dictionary it maintains, traverse this linked list, and publish the message to All subscribers.

• Pub/Sub literally means Publish and Subscribe. In Redis, you can set a key value for message publishing and message subscription. When a message is published on a key value, all Clients who subscribe to it will receive the corresponding message. The most obvious use of this function is to use it as a real-time messaging system, such as ordinary instant chat, group chat and other functions.

6. Usage scenarios

• Real-time messaging system.
• Real-time chat (the channel is used as a chat room, and the information is returned to everyone).
• You can subscribe or follow the system.

Slightly more complex scenarios will use message middleware MQ.

It is not easy to create, if this article is helpful to you, you can give me a like and support it.