Internet giant technology-Redis-cluster model, architecture principle, difficult application scenarios, detailed explanation of high-frequency interview questions

Redis, the English full name is Remote Dictionary Server (remote dictionary service), is an open source written in ANSI C language, supports the network, can be based on memory or persistent log type, Key-Value database, and provides APIs in multiple languages .

Like memcached, data is cached in memory to ensure efficiency. The difference is that redis will periodically write updated data to disk or write modification operations to additional record files, and on this basis, master-slave (master-slave) synchronization is realized.

Unlike the MySQL database, Redis data is stored in memory. Its read and write speed is very fast, and it can handle more than 100,000 read and write operations per second. Therefore, redis is widely used in caching. In addition, Redis is often used as a distributed lock. In addition, Redis supports transactions, persistence, LUA scripts, LRU-driven events, and multiple cluster solutions.

Redis is a high-performance key-value database. The emergence of redis largely compensates for the lack of key/value storage such as memcached, and can play a very good supplementary role in relational databases in some occasions.

Redis supports master-slave synchronization. Data can be synchronized from the master server to any number of slave servers, and a slave server can be a master server associated with other slave servers.

1. Redis cluster model

In service development, a single machine will have a single point of failure problem, and the service is deployed on a server. Once the server is down, the service will not be available. Therefore, in order to make the service highly available, distributed services appear. The same service Deployed to multiple machines, even if several of the servers are down, the service is available as long as one server is available.

The same is true for redis. In order to solve the stand-alone failure, the master-slave mode is introduced, but there is a problem in the master-slave mode: after the master node fails, the service needs to be manually switched from the slave node to the master node before the service is restored. To solve this problem, redis introduces the sentinel mode. The sentinel mode can automatically promote the slave node to the master node after the master node fails, and the service can be restored without manual intervention.

However, neither the master-slave mode nor the sentinel mode has achieved real data sharding storage. Each redis instance stores the full amount of data, so redis cluster was born, realizing real data sharding storage. However, since redis cluster was released relatively late (the official version was released only in 2015), major manufacturers couldn't wait, and successively developed their own redis data fragmentation cluster models, such as: Twemproxy, Codis, etc.

1.1. Master-slave mode

Although a redis single node can persist data to the hard disk through the RDB and AOF persistence mechanisms, the data is stored on a server. If the server has a problem such as a hard disk failure, the data will be unavailable and cannot be read or written. Separation, reading and writing are all on the same server, and I/O bottlenecks will occur when the request volume is large.

In order to avoid single point of failure and non-separation of reading and writing, Redis provides the replication function to realize that after the data in the master database is updated, the updated data will be automatically synchronized to other slave databases.

The characteristics of the master-slave structure of redis above: a master can have multiple salve nodes; a salve node can have slave nodes, and the slave nodes are in a cascade structure.

1.1.1 Advantages and disadvantages of master-slave mode

Advantages: The master-slave structure has the advantages of read-write separation, improved efficiency, data backup, and multiple copies.
Disadvantages: The biggest disadvantage is that the master-slave mode does not have automatic fault tolerance and recovery functions. If the master node fails, the cluster cannot work, and the availability is relatively low. Manual intervention is required to upgrade the slave node to the master node.

Ordinary master-slave mode, when the master database crashes, you need to manually switch the slave database to become the master database:

Use commands in the slave database SLAVE NO ONEto promote the slave database to master data to continue serving.
Start the previously crashed master database, and then use the SLAVEOF command to set it as the slave database of the new master database to synchronize data.

1.2. Sentry mode

The first type of master-slave synchronization/replication mode, when the master server is down, you need to manually switch a slave server to the master server, which requires manual intervention, which is laborious and laborious, and will cause the service to be unavailable for a period of time. That's where Sentry Mode comes in.

The sentinel mode was provided from Redis version 2.6, but the mode of this version was unstable at that time, and the sentinel mode was not stabilized until Redis version 2.8.

The core of the sentinel mode is still master-slave replication, but compared with the master-slave mode, when the master node is down and cannot be written, there is an additional election mechanism: a new master node is elected from all slave nodes. The implementation of the election mechanism depends on starting a sentinel process in the system.

As shown in the figure above, Sentry itself has a single point of failure problem, so in a Redis system with one master and multiple slaves, multiple Sentinels can be used for monitoring. Sentinels will not only monitor the master database and slave databases, but also monitor each other. Each sentinel is an independent process, and as a process, it will run independently.

1.2.1 The role of sentinel mode:

Monitor whether all servers are running normally: Send commands to return the running status of the monitoring server, process and monitor the master server, slave server, and sentinels also monitor each other.

Failover: When the sentinel detects that the master is down, it will automatically switch the slave to the master, and then notify other slave servers through the publish-subscribe mode, modify the configuration file, and let them switch to the master. At the same time, the problematic old master will also become a slave of the new master, that is to say, even if the old master is restored, it will not restore its original master status, but will become a slave of the new master.

1.2.2 Implementation Principle of Sentry

When Sentinel starts the process, it will read the content of the configuration file, and find out the main database to be monitored through the following configuration:

sentinel monitor master-name ip port quorum
#master-name是主数据库的名字
#ip和port 是当前主数据库地址和端口号
#quorum表示在执行故障切换操作前，需要多少哨兵节点同意。

The reason why only the master node needs to be connected here is because the info command of the master node is used to obtain the slave node information, thereby establishing a connection with the slave node, and at the same time knowing the information of the newly added slave node through the info information of the master node.

A sentinel node can monitor multiple master nodes, but this is not recommended because when the sentinel node crashes, multiple cluster switchovers will fail at the same time. After Sentinel starts, two connections are made to the main database.

Subscribe to the master database _sentinel_:hellochannel to get information about sentinels that also monitor that database
Periodically send the info command to the master database to obtain information about the master database itself.

After establishing a connection with the main database, the following three operations will be performed regularly:

(1) Send info commands to master and slave every 10s. The function is to obtain the current database information. For example, when a new slave node is found, a connection will be established and added to the monitoring list. When the role of the master-slave database changes, the information will be updated.

_sentinel_:hello(2) Send your own information to the channel of the master data and the slave database every 2s . The role is to share your own monitoring data with Sentry. Each sentinel will subscribe to _sentinel:hellothe channel of the database. When other sentinels receive the message, they will judge whether the sentinel is a new sentinel. If so, they will be added to the sentinel list and a connection will be established.

(3) Send ping commands to all master-slave nodes and all sentinel nodes every 1s to monitor whether the nodes are alive.

1.2.3 Subjective offline and objective offline

When the sentinel node sends a ping command, if the node does not reply after a certain period of time ( down-after-millisecond), the sentinel considers that it is subjectively offline. Subjective offline means that the current sentinel believes that the node is already down. If the node is the main database, the sentinel will further judge whether it needs to fail over it. At this time, it will send a command ( ) to ask other sentinel SENTINEL is-master-down-by-addrnodes Subjective offline, when the specified number (quorum) is reached, Sentry will consider it objective offline.

When the master node objectively goes offline, a master-slave switchover is required. The steps of the master-slave switchover are:

Elect the lead sentinel.
All slaves of the leader sentinel select the slave database with the highest priority. Priority can be slave-priorityset via options.
If the priority is the same, the greater the offset from the copied command (that is, the more data is copied and synchronized, the newer the data), the higher the priority.
If the above conditions are the same, select the slave database with the smaller run ID.

After a slave database is selected, the sentinel sends slave no onea command to upgrade to the master database, and sends a slaveof command to set the master database of other slave nodes as the new master database.

1.2.4 Advantages and Disadvantages of Sentry Mode

1. Advantages

The sentinel mode is based on the master-slave mode, which solves the problem that the master failure in the master-slave mode cannot automatically switch the fault.

2. Insufficiency - problem

It is a centralized cluster implementation scheme: there is always only one Redis host to receive and process write requests, and write operations are affected by the bottleneck of a single machine.
All nodes in the cluster store the full amount of data, which wastes memory space and does not truly realize distributed storage. When the amount of data is too large, master-slave synchronization seriously affects the performance of the master.
After the Redis host is down, the sentinel mode is not in the case of voting, because no one knows who the host and slave are before the end of the voting. At this time, Redis will also open the protection mechanism to prohibit writing operations until the election New Redis host.

The data stored by each node in the master-slave mode or the sentinel mode is a full amount of data. When the amount of data is too large, the stored data needs to be segmented and stored in multiple redis instances. At this time, Redis Sharding technology will be used.

1.3. Redis cluster solutions of major manufacturers

Redis only supports single-instance mode before version 3.0. Although Redis developer Antirez proposed to add the cluster function in Redis version 3.0 as early as on his blog, version 3.0 did not release the official version until 2015. Major enterprises can't wait any longer. Before the release of version 3.0, in order to solve the storage bottleneck of Redis, they launched their own Redis cluster solutions one after another. The core idea of these solutions is to store data sharding (sharding) in multiple Redis instances, and each shard is a Redis instance.

1.3.1 Client Fragmentation

Client-side sharding is implemented by putting the logic of sharding on the Redis client (for example: jedis already supports the Redis Sharding function, that is, ShardedJedis), through the pre-defined routing rules of the Redis client (using consistent hashing), the The access to Key is forwarded to different Redis instances, and the returned results are collected when querying data. The schema for this scenario is shown in the figure.

Pros and cons of client-side sharding:

Advantages: The advantage of client-side sharding technology using the hash consensus algorithm for sharding is that all logic is controllable and does not depend on third-party distributed middleware. The Redis instances on the server side are independent of each other and not related to each other. Each Redis instance runs like a single server. It is very easy to expand linearly and the system is very flexible. Developers know how to implement sharding and routing rules, so they don't have to worry about stepping into the trap.

1. Consistent hash algorithm:

It is a commonly used algorithm in distributed systems. For example, a distributed storage system needs to store data on a specific node. If a common hash method is used to map the data to a specific node, such as mod(key,d), the key is the key of the data, and d It is the number of machine nodes. If a machine joins or exits the cluster, all data mappings will be invalid.

The consistent hash algorithm solves the problem of poor scalability of the ordinary remainder Hash algorithm, and can ensure that as many requests as possible hit the original routed server when the server is online or offline.

2. Implementation method: consistent hash algorithm, such as MURMUR_HASH hash algorithm, ketamahash algorithm

For example, the Redis Sharding implementation of Jedis uses a consistent hashing algorithm (consistent hashing) to hash the key and node name at the same time, and then perform mapping matching. The algorithm used is MURMUR_HASH.

The main reason for using consistent hashing instead of simple hash-like modulo mapping is that when nodes are added or removed, there will be no rehashing due to re-matching. Consistent hashing only affects the key distribution of adjacent nodes, and the impact is small.

insufficient:

This is a static sharding scheme, which needs to increase or decrease the number of Redis instances, and manually adjust the sharding program.
The operation and maintenance costs are relatively high. Any problem with the cluster data requires the cooperation of operation and maintenance personnel and developers, which slows down the speed of problem solving and increases the cost of cross-departmental communication.
In different client programs, the cost of maintaining the same route sharding logic is huge. For example, the java project and the PHP project share a set of Redis clusters, and the routing fragmentation logic needs to write two sets of the same logic, and there will be two sets of maintenance in the future.

One of the biggest problems with client sharding is that when the topological structure of the server-side Redis instance group changes, each client needs to be updated and adjusted. If the client fragmentation module can be taken out separately to form a separate module (middleware), this problem can be solved as a bridge connecting the client and the server, and proxy fragmentation will appear at this time.

1.3.2 Proxy Fragmentation

Twemproxy is the most widely used redis agent fragmentation, which is an open-source Redis agent by Twitter. Its basic principle is: through the form of middleware, the Redis client sends the request to Twemproxy, and Twemproxy sends it to the correct Redis instance according to the routing rules. Finally, Twemproxy aggregates the results back to the client.

Twemproxy introduces a proxy layer to manage multiple Redis instances in a unified manner, so that the Redis client only needs to operate on Twemproxy, and does not need to care about how many Redis instances there are behind, thus realizing the Redis cluster.

Advantages of Twemproxy:

The client connects to Twemproxy like a Redis instance, without changing any code logic.
Automatic deletion of invalid Redis instances is supported.
Twemproxy maintains a connection with the Redis instance, reducing the number of connections between the client and the Redis instance.

Disadvantages of Twemproxy:

Since each request from the Redis client goes through the Twemproxy proxy to reach the Redis server, there will be a performance loss in this process.
There is no friendly monitoring and management background interface, which is not conducive to operation and maintenance monitoring.
The biggest pain point of Twemproxy is that it cannot expand/shrink smoothly. For operation and maintenance personnel, the workload is very heavy when adding Redis instances due to business needs.

Twemproxy, as the most widely used, proven and stable Redis proxy, is widely used in the industry.

1.3.3 Codes

The problem that Twemproxy cannot smoothly increase Redis instances has brought great inconvenience, so Peapod independently developed Codis, a Redis proxy software that supports smooth increase of Redis instances, which was developed based on Go and C languages, and was launched in November 2014. Open source on GitHub.

In the architecture diagram of Codis, Codis is introduced Redis Server Group, which realizes the high availability of Redis cluster by specifying a master CodisRedis and one or more slave CodisRedis. When a primary CodisRedis hangs up, Codis will not automatically promote a secondary CodisRedis to the primary CodisRedis, which involves data consistency issues (Redis itself uses master-slave asynchronous replication for data synchronization, when data is successfully written to the primary CodisRedis , there is no guarantee whether the slave CodisRedis has read in this data), the administrator needs to manually promote the slave CodisRedis to the master CodisRedis on the management interface.

If manual processing is troublesome, pea pods also provide a tool Codis-hathat will take it offline and promote a secondary CodisRedis to master CodisRedis when it detects that the primary CodisRedis is down.

Codis adopts the form of pre-sharding. When it is started, 1024 slots are created. One slot is equivalent to one box. Each box has a fixed number ranging from 1 to 1024. The slot box is used to store the Key. As for which box the Key is stored in, a number can be obtained through the algorithm " crc32(key)%1024". The range of this number must be between 1 and 1024, and the Key is placed in the slot corresponding to this number.

For example, if crc32(key)%1024the number obtained by a Key through the algorithm " " is 5, put it in the slot (box) coded 5. Only one slot can be placed in one slot Redis Server Group, and one slot cannot be placed Redis Server Groupin multiple slots. A minimum of 1 Redis Server Groupslot can be stored in one slot, and a maximum of 1024 slots can be stored. Therefore, a maximum of 1024 can be specified in Codis Redis Server Group.

The biggest advantage of Codis is that it supports smooth increase (decrease) Redis Server Group(Redis instance), and can migrate data safely and transparently. This is also where Codis is different from static distributed Redis solutions such as Twemproxy. After the addition of Codis Redis Server Group, it involves the migration of slots.

For example, there are two systems Redis Server Group, Redis Server Groupand the corresponding relationship with the slot is as follows.

When a Redis Server Group is added, the slot will be reassigned. There are two ways for Codis to allocate slots:

The first method: manually redistribute through the Codis management tool Codisconfig, and specify the range of slots corresponding to each Redis Server Group. For example, you can specify the new correspondence between Redis Server Group and slots as follows.

The second method: through the rebalance function of the Codis management tool Codisconfig, the slots will be automatically migrated according to the memory of each Redis Server Group to achieve data balance.

1.4、Redis Cluster

Although the sentinel mode of Redis can achieve high availability and read-write separation, there are several deficiencies:

In sentinel mode, each Redis server stores the same data, which is a waste of memory space; the amount of data is too large, and the master-slave synchronization seriously affects the performance of the master.
Sentinel mode is a centralized cluster implementation scheme, each slave machine is highly coupled to the host machine, and the service is unavailable during the period from the master downtime to the recovery of the slave election master.
Sentinel mode always has only one Redis host to receive and process write requests. Write operations are still affected by the bottleneck of a single machine, and a true distributed architecture has not been implemented.

Redis added the Cluster cluster mode on 3.0 to realize the distributed storage of Redis, that is to say, different data is stored on each Redis node. In order to solve the problem of limited Redis capacity of a single machine, the cluster mode distributes data to multiple machines according to certain rules. The memory/QPS is not limited to a single machine, and can benefit from the high scalability of distributed clusters.

Redis Cluster is a server sharding technology (sharding and routing are implemented on the server side), using multiple masters and multiple slaves. Each partition is composed of a Redis host and multiple slaves. In parallel. The Redis Cluster cluster adopts the P2P model and is completely decentralized.

As shown in the figure above, it is officially recommended that at least 3 master nodes are required for cluster deployment, and it is best to use a mode of 3 masters and 3 slaves with six nodes. The Redis Cluster cluster has the following characteristics:

The cluster is completely decentralized and adopts multiple masters and multiple slaves; all redis nodes are interconnected with each other (PING-PONG mechanism), and the binary protocol is used internally to optimize transmission speed and bandwidth.
The client is directly connected to the Redis node without an intermediate proxy layer. The client does not need to connect to all the nodes in the cluster, but can connect to any available node in the cluster.
Each partition is composed of a Redis host and multiple slaves, and the shards and shards are parallel to each other.
Each master node is responsible for maintaining a part of the slots and the key-value data mapped to the slots; each node in the cluster has a full amount of slot information, and through the slots, each node knows which node the specific data is stored on.

Redis cluster is mainly for massive data + high concurrency + high availability scenarios, massive data, if you have a large amount of data, then it is recommended to use redis cluster, when the amount of data is not large, using sentinel is enough. The performance and high availability of redis cluster are better than sentinel mode.

Redis Cluster adopts virtual hash slot partitioning instead of consistent hash algorithm, and pre-allocates some card slots. All keys are mapped to these slots according to the hash function. The master node in each partition is responsible for maintaining a part of the slots and the slots mapped. key-value data.

2. RDB persistence

2.1 RDB full write

2.1.1 Principle of RDB

RDBPersistence is the process of generating a snapshot of the current process data and saving it to the hard disk. RDBThe process of triggering persistence is divided into manual triggering and automatic triggering

process:

1) Execute the bgsave command, and the Redis parent process judges whether there are currently executing child processes, such as RDB/AOF child processes, and returns directly if there is a bgsave command.
2) The parent process executes the fork operation to create a child process. During the fork operation, the parent process will be blocked. View the latest_fork_usec option through the info stats command to obtain the time-consuming of the latest fork operation, in microseconds.
3) After the parent process fork is completed, the bgsave command returns the message "Background saving started" and no longer blocks the parent process, and can continue to respond to other commands.
4) The child process creates an RDB file, generates a temporary snapshot file according to the memory of the parent process, and executes the lastsave command to atomically replace the original file after completion, to obtain the last RDB generation time, which corresponds to the rdb_last_save_time option of info statistics.
5) The process sends a signal to the parent process to indicate completion, and the parent process updates the statistical information. For details, see the rdb_* related options under info Persistence.

2.1.2 rdb mode

SAVE is blocking RDB persistence. When this command is executed, the main process of rdis writes the database state in memory to the rdb file. Redis cannot process any command requests until the file is created.

BGSAVE non-blocking persistence, it will create a child process to write the database state in the memory into the RDB file, and the main process can also process requests from the client, but the child process basically copies the parent process, which It is equal to two redis processes of the same size running on the system, which will cause a substantial increase in memory usage.

2.1.3 rdb trigger situation

1. Manually execute the bgsave or save command 2. Automatically trigger according to the save option of the configuration file 3. In the master-slave structure, the slave node performs a full copy operation, and the master node executes it automatically, and sends the generated RDB file to the slave 4. Execute debug reload When the command reloads Redis 5. By default, when the shutdown command is executed to close redis, if the AOF persistence function is not enabled, it will be executed automatically

2.1.4 rdb advantages and disadvantages

Advantage:

Once this method is adopted, your entire Redis database will only contain one file, and this pair can regularly back up an entire data file every day.
For disaster recovery, RDB is a very good choice. Because we can easily compress a single file and then transfer it to other storage media.
Compared with the AOF mechanism, if the data set is large, the startup efficiency of RDB will be higher.

Disadvantages:

Since RDB assists in the completion of data persistence through fork child processes, if the data set is large, it may cause the entire server to stop serving for hundreds of milliseconds, or even 1 second.
RDB files are saved in a specific binary format. During the evolution of the Redis version, there are RDB versions in multiple formats. There is a problem that the old version of Redis service cannot be compatible with the new version of the RDB format.

2.1.5 rdb file configuration

redis.conf file

#注释所有save行则停止rdb持久化

#900秒（15分钟）内至少1个key值改变（则进行数据库保存--持久化）

save 900 1

#300秒（5分钟）内至少10个key值改变（则进行数据库保存--持久化）

save 300 10
#60秒（1分钟）内至少10000个key值改变（则进行数据库保存--持久化）

save 60 10000
#当RDB持久化出现错误后，再写入数据会报错，用于提示用户出问题了。

#yes是开启，no是关闭，默认开启

stop-writes-on-bgsave-error yes
#是否压缩rdb文件，rdb文件压缩使用LZF压缩算法，压缩会消耗一些cpu，不压缩文件会很大

#yes开启，no关闭，默认开启

rdbcompression yes
#使用CRC64算法来进行数据校验，防止RDB是错误的，但是这样做会增加大约10%的性能消耗

#yes开启，no关闭，默认开启

rdbchecksum yes

copy

2.1.6 rdb command configuration

Block the current Redis server until the RDB process is completed. For instances with relatively large memory, it will cause long-term blocking, and it is not recommended to use it in an online environment.save

The Redis process executes the fork operation to create a child process. The RDB persistence process is in charge of the child process and ends automatically after completion. Blocking only occurs during the fork phase, generally for a short time.bgsave

View the latest_fork_usec option to get the time-consuming of the latest fork operation in microseconds.info stats

2.1.7 rdb data recovery

1. Put the RDB backup in the data directory specified in the configuration file, and start redis and it will be automatically restored. It will block during loading and no other operations can be performed.

2. The above method does not work, or the restored cluster can be restored using the redis-migrate-tool tool.

2.2 AOF persistence: incremental writing

2.2.1 Aof principle

Record each write command in an independent log, and re-execute the command in the AOF file to restore data when restarting. The main function of AOF is to solve the real-time nature of data persistence, which is currently the mainstream method of Redis persistence.

The written data is readable. When synchronizing, it is first written into the buffer and then put into the hard disk. If writing directly to the hard disk, the performance will depend on the disk load, and put it in the buffer, which can provide various synchronization strategies.

process:

1) All write commands will be appended to aof_buf (buffer).
2) The AOF buffer is synchronized to the hard disk according to the corresponding strategy.
3) As the AOF file becomes larger and larger, it is necessary to rewrite the AOF file regularly to achieve the purpose of compression.
4) When the Redis server restarts, the AOF file can be loaded for data recovery.

2.2.2 aof trigger situation

1. Automatically trigger according to the configuration file

2.2.3 aof advantages and disadvantages

Advantage:

This mechanism can bring higher data security , that is, data persistence. Depending on the policy, data security is different, and one can be selected in the performance and security areas.
Since this mechanism adopts the append mode for the writing operation of the log file, even if there is a downtime during the writing process, the existing content in the log file will not be destroyed.
If the log is too large, the rewrite mechanism will be automatically enabled. In the append mode, the modified data is continuously written to the old disk file, and a new file is created to record which modification commands are executed during this period to ensure security.
AOF contains a clearly formatted, easy-to-understand log file for recording all modification operations. In fact, we can also complete data reconstruction through this file.

Disadvantages:

For the same amount of data sets, AOF files are usually larger than RDB files. RDB restores large datasets faster than AOF.
According to different synchronization strategies, AOF is often slower than RDB in terms of operating efficiency. In short, the efficiency of the synchronization strategy per second is relatively high, and the efficiency of the synchronization disable strategy is as efficient as RDB.

2.2.4 aof file configuration

There are three synchronization methods in the Redis configuration file, they are:

#是否开启aof持久化。默认no，要打开

appendonly yes

#位置

appendfilename "appendonly.aof"

#每次有数据修改发生时都会写入AOF文件

#命令写入aof_buf后调用系统fsync操作同步AOF文件，fsync完成后线程返回

appendfsync always

#每秒钟同步一次，该策略为AOF的缺省策略

#命令写入aof_buf后调用系统write操作，write完成后线程返回。fsync同步文件操作由专门线程每秒调用一次

#这个模式兼顾了效率的同时也保证了数据的完整性，即使在服务器宕机也只会丢失一秒内对redis数据库做的修改

appendfsync everysec

#不加入缓冲区，直接写到硬盘，速度最快，不安全

#命令写入aof_buf后调用系统write操作，不对aof文件做fsync同步，同步硬盘操作由操作系统负责，通常同步周期最长30秒

#这种模式下效率是最快的，但对数据来说也是最不安全的，如果redis里的数据都是从后台数据库如mysql中取出来的，属于随时可以找回或者不重要的数据，那么可以考虑设置成这种模式。

appendfsync no

copy

2.2.5 aof command configuration

aof file rewrite trigger manuallybgrewriteaof

Aof file rewrite is automatically triggered, configuration file

#新的aof文件大小是上次的aof文件的大小2倍（100）时，进行重写

auto-aof-rewrite-percentage 100

#表示运行AOF重写时文件最小体积， 默认为64MB

auto-aof-rewrite-min-size 64mb

copy

2.2.6 aof data recovery

Put the AOF backup in the data directory specified in the configuration file, and it will be automatically restored when redis is started. It will block during loading and no other operations can be performed.
If the above method does not work, or the restored cluster can be restored using the redis-migrate-tool tool.
You can use the pipline method to batch hard write, but the efficiency will be low

2.3.RDB&AOF comparison

The standard for choosing between the two is to see whether the system is willing to sacrifice some performance in exchange for higher cache consistency (aof), or is willing to disable backup in exchange for higher performance when writing operations are frequent, and wait until the save is run manually , and then make a backup (rdb).

Start the loading process:

When AOF persistence is enabled and AOF files exist, AOF files are loaded first
When AOF is closed or the AOF file does not exist, load the RDB file
After loading the AOF/RDB file successfully, Redis starts successfully
When there is an error in the AOF/RDB file, Redis fails to start and prints an error message

3. Redis high-level

3.1 Redis high-level cache breakdown

3.1.1 Definition of cache breakdown

缓存击穿, means 缓存中没有但数据库中有的数据, and 某一个key非常热点, while constantly carrying the large concurrency, the large concurrency is concentrated on accessing this point. When the key expires (usually when the cache time expires), the continuous large concurrency will break through the cache. Requesting the database directly is like cutting a hole in a barrier.

Cache breakdown looks a bit similar, but the difference between the two is that cache puncture means that the database is under too much pressure or even goes down, while cache breakdown is just a large number of concurrent requests to the DB database level. It can be considered that the breakdown is a subset of the cached Xueben. Some articles think that the difference between the two is that the breakdown is for a certain hot key cache, while Xueben is for many keys.

3.1.2 Causes of cache breakdown

It can be regarded as a special subset of cache avalanche.

For example, xxx house collapse and xxx product activities. At this time, a large number of users are accessing this hot event, but it may be better than some reason. The hot key of redis has expired, so at this time, a large number of high-concurrency requests for this key have to be If there is no response from redis, the request will be directly sent to the DB server, causing the entire DB to be paralyzed.

3.1.3 Cache breakdown solution

Use a mutex scheme . When the cache fails, instead of loading the db data immediately, use some atomic operation commands with successful returns, such as (Redis's setnx) to operate, and then load the db database data and set the cache when it succeeds. Otherwise, retry to get the cache.
"Never expires" means that no expiration time is set, but when the hotspot data is about to expire, the asynchronous thread updates and sets the expiration time.

3.1.4 Detailed Explanation of Mutex in Cache Breakdown Solution

A common practice in the industry is to use mutex. To put it simply, when the cache fails (judging that the value taken out is empty), instead of immediately loading the db, first use some operations of the cache tool with a successful operation return value (such as Redis's SETNX or Memcache ADD) to set a mutex key, when the operation returns successfully, then perform the load db operation and reset the cache; otherwise, retry the entire get cache method.

SETNX is the abbreviation of "SET if Not eXists", that is, it is only set when it does not exist, and it can be used to achieve the lock effect.

public String get(key) {
      String value = redis.get(key);
      if (value == null) { //代表缓存值过期
          //设置3min的超时，防止del操作失败的时候，下次缓存过期一直不能load db
      		if (redis.setnx(key_mutex, 1, 3 * 60) == 1) {  //代表设置成功
               value = db.get(key);
               redis.set(key, value, expire_secs);
               redis.del(key_mutex);
          } else {  //这个时候代表同时候的其他线程已经load db并回设到缓存了，这时候重试获取缓存值即可
               sleep(50);
               get(key);  //重试
              }
      } else {
          return value;      
   }
 }

3.2 Redis high-level cache penetration

3.2.1 Cache penetration definition

缓存穿透refers to query one 数据库一定不存在的数据. The normal process of using the cache is roughly that the data query is cached first, and if the key does not exist or the key has expired, then the database is queried and the queried object is put into the cache. If the database query object is empty, it will not be put into the cache.

The difference that needs to be paid attention to here 缓存击穿is cache breakdown. Cache breakdown refers to 缓存中没有但数据库中有的数据, and 某一个key非常热点, when the key is in the moment of failure (usually when the cache time expires) Period), continuous large concurrency will break through the cache and directly request the database, just like digging a hole in a barrier.

In layman's terms, when a read request is accessed, neither the cache nor the database has a certain value, which will cause every query request for this value to penetrate to the database, which is cache penetration.

3.2.2 Causes of cache penetration

Unreasonable design of the business, for example, most users do not enable guards, but each of your requests is cached to check whether a certain userid is guarded.
Business/operation and maintenance/development errors, such as cache and database data are deleted by mistake.
Hackers illegally request attacks, for example, hackers deliberately fabricate a large number of illegal requests to read non-existent business data.

3.2.3 Three solutions for cache penetration

If it is an illegal request, we verify the parameters at the API entry and filter illegal values.
If the query database is empty, we can set a null value, or a default value, for the cache. But if a write request comes in, the cache needs to be updated to ensure cache consistency. At the same time, an appropriate expiration time is finally set for the cache. (commonly used in business, simple and effective)
Use Bloom filters to quickly determine whether data exists. That is, when a query request comes in, first judge whether the value exists through the Bloom filter, and then continue to check if it exists.

Bloom filter principle: It consists of a bitmap array with an initial value of 0 and N hash functions. One performs N hash algorithms on a key to obtain N values, hashes the N values in the bit array and sets them to 1, and then checks if the specific positions are all 1, then Bloom filter The device judges that the key exists.

3.2.4 Detailed Explanation of Bloom Filter for Cache Penetration Solution

布隆过滤器Is a bit vector or bit, if we want to map a value to a Bloom filter, we use multiple different hash functions to generate multiple hash values, and set the bit bit pointed to by each generated hash value is 1, the following word baidu sets three positions as 1.

Principle: Perform k hash algorithms on a key to obtain k values, set the k values to 1 after hashing in the bit array, and then check if the specific positions are all 1, then Bloom The filter determines that the key exists.

The word "tencent", the corresponding situation

It can be seen that the bit positions corresponding to different words may be the same. When there are many words, most of the bit positions may be 1. At this time, the corresponding position of the query taobao may be 1, which only shows that the word taobao may exist. , does not necessarily exist, then 1 is covered, which is the misjudgment of the Bloom filter.如果它说不存在那肯定不存在，如果它说存在，那数据有可能实际不存在。

The bitmap of Redis only supports the size of 2^32, which corresponds to 512MB of memory, and the misjudgment rate is 1/10,000. It can store about 200 million data, with high performance and small space occupation, which saves a lot of invalid database connections. .

Therefore, we can use Bloom filters to control Redis cache penetration within an acceptable range.

Using bloom filters:
import dependencies

<dependency>
     <groupId>com.google.guava</groupId>
     <artifactId>guava</artifactId>
     <version>19.0</version>
</dependency>

code:

public class Test {

    private static int size = 1000000;//预计要插入多少数据

    private static double fpp = 0.01;//期望的误判率

    private static BloomFilter<Integer> bloomFilter = BloomFilter.create(Funnels.integerFunnel(), size, fpp);

    public static void main(String[] args) {
        //插入数据
        for (int i = 0; i < 1000000; i++) {
            bloomFilter.put(i);
        }
        int count = 0;
        for (int i = 1000000; i < 2000000; i++) {
            if (bloomFilter.mightContain(i)) {
                count++;
                System.out.println(i + "误判了");
            }
        }
        System.out.println("总共的误判数:" + count);
    }
}

application:

@Cacheable(value="key1")
public String get(String key) {
    String value = redis.get(key);  
    // redis中不存在该缓存
    if (value  == null) {
    //布隆过滤器也没有,直接返回
        if(!bloomfilter.mightContain(key)){
            return null; 
        }else{
            //布隆过滤器中能查到，不代表一定有,查出来放入redis，同样也可以避免缓存穿透
            value = db.get(key);
            redis.set(key, value); 
        }    
    }
    return value；
}

(2) Cache empty objects
When the storage layer misses, even the returned empty objects will be cached, and an expiration time will be set at the same time, and the data will be obtained from the cache when accessed later, protecting the back-end data source .

But this approach has two problems:

● If null values can be cached, it means 缓存需要更多的空间存储更多的键that there may be many keys with null values;

● Even if the expiration time is set for the null value, there will still be inconsistencies between the data in the cache layer and the storage layer for a period of time, which will affect the business that needs to maintain consistency

3.3 Redis high-level cache avalanche

3.3.1 Cache Scheduling Definition

It refers to the large amount of data in the cache until the expiration time, but the query data volume is huge, and all requests directly access the database, causing excessive pressure on the database or even down the machine.

Caching Xueben is generally caused by a large amount of data expiring at the same time. For this reason, it can be solved by setting the expiration time evenly, that is, making the expiration time relatively discrete. For example, use a larger fixed value + a smaller random value, 5 hours + 0 to 1800 seconds.
Redis downtime may also cause cache snowrunning. This requires the construction of a Redis high-availability cluster.

3.3.2 Causes of avalanches

If it is about to arrive at midnight on Double Eleven, there will be a wave of panic buying soon, and this wave of products will be put into the cache at a relatively concentrated time, assuming that the cache is for an hour. Then at one o'clock in the morning, the cache of this batch of products will expire. The access queries for this batch of commodities all fall on the database, and for the database, there will be periodic pressure peaks.

3.3.3 Four Solutions to Cache Avalanche

Use different categories of products and cache different cycles. Items in the same category, plus a random factor. In this way, the cache expiration time can be distributed as much as possible. Moreover, the cache time of products in popular categories is longer, and the cache time of products in unpopular categories is shorter, which can also save the resources of the cache service.
If the cache database is deployed in a distributed manner, evenly distribute the hotspot data in different cache databases.
Set hotspot data to never expire.
Use the locking and current limiting method.

3.3.4 Detailed Explanation of Locking and Current Limiting in Cache Avalanche Solution

Redis Common Interview Questions

What are the features of Redis?

High performance, the reading speed is 100,000 times/s, and the writing speed is 80,000 times/s;
Data persistence, support RDB, AOF;
Support transactions. Wrapped by MULTI and EXEC instructions;
Multiple data structure types;
master-slave replication;
Other features: publish/subscribe, notification, key expiration, etc.;

Why is Redis so fast?

Based entirely on memory, there is no overhead on disk IO, except for asynchronous persistence
Single thread, avoiding the performance loss of multiple thread switching
Non-blocking IO multiplexing mechanism
The underlying data storage structure is optimized, and the native data structure is used to improve performance.

What are the underlying data structures of Redis?

string. Instead of using the traditional string of C language, it implements an abstract type of simple dynamic string SDS by itself, and saves the length
information.
Linked list (linkedlist). Two-way acyclic linked list structure, each node of the linked list is represented by a listNode structure, each node has a prefix and
pointer to the next node
dictionary(hashtable). An abstract data structure that stores key-value pairs. The bottom layer uses a hash table. Each dictionary has two hash tables for normal use.
Used when using and rehash.
Jump table (skiplist). Skip list is one of the underlying implementations of sorted sets. The redis skip table consists of zskiplist and zskiplistNode,
zskiplist is used to save skip list information (header, tail node, length, etc.), zskiplistNode is used to represent table skip nodes, each skip list
The layer height is a random number from 1 to 32. In the same jump table, multiple nodes can contain the same score, but the member objects of each node must
is unique, the nodes are sorted according to the size of the score, and if the scores are the same, they are sorted according to the size of the member objects.
A set of integers (intset). The collection abstract data structure used to store integer values will not have repeated elements, and the underlying implementation is an array.
Compressed list (ziplist). A sequential data structure developed to save memory, can contain multiple nodes, and each node can hold one byte
array or integer value.

What data types does Redis support?

Five commonly used data types: String, Hash, Set, List, SortedSet.
Three special data types: Bitmap, HyperLogLog, and Geospatial. The bottom layer of Bitmap and HyperLogLog is String data type, and the bottom layer of Geospatial is Sorted Set data type;
String object string: int integer, embstr encoded simple dynamic string, raw simple dynamic string
list object list: ziplist, linkedlist
Hash object hash: ziplist, hashtable
Collection object set: intset, hashtable
Ordered collection object zset: ziplist, skiplist

Five commonly used data structures and application scenarios of Redis?

String: cache, counter, distributed lock, etc.
List: linked list, queue, Weibo follower timeline list, etc.
Hash: User information, Hash table, etc.
Set: deduplication, likes, dislikes, mutual friends, etc.
Zset: visits leaderboard, hits leaderboard, etc.

Why does Redis use single thread?
According to the official reply, the CPU will not become the bottleneck of Redis, and Redis is mainly limited by memory and network.

For example, on an average Linux system, you can pass through 1 million requests per second using pipelining, so if your application mostly uses O(N) or O(log(N)) orders, you will hardly use much CPU is an IO-intensive system.

After Redis 6.0, it switched to multi-threading.
The multi-threading of Redis mainly deals with data reading and writing and protocol analysis. Execution commands are still executed in a single-threaded order.
The main reason is that the performance bottleneck of redis lies in the network IO rather than the CPU. Multi-threading is used to perform some peripheral preprocessing, which improves the read and write efficiency of IO, thereby improving the overall throughput. When antirez shared in RedisConf 2019, he mentioned that the multi-threaded IO introduced by Redis 6 has at least doubled the performance.

What are the deletion strategies for Redis expired keys?
There are 3 expiration deletion strategies. Lazy delete, regular delete, timed delete

Lazy deletion : Check only when the key is used, and delete it if it has expired. Disadvantage: If the expired key is not accessed, it cannot be deleted and occupies memory all the time, resulting in a waste of space.
Periodic deletion : check every once in a while, delete expired keys, and just randomly select some keys to check each time.
Timed deletion : set the expiration time for each key and create a timer at the same time. Deletion is performed immediately upon expiration. Disadvantage: If there are many expired keys, it will take up more CPU and have a great impact on the performance of the service.

If the memory space of Redis is insufficient, what is the elimination mechanism?

volatile-lru: Remove the least recently used key from the key with an expiration time set for elimination
allkeys-lru: When the memory is not enough to accommodate the newly written data, in the key space, remove the least recently used key (this is the most commonly used)
volatile-ttl: Remove the key that is about to expire from the key that has an expiration time set
volatile-random: Randomly select keys to be eliminated from the keys whose expiration time has been set
allkeys-random: Randomly select keys from keys for elimination
no-eviction: Disable data eviction. When the memory reaches the threshold, the new write operation reports an error
volatile-lfu: Select the least frequently used data from the data set (server.db[i].expires) with an expiration time set (LFU(Least
Frequently Used) algorithm, that is, the most frequently accessed data is most likely to be accessed in the future)
allkeys-lfu: When the memory is not enough to accommodate the newly written data, in the key space, remove the least frequently used key.

What to do if Redis suddenly hangs up?

From the perspective of system availability, Redis Cluster introduces a master-standby mechanism. When the master node hangs up, it will automatically switch to the backup node to continue to provide services.
The client side introduces a local cache and switches it to prevent Redis from suddenly hanging up and high concurrent traffic from hanging up the database.

What are the methods of Redis persistence?

Snapshot RDBs. Save the database state at a certain point in time to an RDB file, which is a compressed binary file and saved on disk. Can be used to recover data when Redis crashes. Generate RDB files by SAVE or BGSAVE.
1. SAVE: The redis process will be blocked until the RDB file is created. During the process blocking period, redis cannot process any command requests.
2. BGSAVE: A child process will be forked, and the child process will be responsible for generating the RDB file. The parent process can continue to process command requests without blocking the process.
Only append the file AOF. Log every write operation (not read operation). When different nodes synchronize data, read the content of the log file and execute the write command from front to back once to complete data recovery.

Redis Common Scenarios

Cache, there is a saying that is good, "The performance is not enough, the cache can make it up"
Distributed locks, using setnx of Redis
distributed session
counter, via the incr command
Leaderboard, an ordered collection of Redis
other

Seven classic problems that Redis cache should pay attention to?
Listed billion-level systems, what problems may Redis cache encounter under high traffic conditions? and corresponding solutions.

cache set invalidation
cache penetration
cache avalanche
cache hotspot
Cache big key
Cache Data Consistency
Warming up data concurrency competition

What are the Redis cluster solutions?

Master-slave replication mode
Sentinel mode
Redis Cluster mode

Redis master-slave data synchronization (master-slave replication) process?

After the slave starts, send the sync command to the master
After the master receives the sync, it executes bgsave to save the snapshot and generate the full RDB file
The master records the write command of the slave to the cache
After bgsave is executed, send the RDB file to the slave, and the slave executes
The master sends the write command of the buffer to the slave, and the slave receives and executes the command to complete the replication initialization.
After that, every time the master executes a write command, it will be sent to the slave synchronously to maintain the data consistency between the master and the slave.

Advantages and disadvantages of Redis master-slave replication?

1. Advantages: the master can automatically synchronize data to the slave, and can perform read-write separation to share the read pressure of the master. The synchronization between the master and the slave is performed in a non-blocking manner. During the synchronization, the client can still submit queries or update request
Disadvantages: It does not have automatic fault tolerance and recovery functions. After the master node is down, you need to manually specify a new master. If the master is down, if the data is not synchronized before the downtime, there will be data inconsistency after switching IP, and it is difficult to support online Expansion, the capacity of Redis is limited by the stand-alone configuration

Advantages and disadvantages of Redis Sentinel (sentinel) mode?
Sentinel mode is based on the master-slave replication mode, adding sentinels to monitor and automatically handle failures.

Advantages: The sentinel mode is based on the master-slave replication mode, so the master-slave replication mode has some advantages. The sentinel mode also has the master to hang up and can automatically switch, and the system has higher availability.
Disadvantages: The capacity of Redis is limited by the stand-alone configuration, which requires additional resources to start the sentinel process

Advantages and disadvantages of Redis Cluster mode?
The distributed storage of Redis is realized, that is, each node stores different content to solve the problem of online expansion.

advantage:
1. Centralized architecture, data is distributed across multiple nodes according to slots
2. Each node in the cluster is equal, and each node saves its own data and the state of the entire cluster.
3. Each node is connected to all other nodes, and these connections are kept active, which ensures that we only need to connect to any node in the cluster to obtain data from other nodes.
4. It can be linearly expanded to more than 1000 nodes, and nodes can be added or deleted dynamically.
5. It can realize automatic failover, exchange status information between nodes through gossip protocol, and use voting mechanism to complete the role conversion from slave to master.
shortcoming:
1. Data is replicated asynchronously, which does not guarantee strong data consistency.
2. The slave acts as a "cold standby" and does not provide external read and write services, but is only used for failover.
3. Batch operation restrictions, currently only supports batch operations for keys with the same slot value, and is not friendly to operations such as mset, mget, and sunion
4. The support for key transaction operations is limited. Only the transaction operations of multiple keys on the same node are supported. When multiple keys are distributed on different nodes, the transaction function cannot be used.
5. Multiple database spaces are not supported, one redis can support 16 dbs, and only one can be used in cluster mode, namely db0.
6. In Redis Cluster mode, it is not recommended to use pipeline and multi-keys operations to reduce the scenarios generated by max redirect.

How does Redis scale up?
In order to avoid data migration failure, consistent hashing is usually used to achieve dynamic expansion and contraction, effectively reducing the number of keys that need to be migrated.
However, in the Cluster mode, fixed slots (16384) are used to calculate the CRC16 value for each key, and then take the modulus of 16384, and then find the target machine according to the slot value. When expanding, we only need to migrate a part of the slots to the new node That's it.

Redis cluster principle?
A redis cluster is composed of multiple node nodes, and multiple nodes are connected through the cluster meet command to form a cluster.
Data is stored in the form of sharding, and the entire cluster is divided into 16384 slots, and each node is responsible for a part of the slots. The information of the entire slot will be synchronized to all nodes.
The mapping relationship between the key and the slot: the key value is paired with the key, and the CRC16 calculation is performed to calculate a 16-bit value, and the 16-bit value is modulo 16384, and the number from 0 to 16383 is obtained to indicate the hash slot corresponding to the key

How does Redis achieve high availability?
Sentinel mechanism. It has functions such as automatic failover, cluster monitoring, and message notification.
Sentinel can monitor all master and slave servers at the same time. When a master goes offline, it will automatically promote the corresponding slave to master, and then the new master will provide external services.

What are Redis transactions?
A Redis transaction is a collection of commands, packages multiple commands, then adds these commands to the queue in order, and executes these commands in order.
There is no concept of transaction isolation level in Redis transaction like Mysql relational database transaction, it cannot guarantee atomic operation, and there is no rollback operation like Mysql that fails to execute a transaction

Redis transaction execution process?
The transaction mechanism is realized through MULTI, EXEC, WATCH and other commands. The transaction execution process executes a series of multiple commands in order at one time. During the execution, the transaction will not be interrupted, nor will it execute other requests from the client until all The command is executed.
Specific process: the server receives the request from the client, and the transaction starts with MULTI. If it is in the transaction state, it will put the subsequent command into the queue and return it to the client QUEUED. Otherwise, it will directly execute this command when it receives the EXEC command from the client
. Only then will the commands in the queue be taken out and executed sequentially. After execution, the current state will be changed from the transaction state to the non-transaction state.
If the DISCARD command is received, the execution of the commands in the queue will be abandoned, which can be understood as a Mysql rollback operation, and Change the current state from transaction state to non-transaction state
WATCH monitors a key, this command can only be executed before the MULTI command. If the monitored key is modified by other clients, EXEC will give up executing all commands in the queue. UNWATCH Unwatches a key that was previously watched by the WATCH command. The key monitored before executing the EXEC and DISCARD commands will also be unmonitored.

What is the difference between Redis and Guava and Caffeine?
Cache is divided into local cache and distributed cache.

Caffeine, Guava, belong to local cache, features:
1. Direct access to memory, fast speed, limited by memory, unable to store large data.
2. No network communication overhead, higher performance.
3. Only supports local application process access, and the cost of synchronously updating the local cache data of all nodes is high.
4. If the application process is restarted, the data will be lost.
5. Therefore, the local cache is suitable for storing some high-hot data that is not easy to change or changes infrequently.
Redis belongs to distributed cache, features:
1. Cluster mode, supporting large data volume storage
2. Centralized data storage to ensure data consistency
3. The data is transferred across the network, and the performance is lower than that of the local cache. But in the same computer room, it takes 500 microseconds for a request to run back and forth between two servers.
4. Taking advantage of its advantages, this loss is completely negligible, which is why distributed caches are popular.
5. Supports the copy mechanism, which effectively guarantees high availability.

How to implement a distributed lock?

Database table, poor performance
Use Lua script (contains SETNX + EXPIRE two instructions)
SET extended command (SET key value [EX][PX] [NX|XX])
Redlock framework
The Zookeeper Curator framework provides out-of-the-box distributed