Become a Redis master in 5 minutes

Introduction to Redis

Redis is an open source high-performance key-value pair memory database developed in C language. It can be used for database, cache, message middleware and other scenarios. It is a NoSQL (not-only sql, non-relational database) database.

Redis features

Excellent performance, data is stored in memory, read and write speeds are very fast, and can support concurrent 10W QPS.

Single thread and single process, thread-safe, using IO multiplexing
Can be used as a distributed lock
Supports ten data types
Support data persistence

It can be used as message middleware and supports message publishing and subscription.

type of data

The following table lists the characteristics of the five commonly used data types and their usage scenarios:

cache

Data caching is the most important scenario of Redis. It is born for caching. In springboot, there are generally two ways to use it:

Use directly through RedisTemplate
Integrate Redis through Spring Cache (that is, annotation method)

Problems encountered when using cache

(1) Data consistency

In a distributed environment, caches and databases are prone to data consistency problems. If the project requires strong consistency for the cache, then do not use the cache.

We can only use strategies in the project to reduce the probability of cache and database consistency, but cannot guarantee strong consistency between the two. General strategies include a cache update mechanism, updating the cache in a timely manner after updating the database, and adding a retry mechanism when the cache fails.

(2) Cache avalanche

Before understanding the snow crash, we first understand what the cache avalanche phenomenon is. Assume that system A needs to process 5,000 requests per second, but the database can only process 4,000 requests per second. One day, the cache machine goes down and hangs. At this time, all the requests fall on the database at once. The database will definitely not be able to handle it, and the alarm will hang up. If no caching facilities are used at this time, and the database is in a hurry to use it, restart the database. The restart is just completed (it may not be completed). ), the request comes in again, and the database hangs up immediately.

This is an avalanche event, which is one of the most fatal problems in Redis cache (one is penetration). You can take a look at the picture below:

Don’t be anxious or panic after an avalanche event occurs. We can think about solutions from three aspects before, during and after the accident:

Before the accident: redis high availability solution, master-slave + sentinel, cluster solution to avoid total collapse;
In the event of an accident: less pressure on the database, local Ehcache caching + current limiting and downgrading to avoid exceeding the pressure on the database;
After the accident: Make Redis persistent. Once Redis is restarted, data can be quickly restored from the disk.

Let's take a look at the data process after the transformation. Assume that user A sends a request. The system first requests whether the local Ehcache has data. If there is no data, it will then go to Redis to request the data. If there is no data to the database, it will synchronize to Ehcache and Ehcache after obtaining the data. redis.

The role of the current limiting component : you can set the number of requests per second, how many requests pass, and the remaining ones that fail can be downgraded, return some default values, or perform default operations such as friendly reminders. The specific process can be seen in the figure below:

The benefits of doing this are:

Database security : When the current limiting component is available, the database will not hang up. Current limiting ensures how many requests can pass per second;
Some requests can be processed : the database is not down, which means that at least 2/5 of the requests can be processed;
During peak periods, some requests cannot be processed and require the user to click multiple times, because only 2/5 of the requests are processed. For the remaining requests, the user cannot refresh the interface and needs to click several more times;
The cache expiration time set by redis is not set to the same time. The cache time can be flexibly set according to the function, business, and request interface: setRedis (key, value, time+Math.random()*10000);

(3) Cache penetration

Cache penetration refers to data that is neither in the cache nor in the database. The user (hacker) continuously initiates requests, causing the requests to directly query the database. This malicious behavior attack scenario will directly cause the database to hang. The data flow is shown in the following figure:

It is relatively simple to handle this situation. In this case, it bypasses redis or local cache and reaches the database directly. The following solutions can be adopted:

Some verification can be done at the request interface layer, such as user signature authority and parameter verification. Illegal requests can be returned directly;
You can also authenticate or directly intercept valid IDs. Unmatched IDs can be directly filtered or saved to redis using a unified key. The next time an illegal ID is requested, the data will be obtained directly from the cache;
Adopt the Bloom Filter, an advanced interface of redis, and use efficient data structures and algorithms to quickly determine whether your Key exists in the database. If it does not exist, just return. If it exists, check the DB, refresh the KV, and then return.

(4) Cache breakdown

The penetration mentioned above is for large-area data requests, so the penetration is for one point (one key) to cause redis exception, but a certain key is a very hot spot, the requests are very frequent, and it is a centralized access phenomenon. When this key fails (Expiration), a large number of requests will penetrate the cache and directly request the database, just like cutting a hole in the barrier.

Cache breakdown solutions in different scenarios

The data is basically unchanged : when the hot data value is basically not updated, it can be set to never expire.
Data updates are infrequent : When the cache refresh process takes less time, distributed mutex locks or local mutex locks of distributed middleware such as redis and zookeeper can be used to ensure that a small number of requests can request the database and re-update the cache. The process waits for the lock to be released before it can access the new cache.
Frequent data updates : Use timing threads to actively rebuild the cache or extend the expiration time before the cache expires, ensuring that all requests can always access the cache.

Why is Redis so fast?

According to the official introduction of Redis, it can reach 10W+ QPS. This data is no worse than MEMCache. Moreover, Redis is a single-process and single-thread model, which is completely based on memory operation. The CPU is not the bottleneck of Redis. The bottleneck of Redis is memory and network bandwidth. It has the following characteristics:

Using principles similar to HashMap, the time complexity of HashMap queries and operations is O(1), and most requests are purely memory operations, and the data is stored in memory;
The data structure is simple and the data operations are also simple, based on KV;
It is true that the deadlock phenomenon adopts single-thread operation, which avoids unnecessary context switching and competition conditions. There is no CPU switching phenomenon, and there is no problem of considering various locks;
Use non-blocking IO, multiplexed IO model.

Redis elimination strategy

Strategies prefixed with volatile are eliminated from expired data sets.
The strategy prefixed by allkeys is for elimination of all keys.
LRU (least recently used) The least recently used.
LFU (Least Frequently Used) is the least commonly used.
Their trigger conditions are when the memory used by Redis reaches the threshold.

Redis persistence

There are two Redis persistence strategies:

RDB: The snapshot form is to directly save the data in the memory to a dump file, save it regularly, and save the policy.
AOF: Save all the commands to modify the Redis server into a file, a collection of commands. Redis defaults to the persistence mode of snapshot RDB.

If you care a lot about your data but can still afford data loss within minutes, then just use RDB persistence.

AOF appends every command executed by Redis to the disk. Processing huge writes will reduce the performance of Redis. I don’t know if you can accept it.

Database backup and disaster recovery: Regularly generating RDB snapshots is very convenient for database backup, and RDB can restore data sets faster than AOF.

Of course, Redis supports opening RDB and AOF at the same time. After the system restarts, Redis will give priority to using AOF to restore data, so that the data lost will be the least.

Redis master-slave replication

Execute slaveof[masterIP][masterPort] from the slave node to save the master node information;
The scheduled task in the slave node discovers the master node information and establishes a Socket connection with the master node;
The slave node sends a Ping signal, the master node returns Pong, and both parties can communicate with each other;
After the connection is established, the master node sends all data to the slave node (data synchronization);
After the master node synchronizes the current data to the slave node, the replication establishment process is completed;

Next, the master node will continue to send write commands to the slave nodes to ensure master-slave data consistency.

Redis sentry mode

Let’s first talk about the problems with master-slave replication:

Once the master node goes down, the slave node is promoted to the master node. At the same time, the master node address of the application needs to be modified, and all slave nodes need to be ordered to copy the new master node. The whole process requires manual intervention.
The write capability of the master node is limited by the single machine.
The storage capacity of the master node is limited by a single machine.
The disadvantages of native replication will also be more prominent in earlier versions, such as:

After Redis replication is interrupted, the slave node will initiate psync.
If synchronization fails at this time, full synchronization will be performed. While the main database performs full backup, millisecond or second-level lag may occur.

The architectural pattern of Sentinel is as follows:

The system can perform the following four tasks:

Monitoring: Constantly check whether the master server and slave servers are running normally.
Notification: When a problem occurs on a monitored Redis server, Sentinel sends a notification to the administrator or other applications through API scripts.
Automatic failover: When the master node fails to work properly, Sentinel will start an automatic failover operation. It will upgrade one of the slave nodes that has a master-slave relationship with the failed master node to the new master node, and upgrade the other slave nodes. Point to the new master node so that manual intervention is eliminated.
Configuration provider: In Redis Sentinel mode, the client application connects to the Sentinel node collection during initialization to obtain the master node information.

Source: https://www.jianshu.com/p/0a1c9fc23c01