After reading these 20 Redis interview questions, can you make an appointment for an interview with Ant Financial?

1. What is Redis, and what are the characteristics of Redis?

The full name of Redis is: Remote Dictionary Server (remote data service). Redis is a storage system that supports various data structures such as key-value. It can be used in scenarios such as caching, event publishing or subscription, and high-speed queues. Support network, provide string, hash, list, queue, collection structure direct access, based on memory, and can be persisted.

Feature 1: Rich data types

We know that many databases can only handle one data structure:

• Traditional SQL databases process two-dimensional relational data;

• MemCached database, both key and value are strings;

• Document database (MongoDB) is a document composed of Json/Bson.

Of course, it's not that their databases are not good, but that once the data structure provided by the database is not suitable for doing something, the program will be very troublesome and unnatural to write.

Although Redis is also a key-value pair database, it is different from Memcached in that the value of Redis can not only be a string, but also any of the other five data organizations. By choosing different data structures, users can use Redis to solve various problems. When using Redis, if you encounter a problem, the first thing you will think is to choose which data structure to solve which functional problems. With various data structures , to facilitate you to solve the problem.

Feature 2: Memory storage

There are two types of databases: one is hard disk database and the other is memory database.

The hard disk database stores the value on the hard disk, and stores the index in the memory. When the hard disk database wants to access the value of the hard disk, it first finds the index in the memory, and then finds the value. The problem is that when reading and writing to the hard disk, if there are more reads and writes, it will block the IO function of the hard disk.

Memory storage means that all data is stored in memory, and the data reading and writing speed is very fast.

Feature 3: Persistence function

Save the data stored in the memory to the hard disk to ensure data security and facilitate data backup and recovery.

2. What data structures does Redis have?

Redis is a key-value database. The type of key can only be String, but the data type of value is richer, mainly including five types:

• String

• Hash

• List

• Set

• Sorted Set

(1) String string

SET KEY_NAME VALUE

The string type is binary safe. It means that the string of redis can contain any data. Such as jpg images or serialized objects. The string type is the most basic data type of Redis, and a key can store up to 512MB.

(2) Hash

HSET KEY_NAME FIELD VALUE

Redis hash is a collection of key-value (key=>value) pairs. Redis hash is a mapping table of string type field and value, and hash is especially suitable for storing objects.

(3) List list

//在 key 对应 list 的头部添加字符串元素
LPUSH KEY_NAME VALUE1.. VALUEN
//在 key 对应 list 的尾部添加字符串元素
RPUSH KEY_NAME VALUE1..VALUEN
//对应 list 中删除 count 个和 value 相同的元素
LREM KEY_NAME COUNT VALUE
//返回 key 对应 list 的长度
LLEN KEY_NAME

Redis lists are simply lists of strings, sorted by insertion order. You can add an element to the head (left) or tail (right) of the list

(4) Set collection

SADD KEY_NAME VALUE1...VALUEn

Redis's Set is an unordered collection of string types. The collection is implemented through a hash table, so the complexity of adding, deleting, and searching is O(1).

(5) Sorted Set ordered collection

ZADD KEY_NAME SCORE1 VALUE1.. SCOREN VALUEN

Redis zset, like set, is also a collection of string type elements, and duplicate members are not allowed. The difference is that each element will be associated with a score of type double.

Redis uses scores to sort the members of the set from small to large.

The members of zset are unique, but the score (score) can be repeated.

3. What is the maximum capacity that a string type value can store?

Check the official documentation ( https://redis.io/topics/data-types ) and you can see that the maximum supported length of the String type value is 512M, so the correct answer is 512M.

4. Can you tell me about the usage scenarios of each data structure of Redis?

(1) Usage scenarios of String

Usage scenarios of string type: information cache, counter, distributed lock, etc.

Commonly used commands: get/set/del/incr/decr/incrby/decrby

Practical scenario 1: Record the number of visits of each user, or record the number of views of each product

plan:

Commonly used key names: userid:pageview or pageview:userid, if the id of a user is 123, then the corresponding redis key is pageview:123, and the value is the number of visits of the user. To increase the number of times, you can use the command: incr.

Reasons for use: The modification of the number of visits by each user or the number of product views is very frequent. If you use a file system such as mysql, frequent modifications will cause pressure on mysql and the efficiency will be low. There are two advantages of using redis: using memory, very fast; single-threaded, so there is no competition, and the data will not be messed up.

Practical scenario 2: Cache frequently read but infrequently modified information, such as user information and video information

plan:

In terms of business logic: read from redis first, read from redis if there is a value, read from mysql if there is no value, and write a copy to redis as a cache, pay attention to setting the expiration time.

Key-value design:

Directly serialize a mysql record of the user (usually serialized into json) as the value, userInfo:userid as the key, the key name is: userInfo:123, and the value stores the json string corresponding to the user information. For example, the key is "user:id:name:1", and the value is "{"name":"leijia","age":18}".

Practical scenario 3: Limit the number of visits within a certain period of time for an ip

plan:

Use the key to record the IP, and the value to record the number of visits. At the same time, the expiration time of the key is set to 60 seconds. If the key expires, it will be reset. Otherwise, it will be judged. When the visit exceeds 100 times within one minute, the visit will be prohibited.

Practical scenario 4: Distributed session

We know that the session is saved in the server in the form of a file; if your application is load-balanced and the website project is placed on multiple servers, when the user logs in on server A, the session file will be written on server A ; When the user jumps to the page and the request is assigned to the B server, the session file cannot be found, and the user will have to log in again.

If you want multiple servers to share a session, you can store the session in redis. Redis can be independent of all load balancing servers, or it can be placed on one of the load balancing servers; but all the servers where the application is located are connected to the same server. A redis server.

(2) Usage scenarios of Hash

Take the shopping cart as an example, if the user id is set as the key, then all the products in the shopping cart are the values ​​corresponding to the user key, each product has an id and the purchase quantity, and the corresponding hash structure is that the product id is the field, and the product quantity is the value . as the picture shows:

If the product id and product quantity are serialized into a json string, they can also be stored in the string type mentioned above. Let's compare the two data structures:

|

Comparison item

|

string（json）

|

hash

|

in conclusion:

When a property of an object needs to be modified frequently, it is not suitable to use string+json, because it is not flexible enough, and the entire object needs to be re-serialized and assigned for each modification; if the hash type is used, it can be modified separately for a certain property , without serialization and without modifying the entire object. For example, attributes that may change frequently, such as product price, sales volume, number of followers, and number of reviews, are suitable for storage in the hash type.

(3) Usage scenarios of List

A list is essentially an ordered queue with repeatable elements.

Actual combat scenario: Timed leaderboard

The lrange command of the list type can view the data in the queue in pages. The leaderboard calculated every once in a while can be stored in the list type, such as the QQ music mainland leaderboard, which is calculated once a week and stored in the list type. When accessing the interface, the page and size are converted into the lrange command to obtain the leaderboard data .

However, not all leaderboards can be implemented with the list type. Only regularly calculated leaderboards are suitable for storage in the list type. Corresponding to the regularly calculated leaderboards are real-time calculated leaderboards. The list type cannot support real-time calculations. The leaderboard, the implementation of the real-time calculated leaderboard will be introduced in detail when the application scenario of the sorted set is introduced below.

(4) Set usage scenarios

Collections are characterized by being unordered and deterministic (no repetition).

Actual combat scene: Favorites

For example, if you like a song in QQ Music, click "Like" and the song will be placed in your personal favorites. Each user makes a favorite collection, and each favorite collection stores the song ids that the user has favorited.

The key is the user id, and the value is the collection of song ids.

(5) Sorted Set usage scenarios

An ordered collection is characterized by order and no duplicate values. Different from set, each element of sorted set is associated with a score attribute, and redis uses score to sort the members in the set from small to large.

Actual combat scenario: real-time leaderboard

There are many kinds of real-time lists in QQ Music, such as Soaring List, Hot Song List, and New Song List. Redis key can be used to store the list type, score is the number of clicks, value is the song id, and the redis data will be updated every time the user clicks on a song , the sorted set will sort the song ids according to the score, that is, the number of hits.

5. How does Redis do persistence? Can you talk about the implementation principles of RDB and AOF?

What is persistence?

Persistence (Persistence), that is, saving data (such as objects in memory) to a storage device (such as disk) that can be stored permanently. The main application of persistence is to store objects in memory in a database, or in a disk file, in an XML data file, and so on.

[Image upload failed...(image-bcb944-1603435016634)]

Persistence can also be simply understood from the following two levels:

• Application layer: If you shut down (shutdown) your application and restart it, the previous data still exists.

• System layer: If you shut down (shutdown) your system (computer) and restart it, the previous data still exists.

Why should Redis persist?

Redis is an in-memory database, and all operations are done in memory to ensure efficiency. The data is cached in the memory. When you restart the system or shut down the system, the data cached in the memory before will be lost and can no longer be retrieved. Therefore, in order to avoid this situation, Redis needs to implement persistence to store the data in memory.

How does Redis achieve persistence?

Redis officially provides different levels of persistence:

• RDB Persistence: It can store snapshots of your data at specified time intervals.

• AOF Persistence: Record every write operation to the server. When the server restarts, these commands will be re-executed to restore the original data. The AOF command uses the redis protocol to append and save each write operation to the end of the file. Redis can also rewrite the AOF file in the background, so that the size of the AOF file will not be too large.

• Do not use persistence: If you only want your data to exist while the server is running, you can also choose not to use any persistence method.

• Enable RDB and AOF at the same time: You can also enable both persistence methods at the same time. In this case, when redis restarts, the AOF file will be loaded first to restore the original data, because the data set saved by the AOF file is usually It is more complete than the data set saved in the RDB file.

How should we choose so many persistence methods? Before choosing, we need to figure out the difference between each persistence method and their respective advantages and disadvantages.

RDB persistence

RDB (Redis Database) persistence is the process of generating a snapshot of the current memory data and saving it to the hard disk. The process of triggering RDB persistence is divided into manual triggering and automatic triggering.

(1) Manual trigger

Manually triggering the corresponding save command will block the current Redis server until the RDB process is completed. For instances with relatively large memory, it will cause long-term blocking, and it is not recommended to use it in an online environment.

(2) Automatic trigger

The corresponding bgsave command is automatically triggered, and the Redis process executes the fork operation to create a child process. The RDB persistence process is in charge of the child process, and it ends automatically after completion. Blocking only occurs during the fork phase, generally for a short time.

It can be configured in the redis.conf configuration file:

save <seconds> <changes>

Indicates that bgsave is automatically triggered when the data is modified xx times within xx seconds. If you want to turn off automatic triggering, you can add an empty string after the save command, namely:

save ""

There are other common triggers for bgsave, such as:

• If the slave node performs a full copy operation, the master node automatically executes bgsave to generate an RDB file and sends it to the slave node.

• By default, when the shutdown command is executed, bgsave is automatically executed if the AOF persistence function is not enabled.

bgsave working mechanism

[Image upload failed...(image-fedfd7-1603435016634)]

(1) Execute the bgsave command, and the Redis parent process judges whether there is currently an executing child process, such as an RDB/AOF child process. If it exists, the bgsave command returns directly.

(2) The parent process executes the fork operation to create a child process. During the fork operation, the parent process will be blocked. View the latest_fork_usec option through the info stats command to obtain the time-consuming time of the latest fork operation, in microseconds

(3) After the parent process fork is completed, the bgsave command returns the message "Background saving started" and no longer blocks the parent process, and can continue to respond to other commands.

(4) The child process creates an RDB file, generates a temporary snapshot file according to the memory of the parent process, and atomically replaces the original file after completion. Execute the lastsave command to obtain the last RDB generation time, which corresponds to the rdb_last_save_time option of info statistics.

(5) The process sends a signal to the parent process to indicate completion, and the parent process updates the statistical information. For details, see the rdb_* related options under info Persistence.

– RDB persistence complete –

AOF persistence

AOF (append only file) persistence: Record each write command in an independent log, and re-execute the commands in the AOF file to restore data when restarting. The main function of AOF is to solve the real-time nature of data persistence, which is currently the mainstream method of Redis persistence.

AOF persistence working mechanism

To enable the AOF function, you need to configure: appendonly yes, which is not enabled by default.

The AOF file name is set through the appendfilename configuration, and the default file name is appendonly.aof. The storage path is the same as the RDB persistence method, and is specified by dir configuration.

AOF workflow operations: command writing (append), file synchronization (sync), file rewriting (rewrite), restart loading (load).

[Image upload failed...(image-1669b9-1603435016634)]

(1) All write commands will be appended to aof_buf (buffer).

(2) The AOF buffer is synchronized to the hard disk according to the corresponding strategy.

Why does AOF append commands to aof_buf? Redis uses a single thread to respond to commands. If every command to write an AOF file is directly appended to the hard disk, then the performance depends entirely on the current hard disk load. Writing to the buffer aof_buf first has another advantage. Redis can provide multiple buffer synchronization strategies to balance the performance and security.

(3) As the AOF file becomes larger and larger, it is necessary to rewrite the AOF file regularly to achieve the purpose of compression.

(4) When the Redis server restarts, the AOF file can be loaded for data recovery.

AOF rewrite (rewrite) mechanism

Purpose of rewriting:

• Reduce the space occupied by AOF files;

• Smaller AOF files can be loaded and restored by Redis faster.

AOF rewriting can be divided into manual triggering and automatic triggering:

• Manual trigger: call the bgrewriteaof command directly.

• Automatic trigger: Determine the automatic trigger timing according to the auto-aof-rewrite-min-size and auto-aof-rewrite-percentage parameters.

auto-aof-rewrite-min-size: Indicates the minimum file size when running AOF rewrite, the default is 64MB.

auto-aof-rewrite-percentage: represents the ratio of the current AOF file space (aof_current_size) to the AOF file space (aof_base_size) after the last rewrite.

Automatic trigger timing

当aof_current_size>auto-aof-rewrite-minsize 并且（aof_current_size-aof_base_size）/aof_base_size>=auto-aof-rewritepercentage。

Among them, aof_current_size and aof_base_size can be viewed in info Persistence statistics.

Why does the AOF file become smaller after rewriting?

(1) The old AOF file contains invalid commands, such as: del key1, hdel key2, etc. Rewrite write commands that preserve only the final data.

(2) Multiple commands can be combined, such as lpush list a, lpush list b, lpush list c can be directly converted into lpush list abc.

AOF file data recovery

[Image upload failed...(image-c23cde-1603435016634)]

Data recovery process description:

(1) When AOF persistence is enabled and AOF files exist, AOF files are loaded first.

(2) When AOF is closed or the AOF file does not exist, load the RDB file.

(3) After loading the AOF/RDB file successfully, Redis starts successfully.

(4) When there is an error in the AOF/RDB file, Redis fails to start and prints an error message.

– AOF persistence finished –

Advantages and disadvantages of RDB and AOF

Advantages of RDB

• RDB is a very compact file, which saves the data set at a certain point in time, which is very suitable for the backup of the data set. For example, you can save the data in the past 24 hours every hour, and save the past 30 days every day. The data, so that even if something goes wrong, you can restore to a different version of the data set according to your needs.

• RDB is a compact single file, which is easy to transfer to another remote data center and is very suitable for disaster recovery.

• When RDB saves RDB files, the only thing the parent process needs to do is to fork a child process. The next work is all done by the child process. The parent process does not need to do other IO operations, so the RDB persistence method can maximize the performance of Redis .

• Compared with AOF, the RDB method will be faster when restoring large data sets.

Advantages of AOF

• You can use different fsync strategies: no fsync, fsync every second, fsync every time you write. Using the default fsync strategy every second, Redis performance is still very good (fsync is processed by a background thread, the main thread will try its best Handling client requests), once there is a failure, you will lose up to 1 second of data.

• AOF file is a log file that only appends, so there is no need to write seek, even if the complete write command is not executed due to some reasons (disk space is full, downtime during writing, etc.), you can also Use the redis-check-aof tool to fix these issues.

• Redis can automatically rewrite AOF in the background when the size of the AOF file becomes too large: the rewritten new AOF file contains the minimum set of commands required to restore the current dataset. The entire rewriting operation is absolutely safe, because Redis will continue to append commands to the existing AOF file during the process of creating a new AOF file. Even if a shutdown occurs during the rewriting process, the existing AOF file will not be lost. . Once the new AOF file is created, Redis will switch from the old AOF file to the new AOF file and start appending to the new AOF file.

• The AOF file saves all write operations performed on the database in an orderly manner. These write operations are saved in the format of the Redis protocol, so the content of the AOF file is very easy for people to understand, and it is also very easy to analyze (parse) the file. Exporting the AOF file is also very simple: for example, if you accidentally execute the FLUSHALL command, but as long as the AOF file is not overwritten, then just stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, The data set can be restored to the state before FLUSHALL execution.

Disadvantages of RDB

• It is a relatively heavy work for Redis to completely save the entire data set. You usually do a complete save every 5 minutes or longer. In case of an unexpected downtime in Redis, you may lose a few minutes of data.

• RDB needs to frequently fork sub-processes to save the data set to the hard disk. When the data set is relatively large, the fork process is very time-consuming, which may cause Redis to fail to respond to client requests within milliseconds.

Disadvantages of AOF

• For the same data set, the volume of AOF files is usually larger than that of RDB files.

• AOF is slower than RDB during data recovery (load), and usually RDB can provide a more guaranteed maximum delay time.

RDB and AOF simple comparison summary

Advantages of RDB:

• RDB is a compact binary file, more suitable for backup, full copy and other scenarios

• RDB restores data much faster than AOF

Disadvantages of RDB:

• RDB cannot achieve real-time or second-level persistence;

• The old and new versions are not compatible with the RDB format.

Advantages of AOF:

• Can better protect data from loss;

• The writing performance of appen-only mode is relatively high;

• It is suitable for emergency recovery of catastrophic accidental deletion.

AOF disadvantages:

• For the same file, the AOF file is larger than the RDB snapshot;

• After AOF is turned on, it will affect the writing QPS, and the writing QPS will decrease compared with RDB;

• The database recovery is relatively slow, so it is not suitable for cold backup.
  Redis internally uses a file event handler, which is single-threaded, so redis is called a single-threaded model. It uses the IO multiplexing mechanism to monitor multiple sockets at the same time, and selects the corresponding event handler for processing according to the events on the socket.

If the interviewer continues to ask why the redis single-threaded model can be so efficient?

• pure memory operation

• The core is based on non-blocking IO multiplexing mechanism

• Single thread instead avoids the frequent context switching problem of multithreading

In the actual production environment, abnormal scenarios such as cache penetration, cache breakdown, and cache avalanche are sometimes encountered. In order to avoid huge losses caused by abnormalities, we need to understand the causes and solutions of each abnormality to help improve system reliability and High availability.

(1) Cache penetration

What is cache penetration?

Cache penetration means that the data requested by the user does not exist in the cache, that is, there is no hit, and it does not exist in the database at the same time. As a result, every time the user requests the data, he has to query the database once, and then returns empty.

If a malicious attacker keeps requesting data that does not exist in the system, a large number of requests will fall on the database in a short period of time, resulting in excessive pressure on the database and even crashing the database system.

Common solutions for cache penetration

(1) Bloom filter (recommended)

The Bloom Filter (BF) was proposed by Burton Howard Bloom in 1970 and is a probabilistic data structure with high space efficiency.

Bloom filters are designed to detect whether a specific element exists in a collection.

If we usually want to judge whether an element is in a set, we usually use the method of search and comparison. The following analyzes the search efficiency of different data structures:

• Using linear table storage, the search time complexity is O(N)

• Stored in a balanced binary sorting tree (AVL, red-black tree), the search time complexity is O(logN)

• Using hash table storage, considering hash collision, the overall time complexity is also O[log(n/m)]

When it is necessary to determine whether an element exists in a massive data collection, not only the search time is slow, but also a large amount of storage space is occupied. Let's take a look at how Bloom filters solve this problem.

Bloom filter design idea

The Bloom filter is a data structure composed of a bit array (bit array) with a length of m bits and k hash functions (hash function). The bit array is initialized to 0, and all hash functions can hash the input data as evenly as possible.

When an element is to be inserted into the Bloom filter, the element is calculated by k hash functions to generate k hash values, and the hash value is used as the subscript in the bit array, and all k corresponding bit values ​​are represented by 0 is set to 1.

When an element is to be queried, it is also calculated by a hash function to generate a hash value, and then the corresponding k bit values ​​are checked: if any bit is 0, it indicates that the element must not be in the set; if all bits are 1, indicating that the set is likely to be in the set. Why not necessarily in the collection? Because the hash values ​​calculated by different elements may be the same, there will be a hash collision, resulting in a non-existing element that may correspond to a bit of 1, which is the so-called "false positive". In contrast, "false negatives" (false negatives) will never appear in BF.

To sum up: what the Bloom filter thinks is not in the set must not be in the set; what the Bloom filter thinks is in may or may not be in the set.

For example: The figure below is a Bloom filter with a total of 18 bits and 3 hash functions. The three elements x, y, and z in the set are hashed to different bits by three hash functions, and the bit position is set to 1. When querying the element w, through three hash functions, it is found that there is a bit with a value of 0, and it can be determined that the element is not in the set.

Advantages and disadvantages of Bloom filter

advantage:

• Save space: no need to store the data itself, only need to store the hash bits corresponding to the data

• Low time complexity: the time complexity of insertion and search is O(k), k is the number of hash functions

shortcoming:

• There are false positives: Bloom filter judges existence, and elements may not be in the set; the accuracy of judgment depends on the number of hash functions

• Unable to remove elements: If an element is removed, but cannot be removed from the Bloom filter, this is also the cause of false positives

Applicable scenarios of Bloom filter

• Crawler system url deduplication

• spam filtering

• blacklist

(2) Return an empty object

When the cache misses, the query persistence layer is also empty, and the returned empty object can be written to the cache, so that the next time the key is requested, the query returns an empty object directly from the cache, and the request will not fall to the persistence layer database. In order to avoid storing too many empty objects, an expiration time is usually set for empty objects.

There are two problems with this approach:

• If there is a large number of key penetrations, caching empty objects will take up valuable memory space.

• The key of the empty object has an expiration time set, during which time there may be scenarios where data in the cache and the persistence layer are inconsistent.

(2) Cache breakdown

What is cache breakdown?

Cache breakdown means that a key is very hot, and it is constantly carrying large concurrency, and the large concurrency is concentrated on accessing this point. When the key fails at the moment, the continuous large concurrency will break through the cache and directly request the database. , like cutting a hole in a barrier.

Cache Penetration Hazard

The sudden surge in pressure on the database caused a large number of requests to be blocked.

How to solve?

Solution 1: Use a mutex lock (mutex key)

This idea is relatively simple, that is, let one thread write back the cache, and other threads wait for the write-back cache thread to finish executing, and then read the cache again.

At the same time, only one thread reads the database and then writes back to the cache, and other threads are blocked. If it is a high-concurrency scenario, a large number of thread blocking will inevitably reduce throughput. How to solve this situation? You can discuss in the message area.

If it is a distributed application, you need to use distributed locks.

Solution 2: Hotspot data never expires

Never expires actually has two meanings:

• Physical does not expire, no expiration time is set for hotspot keys

• Logical expiration, store the expiration time in the value corresponding to the key, if it is found that it is about to expire, build the cache through a background asynchronous thread

From the perspective of actual combat, this method is very friendly to performance. The only disadvantage is that when building the cache, other threads (those not building the cache) may access old data, which is acceptable for systems that do not pursue strict and strong consistency.

(3) Cache Avalanche

What is cache avalanche?

Cache avalanche means that a large amount of data in the cache reaches the expiration time, and the query data volume is huge, and the request directly falls on the database, causing excessive pressure on the database or even downtime. Different from the cache breakdown, the cache breakdown refers to concurrently querying the same piece of data, and the cache avalanche means that different data have expired, and many data cannot be found, so the database is checked.

Cache Avalanche Solution

Commonly used solutions are:

• uniform expiration

• add mutex

• cache never expires

• Double cache strategy

(1) Uniform expiration

Set different expiration times to make the cache invalidation as uniform as possible. Usually, a random value can be added to the validity period or the validity period can be planned uniformly.

(2) Add mutex

Consistent with the cache breakdown solution, only one thread builds the cache at the same time, and other threads are blocked and queued.

(3) The cache never expires

Consistent with the solution to cache breakdown, the cache never expires physically, and an asynchronous thread is used to update the cache.

(4) Double-layer cache strategy

Use primary and secondary caches:

Main cache: The validity period is set according to the experience value, and it is set as the main read cache. After the main cache fails, the latest value is loaded from the database.

Backup cache: The valid period is long, the cache that is read when the lock acquisition fails, and the backup cache needs to be updated synchronously when the main cache is updated.

(4) Cache preheating

What is cache warming?

Cache warm-up means that after the system goes online, the relevant cached data is directly loaded into the cache system, so as to avoid querying the database first and then writing the data back to the cache when the user requests it.

If no preheating is performed, the initial state data of Redis will be empty. In the early stage of system launch, high concurrent traffic will be accessed to the database, causing traffic pressure on the database.

Operation method of cache warming

• When the amount of data is not large, the loading and caching action is performed when the project starts;

• When the amount of data is large, set up a scheduled task script to refresh the cache;

• When the amount of data is too large, priority is given to ensuring that hot data is loaded into the cache in advance.

Cache downgrade means that when the cache fails or the cache server hangs up, the default data or the memory data of the access service are returned directly without accessing the database.

In actual project combat, some hot data is usually cached in the memory of the service, so that once an exception occurs in the cache, the memory data of the service can be used directly, thereby avoiding huge pressure on the database.

Downgrading is generally a lossy operation, so try to minimize the impact of downgrading on the business.

8. Redis memory elimination mechanism

The Redis memory elimination strategy refers to the strategy of processing new data selection by eliminating old data when the cache memory is insufficient.

How to configure the maximum memory?

(1) Configure through the configuration file

Modify the redis.conf configuration file

maxmemory 1024mb //设置Redis最大占用内存大小为1024M

Note: maxmemory is configured as 0 by default. The maximum memory of redis under 64-bit operating system is the remaining memory of the operating system. The maximum memory of redis under 32-bit operating system is 3GB. (2) Configuration via dynamic commands

Redis supports dynamically modifying the memory size through commands at runtime:

127.0.0.1:6379> config set maxmemory 200mb //设置Redis最大占用内存大小为200M
127.0.0.1:6379> config get maxmemory //获取设置的Redis能使用的最大内存大小
1) "maxmemory"
2) "209715200"

Classification of elimination strategies

After the maximum memory occupied by Redis is used up, if you continue to add data, how to deal with this situation? In fact, Redis has officially defined eight strategies to deal with this situation:

noeviction

The default strategy is to directly return an error for a write request without elimination.

allkeys-lru

lru(less recently used), least recently used. Use approximate LRU algorithm to eliminate from all keys.

volatile-lrulru(less recently used), least recently used. Use the approximate LRU algorithm to eliminate from the key with the expiration time set.

allkeys-random

Randomly eliminated from all keys.

volatile-random

Randomly eliminated from keys with expiration time set.

volatile-ttl

ttl (time to live), in the key with the expiration time set, it will be eliminated according to the expiration time of the key, and the earlier the expiration time, the priority will be eliminated.

allkeys-lfu

lfu(Least Frequently Used), the least frequently used frequency. Use approximate LFU algorithm to eliminate from all keys. Supported from Redis4.0.

volatile-lfu

lfu(Least Frequently Used), the least frequently used frequency. Use the approximate LFU algorithm to eliminate from the key with the expiration time set. Supported from Redis4.0.

Note: When using the three strategies of volatile-lru, volatile-random, and volatile-ttl, if no expired key can be eliminated, an error will be returned like noeviction.

LRU algorithm

LRU (Least Recently Used), the least recently used, is a cache replacement algorithm. When using memory as a cache, the size of the cache is generally fixed. When the cache is full, if you continue to add data to the cache at this time, you need to eliminate some old data and release memory space to store new data. At this time, the LRU algorithm can be used. The core idea is: if a piece of data has not been used in the recent period, it is very unlikely to be used in the future, so it can be eliminated.

Implementation of LRU in Redis

Redis uses an approximate LRU algorithm, which is not the same as the conventional LRU algorithm. The approximate LRU algorithm eliminates data through the random sampling method, randomly selects 5 (default) keys each time, and eliminates the least recently used keys from it.

The number of samples can be modified through the maxmemory-samples parameter, such as: maxmemory-samples 10

The larger the maxmenory-samples configuration is, the closer the elimination result is to the strict LRU algorithm, but the CPU consumption is also high.

In order to implement the approximate LRU algorithm, Redis adds an additional 24-bit field to each key to store the time when the key was last accessed.

Redis3.0 optimization of approximate LRU

Redis 3.0 has made some optimizations to the approximate LRU algorithm. The new algorithm will maintain a candidate pool (size is 16), the data in the pool is sorted according to the access time, the key randomly selected for the first time will be put into the pool, and the key randomly selected each time will only be selected when the access time is less than that in the pool The minimum time will be put into the pool until the candidate pool is filled. When it is full, if there is a new key that needs to be put in, the one with the largest last access time (recently accessed) in the pool will be removed.

When it needs to be eliminated, just select the key with the smallest recent access time (the longest time not accessed) from the pool and eliminate it.

LFU algorithm

LFU (Least Frequently Used) is a new elimination strategy added to Redis4.0. Its core idea is to eliminate keys according to their latest access frequency. was left behind.

The LFU algorithm can better represent the popularity of a key being accessed. If you are using the LRU algorithm, a key has not been accessed for a long time, but is only accessed once in a while, then it is considered as hot data and will not be eliminated, and some keys are likely to be accessed in the future were eliminated. This will not happen if the LFU algorithm is used, because using it once does not make a key a hot data.

9. Does Redis have a transaction mechanism?

• There is a transaction mechanism. Redis transaction life cycle: Open transaction: use MULTI to open a transaction

• Command into the queue: the command for each operation will be added to a queue, but the command will not be actually executed at this time

• Submit a transaction: use the EXEC command to submit a transaction and start sequentially executing the commands in the queue

##10. Are Redis transactions atomic?

First look at the definition of atomicity in relational database ACID: Atomicity: All operations in a transaction are either completed or not completed, and will not end in a certain link in the middle. If an error occurs during the execution of the transaction, it will be restored (Rollback) to the state before the transaction started, as if the transaction had never been executed.

The official document defines a transaction:

• A transaction is a single, isolated operation: all commands within a transaction are serialized and executed sequentially. During the execution of the transaction, it will not be interrupted by command requests sent by other clients.

• A transaction is an atomic operation: either all of the commands in the transaction are executed, or none of them are executed. The EXEC command is responsible for triggering and executing all commands in the transaction: if the client fails to execute EXEC because of disconnection after opening a transaction using MULTI, all commands in the transaction will not be executed. On the other hand, if the client successfully executes EXEC after opening the transaction, all commands in the transaction will be executed.

Officials believe that Redis transactions are an atomic operation, which is considered from the perspective of execution or not. However, from the definition of ACID atomicity, strictly speaking, Redis transactions are non-atomic, because during the sequential execution of commands, once a command execution error occurs, Redis will not stop the execution and roll back the data.

11. Why doesn't Redis support rollback?

Although the Redis command may fail to execute during the transaction, Redis will still execute the remaining commands in the transaction without performing a rollback operation. If you are familiar with mysql relational database transactions, you will be very confused about this. The official reason for Redis is as follows: Only when the called Redis command has a syntax error, this command will fail to execute (after putting this command into the transaction queue During this period, Redis can detect such problems), or perform an operation on a key that does not conform to its data type: in practice, this means that only program errors can cause Redis command execution to fail, which is likely to occur in the program Found during development, generally rarely found in production environments. Supporting transaction rollback capability will lead to complex design, which is contrary to the original intention of Redis. The design goal of Redis is to simplify functions and ensure faster operation speed.

There is a general objection to this official reason: what if the program has bugs? But in fact, regression cannot solve program bugs. For example, a careless programmer plans to update key A, but actually updates key B at the end. The rollback mechanism cannot solve this kind of human error. Just because this kind of human error is unlikely to enter the production system, the official chose a simpler and faster method when designing Redis, and there is no rollback mechanism.

12. What are the commands related to Redis transactions?

(1) WATCH can provide check-and-set (CAS) behavior for Redis transactions. The keys that are WATCHed will be monitored, and it will be found whether these keys have been changed. If at least one of the monitored keys is modified before EXEC executes, the entire transaction is canceled and EXEC returns nil-reply to indicate that the transaction has failed.

（2）MULTI

Used to start a transaction, it always returns OK. After MULTI is executed, the client can continue to send any number of commands to the server. These commands will not be executed immediately, but will be placed in a queue. When the EXEC command is called, all the commands in the queue will be executed.

（3）UNWATCH

Cancel the monitoring of all keys by WATCH command, generally used before DISCARD and EXEC commands. If the EXEC command or the DISCARD command is executed first after executing the WATCH command, then it is not necessary to execute UNWATCH. Because the EXEC command will execute the transaction, the effect of the WATCH command has already been produced; while the DISCARD command cancels the transaction and also cancels all monitoring of the key, so after the execution of these two commands, there is no need to execute UNWATCH.

（4）DISCARD

When the DISCARD command is executed, the transaction will be abandoned, the transaction queue will be emptied, and the client will exit from the transaction state.

（5）EXEC

Responsible for triggering and executing all commands in the transaction:

If the client executes EXEC after successfully opening the transaction, all commands in the transaction will be executed.

If the client fails to execute EXEC because of disconnection after opening the transaction using MULTI, all commands in the transaction will not be executed. It is important to note that even if some/some commands in the transaction fail to execute, other commands in the transaction queue will continue to execute, and Redis will not stop executing the commands in the transaction, unlike the relationship we usually use rollback like a normal database.

13. What is Redis master-slave replication?

Master-slave replication refers to copying the data of one Redis server to other Redis servers. The former is called the master node (master), and the latter is called the slave node (slave); data replication is one-way, only from the master node to the slave node.

The role of master-slave replication

• Data redundancy: master-slave replication implements hot backup of data, which is a data redundancy method other than persistence.

• Fault recovery: When there is a problem with the master node, the slave node can provide services to achieve rapid fault recovery; it is actually a kind of service redundancy.

• Load balancing: On the basis of master-slave replication, combined with read-write separation, the master node can provide write services, and the slave nodes can provide read services to share the server load; Nodes share the read load, which can greatly increase the concurrency of the Redis server.

• The cornerstone of high availability: master-slave replication is also the basis for the implementation of sentinels and clusters, so master-slave replication is the basis for high availability of Redis.

Master-slave replication implementation principle

The master-slave replication process can be divided into three stages: connection establishment stage, data synchronization stage, and command propagation stage.

connection establishment phase

The main function of this stage is to establish a connection between the master and slave nodes to prepare for data synchronization.

Step 1: Save the master node information

The slaveof command is asynchronous. When the slaveof command is executed on the slave node, the slave node immediately returns ok to the client. Two fields are maintained inside the slave node server, namely the masterhost and masterport fields, which are used to store the ip and port information of the master node.

Step 2: Establish a socket connection

The slave node calls the replication timing function replicationCron() once per second. If it finds that there is a master node that can be connected, it will create a socket connection according to the ip and port of the master node.

The slave node establishes a file event handler for the socket that handles the copying work specifically, and is responsible for subsequent copying work, such as receiving RDB files, receiving command propagation, and so on.

After the master node receives the socket connection from the slave node (that is, after accept), it creates a corresponding client state for the socket, and regards the slave node as a client connected to the master node. The master node sends a form of command request to proceed.

Step 3: Send a ping command

After the slave node becomes the client of the master node, it sends a ping command for the first request to check whether the socket connection is available and whether the master node is currently able to process the request.

After the slave node sends the ping command, three situations may occur:

(1) Return pong: indicating that the socket connection is normal, and the master node can currently process the request, and the replication process continues.

(2) Timeout: After a certain period of time, the slave node has not received the reply from the master node, indicating that the socket connection is unavailable, and the slave node disconnects the socket connection and reconnects.

(3) Return results other than pong: If the master node returns other results, such as processing a script that runs overtime, indicating that the master node is currently unable to process the command, the slave node disconnects the socket connection and reconnects.

Step 4: Authentication

If the masterauth option is set in the slave node, the slave node needs to authenticate to the master node; if this option is not set, no authentication is required. The authentication of the slave node is performed by sending the auth command to the master node, and the parameter of the auth command is the value of masterauth in the configuration file.

If the state of the password set by the master node is consistent with the state of the masterauth of the slave node (consistent means that both exist, and the password is the same, or neither exists), the authentication passes and the replication process continues; if they are inconsistent, the slave node disconnects the socket Connect and reconnect.

Step 5: Send slave node port information

After authentication, the slave node will send the port number it listens to (6380 in the preceding example) to the master node, and the master node will save this information in the slave_listening_port field of the client corresponding to the slave node; the port information is not included in the master node It has no effect other than displaying when info Replication is executed.

Data Synchronization Phase

After the connection between the master and slave nodes is established, data synchronization can begin, which can be understood as the initialization of slave node data. The specific execution method is: the slave node sends the psync command to the master node (Redis2.8 was the sync command before), and the synchronization starts.

The data synchronization stage is the core stage of master-slave replication. According to the current state of the master-slave node, it can be divided into full replication and partial replication. These two replication methods and the execution process of the psync command will be explained later, and will not be detailed here. .

Command Propagation Phase

After the data synchronization phase is completed, the master-slave node enters the command propagation phase; at this stage, the master node sends the write command it executes to the slave node, and the slave node receives and executes the command, thereby ensuring the data consistency of the master-slave node.

It should be noted that command propagation is an asynchronous process, that is, the master node does not wait for the reply from the slave node after sending the write command; therefore, it is actually difficult to maintain real-time consistency between the master and slave nodes, and delay is inevitable. The degree of data inconsistency is related to the network status between the master and slave nodes, the execution frequency of the write command of the master node, and the repl-disable-tcp-nodelay configuration in the master node.

14. Can you explain the principle of Sentinel (sentinel mode)?

In the master-slave replication mode of Redis, once the master node cannot provide services due to failure, it is necessary to manually promote the slave node to the master node, and at the same time notify the client to update the address of the master node. This failure handling method is unacceptable to a certain extent. of.

Redis 2.8 and later provides the Redis Sentinel mechanism to solve this problem.

Redis Sentinel is a highly available implementation of Redis. Sentinel is a tool for managing multiple Redis instances, which enables monitoring, notification, and automatic failover of Redis.

The Redis Sentinel architecture diagram is as follows:

The principle of sentinel mode

The main function of Sentinel mode is that it can automatically complete fault discovery and failover, and notify the client, so as to achieve high availability. Sentinel mode usually consists of a set of Sentinel nodes and a set (or sets) of master-slave replication nodes.

heartbeat mechanism

(1) Sentinel and Redis Node

Redis Sentinel is a special Redis node. When creating sentinel mode, you need to specify the relationship between Sentinel and Redis Master Node through configuration, and then Sentinel will obtain the information of all slave nodes from the master node, and then Sentinel will periodically send info commands to the master node and slave nodes to obtain their topology structure and state information.

(2) Sentinel and Sentinel

Based on the Redis subscription publishing function, each Sentinel node will send the Sentinel node's judgment on the master node and the information of the current Sentinel node to the sentinel :hello channel of the master node, and each Sentinel node will also subscribe to this channel to obtain Information about other Sentinel nodes and their judgment on the master node.

Through the above two steps, all Sentinel nodes and all Redis nodes have been aware of each other, and then each Sentinel node will regularly send ping commands to the master node, slave node, and other Sentinel nodes as heartbeat detection to confirm Whether these nodes are reachable.

failover

Each Sentinel will regularly check the heartbeat. When the heartbeat detection timeout of the master node is found, the master node is considered to be unavailable at this time. This judgment is called subjective offline.

Afterwards, the Sentinel node will use the sentinel ismaster-down-by-addr command to ask other Sentinel nodes for the judgment of the master node. When the quorum (quorum) Sentinel nodes all think that the node is faulty, it will perform objective offline, that is, consider The node is no longer available. This also explains why a group of Sentinel nodes is necessary, because a single Sentinel node can easily misjudge the failure state.

Here, the value of quorum is specified when we set up the Sentinel mode. It will be explained later. It is usually the total number of Sentinel nodes/2+1, that is, more than half of the nodes can perform objective offline judgment after making subjective offline judgments.

Because only one Sentinel node is required to complete the failover work, another election will be performed among the Sentinel nodes, and a Sentinel leader is selected based on the Raft algorithm to perform the failover work.

The specific steps for the elected Sentinel leader to fail over are as follows:

(1) Select a node from the slave node list as the new master node

• Filter unhealthy or unsatisfactory nodes;

• Select the slave node with the highest slave-priority (priority), return if it exists, and continue if it does not exist;

• Select the slave node with the largest replication offset, return if it exists, and continue if it does not exist;

• Select the slave node with the smallest runid.

(2) The Sentinel leader node will execute the slaveof no one command to the selected slave node to make it the master node.

(3) The Sentinel leader node will send commands to the remaining slave nodes to let them copy data from the new master node.

(4) The Sentinel leader will update the original master node to a slave node, monitor it, and order it to copy the new master node when it recovers.

15. Can you explain the principle of Cluster?

The reason for introducing the Cluster mode: Whether it is the master-slave mode or the sentinel mode, only one master can write data. In the scenario of massive data and high concurrency, a node is prone to bottlenecks in writing data. The introduction of the Cluster mode can enable multiple nodes to write data at the same time .

Redis-Cluster adopts a non-central structure, each node saves data, and the nodes are connected to each other to know the status of the entire cluster.

As shown in the figure, the Cluster mode is actually a combination of multiple master-slave replication structures. Each master-slave replication structure can be regarded as a node, so there are three nodes in the above Cluster cluster.

16. What are the differences between Memcache and Redis?

The storage method Memecache stores all the data in the memory, and it will hang up after power failure, and the data cannot exceed the memory size.

Part of Redis is stored on the hard disk, which ensures data persistence.

Data support type

Memcache's support for data types is relatively simple.

Redis has rich data types.

Using the underlying model is different

The underlying implementation methods between them and the application protocols for communicating with clients are different.

Redis directly builds the VM mechanism by itself, because if the general system calls system functions, it will waste a certain amount of time to move and request.

17. Suppose there are 100 million keys in Redis, among which 10w keys start with a certain fixed known prefix, how to find them all?

使用keys指令可以扫出指定模式的key列表：
keys pre*

At this time, the interviewer will ask what impact the order will have on the online business, and look directly at the next question.

18. If this redis is providing services for online businesses, what is the problem with using the keys command?

Single-threaded for redis. The keys instruction will cause the thread to block for a period of time, and the online service will be suspended until the instruction is executed before the service can be resumed. At this time, you can use the scan command. The scan command can extract the key list of the specified mode without blocking, but there will be a certain probability of repetition. It is enough to do a deduplication on the client side, but the overall time spent will be longer than using directly. keys Command length.

19. If there are a large number of keys that need to be set to expire at the same time, what should you pay attention to?

If the expiration time of a large number of keys is set too intensively, Redis may experience a short-term freeze at the time of expiration (because redis is single-threaded). If it is serious, it may cause an avalanche of servers, so we generally add a random value to the expiration time to make the expiration time as scattered as possible.

20. What are the commonly used clients of Redis?

Jedis: It is an old-fashioned Redis Java implementation client, which provides comprehensive support for Redis commands.

Redisson: Implements a distributed and scalable Java data structure.

Lettuce: Advanced Redis client for thread-safe synchronous, asynchronous and reactive usage, with support for clusters, Sentinel, pipes and encoders.

advantage:

Jedis: Provides the operating characteristics of Redis more comprehensively.

Redisson: Promotes the separation of users' attention on Redis, and provides many distributed related operation services, such as distributed locks, distributed collections, and supports delay queues through Redis.

Lettuce: An event-driven communication layer based on the Netty framework, whose method calls are asynchronous. Lettuce's API is thread-safe, so you can operate a single Lettuce connection to complete various operations.