Set a single
        hset key subkey subvalue: Set a key-value pair
Get a single
        hget key subkey: Get the value of a subkey
Set multiple
        hmset key subkey1 subvalue1 subkey2 subvalue2 ...: Set multiple key-value pairs
Get multiple
        hmget key subkey1 subkey2 ...: Get the value of multiple subkeys
Get all attributes and attribute values
        hgetall key: Get all the information of the specified key value
delete - subkey
        hdel key subkey1 subkey2 ...
delete
        del key

3 List type (understand) Stirng=list For example: s3=a,a,b,b,c,d

Settings at both ends:
lpush key member1 member2.. : insert to the left
   , for example: lpush l1 abcd d,c,b,a
rpush key member1 member2.. : insert to the right
    , for example: rpush l1 abcd a,b,c,d
lrange key startindex endindex: check for example: lrange key 0 -1: check all

Deletion at both ends:
        lpop key: one pops up on the left
        rpop key: one pops up on the right

4 Collection type (understanding) String=set For example: s4=a,c,d,b

        sadd key member1 member2: set
        srem key member1 member2: delete
        smembers key: view

5 Ordered set type (understand) String=zset For example: s5=a,b,c,d

        zadd key score1 value1 score2 value2...: set to add
        zrem key value1 value2...: delete the specified member
        zcard key: display the length of the element
        zscore key value: get the number of the member

Redis general operation

        1 keys *: query all keys
        2 exists key: determine whether there is a specified key, return 1 if yes, otherwise return 0 3
        rename key new key: rename
        4 type key: determine the type of a key
        5 del key: delete

1. The problem of single-point Redis

1.1 Data loss problem

Redis is an in-memory storage, and data may be lost when the service is restarted.

1.2 Concurrency issues

Although the single-point Redis concurrency capability is good, it cannot satisfy high concurrency scenarios such as 6.18 and 11.11.

1.3 Failback issues

If Redis goes down, the service is unavailable, and an automatic means of failure recovery is needed.

1.4 Storage capacity problem

Redis is based on memory, and the amount of data that can be stored by a single node is difficult to meet the demand for massive data.

2. Problem solving

2.1 Solutions to data loss problems

Redis has two persistence schemes:

2.1.1 RDB persistence

RDB (Redis Database Backup file) Redis data backup file, also known as Redis data snapshot. Simply put, all data in memory is recorded to disk. When the Redis instance fails and restarts, read the snapshot file from the disk and restore the data. Snapshot files are called RDB files, which are saved in the current running directory by default .

1. The timing of RDB persistent execution is executed in the following four situations

Execute the save command
Execute the bgsave command
When Redis is down (shutdown command shutdown)

When Redis is down, Huizhixing executes a save command to achieve RDB persistence.

When an RDB condition is triggered

Redis internally has a mechanism to trigger RDB, which can be found in the redis.conf file.

格式：
# 900s内，如果至少有1个key被修改，则执行bgsave，如果是save ""，则表示禁用RDB
    save 900 1
# 900s内，如果至少有10000个key被修改，则执行bgsave，如果是save ""，则表示禁用RDB
    save 900 10000
    
弊端：898s有10000个key被修改了，但是此时突然断电了，是无法保存的。

#在redis.conf文件中也可以配置如下配置
    #1.是否压缩，建议不开启，压缩会消耗cpu
        rbdcompression no
    #2.RDB文件名称
        dbfilename dunp.rdb
    #3.文件保存的路径目录
        dir ./

2. Think about the difference between save and bgsave

The save command is executed by the main process RDB, and other commands will be blocked during this process. May only be used during data migration.

The bgsave command is used to complete the RDB by an independent process, and the main process can continue to process user requests without being affected.

3. Principle of RDB

At the beginning of bgsave, the main process will be forked to obtain the child process, and the child process will share the memory data of the main process. After the fork is completed, read the memory data and write it to the RDB file

fork uses copy-on-write technology:

When the main process performs a read operation, the shared memory is accessed.
When the main process performs a write operation, a copy of the data is copied and the write operation is performed.

3. Disadvantages of RDB

The RDB execution interval is long, and there is a risk of data loss between two RDB writes.
Fork child process, compress, and write out RDB files are time-consuming

2.1.2 AOF persistence

AOF (Append Only File) appends files. Every write command processed by Redis will be recorded in the AOF file, which can be regarded as a command log file.

1. AOF configuration

AOF is disabled by default, you need to modify the redis.conf configuration file to enable AOF:

#表示每执行一次写命令，立即记录到AOF文件 1281行
appendonly yes
#AOF文件的名称 1256行
appendfilename "appendonly.aof"

The frequency of AOF command recording can also be configured through the redis.conf file

#表示每执行一次写命令，立即记录到AOF文件 1281行
appendfsync always

#写命令执行完先放入AOF缓冲区，然后表示每隔1S将缓冲区数据写到AOF文件，也是默认的方案
appendfsync everysec

#写命令执行完先放入AOF缓冲区，由操作系统决定如何将缓冲区内容写回磁盘。
appendfsync no

Comparison of three recording strategies of AOF
configuration item	Refresh time	advantage	shortcoming
Always	sync refresh	High reliability, almost no data loss	performance impact
everysec	Refresh every second	moderate performance	Up to 1 second of data loss
no	operating system control	best performance	Poor reliability, may lose a large amount of data

2. AOF file rewriting

Because AOF records commands, the AOF file will be much larger than the RDB file. And AOF will record multiple write operations of the same key, but only the last operation is meaningful. By executing the bgrewrite command, the AOF file can be rewritten to achieve the same effect with the fewest commands.

Redis will also automatically rewrite the AOF file when the threshold is triggered. Thresholds are also configurable in redis.conf.

bgrewriteaof

bgrewriteaof

#AOF文件比上次文件增长超过多少百分比则触发重写
auto-aof-rewrite-percentage 100
#Aof文件体积最小多大以上才触发重写
auto-aof-rewrite-min-size 64mb

2.1.3 Comparison between RDB and AOF

RDB and AOF have their own advantages and disadvantages. If the data security is high, the two are often used in combination in actual development.

	RDB	AOF
Persistence	Take snapshots of the entire memory at regular intervals	Log every command executed
data integrity	Incomplete, lost between backups	relatively complete
File size	There will be compression, the file size is small	Record commands, the file size is very large
Downtime recovery speed	soon	slow
Data Recovery Priority	Low, because data integrity is not as good as AOF	High because of higher data integrity
System resource usage	High, heavy CPU and memory consumption	Low, mainly disk IO resources, But AOF will take up a lot when rewriting CPU and memory resources
scenes to be used	Low data security requirements, acceptable Minutes of data loss, the pursuit of faster startup speed	Higher data security requirements

2.2 Solving the problem of concurrency

The concurrency capability of single-node Redis has an upper limit. To further improve the concurrency capability of Redis, a single master-slave cluster is required to achieve read-write separation.

Redis cluster is divided into three ways

master-slave cluster
Sentinel Cluster
Sharded cluster

2.2.1 Redis master-slave cluster

The concurrency capability of single-node Redis has an upper limit. To further improve the concurrency capability of Redis, it is necessary to build a master-slave cluster to achieve read-write separation .

1. Master-slave data synchronization principle

Full sync:

When the master-slave establishes a connection for the first time, it will perform de-full synchronization and copy all the data of the master node to the slave node.

The process is shown in the figure below

Full process description:

The slave node requests incremental synchronization
The master node judges replid, finds inconsistency, and refuses incremental synchronization
Master generates RDB with complete memory data and sends RDB to slave
The slave clears the local data and loads the RDB of the master
The master records the commands during the RDB period in repl_baklog, and continuously sends the commands in the log to the slave
The slave executes the received command and keeps in sync with the master

2. Thinking: How does the master know that salve is connecting for the first time?

First, you need to understand a few concepts:

Replication Id

Replid for short, is the mark of the data set, and the replid consistently indicates that it is the same data set.

Each master has a unique reply, and the slave will inherit the reply of the master node

offset

Offset, as the data recorded in repl_baklog increases and the primary key increases,

When the slave completes the synchronization, it will also record the current synchronization offset.

If the slave's offset is smaller than the master's offset, it means that the slave's data lags behind the master's and needs to be updated.

Therefore, when the slave synchronizes data, it must declare its own replication id and offset to the master, so that the master can determine which synchronization data is needed.

Because the slave is originally a master and has its own reply and offset. When it becomes a slave for the first time and establishes a connection with the master, the offset and reply sent are its own offset and reply. If the master judges that the reply sent by the slave is inconsistent with its own, it means that this is a brand new slave, and it knows to do full synchronization. The master will send its reply and offset to the slave, and the slave will save this information, and the reply of the slave will be consistent with the master in the future

Therefore, the basis for the master to judge whether a node is synchronized for the first time is to see whether the replid is consistent.

2.1 Incremental synchronization

Full synchronization needs to create RDB first, and then transfer the RDB file to a slave through the network, which is too expensive. Therefore, except for the first full synchronization, the slave and the master perform incremental synchronization most of the time .

What is delta sync? It is to update only part of the data that is different between the slave and the master. As shown in the picture:

Thinking: How does the master know where the data difference between the slave and itself is?

This is about the repl_baklog file during full synchronization.

This file is an array with a fixed size, but the array is circular, that is to say, after the subscript reaches the end of the array, it will start reading and writing from 0 again , so that the data at the head of the array will be overwritten.

Repl_baklog will record the command log and offset processed by Redis, including the current offset of the master and the offset that the slave has copied to:

The difference between the offset of the slave and the master is the data that the slave needs to incrementally copy.

As data continues to be written, the offset of the master gradually increases, and the slave keeps copying to catch up with the offset of the master:

until the array is filled:

At this point, if new data is written, the old data in the array will be overwritten. However, as long as the old data is green, it means that the data has been synchronized to the slave, even if it is overwritten, it will have no effect. Because only the red part is not synchronized.

However, if the slave has network congestion, the offset of the master far exceeds the offset of the slave:

If the master continues to write new data, its offset will overwrite the old data until the current offset of the slave is also overwritten:

The red part in the brown box is the data that has not been synchronized but has been overwritten. At this time, if the slave recovers, it needs to synchronize, but finds that its offset is gone, and the incremental synchronization cannot be completed. Can only do full sync.

Note: The size of repl_baklog has an upper limit, and the earliest data will be overwritten when it is full. If the slave has been disconnected for too long and the data that has not been backed up is overwritten, incremental synchronization cannot be performed based on the log, and only full synchronization can be performed again.

Summarize:

Briefly describe the difference between full synchronization and incremental synchronization?

Full synchronization: the master generates an RDB with complete memory data and sends the RDB to the slave. Subsequent commands are recorded in repl_baklog and sent to slave one by one.
Incremental synchronization: the slave submits its own offset to the master, and the master obtains the commands after the offset in the repl_baklog to the slave

When to perform full synchronization?

When the slave node connects to the master node for the first time
The slave node has been disconnected for too long and the offset in the repl_baklog has been overwritten

When is an incremental sync performed?

When the slave node is disconnected and restored, and the offset can be found in the repl_baklog

2.2.2 Redis sentinel cluster

Redis provides a Sentinel mechanism to achieve automatic failure recovery of the master-slave cluster.

The role of the sentinel is as follows:

Monitoring : Sentinel constantly checks that your master and slave are working as expected
Automatic failure recovery : If the master fails, Sentinel will promote a slave to master. When the faulty instance recovers, the new master is also the main one
Notification : Sentinel acts as the service discovery source for the Redis client, and when the cluster fails over, it will push the latest information to the Redis client

Cluster Monitoring Principle

Sentinel monitors the service status based on the heartbeat mechanism, and sends a ping command to each instance of the cluster every 1 second:

• Subjective (subjective) offline: If a sentinel node finds that an instance does not respond within the specified time, it considers the instance to be offline subjectively .

• Objective offline: If more than the specified number (quorum) sentinels think that the instance is offline subjectively, the instance will be offline objectively . The quorum value is preferably more than half of the number of Sentinel instances.

Principles of Cluster Failure Recovery

Once a master failure is found, sentinel needs to select one of the salves as the new master. The selection basis is as follows:

First, it will judge the length of disconnection between the slave node and the master node. If it exceeds the specified value (down-after-milliseconds * 10), the slave node will be excluded
Then judge the slave-priority value of the slave node, the smaller the priority, the higher the priority, if it is 0, it will never participate in the election
If the slave-prority is the same, judge the offset value of the slave node. The larger the value, the newer the data and the higher the priority
The last is to judge the size of the running id (run_id) of the slave node, the smaller the priority, the higher the priority.

When a new master is elected, how to implement the switch?

The process is as follows:

Sentinel sends the slaveof no one command to the candidate slave1 node to make the node the master
Sentinel sends the slaveof 192.168.85.143 7003 command to all other slaves to make these slaves become slave nodes of the new master and start synchronizing data from the new master.
Finally, sentinel marks the failed node as a slave, and when the failed node recovers, it will automatically become the slave node of the new master

summary

What are the three functions of Sentinel?

monitor
failover
notify

How does Sentinel judge whether a redis instance is healthy?

Send a ping command every 1 second, if there is no communication for a certain period of time, it is considered as subjective offline
If most of the sentinels think that the instance is offline subjectively, it is determined that the service is offline

What are the failover steps?

First select a slave as the new master, execute slaveof no one
Then let all nodes execute slaveof new master
Modify the faulty node configuration, add slaveof new master

2.2.3 Redis Fragmentation Cluster

Master-slave and sentry can solve the problem of high availability and high concurrent reading. But there are still two unresolved issues:

Mass data storage problem
The problem of high concurrent writing

Using fragmented clusters can solve the above problems, as shown in the figure:

Sharded cluster features:

There are multiple masters in the cluster, and each master saves different data
Each master can have multiple slave nodes
The master monitors each other's health status through ping
Client requests can access any node in the cluster and will eventually be forwarded to the correct node

3. Avalanche, breakdown and penetration problems

Avalanche: When the data in Redis fails at the same time, when storing data in Redis, you must set the expiration time, two hours, just expired, a large number of requests come, directly access to the database

Solution: make the time of failure into random data

Breakdown: High-frequency access to the same piece of data, this piece of data does not exist in Redis, but in MySQL

Solution: Set Null in Redis

Penetration: Redis does not have a large amount of data, nor does MySQL

Solution: set to Null in Redis, Bloom filter

Bloom filter (BloomFilter):

Bloom filter is a binary vector data structure proposed by Howard Bloom in 1970. It has space and time efficiency and is used to detect whether an element is a member of the set. If the test result is yes, the element is not necessarily in the set; but if the test result is no, the element must not be in the set. Therefore, the Bloom filter has a recall rate of 100% . In this way, each detection request returns two cases: "in the set (possibly wrong)" and "not in the set (absolutely not in the set)". It can be seen that Bloom filter sacrifices accuracy and time to save space.

calculation method

If we need to judge whether an element is in a set, we usually save all the elements, and then know whether it is in the set through comparison. Linked lists and trees are based on this idea. When the number of elements in the set As it grows larger, the space and time we need become linearly larger, and the retrieval speed becomes slower and slower. The Bloom filter uses a hash function method to map an element to a point on an m- length array. When the point is 1 , the element is in the set, otherwise it is not in the set. The disadvantage of this method is that there may be conflicts when there are many detected elements. The solution is to use k hash functions corresponding to k points. If all points are 1 , then the element is in the set. If there are 0 , the element is not in the set.

The biggest problem with the Bloom filter is the false positives rate. This false positive rate is unavoidable, but it should be reduced as much as possible! !

Optimization method: The length of the array and the number of hash functions can be increased as much as possible.

Calculation method:

Application Scenario: Commodity Query

Question: If you keep querying the products whose ID is greater than 10000, you will keep querying the MySQL database. Although you cannot find it, it is still consuming MySQL resources. This is a typical cache penetration phenomenon, so how to solve it?

Bloom Filter can be used

Specific implementation of Bloom Filter

1. Google 's guava framework implements BloomFilter

2. BloomFilter is also implemented in Redis

Redis persistence mode RDB and AOF and Redis cluster introduction

Introduction to Redis

Redis five data types

1 String type (master) String=String For example: s1="aaa"

2 Hash type (understand) String=hash For example: s2=key:value