11.1 Caching Cache⭐️
11.1.1 The concept of caching
-
definition understanding
- A cache is a copy set of original data stored on a computer so that it can be accessed quickly
- Put distant data closer to the program (store data to faster media)
-
cache classification
- CPU cache
- Match the frequency of the CPU
- client cache
- Reduce network access
- Server-side local cache
- Reduce disk IO
- distributed cache
- Reduce time-consuming complex operations
- Accelerate hot data access in DB
- CPU cache
-
Caching principle
- Store copies of data on faster storage devices
- Bring data closer to consumers
-
Common cache scenarios for web projects
-
Caching operation process
11.1.2 Buffer breakdown
- understand
- A single key fails, but there are a large number of requests for access at the same time
- For some keys with an expiration time set, if these keys may be accessed with high concurrency at some point in time, it is a very "hot" data
- Causes of cache breakdown
- When the cache expires at a certain point in time, there are a large number of concurrent requests for this key at this point in time, the key is not hit, and a large number of requests penetrate to the database server
- case understanding
- Commodity spike
- When buying products, the query efficiency of a single product is very high. If the cache of the product fails at this time, a large number of requests will be accessed to the database, resulting in cache breakdown.
- solution
- For hotspot data, carefully consider the expiration time to ensure that the key will not expire during the hotspot period, and some can even be set to never expire
- Use a mutex (such as Java's multi-threaded lock mechanism), lock it when the first thread accesses the key, and after the query database returns, insert the value into the cache and then release the lock. At this time, the cache is already cached
11.1.3 Cache Avalanche
- understand
- A large number of keys expire at the same time, resulting in excessive pressure on the database in an instant, causing a cache avalanche
- A large number of keys are set with the same expiration time, causing all caches to fail at the same time, resulting in a large amount of instantaneous DB requests and a sudden increase in pressure, causing an avalanche
- Causes of cache avalanche
- A large number of keys expire at the same time; a large number of requests fall on the backend DB;
- solution
- For ordinary key caches with low frequency, set a random expiration time to make the cache expiration time as uniform as possible
- High-frequency key cache, the setting will never expire
- Use a highly available distributed cache cluster to ensure high availability of the cache
- As a secondary cache, A1 is the original cache, A2 is the copy cache, when A1 fails, you can access A2
11.1.4 Cache penetration
- understand
- Accessing a key that does not exist bypasses the cache (the cache does not work), and directly accesses the database, causing excessive pressure on the database and causing cache penetration
- Reasons for cache penetration
- The key is highly concurrently accessed; the key is not hit, and it is obtained from the backend DB; a large number of requests penetrate to the database server
- solution
- A null cache should also be stored for non-existent keys to prevent a large number of requests
- Set interceptors to physically isolate some unsafe access, such as Bloom filters
- Bloom filter: use a large enough bitmap to store the keys that may be accessed, and the keys that do not exist are directly filtered to avoid pressure on the underlying data storage system;
11.1.5 Cache Coherence
- understand concepts
- When the real-time requirements of the data are very high, it is necessary to ensure that the data in the cache is consistent with that in the database, and the node needs to ensure that the data in the cache node and the copy are also consistent, and no differences can occur (cluster synchronization)
- Ensure that the cache is consistent with the actual data
- Causes of cache inconsistency
- Inconsistency may occur when reading and writing the same data
- Write MySQL data first, then delete the cache;
- Thread 1 has written MySQL data, but has not come to delete the data urgently. At this time, thread 1 suddenly crashes, and the deletion of the cache fails, and the cache inconsistency occurs.
- Delete the cache first, then write MySQL data,
- Thread 1 deleted the cache, but did not come to write to MySQL in a hurry; another thread 2 happened to read this cache, but did not find it, thread 2 went to MySQL to read data, and finally the original thread 1 updated the data, then thread 1 read dirty data
11.2 Introduction to Redis
11.2.1 Introduction to Redis
-
Introduction to Redis
- Redis is an open source (BSD licensed), in-memory data structure storage system that can be used as a database, cache and message middleware. It supports multiple types of data structures, such as strings , hashes ) , lists , sets , sorted sets and range queries, bitmaps , hyperloglogs and geospatial index radius queries. Redis has built-in replication , LUA scripting (Lua scripting) , LRU driven events (LRU eviction) , transactions (transactions) and different levels of disk persistence (persistence) , and provide high availability (high availability) through Redis Sentinel (Sentinel) and automatic partition (Cluster).
-
Languages supported by Redis
-
Data types supported by Redis
- string 、 hash 、 list 、 set 、`sorted …
11.3 Installation of Redis (SingleNode)
11.3.1 Installation dependencies
yum -y install gcc-c++ autoconf automake
11.3.2 Download and upload
11.3.3 Decompression
tar zxvf redis-5.0.3.tar.gz
11.3.4 Precompilation and installation
-
Switch to the decompression directory
- cd redis-5.0.3/
-
compile source code
- make MALLOC=libc
-
Create the redis installation directory
- mkdir -p /opt/yjx/redis
-
If you need to specify the installation path, you need to add the PREFIX parameter**
- make PREFIX=/opt/yjx/redis/ install
11.3.5 Server and client startup
-
server start
- The foreground starts, and the default port number of the redis service is 6379
前台启动 ./redis-server-
background start
- Copy redis.conf to the installation path
## 创建一个配置文件目录 mkdir -p /opt/yjx/redis/conf ## 拷贝配置文件到目录中 cp ~/redis-5.0.3/redis.conf /opt/yjx/redis/conf- Modify redis.conf under the installation path, and change daemonize to yes
- Background start command
后台启动 ./redis-server /opt/yjx/redis/conf/redis.conf -
Summarize
前台启动 ./redis-server 后台启动 ./redis-server /opt/yjx/redis/conf/redis.conf -
client start
To start the client in the specified directory (command after configuration)
[root@basenode ~]# cd /opt/yjx/redis/bin/ [root@basenode bin]# ./redis-cli -a 123456- Comment out bind 127.0.0.1 to enable all ip access to redis, if you want to specify multiple ip access, but not all ip access, you can bind to set
- Close the protected mode, modify it to no
- Add access authentication
- We can modify the number of default databases to 16 by default, and modify database 32 to 32 databases by default.
- After modification, kill -9 XXXX kills the redis process and restarts redis
- Connection established again -> success
11.4 Redis commands
11.5 Redis persistence and transaction mechanism
11.5.1 Redis Persistence Mechanism
definition understanding
- Redis is an in-memory database, and the data is stored in the memory. Although the data in the memory is read quickly, it is easy to lose
- Redis also provides us with persistence mechanisms, namely RDB (Redis DataBase) and AOF (Append Only File)
1. RDB(Redis DataBase)
-
understand
- RDB persistence can generate a point-in-time snapshot of the dataset within a specified time interval, which is the default persistence method
-
features
- This method is to write the data in the memory to the binary file as a snapshot, and the default file name is dump.rdb
-
Three trigger mechanisms of RDB
-
save trigger method
- This command will block the current Redis server. During the execution of the save command, Redis cannot process other commands until the RDB process is completed.
-
bgsave trigger mode
- When executing this command, Redis will perform snapshot operations asynchronously in the background, and snapshots can also respond to client requests
- The Redis process executes the fork operation and creates a child process. The RDB persistence process is in charge of the child process, which is automatically introduced after completion, and the blocking only occurs in the fork phase, and the general time is very short.
-
automatic trigger
- Automatic triggering is done by our configuration file.
#配置文件 # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed save 900 1 save 300 10 save 60 10000 #配置文件的意义 服务器在 900 秒之内，对数据库进行了至少 1 次修改，就对操作保存 服务器在 300 秒之内，对数据库进行了至少 10 次修改，就对操作保存 服务器在 60 秒之内，对数据库进行了至少 10000 次修改，就对操作保存- stop-writes-on-bgsave-error
# However if you have setup your proper monitoring of the Redis server # and persistence, you may want to disable this feature so that Redis will # continue to work as usual even if there are problems with disk, # permissions, and so forth. # 默认值为yes。当启用了RDB且最后一次后台保存数据失败，Redis是否停止接收数据。 stop-writes-on-bgsave-error yes- rdbcompression
# Compress string objects using LZF when dump .rdb databases? # For default that's set to 'yes' as it's almost always a win. # If you want to save some CPU in the saving child set it to 'no' but # the dataset will likely be bigger if you have compressible values or keys. # 默认值是yes。对于存储到磁盘中的快照，可以设置是否进行压缩存储。 rdbcompression yes- rdbchecksum
# RDB files created with checksum disabled have a checksum of zero that will # tell the loading code to skip the check. # 默认值是yes。在存储快照后，我们还可以让redis使用CRC64算法来进行数据校验，但是 这样做会增加大约10%的性能消耗 rdbchecksum yes- dbfilename
# The filename where to dump the DB # 设置快照的文件名，默认是 dump.rdb dbfilename dump.rdb- dir
# The working directory. # The DB will be written inside this directory, with the filename specified # above using the 'dbfilename' configuration directive. # The Append Only File will also be created inside this directory. # Note that you must specify a directory here, not a file name. # 设置快照文件的存放路径，这个配置项一定是个目录，而不能是文件名 dir ./
-
-
Advantages and disadvantages of RDB
- Advantage
- RDB files are compact and fully backed up, ideal for backup and disaster recovery
- When generating RDB files, the redis main process will fork() a child process to handle all the saving work, and the main process does not need to perform any disk IO operations
- RDB is faster than AOF when restoring large data sets
- disadvantage
- When performing snapshot persistence, a child process will be started to be responsible for snapshot persistence. The child process will have the memory data of the parent process. When the parent process performs other operations later, the child process cannot persist this part of the data to disk, so Data modified during snapshot persistence will not be saved and data may be lost
- Advantage
2. AOF(Append Only File)
-
Improvements for the disadvantages of RDB
- Full backup is always time-consuming. Sometimes we provide a more efficient way AOF. The working mechanism is very simple. Redis will append each received write command to the file through the write function.
- Popular understanding is logging
-
Rwrite strategy
- Rewrite the data in the memory with commands to reduce the size of the log file. Redis provides the bgrewriteaof command
- Save the data in the memory to a temporary file in the form of a command, and at the same time fork a new process to rewrite the file
- Rewrite the contents of the database in the entire memory to a new aof file by command
-
In-depth understanding of AOF file rewriting (Rwrite)
- When the client enters the BGREWRITEAOF command, Redis will create a child process to rewrite the AOF file, and create an AOF rewrite buffer, so that the child process rewrites the AOF file, and the parent process can continue to accept the client's command request. When the child process rewrites After the AOF file is completed, the parent process will be notified to add the contents of the AOF rewrite buffer to the rewritten AOF file. The appending process is blocked and the client's command request is not accepted.
- When the child process rewrites the AOF file, the parent process ignores the client's BGREWRITEAOF command, and all client write operations will proceed normally, and the write operations will be recorded in the AOF buffer and the AOF rewrite buffer
- After each event cycle, the contents of the AOF buffer will be written to the AOF file. If the child process is rewriting the AOF file, the AOF file written by the parent process is the original AOF file. Whether it is synchronized to the disk depends on the appendfsync parameter and no -appendfsync-on-rewrite parameter, if no-appendfsync-on-rewrite is set to no, it will not sync
-
AOF configuration information
############################## APPEND ONLY MODE ############################### # By default Redis asynchronously dumps the dataset on disk. This mode is # good enough in many applications, but an issue with the Redis process or # a power outage may result into a few minutes of writes lost (depending on # the configured save points). # # The Append Only File is an alternative persistence mode that provides # much better durability. For instance using the default data fsyncpolicy -
AOF trigger strategy
- Every time the synchronization policy is modified: always
- Synchronous persistence Every time a data change occurs, it will be immediately recorded to the disk. The performance is poor but the data integrity is better.
- Synchronization strategy per second: everysec
- Asynchronous operation, record every second If there is a downtime within one second, there will be data loss
- Out of sync policy: no
- never sync
- Every time the synchronization policy is modified: always
-
AOF advantages and disadvantages
- Advantage
- AOF can better protect data from loss. Generally, AOF will perform an fsync operation through a background thread every 1 second, and at most 1 second of data will be lost.
- AOF log files do not have any disk addressing overhead, the writing performance is very high, and the files are not easy to be damaged.
- Even if the AOF log file is too large, the background rewrite operation will not affect the read and write of the client.
- The commands of the AOF log file are recorded in a very readable way. This feature is very suitable for emergency recovery of catastrophic accidental deletion.
- disadvantage
- For the same piece of data, AOF log files are usually larger than RDB data snapshot files
- The write QPS supported by AOF will be lower than the write QPS supported by RDB
- QPS: Queries Per Second means "query rate per second"
- TPS: is the abbreviation of TransactionsPerSecond, which is the number of transactions per second
- Advantage
3. Selection and comparison of RDB and AOF
-
Adults do not do multiple choice questions
-
If you use AOF and RDB at the same time, then use AOF as the template for restoring data at startup
-
If you choose, the combination of the two is better.
-
Because you understand the two persistence mechanisms, the rest depends on your own needs. You may choose different needs, but they are usually used in combination.
-
-
comparison of the two
11.6 Master-slave replication cluster
-
The background of the master-slave cluster
-
Although Redis reads and writes very fast, it also creates situations where the reading pressure is particularly high. In order to share the reading pressure, Redis supports master-slave replication. The master-slave structure of Redis can adopt a master-slave or cascade structure.
-
Redis master-slave replication can be divided into full synchronization and incremental synchronization according to whether it is full or not
-
11.6.1 Build master-slave server
-
Create a config file in the Redis main config file folder
-
Master node configuration file
## 导入一个通用配置文件 include /opt/yjx/redis/conf/redis.conf ## 当前主服务器端口 port 7100 ## 设置主服务密码 requirepass 123456 ## 当前主服务进程ID pidfile /var/run/redis_7100.pid ## 当前主服务RDB文件名称 dbfilename dump7100.rdb ## 当前主服务文件存放路径 dir /opt/yjx/redis/conf/ -
Slave nodes need to be configured
- Permanent slave nodes (added directly in the configuration file)
## 导入一个通用配置文件 include /opt/yjx/redis/conf/redis.conf ## 当前主服务器端口 port 7100 ## 当前主服务进程ID pidfile /var/run/redis_7200.pid ## 当前主服务RDB文件名称 dbfilename dump7200.rdb ## 当前主服务文件存放路径 dir /opt/yjx/redis/conf/ ## 同步master节点的网络信息（低版本必须使用slaveof,高版本推荐使用replicaof） replicaof 192.168.88.101 7100 ## 设置master节点的密码信息 masterauth 123456 ## 从节点只做读的操作，保证主从数据的一致性 slave-read-only yes- Temporary slave node (input on the redis client command line)
- replicaof 192.168.88.101 7100
- config set masterauth 123456
-
End the fate of the slave server (end)
- slaveof no one
11.6.2 Data Synchronization Mechanism
-
understand
- The master-slave structure of Redis can adopt a master-slave or cascade structure. Redis master-slave replication can be divided into full synchronization and incremental synchronization according to whether it is full or not.
- When the master-slave is just connected, perform full synchronization;
- After the full synchronization is complete, perform incremental synchronization
-
full sync
-
synchronization timing
- Redis full replication generally occurs during the Slave initialization phase. At this time, the Slave needs to copy all the data on the Master.
-
synchronization steps
- The slave server connects to the master server and sends the SYNC command;
- After the master server receives the SYNC command, it starts to execute the BGSAVE command to generate the RDB file and uses the buffer to record all write commands executed thereafter;
- After the master server BGSAVE is executed, it sends snapshot files to all slave servers, and continues to record the executed write commands during the sending;
- After receiving the snapshot file from the server, discard all old data and load the received snapshot;
- After the master server snapshot is sent, it starts to send the write command in the buffer to the slave server;
- The slave server finishes loading the snapshot, starts receiving command requests, and executes write commands from the master server buffer;
-
-
Incremental synchronization
-
synchronization timing
- Redis incremental replication refers to the process of synchronizing the write operations of the master server to the slave server when the slave is initialized and starts to work normally.
-
synchronization process
- The process of incremental replication is mainly that the master server sends the same write command to the slave server every time it executes a write command, and the slave server receives and executes the received write command
-
-
Asynchronous nature of master-slave replication
- Master-slave replication is non-blocking for incoming books from the master Redis server
- This means that when the slave server is in the process of master-slave replication synchronization, the master Redis can still handle external access requests
- Master-slave replication is non-blocking for incoming books from the Redis server
- This means that even if redis is in the process of master-slave replication, it can accept external query requests, but at this time, the old data returned from redis
- Master-slave replication is non-blocking for incoming books from the master Redis server
11.6.3 Server disconnection and reconnection
-
Disconnect and reconnect
- Starting from Redis 2.8, if you encounter a disconnection, you can continue to replicate from the interruption after reconnecting without having to resynchronize
-
Reconnection process
- The realization of partial synchronization depends on maintaining a synchronization log and synchronization identification for each slave server in the memory of the master server
- Each slave server will carry its own synchronization identifier and the last location of the last synchronization when synchronizing with the master server
- When the master-slave connection is broken, the slave server interval time (default 1s) actively tries to connect with the master server
- If the offset identifier carried by the slave server is still in the synchronization backup log on the master server
- Then continue the last synchronization operation from the offset sent by slave
- If the offset sent by the slave is no longer in the master's synchronous backup log, a full update must be performed
11.7 Redis Sentry
- reason for sentry
- In the master-slave replication mode of Redis, once the master node cannot provide services due to failure, it is necessary to manually promote the slave node to the master node, and at the same time notify the client to update the address of the master node
- Sentinel Sentinel is a high-availability solution officially provided by Redis, which can be used to monitor the operation of multiple Redis service instances
11.7.1 Sentinel functions and functions
-
monitoring:
- Sentinel will constantly check whether your master server and slave server are working properly
- Sentinel belongs to many-to-many monitoring, a sentinel can monitor all nodes
- ZKFC belongs to one-to-one monitoring
- A data node can also be monitored by multiple sentinels
-
Reminder (Notification):
- When a node fails, the operation and maintenance personnel will be reminded
- When there is a problem with a monitored Redis server, Sentinel can send a notification to the administrator or other applications through the API.
-
Automatic failover:
-
When Sentinel finds that one of the nodes is down, if the node is down, it will automatically elect a backup node to become the new master node
-
When a master server fails to work properly, Sentinel will start an automatic failover operation, which will upgrade one of the slave servers of the failed master server to the new master server, and let the other slave servers of the failed master server replicate the new one main server;
-
When the client tries to connect to the failed master server, the cluster will also return the address of the new master server to the client, so that the cluster can use the new master server to replace the failed server.
-