Climbing the ladder: Redis full analysis (2)

The learning resources are organized from: The
book "The Story of Mad God" at Station B was continued last

11. Detailed explanation of Redis configuration file

The following are the default configurations in redis.conf

######### INCLUDES #########
# 包含，可以想properties一样包含其他配置文件
# include /path/to/local.conf
# include /path/to/other.conf
######### NETWORK ##########
# 绑定地址，可以接收请求的链接，改成 0.0.0.0 或者 * 表示通配
bind 127.0.0.1

# 保护模式，非本机访问需要将其关闭
protected-mode yes

port 6379 # 监听端口
######### GENERAL ########
# 以守护进程的方式运行，默认no，意思是是否可以后台运行redis
daemonize no

# 如果以后台方式运行，我们需要制定一个pid文件
pidfile /var/run/redis_6379.pid

# 日志级别
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably) 生产环境适用
# warning (only very important / critical messages are logged)
loglevel notice

logfile "" # 日志文件名

databases 16 # 数据库数量，默认16个

#是否显示logo，就是运行redis-server的时候那个redis logo
always-show-logo yes

########## SNAPSHOTTING  ########
# 快照 

# 持久化的执行规则，复合条件则执行一次数据持久化 .rdb .aof
# 如果900秒内，有至少1个key修改过
# 如果300秒内，有至少10个key修改过
# 如果60秒内，至少有10000个key修改过
save 900 1
save 300 10
save 60 10000

# 持久化出错，是否继续工作
stop-writes-on-bgsave-error yes

rdbcompression yes # 是否压缩rdb文件（压缩会消耗cpu资源）

rdbchecksum yes # 保存rdb文件的时候，进行错误的检查校验

dbfilename dump.rdb # rdb文件名

dir ./   # rdb文件保存的目录 

########## SECURITY #########
# 密码，默认是注释掉，没有密码的
# requirepass foobared
# 也可以通过命令修改密码：config set requirepass 123

######### CLIENTS ##########
# 最大连接数，默认不限制，指的是同时存活的redis客户端
# maxclients 10000

######## MEMORY MANAGEMENT ########
# 最大内存容量
# maxmemory <bytes>

# 内存满了之后的处理策略
# maxmemory-policy 六种方式
# 1、volatile-lru：只对设置了过期时间的key进行LRU（默认值） 
# 2、allkeys-lru ： 删除lru算法的key   
# 3、volatile-random：随机删除即将过期key   
# 4、allkeys-random：随机删除   
# 5、volatile-ttl ： 删除即将过期的   
# 6、noeviction ： 永不过期，返回错误
# maxmemory-policy noeviction

####### APPEND ONLY MODE #######
# aof
# 默认不开启aof模式，默认使用rdb模式
appendonly no    
    
# aof持久化文件名
appendfilename "appendonly.aof"
    
# aof同步模式 
# appendfsync always  # 每次修改都会 sync，比较消耗性能
appendfsync everysec  # 每秒执行一次sync，可能会丢失1秒的数据
# appendfsync no # 不执行 sync，操作系统自己同步数据

12. Redis persistence

concept

Redis is an in-memory database, and data is lost after power failure, so it needs persistence.

Redis has two persistence strategies, RDB and AOF, and uses RDB mode by default.

Trigger mechanism:

1. Save rules, if satisfied, execute rdb persistence

2. Execute fluxhall

3. Exit redis

Data Recovery:

When redis is started, it will automatically check the persistence rules configured in redis.conf to find out whether there is a corresponding persistence file, and if so, it will reply to the data in the file.

rdb

redis database

When the specified persistence conditions are met, the data set in the memory is written to the disk.

Redis will create a fork child process for persistence operations:

1. First, write the data into a temporary file;

2. After all the data is written into the temporary file, replace the last persistent rdb file with this temporary rdb file to become the official rdb file.

In the entire persistence process, all io operations are completed by the fork process, ensuring the high performance of the main process.

redis.conf

dbfilename dump.rdb # rdb文件名
dir ./   # rdb文件保存的目录 

Advantages : high performance, suitable for large-scale data recovery

Disadvantages : the last persisted data may be lost (downtime during persistence, etc.), the fork process will occupy certain system resources

aof

append only file append the command to the file

The aof strategy means that each write instruction is stored in the aof file in the form of a log, and redis will execute the write instructions in the aof file line by line when recovery is needed.

Aof file repair:

If the AOF file is damaged, redis in AOF mode will not start. Use redis-check-aof tool to repair:

redis-check-aof --fix appendonly.aof

redis.conf

##### APPEND ONLY MODE #####
# 默认不开启，修改为yes重启生效
appendonly no
appendfilename "appendonly.aof"

# appendfsync always
appendfsync everysec
# appendfsync no

# 是否不需要重写aof文件
no-appendfsync-on-rewrite no

# 当aof文记达到64兆的100%时，重写一个新的文件来继续存储指令
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Advantages :

1. Data integrity will be better

2. Up to 1 second of data loss

3. If synchronization is not required, no fork process cycle is required for persistence

Disadvantages :

1. The file size of aof is much larger than that of rdb

2. The speed of data recovery is slower than rdb

Persistence summary extension

1. The RDB persistence method can perform snapshot storage of data within a specified time interval
2. The AOF persistence method records each write operation to the server. When the server restarts, these commands are executed again to restore the original data. The AOF command uses the Redis protocol to append the save command to the end of the file, and Redis can also rewrite the AOF file in the background , So that the volume of the AOF file is not too large;
3. Make a cache, if you only want your data to exist when the server is running, you can also not do any persistence
4. Turn on two caching methods at the same time
   • In this case, when Redis restarts, it will load the AOF file first to restore the original data, because under normal circumstances the data set saved by the AOF file is more complete than the RDB file;
   • The RDB data is not real-time. When colleagues use both, the server will only find the AOF file when restarting. Even so, it does not mean that only the AOF mode can be turned on, because RDB is more suitable for backing up the database. ), restart quickly, and there is no guarantee that AOF will not have a statement error, leave a back hand;
5. Performance recommendations
   • Because the RDB file is only used for backup purposes, it is recommended to only persist the RDB file on the Slave, and it only needs to be backed up every 15 minutes, and only the 900 1 rule is retained;
   • If AOF is turned on, the advantage is that under severe conditions, only data of no more than two seconds will be lost. The startup script is simpler and only loads the AOF file. The price is: 1. Continuous IO 2. AOF rewrite will rewrite at the end It is almost inevitable that the new data generated in the process is written to the new file and it is blocked. Therefore, as long as the size of the hard disk allows, the frequency of AOF rewrite should be reduced as much as possible. The default size of AOF rewrite is 64m, which is too small and can be set to more than 5G. , The default rewrite can be changed to an appropriate value when it exceeds 100% of the original size;
   • If you do not start AOF, you can only rely on Master-Slave Replication to achieve high availability, which can save most of the IO and reduce the system fluctuations caused by rewrite. The price is that if the Master/Slave colleague crashes, you will lose more than ten minutes of data. The startup script must also compare the RDB files in the two Master/Slave and load the newer one (Weibo uses this architecture).

14. Redis publish and subscribe

Quoting rookie information

Subscribe/publish message graph

Demo

Subscriber:

127.0.0.1:6379> subscribe ssxChennel
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "ssxChennel"
3) (integer) 1
#进入等待阶段，等待频道消息推送
1) "message" 	#消息
2) "ssxChennel" #频道名
3) "helloCust" 	#消息内容

Sending end:

127.0.0.1:6379> publish ssxChennel helloCust
(integer) 1

Application scenarios

1. Real-time messaging system

2. Real-time chat (the channel is used as a chat room, and the information is echoed back to the chatter)

3. Subscribe, follow the system

   Slightly more complex scenarios are implemented using message queues.

15. Redis master-slave replication

concept

Master-slave replication refers to copying the data of one Redis server to other Redis servers. The former is called the master node (Master/Leader), and the latter is called the slave node (Slave/Follower). Data replication is one-way, and it can only be from the master node to the slave node (master-slave replication, separation of read and write) Master is mainly writing, and Slave is mainly reading.

PS: By default, each Redis server is the master node, and a master node can have multiple slave nodes (or none), and a slave node can only have one master node.

The main functions of master-slave replication include:

1. Data redundancy: Master-slave replication realizes hot backup of data, which is a data redundancy method besides persistence;
2. Failure recovery: When the master node has a problem, the slave node can provide services to achieve rapid failure recovery (service redundancy);
3. Load balancing: On the basis of master-slave replication, with the separation of reads and writes, the master node can provide write services, and the slave nodes provide read services to share the server load; the general system is that the read pressure is greater and can be shared by multiple slave nodes "Reading" pressure;
4. High availability: Master-slave replication is still the basis for the realization of sentinels and clusters, so master-slave replication is the basis for Redis high availability.

The general production environment cannot use a single Redis;

Environment configuration

By default, each Redis server is the master node

127.0.0.1:6379> info replication
# Replication
role:master #主节点
connected_slaves:0
master_replid:d1ca96e10d765ebb85d9db44475211bade81534d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

Configure the cluster:

Three configuration files: Example: 6379 6380 6381

1. Port: port 6379

2. Background service: daemonize yes

3. pid file: pidfile /var/run/redis_6379.pid

4. rdb file: dbfilename dump.rdb

5. Log file: logfile "6379.log"

6. Start three redis separately
   

Cluster configuration: one master and two slaves

Let the slave recognize the boss

Use command to configure master-slave cluster

Configuration 6380:

127.0.0.1:6380> slaveof 127.0.0.1 6379	#配置主机的host和port  （我使用localhost能连接成功，但主机没找到从机....）
OK
127.0.0.1:6380> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:0
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:d507237df1352cd264dfe399b439c2872dca4977
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:0

Host 6379:

127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6380,state=online,offset=14,lag=1
master_replid:d507237df1352cd264dfe399b439c2872dca4977
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:14
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:14

The general production environment is directly configured in the configuration file, permanent configuration

# replicaof <masterip> <masterport>
# masterauth <master-password>

Master write slave read

#主机写入
127.0.0.1:6379> set key1 v1
OK

#从机读取
127.0.0.1:6380> get key1
"v1"
127.0.0.1:6380> set key2 v2
(error) READONLY You can't write against a read only replica. #从机不能进行写操作

Reconnect when the host is down

127.0.0.1:6379> shutdown
not connected> exit

At this time, the host is turned off,

and then the host restarts the setting

[root@localhost redis-5.0.8]# src/redis-server redis.conf
[root@localhost redis-5.0.8]# src/redis-cli -p 6379
127.0.0.1:6379> set key2 v2
OK

Read value from machine

127.0.0.1:6380> get key2  
"v2"   #还是可以读取到，意味着恢复了主从链接关系

Reconnect from machine down

If the master-slave relationship configured by the command is used, the slave will lose the associated master when restarting (the master-slave structure cannot be restored after restarting)

But if you use the command to associate to the master slaveof, all the data in the master will be restored to the slave.

Copy principle

After the Slave starts and successfully connects to the Master, it will send a sync command.

After receiving the command, the Master starts the background save process and collects all the received write commands. After the background process is executed, the Master will transfer the entire file to the Slave to complete a data synchronization;

Official noun:

• Full copy: After the Slave server receives the data file, it saves it and loads it into the memory;
• Incremental replication: When the Master receives the write command, it sends the commands to the Slave to complete synchronization;

Only if the slave reconnects to the master, a full copy will be completed.

Cluster configuration: link cluster

In this mode, when the Master goes down, you need to manually execute commands to make Slave1 the boss:

slaveof on one

If the Master comes back at this time, you need to manually configure the cluster structure.

16. Redis sentry mode

Automatically select boss

Overview

The method of master-slave switching technology is: when the master server goes down, manually switching a slave server to the master server requires manual intervention, which is time-consuming and laborious and will cause service unavailability for a period of time. This is not a recommended method, more often we give priority to sentinel mode. Redis 2.8 began to provide Sentinel (sentinel) architecture to solve this problem.

The automatic version of seeking to usurp the throne can monitor whether the host is faulty in the background. If there is a fault, it will automatically convert a slave to the host according to the voting decision .

The sentinel mode is a special mode. First of all, Redis provides the command of the sentinel, and the sentinel is an independent process. Process principle: The sentinel can monitor multiple Redis instances running by sending commands and waiting for the Redis server to respond.

Single sentinel mode

Multi-sentinel mode

1. When the main server goes down, suppose Sentinel 1 detects this result first. At this time, the main service will be judged as "subjectively offline" and will not immediately failover.
2. When the other sentinels detect that the main server is down and the number reaches a certain value, then a vote will be conducted among the sentries, and the voting behavior is initiated by one of the sentries, that is, a failover operation is performed.
3. After the switch is successful, each sentinel will switch the host operation of the slave server he monitors through the publish and subscribe mode. This process becomes "objective offline".

Practice

The current cluster structure is: one master and two slaves

1. Configure the sentinel file sentinel.conf

#sentinel monitor 监听名 host port 票数
sentinel monitor myredis 127.0.0.1 6379 2

The number of votes in the back means that after the host is down, Slave will vote who becomes the host, and the one with more votes becomes the host.

2. Start the sentinel process
   
   When the host is down, the sentinel will elect another as the host after a certain period of time.

advantage

1. Sentinel cluster, based on master-slave replication mode, all the advantages of master-slave configuration
2. Master-slave can be automatically switched, failover automatically, Redis achieves high availability
3. Upgrade from manual mode to master-slave mode, more robust and real-time

Disadvantage

1. Redis is not easy to expand online, once the cluster capacity reaches the upper limit, online expansion is very troublesome
2. It is more troublesome to implement the full set of sentinel mode

The full set of sentinel mode configuration:

bind 172.31.11.235
port 26380
daemonize yes
logfile "/usr/local/redis-4.0.9/sentinel.log.26380"

#master1
# 哨兵监控这个master，在至少1个哨兵实例都认为master down后把master标记为odown
sentinel monitor master1 172.31.11.235 6380 1#多少毫秒后，sentinel 认定redis master 服务器已掉线sentinel down-after-milliseconds master1 5000
# 若sentinel在该配置值内未能完成failover操作（即故障时master/slave自动切换），则认为本次failover失败
sentinel failover-timeout master1 10000
#sentinel can-failover master1 yes
sentinel parallel-syncs master1 2
# Generated by CONFIG REWRITE
dir "/usr/local/redis-4.0.9"
sentinel auth-pass master1 xxxxx

17. Redis cache penetration, breakdown and avalanche

The use of Redis cache greatly improves performance and efficiency, especially in terms of data query. But it also brings some problems, the most fatal of which is the problem of data consistency, which is unsolvable in a strict sense. If the data consistency requirements are high, then the cache cannot be used.

Other typical problems are: cache penetration, cache avalanche, and cache breakdown.

Cache penetration

(Some kind of data that bypasses the cache to query the database)

Cache penetration means that when a user wants to query a piece of data, and there is no data in Redis, so the cache does not hit, then it will enter the persistence layer to check the database, and find that there is no, so this query fails.

Then when this type of query is executed countless times (malicious attacks, spikes, etc.), it has not been stored in the cache, and the database has been queried all the time, which will cause the database to be attacked, which is called cache breakdown.

Bloom filter

Bloom filter is a data structure that stores all possible query parameters in the form of hash, which is verified at the control layer first, and discarded if they do not match, thus avoiding query attacks on the underlying storage system.

Cache empty objects

When the storage layer misses, even the returned empty object will be cached, and an expiration time will be set at the same time. Later access to this data will be obtained from the cache, protecting the back-end data source.

There are two problems:

1. If the null value is stored, it also means that more memory space is used;
2. Although the expiration time is set for the null value, there will still be a problem of data inconsistency between the cache layer and the storage layer for a period of time. At this time, when the business has high consistency requirements, it will cause unnecessary impact.

Cache breakdown

(A hotspot cache continues to have high concurrent queries)

It refers to a key that is very popular and has been intensively queried by large concurrency. Then when the key expires instantly, the continuous large concurrency will penetrate the cache and directly impact the database (similar to a wall, in an empty Through a hole)

Set hotspot data not to expire

From the perspective of caching, no expiration time is set, so there will be no hot key expiration problem, avoiding cache breakdown

Add mutex

Distributed lock: Use distributed lock to ensure that there is only one thread to query the back-end service for each key at the same time, and other threads cannot query without the permission of the distributed lock. This implementation transfers the pressure of high concurrency to distributed locks, so the test for distributed locks is greater.

Cache avalanche

It refers to the collective failure of the cache at a certain point in time. (Redis downtime, collective expiration)

For example: on Double Eleven, popular products are put into the cache, and the expiration is set for 1 hour. Then at one o'clock in the morning, the cache collectively expires, which will cause a pressure peak on the database, which is very likely to destroy the persistence layer;

centralized expiration It's not very dangerous. What's more deadly is that a node of the cache server is down and disconnected, and all the pressure is directly offloaded to the persistence layer.

Redis high availability

Redis cluster, when one is down, there will be other redis on top. (Live more in different places)

Current limit downgrade

• Distributed lock current limit, control only one or a few threads can access the background;
• Downgrade: temporarily shut down other non-urgent services to give up server resources for the main service;

Data warm-up

This means that before the official launch, the data that may be accessed is accessed in advance, and most of the data is stored in the cache.

Before a large concurrent access is about to occur, manually trigger the loading of different keys in the cache, and set different expiration times to make the time points of cache invalidation evenly distributed.