Preface

Redis is a memory-based key-value database, and the size of memory is limited
In a production environment, I occasionally encounter the situation of insufficient Redis server memory. How does Redis memory reclaim for this situation? What will Redis do if the memory is full

1. View Redis memory usage

#使用 192.168.126.15:6379 连接本机上的 Redis 数据库
[root@localhost ~]# redis-cli -h 192.168.126.15 -p 6379
192.168.126.15:6379> INFO memory    #查看内存使用情况
# Memory
used_memory:853728                  #由 Redis 分配器分配的内存总量，包含了 redis 进程内部的开销和数据占用的内存，以字节（byte）为单位
used_memory_human:833.72K           #已更直观的单位展示分配的内存总量
used_memory_rss:16302080            #向操作系统申请的内存大小，与 top 与 ps 等命令的输出一致
used_memory_rss_human:15.55M        #已更直观的单位展示向操作系统申请的内存大小
used_memory_peak:853728             #redis 的内存消耗峰值(以字节为单位)
used_memory_peak_human:833.72K      #以更直观的格式返回 redis 的内存消耗峰值
used_memory_peak_perc:100.00%       #使用内存达到峰值内存的百分比，即(used_memory / used_memory_peak) *100%
used_memory_overhead:841390         #Redis 为了维护数据集的内部机制所需的内存开销，包括所有客户端输出缓冲区、查询缓冲区、AOF 重写缓冲区和主从复制的 backlog
used_memory_startup:791416          #Redis 服务器启动时消耗的内存
used_memory_dataset:12338           #数据占用的内存大小，即 used_memory - sed_memory_overhead
used_memory_dataset_perc:19.80%     #数据占用的内存大小的百分比，100%*(used_memory_dataset/(used_memory - used_memory_startup))
allocator_allocated:1304352
allocator_active:1634304
allocator_resident:11395072
total_system_memory:3958075392      #整个系统内存
total_system_memory_human:3.69G     #以更直观的格式显示整个系统内存
used_memory_lua:37888               #Lua 脚本存储占用的内存
used_memory_lua_human:37.00K        #以更直观的格式显示 Lua 脚本存储占用的内存
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0                         #Redis 实例的最大内存配置
maxmemory_human:0B                  #以更直观的格式显示 Redis 实例的最大内存配置
maxmemory_policy:noeviction         #当达到 maxmemory 时的淘汰策略
allocator_frag_ratio:1.25
allocator_frag_bytes:329952
allocator_rss_ratio:6.97
allocator_rss_bytes:9760768
rss_overhead_ratio:1.43
rss_overhead_bytes:4907008
mem_fragmentation_ratio:20.06       #碎片率，used_memory_rss/ used_memory，>1表示有碎片，<1表示部分Redis的内存被系统交换到硬盘（此时Redis性能变差）
mem_fragmentation_bytes:15489376
mem_not_counted_for_evict:96
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:49694
mem_aof_buffer:96
mem_allocator:jemalloc-5.1.0        #内存分配器
active_defrag_running:0             #表示没有活动的 defrag 任务正在运行，1 表示有活动的 defrag 任务正在运行（defrag:表示内存碎片整理）
lazyfree_pending_objects:0          #0表示不存在延迟释放的挂起对象

Second, the calculation of memory fragmentation rate

Calculated by dividing the memory value used_memory_rss allocated by the operating system by the memory value used_memory used by Redis

used_memory_rss：是Redis向操作系统申请的内存
used_memory：是Redis中的数据占用的内存
mem_fragmentation_ratio：内存碎片率

mem_fragmentation_ratio = used_memory_rss / used_memory

Three, the generation of memory fragmentation

Memory fragmentation is caused by the inefficient allocation/reclamation of physical memory by the operating system (discontinuous physical memory allocation):

Redis has its own memory manager inside, in order to improve the efficiency of memory use, to manage the application and release of memory
When the value in Redis is deleted, the memory is not released directly and returned to the operating system, but is handed over to the internal memory manager of Redis
When applying for memory in Redis, first check whether there is enough memory available in your memory manager
This mechanism of Redis improves the memory usage rate, but it will make some of the memory in Redis that is not used but not released, which leads to the occurrence of memory fragmentation.

Third, the meaning of memory fragmentation rate

Tracking the memory fragmentation rate is very important to understand the resource performance of the Redis instance:

It is reasonable that the memory fragmentation rate is slightly greater than 1, this value indicates that the memory fragmentation rate is relatively low
The memory fragmentation rate exceeds 1.5, indicating that Redis consumes 150% of the actual physical memory needed, 50% of which is the memory fragmentation rate. It is necessary to consider whether to clean up the memory fragmentation, which should be paid attention to.
If the memory fragmentation rate is lower than 1, it means that the Redis memory allocation exceeds the physical memory, and the operating system is swapping memory, that is, the hard disk is being used; it is necessary to increase the available physical memory (expanded memory) or reduce the memory footprint of Redis

Fourth, how to solve the phenomenon of excessive memory fragmentation

1. Redis version less than 4.0

shutdown save
#在redis-cli 工具上输入，随后重启 Redis 服务器
#Redis 服务器重启后，Redis 会将没用的内存归还给操作系统，碎片率会降下来

2. Redis version higher than 4.0

#Redis 4.0版本开始，可以在不重启的情况下，线上整理内存碎片

CONFIG SET activedefrag yes
#内存自动清理

memory purge
#手动直接清理
INFO memory
#再次查看

Five, memory usage

The memory usage of the Redis instance exceeds the maximum available memory, and the operating system will start swapping memory and swap space
Ways to avoid memory swap:
- Choose to install the Redis instance for the size of the cached data (for each key, less memory will be used, that is, the pointer occupies less bytes)
- Use Hash data structure storage as much as possible (Redis has very high storage efficiency in storing Hash structures with less than 100 fields)
- Set the expiration time of the key (reduce the number of keys)

Six, internal recovery key (memory eviction strategy)

Ensure reasonable allocation of redis's limited memory resources
When the maximum memory limit of the configuration file is reached, a key recycling strategy needs to be selected, that is, Redis has several strategies to deal with this situation: by default, the recycling strategy is to prohibit deletion
Modify the maxmemory-policy attribute value in the configuration file (just modify the corresponding part directly):

vim /etc/redis/6379.conf
#598行
maxmemory-policy noeviction
#noenviction（默认策略）：禁止淘汰数据，对于写请求不再提供服务，直接返回错误（DEL请求和部分特殊请求除外）
#volatile-lru：从设置了过期时间的 key 中使用 LRU 算法进行淘汰
#volatile-ttl：在设置了过期时间的 key中，根据 key 的过期时间进行淘汰，越早过期的越优先被淘汰
#volatile-random：从设置了过期时间的 key 中随机淘汰
#allkeys-lru：从所有 key 中使用 LRU 算法进行淘汰
#allkeys-random：从数据集合中任意选择数据淘汰

How does the eviction policy take effect:
- When Redis receives a command that will add data
- Redis checks the memory usage, whether it exceeds the maximum memory limit
- If exceeded, execute the memory eviction strategy, and then execute the command

tips: If a command uses a lot of memory, the memory usage may exceed the maximum memory limit

Redis of NotSQL (4) (Memory Optimization of Performance Management)

Article Directory