Redis is an in-memory database, the data stored in memory, reading and writing efficiency of the database stored on disk is much faster than traditional data. So, Redis monitor memory consumption and learn Redis Redis memory model for efficient and long-term stability is essential.
Memory usage statistics
Redis can get relevant info memory by memory index command. More important indicators shown and explained as follows:
| Property name | Property Description |
|---|---|
| used_memory | Redis total memory allocator, that is, all the data stored in the internal memory footprint |
| usedmemoryhuman | Back in a readable format used_memory |
| usedmemoryrss | It shows the total amount of physical memory occupied by Redis process from the perspective of the operating system |
| usedmemoryrss_human | usedmemoryrss user should read the display format |
| usedmemorypeak | Maximum memory usage, represents the peak of used_memory |
| usedmemorypeak_human | Usedmemorypeak return value in a readable format |
| usedmemorylua | Lua engine consumed by memory size. |
| memfragmentationratio | Ratio usedmemoryrss / used_memory, the rate may represent memory fragmentation |
| maxmemory | Redis limit the maximum memory that can be used, 0 means no limit, in bytes. |
| maxmemory_policy | Redis memory recovery strategies used, can be noeviction, allkeys-lru, volatile-lru, allkeys-random, volatile-random or volatile-ttl. The default is noeviction, that is not recycled. |
When memfragmentationratio> 1, and no explanation has some memory for data storage, but is consumed by memory fragmentation, if the value is large, indicating a serious breakage rates. When memfragmentationratio <1, this situation usually occurs in the operating system memory Redis exchange (swap) to lead to hard disk, to pay special attention to this situation, because the hard disk speed is much slower than memory, Redis performance becomes poor, even dead.
When the Redis memory beyond the available memory, the operating system will swap, the old page is written to disk. From the hard disk read and write from memory read and write about than five orders of magnitude slower. used_memory indicators can help determine whether the risk of being Redis swap or it has been swap.
In Redis Administration article (link at end of the text) is recommended swap to set and memory of the same size, if not swap, once Redis suddenly need more memory than the current operating system's available memory, Redis because of out of memory to be Linix Kernel of OOM Killer directly kill. Although when Redis data is swapped out (swap out), Redis performance will be worse, but better than directly kill good.
Redis use maxmemory parameters limit the maximum available memory. The main purpose of memory limitations are:
- For caching scenarios, such as using the LRU strategy deleted to free up space when exceeded memory limit maxmemory.
- Prevent the use of server memory than physical memory, resulting OOM system after the process is killed.
maxmemory limit the amount of memory Redis is actually used, that is, used_memory statistical item corresponding memory. The actual memory consumption may be larger than maxmemory set to be careful because this memory leads to OOM. So if you have 10GB of memory, the best maxmemory to 8 or 9G
Memory consumption divided
The Redis process consumes including: own memory + Object Memory + buffer memory + memory fragmentation, which Redis empty process itself the memory consumption is very low, typically usedmemoryrss at around 3MB, used_memory generally about 800KB, an empty Redis process consumes memory can can be ignored.
Objects Memory
Objects Memory is the biggest piece of Redis memory footprint, storage of all data users. When all the data are used Redis key-value data types, every time you create key-value pairs, create at least two types of objects: key objects and value objects. Object memory consumption can be simply understood as both objects and memory consumption of (and similar information expired and the like). Key objects are strings, the impact on memory consumption is easy to overlook key while using Redis, should avoid the use of long keys. For more information about Redis object systems, see my previous article twelve view of a band you know Redis data structures and object system .
Buffer Memory
Buffer memory including: client-side caching, replication backlog AOF buffer and buffer.
The client buffer means all access to the server TCP connections Redis input output buffer.
You can not control the input buffer, the maximum space 1G, and if it exceeds disconnected. And the input buffer is not maxmemory control, assuming a Redis example provided maxmemory to 4G, 2G data has been stored, but at this time if the input buffer using 3G, maxmemory has exceeded the limit, may result in loss of data, key eliminated or OOM.
The input buffer is too large mainly because the processing speed of Redis can not keep up the speed input buffer, and every time you enter a command input buffer contains a large number of bigkey.
The output buffer parameter client-output-buffer-limit control, the following format.
client-output-buffer-limit [hard limit] [soft limit] [duration]
hard limit 是指一旦缓冲区大小达到了这个阈值，Redis 就会立刻关闭该连接。而 soft limit 和时间 duration 共同生效，比如说 soft time 为 64mb、duration 为 60，则只有当缓冲区持续 60s 大于 64mb 时，Redis 才会关闭该连接。
普通客户端是除了复制和订阅的客户端之外的所有连接。Reids 对其的默认配置是 client-output-buffer-limit normal 0 0 0 , Redis 并没有对普通客户端的输出缓冲区做限制，一般普通客户端的内存消耗可以忽略不计，但是当有大量慢连接客户端接入时这部分内存消耗就不能忽略，可以设置 maxclients 做限制。特别当使用大量数据输出的命令且数据无法及时推送到客户端时，如 monitor 命令，容易造成 Redis 服务器内存突然飙升。相关案例可以查看这篇文章美团在Redis上踩过的一些坑-3.redis内存占用飙升。
从客户端用于主从复制，主节点会为每个从节点单独建立一条连接用于命令复制，默认配置为 client-output-buffer-limit slave 256mb 64mb 60。当主从节点之间网络延迟较高或主节点挂载大量从节点时这部分内存消耗将占用很大一部分，建议主节点挂载的从节点不要多于 2 个，主从节点不要部署在较差的网络环境下，如异地跨机房环境，防止复制客户端连接缓慢造成溢出。与主从复制相关的一共有两类缓冲区，一个是从客户端输出缓冲区，另外一个是下面会介绍到的复制积压缓冲区。
订阅客户端用于发布订阅功能，连接客户端使用单独的输出缓冲区，默认配置为 client-output-buffer-limit pubsub 32mb 8mb 60，当订阅服务的消息生产快于消费速度时，输出缓冲区会产生积压造成内存空间溢出。
输入输出缓冲区在大流量场景中容易失控，造成 Redis 内存不稳定，需要重点监控。可以定期执行 client list 命令，监控每个客户端的输入输出缓冲区大小和其他信息。
| 属性名 | 属性说明 |
|---|---|
| qbuf | 查询缓冲区的长度（字节为单位， 0 表示没有分配查询缓冲区） |
| qbuf-free | 查询缓冲区剩余空间的长度（字节为单位， 0 表示没有剩余空间） |
| obl | 输出缓冲区的长度（字节为单位， 0 表示没有分配输出缓冲区） |
| oll | 输出列表包含的对象数量（当输出缓冲区没有剩余空间时，命令回复会以字符串对象的形式被入队到这个队列里） |
127.0.0.1:6379> client listid=3 addr=127.0.0.1:58161 fd=8 name= \age=1408 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 \qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 \events=r cmd=client
client list 命令执行速度慢，客户端较多时频繁执行存在阻塞redis的可能，所以一般可以先使用 info clients 命令获取最大的客户端缓冲区大小。
127.0.0.1:6379> info clients# Clientsconnected_clients:1client_recent_max_input_buffer:2client_recent_max_output_buffer:0blocked_clients:0
复制积压缓冲区是Redis 在 2.8 版本后提供的一个可重用的固定大小缓冲区，用于实现部分复制功能。根据 repl-backlog-size 参数控制，默认 1MB。对于复制积压缓冲区整个主节点只有一个，所有的从节点共享此缓冲区。因此可以设置较大的缓冲区空间，比如说 100MB，可以有效避免全量复制。有关复制积压缓冲区的详情可以看我的旧文章 Redis 复制过程详解。
AOF 重写缓冲区：这部分空间用于在 Redis AOF 重写期间保存最近的写入命令。AOF 重写缓冲区的大小用户无法控制，取决于 AOF 重写时间和写入命令量，不过一般都很小。有关 AOF 持久化的详情可以看我的旧文章 Redis AOF 持久化详解。
Redis 内存碎片
Redis 默认的内存分配器采用 jemalloc，可选的分配器还有：glibc、tcmalloc。内存分配器为了更好地管理和重复利用内存，分配内存策略一般采用固定范围的内存块进行分配。具体的分配策略后续会具体讲解，但是 Redis 正常碎片率一般在 1.03 左右(为什么是这个值)。但是当存储的数据长度长度差异较大时，以下场景容易出现高内存碎片问题：
- 频繁做更新操作，例如频繁对已经存在的键执行 append、setrange 等更新操作。
- 大量过期键删除，键对象过期删除后，释放的空间无法得到重复利用，导致碎片率上升。
这部分内容我们后续再详细讲解 jemalloc，因为大量的框架都会使用内存分配器，比如说 Netty 等。
子进程内存消耗
子进程内存消耗主要指执行 AOF 重写 或者进行 RDB 保存时 Redis 创建的子进程内存消耗。Redis 执行 fork 操作产生的子进程内存占用量表现为与父进程相同，理论上需要一倍的物理内存来完成相应的操作。但是 Linux 具有写时复制技术 (copy-on-write)，父子进程会共享相同的物理内存页，当父进程处理写请求时会对需要修改的页复制出一份副本完成写操作，而子进程依然读取 fork 时整个父进程的内存快照。
如上图所示，fork 时只拷贝 page table，也就是页表。只有等到某一页发生修改时，才真正进行页的复制。
但是 Linux Kernel 在 2.6.38 内存增加了 Transparent Huge Pages (THP) 机制，简单理解，它就是让页大小变大，本来一页为 4KB，开启 THP 机制后，一页大小为 2MB。它虽然可以加快 fork 速度( 要拷贝的页的数量减少 )，但是会导致 copy-on-write 复制内存页的单位从 4KB 增大为 2MB，如果父进程有大量写命令，会加重内存拷贝量，都是修改一个页的内容，但是页单位变大了，从而造成过度内存消耗。例如，以下两个执行 AOF 重写时的内存消耗日志：
// 开启 THPC * AOF rewrite: 1039 MB of memory used by copy-on-write// 关闭 THPC * AOF rewrite: 9MB of memory used by copy-on-write
The two logs from the same Redis process, the total used_memory is 1.5GB, written during the second sub-process execution command in the amount of about 200. When are turned on and off THP, sub-process memory consumption are worlds apart. So turn at high concurrent write scenes THP, sub-process memory consumption may be several times the parent process, causing the machine physical memory overflow.
So, the child process does not need to consume Redis produce 1 times the parent process of memory, the actual amount determined based on consumption during the write command, you need to set aside some memory to prevent overflow. And THP advised to turn off the system to prevent excessive memory consumption during the copy-on-write. Not only is Redis, the machine will deploy MySQL generally closed THP.
Original micro-channel public number
Reference article
-
https://www.datadoghq.com/pdf/Understanding-the-Top-5-Redis-Performance-Metrics.pdf
-
Redis Administration https://redis.io/topics/admin