In the second half of the year, I used my spare time to study and analyze some of the Redis source code. This article analyzes the design ideas of redis from the four perspectives of network model, data structure and memory management, persistence and multi-machine collaboration. If there are any inaccuracies, I hope All the great gods pointed out.
Redis is a widely used cache component in the industry. The most intuitive way to study a component framework is to consider each step from the perspective of the application side. Starting from these steps can often be the fastest way to understand a component. Framework's design philosophy. Taking Redis as an example, whenever a request is initiated, how does Redis manage the network request, what data structure is used to organize and operate the memory after receiving the request, and how does the data dump to disk for persistence? Then how to synchronize and ensure consistency in a multi-machine environment... This article briefly describes the design of redis and my own experience from the four perspectives of network model, data structure design and memory management, persistence method and multi-machine.
1. Network Model
Redis is a typical Reactor-based event-driven model, single-process, single-threaded, and efficient frameworks are always similar. The network model is almost identical to spp's asynchronous model.
The Redis process is divided into three synchronization modules: accept request processor, response processor, and response processor. Each request needs to go through these three parts to obtain the download address .
Redis integrates a libevent/epoll/kqueue/selectvariety of event management mechanisms, and can freely choose the appropriate management mechanism according to the operating system version, among which libevent is the optimal mechanism.
The network model of Redis has all the advantages of the event-driven model, which is efficient and low-consumption. However, in the face of a long operation, the request cannot be processed, and the response can only be waited until the event is processed. I have also encountered such a scenario in the business before, deleting the full key-value in redis, the entire operation time Longer, all requests cannot be responded to during the operation. Therefore, a clear understanding of the network model will help you to avoid weaknesses in your business, reduce long-time-consuming requests, and use as many simple short-time-consuming requests as possible to maximize the power of the asynchronous model. In fact, this is also reflected in the design of Redis many times. a little.
2. Data structures and memory management
1. String
1.1 Structure
The Redis string is a secondary encapsulation of the original string in the C language. The structure is as follows:
struct sdshdr {
long len;
long free;
char buf[]; }; It can be seen that whenever a string is defined, in addition to the space for saving characters, Redis also allocates additional space for managing attribute fields.
1.2 Memory management method
Dynamic memory management mode, the biggest advantage of dynamic mode is that it can make full use of memory space and reduce memory fragmentation. At the same time, the disadvantage is that it is easy to cause frequent memory jitter. Usually, "space pre-allocation" and "lazy space" are used. Release" two optimization strategies to reduce memory jitter, and redis is no exception.
Every time the content of the string is modified, first check whether the memory space meets the requirements, otherwise it will be doubled or increased by M; when the content of the string is reduced, the memory will not be reclaimed immediately, but will be reclaimed on demand.
Regarding the optimization of memory management, the most basic starting point is the trade-off between wasting a little space and sacrificing some time. The core idea of STL, tcmalloc, and the arena mechanism of protobuf3 is "pre-allocation and late recovery", and Redis is the same.
1.3 Binary Safety
The identifier to determine whether the string ends or not is the len field, not the '\0' of the C language, so it is binary safe.
Safely store the serialized binary string of pb into redis.
In short, through the simple encapsulation of redis, the operation of redis strings is more convenient, the performance is more friendly, and some problems that users need to care about in C language strings are shielded.
2. Dictionary (Hash)
The bottom layer of the dictionary must be hash, and when it comes to hash, it must involve hash algorithm, conflict resolution, and hash table expansion and contraction.
2.1 hash algorithm
Redis uses the commonly used Murmurhash2. The Murmurhash algorithm can give the hash distribution under any input sequence, and the calculation speed is very fast. The previous requirement of Local-Cache for shared memory also took advantage of the advantages of Murmurhash to solve the problem of poor hash distribution of the hash function of the original structure.
2.2 hash conflict resolution
链地址法解决hash冲突，通用解决方案没什么特殊的。多说一句，如果选用链地址解决冲突，那么势必要有一个散列性非常好的hash函数，否则hash的性能将会大大折扣。Redis选用了Murmurhash，所以可以放心大胆的采用链地址方案。
2.3 hash扩容和缩容
维持hash表在一个合理的负载范围之内，简称为rehash过程。
rehash的过程也是一个权衡的过程，在做评估之前首先明确一点，不管中间采用什么样的rehash策略，rehash在宏观上看一定是：分配一个新的内存块，老数据搬到新的内存块上，释放旧内存块。
老数据何时搬？怎么搬？就变成了一个需要权衡的问题。
第一部分的网络模型上明确的指出Redis的事件驱动模型特点，不适合玩长耗时操作。如果一个hashtable非常大，需要进行扩容就一次性把老数据copy过去，那就会非常耗时，违背事件驱动的特点。所以Redis依旧采用了一种惰性的方案：
新空间分配完毕后，启动rehashidx标识符表明rehash过程的开始；之后所有增删改查涉及的操作时都会将数据迁移到新空间，直到老空间数据大小为0表明数据已经全部在新空间，将rehashidx禁用，表明rehash结束。
将一次性的集中问题分而治之，在Redis的设计哲学中体现的淋漓尽致，主要是为了避免大耗时操作，影响Redis响应客户请求。
3.整数集合
变长整数存储，整数分为16/32/64三个变长尺度，根据存入的数据所属的类型，进行规划。
每次插入新元素都有可能导致尺度升级（例如由16位涨到32位），因此插入整数的时间复杂度为O（n）。这里也是一个权衡，内存空间和时间的一个折中，尽可能节省内存。
4.跳跃表
Redis的skilplist和普通的skiplist没什么不同，都是冗余数据实现的从粗到细的多层次链表，Redis中应用跳表的地方不多，常见的就是有序集合。
Redis的跳表和普通skiplist没有什么特殊之处。
5.链表
Redis的链表是双向非循环链表，拥有表头和表尾指针，对于首尾的操作时间复杂度是O(1)，查找时间复杂度O(n)，插入时间复杂度O(1)。
Redis的链表和普通链表没有什么特殊之处。
三.AOF和RDB持久化
AOF持久化日志，RDB持久化实体数据，AOF优先级大于RDB。
1.AOF持久化
机制：通过定时事件将aof缓冲区内的数据定时写到磁盘上。
2.AOF重写
为了减少AOF大小，Redis提供了AOF重写功能，这个重写功能做的工作就是创建一个新AOF文件代替老的AOF，并且这个新的AOF文件没有一条冗余指令。（例如对list先插入A/B/C，后删除B/C，再插入D共6条指令，最终状态为A/D，只需1条指令就可以）
实现原理就是读现有数据库的状态，根据状态反推指令，跟之前的AOF无关。同样，为了避免长时间耗时，重写工作放在子进程进行。
3.RDB持久化
SAVE和BGSAVE两个命令都是用于生成RDB文件，区别在于BGSAVE会fork出一个子进程单独进行，不影响Redis处理正常请求。
定时和定次数后进行持久化操作。
简而言之，RDB的过程其实是比较简单的，满足条件后直接去写RDB文件就结束了。
四.多机和集群
1.主从服务器
避免单点是所有服务的通用问题，Redis也不例外。解决单点就要有备机，有备机就要解决固有的数据同步问题。
1.1 sync——原始版主从同步
Redis最初的同步做法是sync指令，通过sync每次都会全量数据，显然每次都全量复制的设计比较消耗资源。改进思路也是常规逻辑，第一次全量，剩下的增量，这就是现在的psync指令的活。
1.2 psync
部分重同步实现的技术手段是“偏移序号+积压缓冲区”，具体做法如下：
（1）主从分别维护一个seq，主每次完成一个请求便seq+1，从每同步完后更新自己seq；
（2）从每次打算同步时都是携带着自己的seq到主，主将自身的seq与从做差结果与积压缓冲区大小比较，如果小于积压缓冲区大小，直接从积压缓冲区取相应的操作进行部分重同步；
（3）否则说明积压缓冲区不能够cover掉主从不一致的数据，进行全量同步。
本质做法用空间换时间，显然在这里牺牲部分空间换回高效的部分重同步，收益比很大。
2.Sentinel
本质：多主从服务器的Redis系统，多台主从上加了管理监控，以保证系统高可用性。
3.集群
Redis的官方版集群尚未在工业界普及起来，下面主要介绍一下集群的管理体系和运转体系。
2.1 slot-集群单位
集群的数据区由slot组成，每个节点负责的slot是在集群启动时分配的。
2.2 客户请求
客户请求时如果相应数据hash后不属于请求节点所管理的slots，会给客户返回MOVED错误，并给出正确的slots。
从这个层面看，redis的集群还不够友好，集群内部的状态必须由客户感知。
2.3 容灾
主从服务器，从用于备份主，一旦主故障，从代替主。
通过Redis的研究，深刻体会到的一点就是：所有设计的过程都是权衡和割舍的过程。同样放到日常的工作和开发中也是如此，一句代码写的好不好，一个模块设计的是否科学，就从速度和内存的角度去衡量看是否需要优化，并去评估每一种优化会收益到什么，同时会损失什么，收益远大于损失的就是好的优化，这样往往对于开发和提升更有针对性，更能提高效率。