Preface
In the open source world, the typical representatives of high-performance services are Nginx and Redis. Looking at the source code of these two softwares, they are very concise and efficient, and they are also based on asynchronous network I/O mechanism, so for programmers or enthusiasts who want to learn high-performance services, study these two network services The source code is very necessary.
Nginx currently has many books on the market, but Redis is indeed very few. The Redis version has developed very fast in recent years, from the stable 2.x version to the 5.0 version which has added many excellent features. There is no data for systematic explanation of these features. However, the book "Redis 5 Design and Source Code Analysis" brought by the editor next to you fills a major gap in Redis 5.0 technology learning, and is an effective way for technical colleagues to deeply understand the Redis kernel implementation mechanism.
The editor will explain the document "Redis 5 Design and Source Code Analysis" from the three parts of the preface, the table of contents, and the main content. At the same time, I hope this article can help you learn, and hope that you will like it! !
Let's take a look at this Redis catalog first
main content
The content of this book is divided into three parts, a total of 22 chapters
The first article : Chapter 1 briefly introduces Redis, as well as the way of compiling, installing and studying Redis; Chapters 2 to 8 focus on SDS, jump tables, end-pressing lists, dictionaries, integer sets, quicklist and Stream data structures achieve.
The second : Chapter 9 Redis explains the life cycle of the process of implementation of the order, it is important to read; Chapters 10 through 19, respectively, explained the keys, strings, hash tables, lists, sets, ordered set, GEO, Implementation of HyperLog data flow related commands.
Part III : The first 20 to 22 simple chapter explains the persistence to achieve master-slave replication and clustering, did not begin in detail, hoping to bring the reader entry
Chapter 1, Chapter 1~Chapter 8
Chapter 1. This chapter first introduces the development process of Redis and the new features of Redis 5.0. Then it focused on how to read the Redis source code, and briefly introduced the installation and debugging methods of the Redis source code to lay the foundation for everyone to learn the subsequent chapters.
Chapter 2. This chapter introduces the data structure of SDS and the implementation of basic API. In the source code analysis process, we can know how the features of SDS are realized.
l) How is SDS compatible with C language strings? How to ensure binary security?
The buf in the SDS object is a flexible array. When called by the upper layer, the SDS directly returns buf. Since buf is a pointer directly to the content, it is compatible with C language functions. When the content is actually read, SDS will limit the read length through len instead of "i0", ensuring binary security.
2) What is special about sdshdr5?
sdshdr5 is only responsible for storing strings smaller than 32 bytes. In general, the storage of small strings is more common, so Redis further compresses the data structure of sdshdr5, puts the type and length of sdshdr5 into the same attribute, and uses the low 3 bits of flags to store the type and the high 5 bits to store the length. . When creating an empty string, sdshdr5 will be replaced by sdshdr8.
Due to space limitations, the editor will not introduce the content of the article in detail one by one, but will intercept the main content. Friends who need this "Redis 5 Design and Source Analysis" document will pick it up at the end of the article!
Chapter 3 , this chapter introduces the basic principles and implementation process of jump tables. The principle of the jump table is simple, and the average complexity of its query, insertion, and deletion is O(logN). The jump list is mainly used in the underlying implementation of ordered sets.
Chapter 4 , this chapter first introduces the storage structure of the compressed list, and then analyzes the basic operations of the compressed list in detail from the source level: creating a compressed list, inserting elements, deleting elements and traversing the compressed list, and finally analyzes the reasons for the chain update of the compressed list And the solution. Through the study of this chapter, everyone can have a deeper understanding of compressed lists.
Chapter 5 , this chapter will introduce one of the important data structures of Redis database-dictionary. What is a dictionary? How does Redis implement a dictionary? What are the basic operations and applications of a dictionary? Here are three questions to explain step by step.
Chapter 6, this chapter first introduces the storage structure of intset and verifies the actual storage method when a collection type is stored as intset through GDB, and then introduces the methods of adding, deleting and finding elements in intset. Finally, some common APIs and operational complexity of intsct are introduced.
Chapter 7 , this chapter mainly introduces the basic data structure quicklist commonly used in Redis, mainly introduces the general situation of quicklist and the underlying storage of compression. In addition, we introduced the basic operations of quicklist in detail, and described the changes in data storage in various situations. Finally, we give the common API interface of quicklist and its complexity.
Chapter 8 , this chapter mainly introduces the underlying implementation of Stream. First, it explains the two data structures Listpack and Rax that the Stream structure needs to rely on, and introduces the basic operations of these two structures in detail. After that, it further explained how Stream uses these two structures.
Part Two, Chapter 9~Chapter 19
Chapter 9 , this chapter first introduces some basic structures, such as object structure robj, client structure client, server structure redisServer, and command structure redisCommand. Finally, this chapter introduces the entire process of the server processing client command requests, including server startup monitoring, receiving command requests and breaking, executing command requests and returning command responses, etc., to lay the foundation for everyone to learn the subsequent chapters.
Chapter 10 , the commands introduced in this chapter do not need to determine the specific type, and can act on any type of key. Note that: the move command cannot work in cluster mode; some functions of the sort command (subcommand byigct) are limited, dcl and The use of lunlink should be differentiated. Like del, the commands that may cause the server to block when used include hgetall, lrange, smembers, flushall, flushdb, keys, etc., among which the first three commands are compared with the del command. Easily overlooked; flushall and flushdb have parameters that can be operated asynchronously. For details, please refer to the corresponding chapters.
Chapter 11 , this chapter introduces Redis string commands. The sct and get commands are the most commonly used commands in Redis. The bottom layer of the string command is realized by means of sds, and the data setting and acquisition are realized through the robj structure. The string key-value and timeout period are stored in the redisDb dictionary.
Chapter 12 , this chapter mainly introduces the underlying implementation of the hash-related commands provided by Redis. We first introduce the way Redis stores the hash structure, that is, ziplist or hash table. When the length of field-valuc is short and the number of fiecld-values is small, Redis uses ziplist for storage, otherwise it uses hash table. After that, I summarized the ease of use of Redis and the unified interface provided to the outside world after integrating these two structures. Finally, it explains in detail how Redis uses the above integrated interface to implement hash-related commands.
Chapter 13 , This chapter describes the implementation of the list commands in Redis. The data structure of the list account layer uses quicklist. First of this chapter
Introduced the basic concepts of stacks and queues, and how to implement stacks and queues through push/pop; secondly, introduced list blocking
The realization of the command, through the blpop command, explained the client blocking process and unblocked the client process, the process also
It is more complicated and requires readers to study carefully; finally introduces some common list operations and query commands.
Chapter 14 , this chapter introduces the various commands of collections in Redis. Commands include operations on a single set and operations between multiple sets. From the source code, we can see that the bottom layer of the collection is based on the two basic data structures of dict and intset. Most operations are discussed separately. The efficiency of insertion and deletion also depends on dict and intset. When learning the source code of a collection of commands, you may wish to combine the data structure learning in Chapter 5 and Chapter 6.
Chapter 15 , this chapter mainly explains the commands related to ordered sets. According to the size of the elements, the underlying implementation of ordered sets is divided into two types, one is ziplist, the other is dict and skiplist. Based on these three data structures, the basic operations of ordered sets, batch operations and set operations are analyzed. Hope to help readers understand and master the commands and principles related to ordered sets.
Chapter 16 , this section first introduces the development history of geohash algorithm, and then explains in detail the implementation of GEO related commands in Redis. In the process of introducing the implementation of Redis GEO-related commands, I also explained the wonderful bit manipulation algorithms, hoping to inspire readers.
Chapter 17 , the first section discusses the evolution of the base counting algorithm, from the initial LC algorithm to the LLC algorithm, and then to the HLL algorithm. The LC algorithm is more accurate when the base is small, LLC has an advantage when the base is large, and AC is a simple combination of LC and LLC, and HLL is a number of optimization improvements based on LLC. The second section explains the implementation of Redis's HLL algorithm, and the third section explains the implementation of the HypcrLogLog command.
Chapter 18 , this chapter explains the source code implementation of Stream-related commands. Due to space limitations, the implementation of some commands only gives a general overview. If you want to learn more about the commands, you can automatically view the source code of Redis 5.0.
Chapter 19 , this chapter introduces the implementation of transactions, publish-subscribe, and Lua scripts in Redis. Both things and Lua scripts can achieve atomicity, but Lua scripts are more powerful; publish and subscribe functions can also be implemented using the newly introduced Stream in Redis 5.0. For details, please refer to the introduction in the Strcam chapter of this book.
Through this chapter, you can have a deeper understanding of transactions, publish and subscribe and Lua scripts in Redis, so that you can better apply them to practice.
Part Three, Chapter 20~Chapter 22
Chapter 20 , this chapter introduces the two ways of Redis to achieve persistence, RDB and AOF. First, it introduces the implementation method of RDB and the specific format of RDB file, and analyzes the RDB file through an example. Secondly, it introduces the realization method of AOF and the realization of AOF rewriting. By comparing the advantages and disadvantages of AOF and RDB, finally introduced the implementation of Redis hybrid persistence.
Through the study in this chapter, we can understand the principle of Redis persistence, and make trade-offs on data security and performance according to the actual situation, and reasonably configure Redis persistence parameters.
Chapter 21 , this chapter first introduces the realization of the function of master-slave replication, from which you can learn Redis's optimized design ideas for master-slave replication. When introducing the implementation of master-slave replication source code, first introduces the definition of its main data variables, and finally introduces the realization of the main seven processes of master-slave replication in detail. I believe that through the study of the article, readers should have a deeper understanding of master-slave replication.
Chapter 22 , this chapter first introduces several problems that the cluster needs to solve, and then specifically introduces how Redis cluster solves these problems. It focuses on how to implement master-slave switching in a Redis cluster, the background and principles of replica drift, and the specific ideas of shard migration. Finally, 9 data packet formats for communication between Redis clusters are described in detail.
Friends who need this "Redis 5 Design and Source Code Analysis" document can receive it for free with the assistant VX below!
