Source: www.cnblogs.com/wyc1994666/p/10669212.html
When I was looking for a job interview before, I got stuck on Redis related issues.
Before the interview, I prepared according to the usual routines, such as the five commonly used data structures of Redis, Redis persistence strategy, Redis implementation of distributed locks, simple publish and subscribe, and so on. I thought I was capsized on the first question asked in Redis after all~~
Interviewer: Look at your resume that you are familiar with commonly used data structures. What are there to talk about?
I: There are 5 commonly used, string, list, set, zset, and hash (very happy inside)
Interviewer: Then tell me which data structure you have used_I
: string is the most used, and json string is usually stored in_
Interviewer: Do you know how Redis implements its string internally? _
I: uh ~ I understand Redis is written in C language, as to the specific implementation is not clear ~
I'm here to die~~~
Do you have friends with the same experience?
After I went back, I made up for the knowledge of Redis about the principle. It happened to be the most recent summary of the interview experience, so I have today's article.
This article will talk about the following:
-
Implementation of Redis string
-
Performance advantages of Redis strings
Implementation of Redis string
Although Redis is written in C language, it does not directly use C language strings, but implements a set of strings by itself. The purpose is to improve speed and performance. It can be seen that Redis also takes great pains for high performance.
Redis built a simple dynamic string (Simple Dynamic String), referred to as SDS
1. SDS code structure
struct sdshdr{
// 记录已使用长度
int len;
// 记录空闲未使用的长度
int free;
// 字符数组
char[] buf;
};
SDS? What the hell? Friends who are unfamiliar may have doubts about this name. It's just a noun and don't care, we should focus on appreciating the design ideas of Redis. Draw a picture below to illustrate it at a glance.
Redis strings will also comply with the implementation rules of C language strings, that is, the last character is a null character. However, this null character will not be counted in len.
2. SDS dynamic expansion characteristics
The most powerful and wonderful thing about SDS is its Dynamic. Dynamically change the length. for example
As shown in the figure above, there are only 5 free seats in s1 at the beginning, and 6 characters of'world' need to be added later, which is obviously not enough. what is it now? Redis will do the following three operations:
-
Calculate whether the size is enough
-
Open up the space to meet the required size
-
Open up free free space with the same length as the used size len (if len <1M) Open up free free space with 1M length (if len >= 1M)
Have any friends seen here think that this implementation is a bit similar to Java's List implementation? After reading it, you will feel more like it.
Performance advantages of Redis strings
-
Get string length quickly
-
Avoid buffer overflow
-
Reduce the number of space allocations and improve memory usage efficiency
1. Get the string length quickly
Look at the SDS structure above:
struct sdshdr{
// 记录已使用长度
int len;
// 记录空闲未使用的长度
int free;
// 字符数组
char[] buf;
};
Since the used character length len is stored in the SDS, you can directly return len when you want to obtain the length of the string, and the time complexity is O(1). If you use a C language string, its string length acquisition function time complexity is O(n), n is the number of characters, because it is traversed and added from beginning to end (to the empty character'\0').
2. Avoid buffer overflow
When adding a string to a C language string with strcat, you need to open up the required space in advance. If you don't open up the space, it may cause buffer overflow and affect other codes of the program. As shown in the figure below, there is a string s1="hello" and a string s2="baby". Now strcat(s1,"world") is to be executed, and no space has been made for s1 before execution, which caused a buffer overflow.
For Redis, because every time a string is appended, it checks whether the space is enough, so there is no buffer overflow problem. Before each append operation, the following operations will be done:
-
Calculate whether the size is enough
-
Open up the space to meet the required size
3. Reduce the number of space allocations and improve memory usage efficiency
The append operation of strings will involve memory allocation problems, but memory allocation problems will involve memory partitioning algorithms and system calls, so if they occur frequently, performance will be affected, so this is absolutely unbearable for performance-oriented Redis.
So the following two optimization measures have been taken
-
Space and allocation
-
Inert space recycling
1. Pre-allocation of space
For append operations, Redis will not only open up enough space but also pre-allocate unused space (free) for the next operation. As for the size of the unused space (free), it is determined by the modified string length.
When the modified string length len <1M, unused space with the same length as len will be allocated (free)
When the modified string length len >= 1M, 1M unused space (free) will be allocated
With this pre-allocation strategy, the number of memory allocations will be reduced, because before the allocation will check whether the existing free space is enough, if it is enough, it will not be opened~
2. Inert space recovery
Contrary to the above situation, lazy space reclamation is suitable for string reduction operations. For example, if there is a string s1="hello world", perform the sdstrim(s1," world") operation on s1. After the operation is completed, Redis will not immediately reclaim the reduced part, but will allocate it to the next program that needs memory . Of course, Redis also provides an API for reclaiming memory, which can be manually called back and forth to shrink and reduce some of the memory.
It's over here~
Next time you encounter this problem, you can talk about it, hahaha~
------------------------------------------- < END >---------------------------------------
Xiaomu's blog