Hello Redis have a few questions I want to ask you
Hello, Redis! We have been together for many years, from the vague realization that now we have deeply integrated, I have always known and always remember your goodness, can you ask me more questions to let me go deeper To get to know you.
1. What is the communication protocol of redis
The communication protocol of redis is a text protocol. Yes, the Redis server and the client communicate through RESP (REdis Serialization Protocol) protocol. Yes, the text protocol does waste traffic, but its advantages are intuitive, very simple, and parsing performance Fortunately, we don't need a special redis client to communicate with redis only by telnet or text stream.
Command format of the client:
- Simple Strings, start with "+" plus sign
- Errors, start with "-" minus sign
- Integer, starting with ":" colon
- Bulk Strings, start with "$" dollar sign
- Arrays of type Arrays, start with "*" asterisk
set hello abc
一个简单的文本流就可以是redis的客户端
复制代码
Brief summary
For details, see redis.io/topics/prot... The redis document believes that simple implementation, fast analysis, and intuitive understanding are the most important aspects of adopting the RESP text protocol. It is possible that the text protocol will cause a certain amount of traffic waste, but it is Performance and operation are fast and simple, and this is also a process of weighing and coordination.
2. Does redis have ACID transactions?
To find out if redis has affairs, it is actually very simple. Go to the official website of redis to check the document and find:
Redis does have transactions, but according to the traditional transaction definition ACID, does redis have the characteristics of ACID, ACID refers to 1. Atomicity 2. Consistency 3. Isolation 4. Durability, we will use The above redis transaction commands are used to verify whether redis has all the characteristics of ACID.
Atomicity
The atomicity of the transaction means that the database executes multiple operations in the transaction as a whole, and the service either executes all the operations in the transaction or does not execute one operation.
Transaction queue
First of all, after redis starts the transaction multi command, redis will generate a queue for this transaction, and the commands for each operation will be inserted into this queue in order. The commands in this queue will not be executed immediately, until the exec command commits the transaction. All commands in the queue will be executed once and exclusively.
Correspondence ->
As can be seen from the above example, when a successful transaction is executed, the commands in the transaction are executed sequentially and exclusively in the queue. But another feature of atomicity is that either all succeed or all fail, which is what we call rollback in traditional DB.
When we execute a failed transaction:
It can be found that even if there is a failure in the middle, the operation of set abc x has been executed, and no rollback has been performed. In a strict sense, redis is not atomic.
Why does redis not support rollback
This is actually related to the positioning and design of redis. First, let's see why our mysql can support rollback. This is still related to writing log. Redis will perform AOF logging after completing the operation. The positioning of AOF log is only recording operations. Command record, and mysql has a complete redolog, and redolog and binlog are written before the transaction commits
You must know that MySQL has spent a lot of money in order to be able to rollback. Redis application scenarios are more high-performance against high concurrency, so redis chooses a simpler and faster way to process transactions without rollback is also in line with the scenario.
consistency
Transaction consistency means that if the database is consistent before the transaction is executed, then after the transaction is executed, the database should be consistent regardless of whether the transaction is successful.
From the perspective of redis, there are two levels, one is whether the execution error is to ensure consistency, and the other is whether redis has a mechanism to ensure consistency when it is down.
Are execution errors ensured consistency
Still going to execute a wrong transaction, it will identify and deal with the error during the execution of the transaction. These errors will not modify the database, nor will it affect the consistency of the transaction.
The impact of downtime on consistency
Don't consider the distributed high-availability redis solution for the time being. First look at the single machine to see if the recovery from the downtime can satisfy the data integrity constraints.
Regardless of the RDB or AOF persistence solution, you can use the RDB file or AOF file to restore data, thereby restoring the database to a consistent state.
Discuss consistency again❓❓
The above view of the impact of execution errors and downtime on consistency is taken from Huang Jianhong's "Redis Design and Implementation". When reading this chapter, there are still some doubts. In the final analysis, redis is not a relational database. If only the ACID expression comes up In other words, consistency means that from the state A to the state B through the transaction without destroying various constraints, only redis is not discussed about the realized business, it is obviously satisfactory consistency.
But if you add business to talk about consistency, for example, A transfers money to B, A reduces 10 yuan, and B increases 10 yuan, because redis does not have rollback, it does not have the atomicity in the traditional sense, so from Redis should not have traditional consistency.
In fact, here is just a brief discussion on how to connect redis to the traditional ACID concept. Perhaps, it may be that I think too much. It is meaningless to use the ACID of the traditional relational database to audit redis. Redis has no intention to implement it. ACID's affairs.
Isolation
Isolation refers to the concurrent execution of multiple transactions in the database, each transaction will not affect each other, and the results of the transaction executed in the concurrent state and the transaction executed serially are exactly the same.
Because redis is a single-threaded operation, it has a natural isolation mechanism in terms of isolation. When redis executes a transaction, the redis server guarantees that the transaction will not be interrupted during the execution of the transaction, so the redis transaction is always serial The transaction is also isolated.
Endurance
The durability of a transaction means that when a transaction is executed, the results obtained by executing the transaction are stored in the persistent storage. Even if the server stops after the transaction is executed, the result of the executed transaction will not be lost. .
Whether redis has persistence depends on the persistence mode of redis
- Pure memory operation, no persistence, once the service is down, all data will be lost
- The RDB mode depends on the RDB strategy. bgsave will only be executed when the strategy is met. Asynchronous execution does not guarantee redis persistence
- Aof mode, only when appendfsync is set to always, the program will execute the command and save it to the disk synchronously. In this mode, redis has persistence
(Set appendfsync to always, but persistence is feasible in theory, but generally not in this way)
Brief summary
- Redis has a certain degree of atomicity, but does not support rollback
- Redis does not have the concept of consistency in ACID (or redis ignores this in its design)
- Redis has isolation
- Redis can ensure durability through certain strategies
Redis and ACID are purely thinking from the perspective of users. Redis design is more about pursuing simplicity and high performance, and will not be constrained by traditional ACID.
3. How is redis' optimistic lock watch implemented?
When we mention optimistic locking, we think of CAS (Compare And Set). The CAS operation contains three operands-the value of the memory location (V), the expected original value (A) and the new value (B). If the value of the memory location matches the expected original value, the processor will automatically update the location to the new value. Otherwise, the processor does nothing.
Use watch in redis transactions. Watch will stare at one or more key variables before the transaction starts. When the transaction is executed, that is, when the server receives the exec instruction to execute the cached transaction queue sequentially, Redis will check the key variables Has it been modified since the watch?
Optimistic locking mechanism of java's AtomicXXX
In Java, we often use some optimistic locking parameters, such as AtomicXXX. How are these mechanisms implemented? Whether redis is also the same as the implementation mechanism of Java CAS. Let’s take a look at the Java Atomic class first. Follow the source code, you can see that behind it is actually Unsafe_CompareAndSwapObject
You can see that compareAndSwapObject is a native method, you need to continue to track down, you can download the source code or open hg.openjdk.java.net/jdk8u/
cmpxchg
It can be found that tracing the final cas, "compare and modify", was originally two semantics, but in the end a cpu instruction cmpxchg was completed. cmpxchg is a CPU instruction command instead of multiple cpu instructions, so it will not be multi-threaded The scheduling is interrupted, so the operation of CAS can be guaranteed to be an atomic operation. Of course, the mechanism of cmpxchg actually has the problem of ABA and multiple retries, which is not discussed here.
Redis watch mechanism
Does redis watch also use cmpxchg? There are similarities between the two and there are some differences in usage. Redis watch does not have aba problems, and there is no multiple retry mechanism. One of the most important differences is:
The redis transaction execution is actually serial . Simply follow the source code: The extracted source code may be a bit messy, yes, you can simply summarize the data structure diagram and simple flowchart, and then look at the source code and it will be much clearer
storage
redisDb stores a watched_keys dcit structure. The value of each watched key is a linked list structure, which stores a set of redis client flags.
Process
This watched_keys structure will be queried for judgment every time watch, multi, exec, and every time the watched key is touched, it will be marked as CLIENT_DIRTY_CAS
Because all transactions in redis are serial, suppose that both client A and client B watch the same key. When client A performs touch modification or A is executed first, client A will be removed from this watched_keys Delete this key list and set all clients in this list to CLIENT_DIRTY_CAS. When the subsequent client B starts to execute, it judges that its state is CLIENT_DIRTY_CAS, and discardTransaction terminates the transaction.
Brief summary
The implementation of cmpxchg mainly uses cpu instructions. It seems that two operations are completed using one cpu instruction, so it will not be interrupted by multiple threads. The watch mechanism of redis uses more of the single-threaded mechanism of redis itself, and uses the watched_keys data structure and serial process to implement the optimistic lock mechanism.
4. How does redis persist?
There are two mechanisms for redis persistence. One is RDB, which is a snapshot. A snapshot is a full backup, which will serialize all redis memory data to disk. The other is the AOF diary. The AOF log records the instruction log of data operation modification, which can be analogous to the binlog of mysql. The AOF date will only increase infinitely over time.
When restoring redis, the RDB snapshot can be restored by directly reading the disk, while aof needs to replay all the operating instructions to restore. This process may be very long.
RDB
There are two methods for redis to generate RDB snapshots. One is save. Since redis is a single process and single thread, use save directly, redis will perform a huge file IO operation, because the single process and single thread will inevitably block online For business, generally speaking, save is not used directly, but bgsave is used. It has been said that redis is single-process and single-threaded. In fact, it is not. When using bgsave, redis will fork a child process, and the persistence of the snapshot is handed over to the child process. To process, and the parent process continues to process online business requests.
fork mechanism
If you want to figure out the principle of RDB snapshot generation, you must figure out the fork mechanism. The fork mechanism is a process mechanism of the Linux operating system. When the parent process forks a child process, the child process and the husband process have a common memory data structure. The child process just When generated, it shares the code and data segments in the memory with the parent process.
At the beginning, the two processes have the same memory segment. When the child process does data persistence, it will not modify the current memory data, but will use cow (copy on write) to separate the data segment pages. When the parent process modifies a certain data segment, the shared page will be copied and separated, and then the parent process will modify the new data segment.
Split
This process has also become a split process. Originally, the parent and child processes all point to many identical memory blocks, but if the parent process modifies one of the memory blocks, it will be copied out, split, and then performed on the new memory block. modify.
Because the child process can fix the memory when it forks, the data at this time point will not change, so we can safely generate the snapshot without worrying about the content of the snapshot being affected by the business request of the parent process. In addition, we can imagine that if During the bgsave process, redis does not have any operations, the parent process does not receive any business requests, and there is no backing such as expired removal. The parent process and the child process will use the same memory block.
AOF
AOF is the log storage of redis operation instructions, similar to the binlog for mysql. Assuming that AOF has been executed since redis was created, then AOF records the records of all redis instructions. If you want to restore redis, you can reorder AOF. The entire redis instance can be repaired, but the AOF log also has two major problems. One is that the AOF log will increase over time. If a large amount of data runs for a long time, the AOF log volume will become extremely large. Another problem is that when AOF is doing data recovery, the recovery time will be very long due to the huge amount of replay.
AOF write operation is that after redis has processed the business logic, some AOF logs will be saved according to a certain strategy. This is very different from mysql's redolog and binlog. In fact, for this reason, redis is because the processing logic is first After recording the operation log, it is also a reason that redis cannot be rolled back.
bgrewriteaof
In response to the above problems, redis also used bgrewriteaof to slim down the aof log after 2.4. The bgrewriteaof command is used to perform an AOF file rewriting operation asynchronously. Rewriting will create an optimized version of the current AOF file.
RDB and AOF mix and match model
When restoring redis, if we use the RDB method, because of the bgsave strategy, we may lose a lot of data. If we adopt the AOF model and recover through AOF operation log replay, replaying AOF logs will take much longer than RDB.
After redis4.0, in order to solve this problem, a new persistence mode was introduced, hybrid persistence, which combines rdb files and partially incremental AOF files. Rdb can use a longer-term storage strategy, and aof does not require It is a full amount of logs, and only needs to save the incremental aof log from the beginning of the previous RDB storage to the period of time. Generally speaking, this log amount is very small.
5. How does redis increase revenue and reduce expenditure in memory usage
Redis is different from other traditional databases. Redis is a pure memory database and stores data with some data structures. If memory is not controlled, redis is likely to cause the system to crash due to excessive data volume.
ziplist
127.0.0.1:6379> hset hash_test abc 1
(integer) 1
127.0.0.1:6379> object encoding hash_test
"ziplist"
127.0.0.1:6379> zadd z_test 10 key
(integer) 1
127.0.0.1:6379> object encoding z_test
"ziplist"When I first tried to open a small data hash structure and a zset structure, I found that their real structure type in redis is a ziplist. The ziplist is a compact data structure. Each element is a continuous memory. If the data structure enabled by redis is very small in redis, redis will switch to using compact storage for compressed storage.
For example, in the above example, we used a hash structure for storage. The hash structure is a two-dimensional structure, which is a typical structure in which space is exchanged for time. However, if the amount of data used is very small, the use of a two-dimensional structure will waste space, and the performance of time has not been greatly improved. It is better to directly use a one-dimensional structure for storage. When looking up, although the complexity It is O(n), but because the amount of data is small, it is also very fast to traverse, and it is faster than the query of the hash structure itself.
If the elements of the collection object continue to increase, or the value of a certain value is too large, this small object storage will also be upgraded to generate a standard structure. Redis can also define the conversion parameters of compact structure and standard structure in the configuration:
hash-max-ziplist-entries 512 # hash的元素个数超过512就必须用标准结构存储
hash-max-ziplist-value 64 # hash的任意元素的key/value的长度超过 64 就必须用标准结构存储
list-max-ziplist-entries 512
list-max-ziplist-value 64
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
set-max-intset-entries 512 quicklist
127.0.0.1:6379> rpush key v1
(integer) 1
127.0.0.1:6379> object encoding key
"quicklist"The quicklist data structure is a doubly linked list data structure introduced by redis in 3.2, and it is indeed a doubly linked list of ziplist. Each data node of the quicklist is a ziplist, and the ziplist itself is a compact list. If the quicklist contains 5 ziplist nodes, and each ziplist list contains 5 data, then from the outside, this quicklist contains 25 data items.
The structure of quicklist is simply summed up as a compromise between space and time:
- A doubly linked list can perform push and pop operations at both ends, but in addition to saving its own data at each node, it also saves two pointers, which adds additional memory overhead. Secondly, because each node is independent, the memory address is not continuous, and more nodes are prone to memory fragmentation.
- The ziplist itself is a contiguous memory, which has high storage and query efficiency. However, it is not conducive to modification operations. Each time the data changes, it will trigger a memory realloc. If the length of the ziplist is very long, a realloc will cause a large number of data copies.
Therefore, combining the advantages of ziplist and doubly linked list, quciklist was born.
Object sharing
Redis has built a reference counting method in its own object system, through which the reference counting information of the object can be tracked. In addition to releasing the object at an appropriate time, it can also be used as an object to share. For example, if key A creates a string with an integer value of 100 as a value object, at this time key B also creates a string object with the same integer value of 100 as a value object, then during the redis operation:
- Speaking of the database key pointer points to an existing value object
- Talk about the shared value object reference count plus one
If there are hundreds of keys pointing to the integer value 100 in our database, there are not only keys A and B, but a few hundred, then the redis server only needs the memory of one string object to save the original hundreds of string objects. Data that can only be stored in memory.
6. How does redis realize master-slave replication
Several definitions
- runID ID of the server running
- offset The replication offset of the master server and the offset of the replication from the server
- replication backlog The replication backlog buffer of the primary server
After redis2.8, use the psync command instead of the sync command to perform the synchronization operation of the replication. The psync command has two modes of complete resynchronization and partial resynchronization:
- The complete synchronization is used to handle the initial replication situation: the execution steps of the complete resynchronization are the same as the execution steps of the sync command. Both are synchronized by letting the master server create and send the rdb file, and send the write command saved in the buffer to the slave server .
- Partial resynchronization is used to handle re-replication after disconnection: when the slave server reconnects to the master server after the disconnection, the master service can send the write command executed during the disconnection of the master server to the slave server, and the slave server only needs to After receiving and executing these write commands, the database can be updated to the current state of the main server.
Complete resynchronization:
- The slave sends psync to the master, because it is the first time to send, without runid and offset
- The master receives the request and sends the runid and offset of the master to the slave node
- master generates and saves the rdb file
- The master sends the rdb file to the slave
- While sending the RDB operation, the write operation will be copied to the buffer replication backlog buffer, and sent from the buffer area to the slave
- The slave loads the data of the rdb file and updates its own data
If the network is jittered or the short-term disconnection also requires complete synchronization, it will cause a lot of overhead. These overheads include the time of bgsave, the time of RDB file transfer, the time of slave reloading RDB, if the slave has aof, also Will lead to aof rewriting. These are a lot of overhead, so a partial resynchronization mechanism is also implemented after redis2.8.
Partial resynchronization:
- An error occurred in the network, and the master and slave lost connection
- The master still writes data to the buffer buffer
- The slave reconnects to the master
- The slave sends its current runid and offset to the master
- The master will determine whether the offset sent by the slave to itself is in the buffer queue, if it exists, it will send continue to the slave, if it does not exist, it means that too much data may be wrong, the buffer has been emptied, and it needs to be repeated at this time Full copy
- The master sends the buffer data offset from the offset to the slave
- Slave gets data to update its own data
7. How does redis formulate an expired deletion strategy
When a key is in an expired state, in fact, the memory in redis is not removed from the memory in real time, but redis uses a certain mechanism to remove some expired keys, and then releases the memory. When a key is expired, when will redis delete it? There are three possibilities when it is deleted, and these three possibilities also represent the three different deletion strategies of redis.
- Timed deletion: While setting the elapsed time of the key, create a timer so that the key will be deleted immediately when the key expires.
- Lazy deletion: Let the key expire regardless, but every time the key is obtained from the key space, it will check whether the key has expired, and if it expires, delete the key.
- Periodic deletion: Every once in a while, the program will check the database and delete the expired keys in it. As for the number of expired keys to be deleted, it depends on the algorithm.
Timed delete
Set the expiration time of the key and create a timer. Once the expiration time comes, immediately operate the key. This is memory-friendly, but the cpu time is the most unfriendly, especially when the business is busy and expired keys In many cases, the operation of deleting expired keys will take up a large part of the CPU time. You must know that redis is a single-threaded operation. When the memory is not tight and the CPU is tight, the CPU time is wasted on deleting expired keys that have nothing to do with business. The above will affect the response time and throughput of the redis server. In addition, the creation of a timer needs to use the time event in the redis server, and the implementation of the time event is an unordered linked list, and the time complexity is O(n). Let the server create a large number of timers to implement the timing deletion strategy. It will have a greater performance impact, so regular deletion is not a good deletion strategy.
Lazy deletion
Contrary to timing deletion, the lazy deletion strategy is the most friendly to the cpu. The program will only check when the key is retrieved, which is a passive process. At the same time, lazy deletion is the least friendly to memory. When a key expires, as long as it is no longer taken out, the expired key will not be deleted, and the memory it occupies will not be released. Obviously, lazy deletion is not a good strategy. Redis is very dependent on memory and good memory. If some long-term keys are not accessed for a long time, it will cause a lot of memory garbage, and even become a memory leak.
When writing the execution data, the written key is expired through the expireIfNeeded function. Among them, expireIfNeeded does three things internally, namely:
- Check whether the key has expired
- Propagate the action of executing the past key to the slave node
- Delete expired key
Delete regularly
The above two deletion strategies, whether it is timed deletion or lazy deletion, have obvious defects in a single use. They either take up too much CPU time or waste too much memory. Periodic deletion strategy is an integration and compromise of the first two strategies
- The regular deletion strategy executes the delete expired key operation every once in a while, and reduces the impact of the delete operation on the cpu time by limiting the time and frequency of the deletion operation
- A reasonable deletion expiration key can be achieved by reasonable deletion execution time and frequency
to sum up
It can be said that redis can be described as broad and profound, simple seven-question or just blind people touching the elephant, or this time just touched an elephant trunk, or you should touch down the nose, next time you may touch an elephant ear, as long as you are willing to go down In-depth understanding and exploration, instead of just applying and not thinking, one day you will find out the elephant like redis.
[Note: Some chapters refer to and quote Huang Jianhong "Redis Design and Implementation"]
Author: Chen Yu Zhe
link: https: //juejin.im/post/5d29ac845188252cc75e2d5c
Source: Nuggets
copyright reserved by the authors. For commercial reprints, please contact the author for authorization. For non-commercial reprints, please indicate the source.