Table of contents
Expansion: What is a distributed lock?
Expansion: Mainstream implementation of distributed locks?
1. setnx command + expire command + del command
2. set key value EX seconds/PX milliseconds NX command + del command
3. set key value EX seconds/PX milliseconds NX command + lua script
5. Redisson framework + RedLock red lock
Expansion: Common commands of Redis:
Problem phenomenon:
Recently, I am learning about distributed locks, and learned the idea and related technologies of using Redis as a distributed lock, so I wrote this article based on my own understanding, and hereby record it. I hope it will be useful to everyone. The questions are as follows:
Redis distributed lock - with implementation principle and optimization process (Common command of Redis)!
Expansion: What is a distributed lock?
A lock for controlling and solving distributed transaction concurrency problems in distributed systems!
Expansion: Mainstream implementation of distributed locks?
[tips: If you are not interested in the content of this expanded node, just read point 3 (redis distributed lock) related to the topic of this article. If you are interested, you can find out. I will also add relevant knowledge articles in the future. You are welcome to leave a message to discuss or share your views with me at any time, thank you! ! ! 】
Currently, there are three mainstream implementations of distributed locks:
- Database lock : Databases basically have corresponding lock rules for read and write operations, which are used to ensure the concurrency of database transactions.
- Zookeeper distributed lock : A temporary sequential node tree is maintained in zookeeper , and the order of nodes in the node tree has the exclusive characteristics of locks. Before the thread acquires the locked resource, it first visits the temporary sequential root node , generates child nodes in order from the root node, and records the serial number and distributes it to the thread. If the serial number is the smallest among the child nodes, then the lock is successful. After the business logic is operated, the node corresponding to the midline program number of the temporary sequential node tree is deleted; otherwise, it waits and monitors the previous serial number node until the previous When a serial number node is deleted, it can be locked.
- Redis distributed lock: There is a rule (setnx) for inserting a key in redis: first determine whether a key exists, if it does not exist, you can insert data, and return 1, if it exists, you cannot insert data, and return 0. This rule has the exclusive characteristics of locks. Before the thread acquires the locked resource, it first goes to setnx for a key. If it returns 1, the lock is successfully operated and the business logic is completed, and the key is deleted; otherwise, it waits until it returns 1, and the lock can be added.
problem analysis:
OK, I believe everyone has understood why redis can be used as a distributed lock. Let me talk about the experimental process based on this idea:
[tips: Before starting the following content, I would like to remind everyone: In order not to affect the look and feel of the article as much as possible, I provide some common commands of Redis recorded by myself at the end of the article. Of course, there are also the usage of setnx command and expire command . Friends in need, please mention it yourself, it will be of great help to understand the following content, so I personally strongly recommend reading it in advance, of course, if you are a redis boss, please pretend I didn’t say it. 】
First of all, the simplest implementation of Redis distributed locks is nothing more than:
1. setnx command + expire command + del command
First use the setnx command to set a key. Since the key does not exist in the redis library, it returns 1 (success). Therefore, the setting is successful, which also means that the locking is successful. If the subsequent thread goes to lock (setnx the same key), it will return 0 (because the key already exists).
Then use the expire command to add an expiration time to the key. Otherwise, once the redis service goes down, the key will not expire forever, that is, it will be deadlocked.
At this point, you can start executing distributed transactions in the program to complete the business logic.
Finally, use the del command to delete the key we set, that is, to release the lock resource, so that other threads can acquire the lock.
Summary : It looks good, but there is obviously something wrong? Don't worry, come one by one!
Disadvantages : 1. Deadlock : The deadlock problem has not been completely solved, because setnx and expire are not an atomic (integral) operation, that is to say, it is possible that redis will be down after the setnx command and before the expire command, so If so, it will still cause a deadlock problem.
2. set key value EX seconds/PX milliseconds NX command + del command
In order to completely solve the deadlock problem, it is necessary to solve the atomicity of the two operations of setting the key and setting the expiration time . Redis provides a combined sentence pattern to realize this idea.
[tips: Let me remind you again that the usage of this combined sentence pattern is mentioned in the common commands of Redis at the end of the article . Interested friends, please mention it yourself. 】
OK, so far we can solve the atomicity problem by using this merged sentence instead of the above two-step operation, that is, solve the deadlock problem?
Summary : So what else is wrong? Of course there is, let’s move on to the next shortcoming!
Disadvantages : 2. Lock expiration and accidental deletion : When I get the business logic done, I need to del the lock. If the lock expires before I get the del command (you can't determine how long the expiration time needs to be added to the lock. Business processing, Network fluctuations, cpu performance, disk io, etc. are all unstable factors), and then this lock is acquired by other threads, and then you execute the del command, which will lead to: accidentally delete the lock set by other threads! ! !
Some people say that this problem can be solved by setting an appropriate expiration time for the lock. This is indeed true, but this time is not easy to grasp. There is always a feeling of hidden danger, because if you think about it, if this business The actual completion time does take a long time, right... so is there an absolute solution?
Of course there are, that is: first determine whether the lock is owned by the current thread, and then consider whether to execute the del command! For example, I set the value of this key to a random number (as long as it is different from the value set by other threads), then when I go to del it, first go to get the key to get the value and then compare it to see if it is me The value entered in the original set can be the value, whether the key is owned by the current thread, the specific java code implementation logic is as follows:
//判断锁是否为当前线程所持有
if (random.equals(jedis.get(key))) {//random：之前setnx key时的value值
jedis.del(key); //释放锁：同一个线程设置的key，才可删
}At this point , there is an absolute solution to the problem of accidental deletion of expired locks , but there is still a pit that has been stepped on that needs to be prevented. What is it?
That's the atomicity problem! Atomic problems are extreme problems. Although the probability is extremely small, there are hidden dangers. As logically rigorous science students, we naturally cannot ignore them. Same problem as mentioned above:
If I if judged that the current thread does hold the key, and before executing the del operation, the lock just expired and was acquired by other threads, then del it at this time still cannot solve the problem of lock expiration and accidental deletion .
3. set key value EX seconds/PX milliseconds NX command + lua script
Since there is no merged sentence pattern in redis that can realize the above two operations, in order to solve this atomicity problem, a powerful lua script is needed.
The role of the lua script is to allow users to write a set of redis commands, and then when the script is executed, the commands inside will ensure atomicity (integrity), either succeed or fail! ! !
Seeing this, it is estimated that some friends will find out: Isn't this just a business?
Yes, you can fully understand the function of lua script as realizing redis transactions, although its function is far more than that.
OK, then the current code needs to be modified as follows:
String luaScripts = "if redis.call('get',KEYS[1]) == ARGV[1] then redis.call('del',KEYS[1]) return 1 else return 0 end";
Object result = jedis.eval(luaScripts, Collections.singletonList(key), Collections.singletonList(random));
//判断是否成功
return result.equals(1L);So far, the problem of lock expiration and accidental deletion has been completely solved .
Summary: Stop talking, there must be another question.
Disadvantages: 3. The business logic has not been executed : Obviously, if the lock time setting is unreasonable, it will cause the business to expire before the execution is completed. Although this is not a technical problem, it is a real headache because the business is not After execution, it is equivalent to doing nothing, not paying wages, and wasting resources. Obviously, this is another problem caused by lock expiration.
Solution:
OK, anyone who knows me or has read my article should know that this part is the answer to the question in the article. So the answers to the questions in this issue, there are two solutions, which are applicable to two situations respectively , without further ado, let’s continue:
4. Redisson framework
OK, in order to completely solve the problem caused by the lock expiration time that has plagued everyone for a long time , today's highlight is here: There is a WatchDog (watchdog) function mechanism
in the Redisson framework , and its function is actually very simple; it is to lock After that, the default is to judge "whether the current thread holds a key (lock)" every 10s (it is said). If you still hold the key, then extend the expiration time of the key (how long? It is said that it can be set).
It's that simple, I feel that I can write a daemon thread in the locking thread to monitor the thread's ownership of the lock in real time, and then extend the expiration time accordingly! Just kidding, sorry, this WatchDog should have some underlying logic that I don't know to avoid some pitfalls that I didn't notice, but I don't have time to use the Redisson framework recently, you can see the words I wrote in this part of the copy (all use I heard about this word in the Internet, I have never used this word in my previous articles) and I know it.
Well, I have always adhered to the principle of practicing true knowledge, so I will definitely practice this Redisson framework when I have time later, and experience its power. Now you can only ask your friends to practice by themselves, so the code will not be provided.
Of course, since a recognized technical framework has been formed, the atomicity problems, deadlocks, and lock expiration and accidental deletion problems mentioned above must have been solved, and I am still confident in drawing conclusions.
So far, all the problems mentioned above can be completely solved through the Redisson framework!
Summary: So is it perfect? If you ask that, the answer is definitely not.
Disadvantages: 4. Lock duplication : This is the last question, of course, this question depends on the situation. In fact, in redis stand-alone mode, the Redisson framework is already perfect, but do you still remember our topic today? It is a distributed lock! In large-scale distributed projects, Redis must build a cluster. Since it is a cluster, this may happen: If I add a lock on the master node, the master will go down again. What is even more sad is The locked data has not yet been synchronized to other slave nodes. After the master-slave switch, the slave becomes the new master node. This means that there are other threads to add the same lock, and because the new master does not have the previous Lock information, the database queries the key, so the lock is successful again. Therefore, it will lead to: the current thread has successfully locked on the master before; after the master-slave switch, other threads also lock successfully on the new master and repeat the problem.
5. Redisson framework + RedLock red lock
In order to solve the lock duplication problem, you need to use RedLock red lock!
RedLock is a kind of joint lock. Its principle is a lot of words, so I won’t go into details. Interested partners can Baidu by themselves. Let me talk about my personal understanding and simplified principles here:
First, let's build a redis cluster (multiple master nodes + multiple slaves):
1. When a thread needs to be locked, it will initiate a lock request to all master nodes in the cluster in order.
2. Each master node will express whether it agrees by locking (equivalent to voting, with a timeout abstention mechanism).
3. Finally, if the number of votes > 1/2 of the total number of masters (as shown in the figure, the number of votes must be greater than 2.5, that is, at least 3 masters must agree), then the lock is successful, otherwise the lock fails, and if the lock fails, all masters must be unlocked.
The process should be very simple. Of course, this method will consume a lot of resources. After all, the greater the power, the greater the responsibility.
Look forward to its continuous optimization in the future! ! !
Summary: So is it really perfect now?
Disadvantages: In fact, I personally think that it is already very perfect. If I have to say it, it is resource consumption and technical cost .
Expansion: Common commands of Redis:
set key value：
Set the value of the key to value, overwrite if the key exists, ignore the type, and return no value.
setex key seconds value：
Set the value of the key to value, overwrite if the key exists, ignore the type, and set the expiration time in seconds, return OK if successful, and return error if failed.
psetex key milliseconds value：
Set the value of the key to value, overwrite if the key exists, ignore the type, and set the expiration time in milliseconds, return OK if successful, and return error if failed.
setnx (abbreviation for set if not exist) key value:
If the key exists, no operation will be performed, and 0 will be returned directly;
If the key does not exist, set the value of the key to value and return 1.
Due to the unique exclusivity of setnx, it meets the characteristics of a lock.
ttl key：
If the key does not exist, return -2;
If the key exists and is permanent/no expiration time, return -1;
If the key exists and has an expiration time, return the expiration time in seconds.
expire key seconds：
Set the expiration time for the key.
persist key：
Clear the key's expiration time to make it permanent.
set key value EX seconds/PX milliseconds NX/XX (default XX)
Set the value for the key and set the expiration time at the same time.
EX seconds/PX milliseconds: The unit used to specify the expiration time.
NX/XX: NX means that if the key does not exist, it will return OK, and if it exists, it will return nil. XX is the opposite of NX, so the NX mode is equivalent to the combination of setnx+expire commands.
Example:
Set the key to abc, the value is " 123 " , and the expiration time is 60 seconds:
set abc “123” EX 60
Set the key to abc, the value is " 123 " , and the expiration time is 60 seconds:
set abc “123” PX 60000
If the key does not exist, set the key to abc, the value is " 123 " , and the expiration time is 60 seconds:
set abc “123” EX 60 NX
If the key exists, set the key to abc, the value is " 123 " , and the expiration time is 60 seconds:
set abc “123” EX 60 XX
get key：
If it can exist and is of String type, return the value of key.
If the key exists but is not Stiring type, return an error error.
Returns nil if key does not exist.
getset key value：
Set the value of the key to value, overwrite if the key exists, and return the old value of the key.
Returns null if key does not exist.
If the value of the key is not a String type, an error error is returned.
unlink key...：
Delete one or more keys asynchronously (space separates multiple keys). It will not block the redis server (redis defaults to single-threaded), first delete the key in the namespace (database) and then return (at this time, the key value is null through get), and then release the space in the background (really delete from the memory, So this period is a memory leak).
del key...：
Synchronously delete one or more keys. This will cause blocking, so it is not recommended to use the del command to delete large keys (such as collections and advanced types of key values that may occupy several GB). In most scenarios, unlink will be used instead of del, unless you want to delete a key as soon as possible. The key whose space is rapidly increasing.