A text distributed lock and get to know the business scene

Reprinted chunqiu3351 Last post on 2019-07-15 23:55:00 reading number 1178 Collect

Why Distributed Lock?

Before discussing this issue, we look at a business scenario:

A system is an electricity supplier system, a machine currently is deployed, the system has a user interface to the next order, but must go to check stock before orders of users to ensure sufficient inventory will give the user orders.

Because the system has a certain degree of concurrency, it will advance the stock is stored in redis, a single user when updates redis inventory.

At this point the system architecture is as follows:

But this way causes a problem: If a certain time, a commodity inventory inside Redis is 1, and the two simultaneously incoming requests, wherein a request to perform the steps of FIG. 3, the inventory database is updated to 0 but step 4 has not been executed.

And another request to perform the step 2, or 1 inventory found, continue to step 3.

This result led to two of goods sold, but in fact, only one stock.

Obviously not ah! This is a typical problem oversold stocks

At this point, it is easy to think of solutions: with lock to lock-step 2,3,4, let them after the execution of another thread can come to step 2.

FIG accordance with the above, at step 2, using synchronized or ReentrantLock Java provided to lock before the lock is released after executing the step 4.

Thus, 2,3,4 step 3 is "locked" live, serialization can be performed between a plurality of threads.

But it did not last long, complicated by the entire system soared, a machine could not carry. A machine now increases, as shown below:

After increasing the machine, the system becomes as shown in the figure, my God!

Assuming that the arrival of two simultaneous user requests, but fell on different machines, then the two requests can be performed simultaneously, or the problem will appear oversold stocks.

why? A figure above because the two systems, running on two different JVM, and they have added locks only for their own thread inside the JVM effective for other JVM thread is invalid.

So the question here is: Java provides native locking mechanism failure in a multi-machine deployment scenarios

This is because the two machines is not the same locks plus a lock (two locks in different JVM inside).

Well, we just added a lock to ensure that both machines is the same lock, the problem is not solved it?

At this point, it's time distributed lock grand debut, distributed lock idea is:

Throughout a global system, the only acquire lock "stuff" and then locking each system when needed, to ask all this "stuff" to get a lock, so that different systems can be considered to get the same lock.

As for the "something" can be Redis, Zookeeper, can also be a database.

Not intuitive description of the text, we look at the chart:

Through the above analysis, we know that the stock is oversold scene using native Java lock mechanism in case of distributed deployment of the system can not guarantee thread-safe, so we need to use distributed lock solution.

So, how distributed lock it? Then read on!

Based on Redis achieve Distributed Lock

The above analysis of why you want to use distributed lock, and here we look at the specific distributed lock when landing should be how to deal with.

The most common type of scheme is to use Redis do Distributed Lock

Redis do use distributed lock thinking something like this: Set a value in redis said, locking and releasing the lock time to put the delete key.

Such a specific code is:

// 获取锁
// NX是指如果key不存在就成功，key存在返回false，PX可以指定过期时间
SET anyLock unique_value NX PX 30000
// 释放锁：通过执行一段lua脚本
// 释放锁涉及到两条指令，这两条指令不是原子性的
// 需要用到redis的lua脚本支持特性，redis执行lua脚本是原子性的
if redis.call("get",KEYS[1]) == ARGV[1] then
return redis.call("del",KEYS[1])
else
return 0
end

This approach has several major issues:

You must use the SET key value NX PX milliseconds command

If you do not, the first set value, and then set the expiration time, this is not atomic operations, it is possible to set the expiration time before the downtime will result in a deadlock (key perpetuate)
value should be unique

This is the time to unlock, need to verify the value is locked and only consistent delete key.

This is to avoid a situation: Suppose A lock is acquired, the expiration time 30s, this time after 35s, the lock has been automatically released, A to release the lock, but this time B may obtain the lock. A client can not remove the lock of the B.

In addition to considering how to achieve the client distributed lock, but also need to consider the deployment of redis.

redis There are three deployment methods:

Stand-alone mode
master-slave + sentinel election mode
redis cluster mode

Redis make use of distributed lock drawback: If a single deployment model, there will be a single point of the problem, as long as the redis failed. Lock to die.

The use of master-slave mode, the lock only when a node locked, even if done through sentinel high availability, but the master if master node fails, the switch occurred from this time there will be a possible loss of lock problems.

Based on the above considerations, in fact redis authors also take this into consideration, he proposed a RedLock the algorithm which means something like this:

The deployment model is the assumption redis redis cluster, a total of five master node, obtain a lock by following these steps:

Get the current timestamp in milliseconds
Attempts to set the lock on each master node, set the expiration time is short, generally tens of milliseconds
Attempt to establish a lock on the majority of the nodes, such as node 5 is three nodes requires (n / 2 +1)
Client computing time to establish a good lock, the lock time is less than if the establishment of the timeout, even if successful establishment
If the establishment fails lock, then click Delete this lock
As long as someone else has established a distributed lock, you have to continue to try to acquire the lock poll

But such this algorithm is still quite controversial, it will still exist many problems, we can not guarantee that locking process necessarily correct.

Another way: Redisson

In addition, implementation of distributed lock Redis, in addition to their own native api based redis client to achieve, you can also use open-source framework: Redission

Redisson is an enterprise-class open source Redis Client, also provides support for distributed lock. I also recommend that you use, and why?

Recall that the above said, if they write code by redis sets a value that is set by the following command.

SET anyLock unique_value NX PX 30000

Timeout set here is 30s, over 30s if I still without a complete business logic, key will expire, there may be other threads to acquire the lock.

As a result, it does not end the first thread execution of business logic, came in second thread thread-safety issues will arise. So we also need to maintain this extra expiration time, too much trouble ~

Redisson we look at is how to achieve? First use redission feel cool:

Config config = new Config();
config.useClusterServers()
.addNodeAddress("redis://192.168.31.101:7001")
.addNodeAddress("redis://192.168.31.101:7002")
.addNodeAddress("redis://192.168.31.101:7003")
.addNodeAddress("redis://192.168.31.102:7001")
.addNodeAddress("redis://192.168.31.102:7002")
.addNodeAddress("redis://192.168.31.102:7003");
RedissonClient redisson = Redisson.create(config);
RLock lock = redisson.getLock("anyLock");
lock.lock();
lock.unlock();

It is that simple, we only need to complete the distributed lock through its api in the lock and unlock, he helped us a lot of details to consider:

redisson all instructions are executed, redis support lua lua script by script execution atomicity
redisson a key set of default expiration time for the 30s, if a client holds a lock exceeds the 30s how to do?

redisson there is a watchdogconcept, which translates to the watchdog, it will lock after you get, every 10 seconds to help you put the key timeout to 30s

In this case, even if the lock would not have been holding key has expired, other threads to acquire a lock problem.
redisson of "watchdog" logic ensures that no deadlock occurs.

(If the machine is down, the watchdog also gone. At this point it will not extend the expiration date of the key, to automatically expire after 30s, other threads can get to lock)

Which slightly stick out here implementation code:

// 加锁逻辑
private <T> RFuture<Long> tryAcquireAsync(long leaseTime, TimeUnit unit, final long threadId) {
if (leaseTime != -1) {
return tryLockInnerAsync(leaseTime, unit, threadId, RedisCommands.EVAL_LONG);
}
// 调用一段lua脚本，设置一些key、过期时间
RFuture<Long> ttlRemainingFuture = tryLockInnerAsync(commandExecutor.getConnectionManager().getCfg().getLockWatchdogTimeout(), TimeUnit.MILLISECONDS, threadId, RedisCommands.EVAL_LONG);
ttlRemainingFuture.addListener(new FutureListener<Long>() {
@Override
public void operationComplete(Future<Long> future) throws Exception {
if (!future.isSuccess()) {
return;
}
Long ttlRemaining = future.getNow();
// lock acquired
if (ttlRemaining == null) {
// 看门狗逻辑
scheduleExpirationRenewal(threadId);
}
}
});
return ttlRemainingFuture;
}
<T> RFuture<T> tryLockInnerAsync(long leaseTime, TimeUnit unit, long threadId, RedisStrictCommand<T> command) {
internalLockLeaseTime = unit.toMillis(leaseTime);
return commandExecutor.evalWriteAsync(getName(), LongCodec.INSTANCE, command,
"if (redis.call('exists', KEYS[1]) == 0) then " +
"redis.call('hset', KEYS[1], ARGV[2], 1); " +
"redis.call('pexpire', KEYS[1], ARGV[1]); " +
"return nil; " +
"end; " +
"if (redis.call('hexists', KEYS[1], ARGV[2]) == 1) then " +
"redis.call('hincrby', KEYS[1], ARGV[2], 1); " +
"redis.call('pexpire', KEYS[1], ARGV[1]); " +
"return nil; " +
"end; " +
"return redis.call('pttl', KEYS[1]);",
Collections.<Object>singletonList(getName()), internalLockLeaseTime, getLockName(threadId));
}
// 看门狗最终会调用了这里
private void scheduleExpirationRenewal(final long threadId) {
if (expirationRenewalMap.containsKey(getEntryName())) {
return;
}
// 这个任务会延迟10s执行
Timeout task = commandExecutor.getConnectionManager().newTimeout(new TimerTask() {
@Override
public void run(Timeout timeout) throws Exception {
// 这个操作会将key的过期时间重新设置为30s
RFuture<Boolean> future = renewExpirationAsync(threadId);
future.addListener(new FutureListener<Boolean>() {
@Override
public void operationComplete(Future<Boolean> future) throws Exception {
expirationRenewalMap.remove(getEntryName());
if (!future.isSuccess()) {
log.error("Can't update lock " + getName() + " expiration", future.cause());
return;
}
if (future.getNow()) {
// reschedule itself
// 通过递归调用本方法，无限循环延长过期时间
scheduleExpirationRenewal(threadId);
}
}
});
}
}, internalLockLeaseTime / 3, TimeUnit.MILLISECONDS);
if (expirationRenewalMap.putIfAbsent(getEntryName(), new ExpirationEntry(threadId, task)) != null) {
task.cancel();
}
}

In addition, redisson also provides support for redlock algorithm,

Its usage is very simple:

RedissonClient redisson = Redisson.create(config);
RLock lock1 = redisson.getFairLock("lock1");
RLock lock2 = redisson.getFairLock("lock2");
RLock lock3 = redisson.getFairLock("lock3");
RedissonRedLock multiLock = new RedissonRedLock(lock1, lock2, lock3);
multiLock.lock();
multiLock.unlock();

summary:

This section analyzes the concrete floor scheme used as a distributed lock redis

As well as some of its limitations

Then introduces the client framework redisson of a redis,

This is what I recommend everyone to use,

Than it would be less care to write code to achieve a lot of details.

Based zookeeper achieve Distributed Lock

Common implementations of distributed lock which, in addition to using redis be implemented outside the zookeeper can also be achieved using a distributed lock.

Before mechanism introduced zookeeper (below instead of using zk) implement a distributed lock, first rough zk explain what it is:

Zookeeper is a method of providing configuration management, distributed collaboration and centralized naming service.

zk model is this: zk contains a series of nodes, called znode, as if each znode file system represents a directory, then znode there are some features:

Ordered node: If there is a current parent node /lock, we can create a child node in the parent node below;

zookeeper provides an optional order characteristic, for example, we can create a child node "/ lock / node-" and specify the order, then the zookeeper will automatically add an integer number based on the current number of child node when the child node is generated

That is, if a child is the first node is created, then the child node is generated /lock/node-0000000000, compared to the next node /lock/node-0000000001, and so on.
Temporary nodes: the client can establish a temporary node, at the end of the session or the session times out, zookeeper will automatically delete the node.
Event listeners: When reading the data, we can set the node while an event listener, when the node data or structural changes, zookeeper will notify the client. Current zookeeper following four events:

- Node Creation
- Delete Node
- Node data modifications
- Child node change

Based on the above, some of the characteristics zk, we can easily draw the floor plan to use zk implement a distributed lock:

Zk use of temporary nodes and ordered nodes, each thread gets the lock is to create a temporary order of nodes zk, such as in / lock / directory.
After you create a node is successful, get / all nodes under lock temporary directory, then judge whether the current node is the number of threads created all the nodes of the lowest node
If the current node is the smallest of threads created all the node number node, the lock acquisition success.
If the node is not the current thread to create all the smallest node number, the number of nodes on a node before adding an event listener.

For example, the current thread to get the node number is /lock/003, then a list of all nodes [/lock/001,/lock/002,/lock/003], then to /lock/002add an event listener to this node.

If the lock is released, it will wake up the next number of nodes, and then execute step 3 again, to determine whether their node number is minimal.

Such /lock/001release, /lock/002listening to time, then the node is set [/lock/002,/lock/003], the /lock/002minimum node ID, the acquired lock.

The whole process is as follows:

Concrete realization of ideas is such, as to how to write code, is more complicated here is not posted out.

Curator Introduction

Curator is a zookeeper open source client, but also provides an implementation of a distributed lock.

His use is relatively simple:

InterProcessMutex interProcessMutex = new InterProcessMutex(client,"/anyLock");
interProcessMutex.acquire();
interProcessMutex.release();

Distributed Lock achieve its core source code is as follows:

private boolean internalLockLoop(long startMillis, Long millisToWait, String ourPath) throws Exception
{
boolean haveTheLock = false;
boolean doDelete = false;
try {
if ( revocable.get() != null ) {
client.getData().usingWatcher(revocableWatcher).forPath(ourPath);
}
while ( (client.getState() == CuratorFrameworkState.STARTED) && !haveTheLock ) {
// 获取当前所有节点排序后的集合
List<String> children = getSortedChildren();
// 获取当前节点的名称
String sequenceNodeName = ourPath.substring(basePath.length() + 1); // +1 to include the slash
// 判断当前节点是否是最小的节点
PredicateResults predicateResults = driver.getsTheLock(client, children, sequenceNodeName, maxLeases);
if ( predicateResults.getsTheLock() ) {
// 获取到锁
haveTheLock = true;
} else {
// 没获取到锁，对当前节点的上一个节点注册一个监听器
String previousSequencePath = basePath + "/" + predicateResults.getPathToWatch();
synchronized(this){
Stat stat = client.checkExists().usingWatcher(watcher).forPath(previousSequencePath);
if ( stat != null ){
if ( millisToWait != null ){
millisToWait -= (System.currentTimeMillis() - startMillis);
startMillis = System.currentTimeMillis();
if ( millisToWait <= 0 ){
doDelete = true; // timed out - delete our node
break;
}
wait(millisToWait);
}else{
wait();
}
}
}
// else it may have been deleted (i.e. lock released). Try to acquire again
}
}
}
catch ( Exception e ) {
doDelete = true;
throw e;
} finally{
if ( doDelete ){
deleteOurPath(ourPath);
}
}
return haveTheLock;
}

In fact, analysis of distributed lock curator achieve the underlying principles and above is about the same. Here we use a chart describing in detail its principles:

summary:

This section describes the basic use of distributed lock zookeeperr realization of programs and open source client zk, a brief introduction to the principles of its implementation.

Advantages and disadvantages of the two schemes

After school finished two schemes to achieve distributed lock, this section needs to be discussed is the implementation redis and zk of their advantages and disadvantages.

For distributed lock redis is concerned, it has the following disadvantages:

It is simple and crude way to acquire the lock, the lock can not directly acquire constantly trying to acquire the lock, more consumption performance.
Another is the case, redis design positioning determines its strong consistency of the data is not, in some extreme cases, it may cause problems. The model is not sufficiently robust lock
Even with redlock algorithm to achieve, in some complex scenes, can not guarantee 100% no problem, the discussion on redlock can see How to do distributed locking
redis distributed lock, in fact, need to continue to try to acquire the lock yourself, compare consumption performance.

But on the other hand the use of distributed lock redis achieve very common in many companies, and have never suffered from the so-called "extremely complex scene" in most cases

So redis use as a distributed lock may well be a good program, the most important point is that the redis high performance, high concurrency can get support, release the lock operation.

For zk Distributed Lock terms:

zookeeper born design orientation is distributed coordination, strong consistency. Lock model robust, easy to use, suitable for distributed lock.
If you can not get a lock, just need to add a listener on it, do not have polling, small performance overhead.

But zk also has its disadvantages: If you have more frequent client application lock, the lock is released for zk clusters pressure will be relatively large.

summary:

In summary, redis and zookeeper has its advantages and disadvantages. We make technology selection as a reference when you can factor in these issues.

Author of some of the recommendations

By the previous analysis, two common program to achieve a distributed lock: redis and zookeeper, their strengths and weaknesses. How should the selection of it?

Personally, I achieved more respected lock zk:

Redis is possible because there are hidden dangers that may result in data wrong situation. However, how the choice depends on the specific scene in the company.

If the company there are zk clusters conditions, preferred zk achieved, but only if the company which redis clusters, no condition to build zk cluster.

In fact, it can also be achieved by redis, in addition to system designers may also take into account the system has been redis, but do not want reintroduced some cases external dependencies, you can choose redis.

This is based on the architecture of the system designer to consider the

Reproduced in: https: //my.oschina.net/woniuyi/blog/3074572