[Distributed] 02-Distributed lock principle and selection

When discussing the principle of distributed locks, we enter today's topic with the following thoughts:

  • How to choose a distributed lock model according to the business scenario: CP or AP?
  • Are read locks and locks atomic operations?
  • Whether the lock can be released normally to avoid deadlock (such as adding expiration time)
  • Ensure high availability of distributed locks: how to ensure that lock resources are not lost (the machine that holds the lock may be down)

The essence of locks is the serialization of shared resources. In a single-process environment, the Java JDK provides two implementations of mutual exclusion locks: Lock and Synchronized. These two locks add and unlock before and after the operation of the shared resource, ensuring that different threads can mutually exclusive and orderly operate the shared resource.

In a distributed environment, since different hosts cannot directly access shared resources, we need to implement distributed locks ourselves to ensure that there will be no resource preemption between different JVMs and different hosts.

First, the basic conditions of distributed locks

The basic conditions for implementing distributed locks are as follows:

  • Storage space: The realization of the lock requires a space where the lock can be stored. In multithreading, you can use memory to save locks; in multi-processes, you can use shared memory or files in disks as locks; in a distributed environment, different hosts cannot access each other's memory or disk files, so use external storage space for storage lock. Common ones include databases and caches such as Redis, MongoDB, ZK, etc.;
  • Unique identification: In a distributed environment, it is necessary to ensure that the name of the lock is globally unique;
  • Two states: the lock needs to have two different states, such as locked and unlocked; existing, non-existent, etc.;

The simplest implementation can use a database to implement a simple distributed lock:

Build a table:

CREATE TABLE `tb_lock` (
	`id` BIGINT NOT NULL AUTO_INCREMENT,
	`resource` bigint(64) NOT NULL COMMENT '锁定的资源',
  `status` tinyint(2) NOT NULL DEFAULT '0' COMMENT '锁资源状态:(0:解锁,1:加锁)',
	`desc` varchar(120) NOT NULL DEFAULT "" COMMENT '描述',
	`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
	PRIMARY KEY (`id`),
	UNIQUE KEY `uk_idx_resource` (`resource`) 
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='数据库分布式锁表';

Fake code:

// 1、读锁 (resourceId对应数据库resource字段,可以为用户编号、订单编号等其他场景)
lock = mysql.getByResourceId(resourceId);
// 2、判断锁状态(lock.status = 1是否为加锁状态 ,是表示锁被占用, sleep后再尝试)
while(lock.status == 1) {
    
    
    sleep(100);
}
// 3、加锁(更新lock.status = 1,表示占有锁)
mysql.update(lock.status = 1);
// 4、执行业务逻辑
doBizSomething();
// 5、解锁(更新lock.status = 1,表示解锁)
mysql.update(lock.status = 0);

The above is a distributed lock based on a simple implementation of the database. What might be the problem with such a distributed lock?

  • Locking non-atomic operation: Assume that request 1 performs steps 1 and 2 to read the lock and finds that lock.status != 1 indicates that the lock is not occupied, and then prepares to execute step 3 to prepare for locking (not yet executed), and assume at the same time Request 2 has also executed steps 1 and 2 to read the lock and found that the lock is not occupied. At this time, request 2 is also ready to execute the third step of the lock logic; then request 1 executes the third operation and lock.status = 1 means that the lock is locked. If it succeeds, request 2 also executes the third step to lock successfully, so that two requests get the lock at the same time, which violates the rules of distributed locks. The fundamental reason is that the two operations of read lock and lock are not atomic operations , so there is a situation where the same lock will be occupied by multiple requests.

  • Can the lock be released in time: Suppose that after request 1 executes step 3 and successfully obtains the distributed lock, the service is down when the business logic is processed in step 4, and the lock is not released in step 5. In this way, the distributed lock status is always occupied. Therefore, we need to consider that the host or service holding the lock can release the lock in time when there is a downtime or abnormality, to ensure that subsequent requests can acquire the lock normally, and to ensure the fairness of the lock.

  • Correctness of unlocking:

  • How to ensure that the stored lock resources are not lost: suppose that when request 1 successfully acquires a distributed lock, the data is lost after the database goes down after recovery, and request 2 can successfully acquire the same distributed lock;

    To solve this situation, we must consider using clusters to store distributed lock resources and synchronize lock resource data to prevent data loss. **However, there may be scenarios where data is inconsistent, which involves the use of AP or CP distributed locks.

Second, the typical implementation of distributed locks

  • 1、MySQL

  • 2 、 ZK

  • 3 、 Redis

  • 4、etcd

1. Based on MySQL database

When discussing the implementation of distributed locks, it is generally not considered to implement distributed locks based on MySQL database. The reason is that it depends on the database and the data needs to fall on the hard disk. Frequent reading of data will cause high IO overhead and low performance. It is suitable for business scenarios with low concurrency and low performance requirements. The advantage is that there is no need to introduce third-party components such as ZK and Redis.

What kind of distributed lock implementation can not be separated from the premise of the business scenario, so the distributed lock based on the MySQL database can be applied to some scenarios where the data consistency requirements are not very high, the request volume is not large, and the performance requirements are not high. . For example, I have done a background system for operations before. One of the functions requires that when one operation edits a certain page, other operations cannot be edited. In this scenario, the amount of concurrency is not high, and the requirements for data consistency are not high. You can implement a simple distributed lock based on the MySQL database, which is simple and easy to understand.
Insert picture description here
Optimistic lock based on MySQL:

In most cases, the optimistic lock task will not conflict. Only when the data is updated, it is compared with the expected data. If the data is consistent, the data is updated, otherwise it returns to fail.

Optimistic lock adds a version field to each data, reads the version number (version 1) when reading, first reads the version number (version 2) of the data to be updated when updating, and compares the two version numbers. Update if there is a change, otherwise the update will fail.

Let's look at the simple implementation of optimistic locking, add the version field to the business table

CREATE TABLE `tb_lock` (
	`id` BIGINT NOT NULL AUTO_INCREMENT,
	`goodId` bigint(64) NOT NULL COMMENT '商品id',
  `count` bigint(64) NOT NULL COMMENT '商品数量',
  `version` bigint(20) NOT NULL DEFAULT '0' COMMENT '版本号',
	`desc` varchar(120) NOT NULL DEFAULT "" COMMENT '描述',
	`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
  `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
	PRIMARY KEY (`id`),
	UNIQUE KEY `uk_idx_resource` (`resource`) 
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='数据库分布式锁表';

After the version field is introduced, the specific operations are as follows:

  • Thread 1 acquires the lock: Select resource, version from tb_lock where resource = xxx; (assuming version = 0 at this time)
  • At this stage, thread 2 may obtain the lock of commodity id=2 and +1 the quantity of commodity id=2. After execution, version+1, at this time version=1;
  • Thread 1 intends to also +1 the number of commodities with commodity id=2. The execution statement is update tb_lock set count= count+1 where goodId=2 and version = 0that because the version of the record of goodId=2 has been updated by thread 2, the update of thread 1 fails.

The above is the simple realization of optimistic locking. Of course, in addition to the version field, you can also use the timestamp: when updating, it is judged whether the obtained update timestamp is consistent with the update timestamp of the data to be updated.

The advantage of optimistic locking is that it does not need to rely on third-party components, nor does it rely on the locking mechanism of the database itself, and has less impact on the performance of the request, and it can directly fail when concurrency occurs.

The disadvantages are: business tables need to add additional fields to increase database redundancy; when the amount of concurrency is high, a large number of requests may request row locks for the same record, and the old database will generate greater read and write pressure.

Therefore, the database is suitable for scenarios with small concurrency and infrequent write operations.

2. Distributed lock based on ZK implementation

Zookeeper (ZK for short) is a high-performance, highly available distributed coordination service that can be used to solve distributed data consistency problems. Based on ZK, functions such as data publishing/subscription, load balancing, naming services, cluster management, distributed locks, etc. can be realized.

The ZK cluster is a CP model, which has the characteristics of atomicity, reliability, and sequential consistency.

Each data node in the distributed lock implemented based on ZK represents a lock, and the temporary sequence node that the client requests to successfully create represents the successful acquisition of the lock; when the client disconnects from the ZK cluster, the node will be automatically deleted. Represents the release of the lock. The following is the implementation logic of ZK distributed lock (shared lock):

Lock acquisition

  • All clients try to call create () method in the /distributed-locks/lockname/attempt to create a node under a temporary order of child nodes : such as /distributed-locks/order/00000001, /distributed-locks/order/00000002;
  • All clients will call the getChildren("lockname") method to get /distributed-locks/lockname/all the child nodes that have been created under the node;
  • After the client obtains the path of all child nodes, if it finds that the node created in step 1 has the smallest number of all child nodes, it means that the client has acquired the lock (read and write requests will be distinguished); otherwise, it will watch The largest node smaller than its own serial number, enters and waits until the next monitoring node changes, and then perform the above steps;

Lock release

The node created by the above client when acquiring the lock (such as:) /distributed-locks/order/00000001is a temporary sequence node. When the lock is released, the temporary sequence node is deleted and /distributed-locks/all registered child nodes under the node are notified . After the child node receives the change notification, it will initiate the above lock acquisition process again.

The lock release has the following two situations:

  • Due to the network interruption or downtime of the client machine that has acquired the lock, the temporary sequence child nodes created by the client on the ZK cluster will be deleted and the lock will be released;
  • After the client acquires the lock and executes the business logic normally, it actively deletes the created temporary sequence child nodes and releases the lock;

to sum up

The temporary node of ZK can avoid the problem that the client and ZK cluster cannot release the lock due to network interruption or downtime of the client host;

ZK's sequential nodes can avoid the herd effect (the ZK server sends a large number of event notifications to the client in a short time). Each lock requester will only watch the largest node with a smaller sequence number. When the lock is released, there will only be one lock requester. Will be notified.

3. Distributed lock based on Redis

Redis is a memory-based cache, and its distributed cache feature makes it one of the more common implementations of distributed locks.

The principle of Redis to implement distributed locks is: multiple processes set the same key concurrently, and only one process can set the lock successfully on behalf of the process to obtain the lock, and the other processes fail to set the same key and continue to try.

(1) How to ensure that the read lock and lock are atomic operations?

Redis provides the SetNX command, which means that when the specified key does not exist, the specified value is set for the key. If the setting is successful, a return of 1 means the locking is successful; if the setting fails, a return of 0 means the locking failed.

This process is atomic.

(2) How to ensure that the lock can be released normally and avoid deadlock?

When the client has a network abnormality or downtime, Redis cannot clear the lock state like ZK, but provides a way to ensure that the lock can be released in time and ensure the fairness of the lock so that other clients can have a chance. Apply for this lock.

Redis 2.6.12 and above provide the set key value NX PX millisecondsguarantee that the two steps of locking and setting the lock expiration time can be executed successfully at the same time.

NX represents the value specified when the specified key does not exist; PX represents the expiration time of the lock, in ms.

  • key: unique value, used as a lock, such as couponPool.id
  • value: Equivalent to the owner of the lock. Only the ovwner can unlock it when unlocking, preventing A from unlocking B's lock.
  • NX: set if not exist. That is, when the key does not exist, lock & set the validity period; if it exists, do not operate;
  • PX: indicates the expiration time of the lock, the unit is ms;
  • milliseconds: the value of the expiration time

(3) How to ensure fault tolerance, that is, lock resources are not lost?

If the Redis single-point mode is adopted, assuming that service S1 and service S2 apply for lock lock1 at the same time, Redis distributed lock will ensure that only one service applies for the lock, and the other application fails.

If Redis is down at this time and all the locks in the memory are lost, the service S2 successfully re-applies for the lock after restarting, but the business S1 still holds the lock. In this way, there is a situation where the same lock is occupied by multiple clients.

To solve the problem of data loss in Redis stand-alone mode is through the master-slave mode of Redis cluster. The Redis master-slave cluster will asynchronously synchronize the data of the master node to the Redis slave node. When the Redis master node goes down, the Sentinel sentinel mechanism and master-slave switching mechanism of the Redis cluster can ensure that the slave node is elected as the master node of the Redis cluster and continue to provide lock services to the outside world.

However, the distributed lock implemented by the Redis cluster mode has such a problem: the service S1 acquires the lock on the master node, and the service S2 cannot acquire the lock anymore. In extreme cases, the master node is down, and it happens that the lock data on the master node will not be synchronized to the slave node in the future. Then the Redis Sentinel sentinel mechanism elects the slave node to become the new master node, but there is no previous lock data on the new master node, which causes the service S2 to obtain the lock successfully. In this way, two clients have the same lock at the same time.

Sharing from the architectural level, the distributed lock is a CP model, and in any case, it is necessary to ensure that the data on all nodes is consistent. However, the distributed lock implemented by the master-slave mode of the Redis cluster is the AP model, so the above problems will occur.

4. Distributed lock based on etcd

etcd is a highly available distributed KV system. It adopts the consensus algorithm raft protocol and is implemented based on the Go language. It can be used to implement various distributed collaborative services.


reference

https://mp.weixin.qq.com/s/dnNUmlHR7-sjzjMYXID_Cw

https://mp.weixin.qq.com/s/Uya33qfxO0Xy3B76GmAHZQ

https://tech.meituan.com/2016/09/29/distributed-system-mutually-exclusive-idempotence-cerberus-gtis.html

https://segmentfault.com/a/1190000018826044?utm_source=tag-newest

Guess you like

Origin blog.csdn.net/noaman_wgs/article/details/106463981