1. Why choose Etcd
According to the official website, Etcd is a distributed and reliable Key-Value storage system, which is mainly used to store key data in distributed systems. At first glance, Etcd is somewhat similar to NoSQL database systems, but as a database, Etcd is by no means its strong point, and its read and write performance is far inferior to MongoDB, Redis and other Key-Value storage systems. "Let professional people do professional things!" As a highly available key-value storage system, Ectd has many typical application scenarios. This article will introduce one of Etcd's excellent practices: distributed locks.
1.1 Advantages of Etcd
At present, there are many open source software that can implement distributed locks, among which ZooKeeper is the most widely used and familiar to everyone, and there are also databases, Redis, Chubby, etc. However, if comprehensive consideration is given to reading and writing performance, reliability, usability, security, and complexity, Etcd, as a rising star, is undoubtedly the "leader" among them. It is completely comparable to the "famous" ZooKeeper in the industry. In some respects, Etcd even surpasses ZooKeeper. For example, the Raft protocol adopted by Etcd is simpler and easier to understand than the Zab protocol adopted by ZooKeeper.
As a CoreOS open source project, Etcd has the following characteristics.
- Simple: written in Go language, easy to deploy; supports cURL user API (HTTP+JSON), easy to use; open source Java client is easy to use;
- Security: optional SSL certificate authentication;
- Fast: While ensuring strong consistency, the read and write performance is excellent. For details, please refer to the official Benchmark data;
- Reliable: The Raft algorithm is used to achieve high availability and strong consistency of distributed system data.
1.2 Basic principles of distributed locks
In a distributed environment, multiple processes on multiple machines operate on the same shared resource (data, files, etc.). If mutual exclusion is not performed, there may be "negative balance deductions" or "commodity oversold". Condition. In order to solve this problem, a distributed lock service is required. First, let’s take a look at what conditions a distributed lock should have.
- Mutual exclusion: at any time, only one client can hold the same lock, thus ensuring that a shared resource can only be operated by one client at a time;
- Security: that is, no deadlock will be formed. When a client crashes while holding the lock and does not actively unlock it, the lock it holds can also be released correctly, and other clients can be locked later;
- Availability: When the node providing the lock service has an unrecoverable failure such as downtime, the "hot standby" node can take over the failed node and continue to provide services, and ensure that the data it holds is consistent with the failed node.
- Symmetry: For any lock, it must be locked and unlocked by the same client, that is, client A cannot unlock the lock added by client B.
1.3 Etcd implements the basis of distributed locks
Needless to say, Etcd’s high availability and strong consistency have been clarified in the previous chapters. This section mainly introduces the following mechanisms supported by Etcd: Watch mechanism, Lease mechanism, Revision mechanism and Prefix mechanism. It is these mechanisms that enable Etcd to achieve distributed ability to lock.
- Lease mechanism: the lease mechanism (TTL, Time To Live). Etcd can set a lease for the stored Key-Value pair. When the lease expires, the Key-Value will be invalidated and deleted; it also supports renewal. Through the client, the lease can be Renew before expiration to avoid Key-Value pair expiration. The Lease mechanism can ensure the security of distributed locks. A lease is configured for the key corresponding to the lock. Even if the lock holder fails to actively release the lock due to a fault, the lock will be automatically released when the lease expires.
- Revision mechanism: Each Key has a Revision number, which is incremented by one every time a transaction is performed, so it is globally unique. If the initial value is 0, if a put(key, value) is performed, the Revision of the Key becomes 1, and the same Perform the operation again, and the Revision will become 2; replace it with key1 and perform the put(key1, value) operation, and the Revision will become 3; this mechanism has a function: the order of the write operation can be known through the size of the Revision. When implementing distributed locks, multiple clients grab locks at the same time, and obtain locks sequentially according to the size of the Revision number, which can avoid the "herd effect" (also known as "shocking herd effect") and achieve fair locks.
- Prefix mechanism: the prefix mechanism, also known as the directory mechanism. For example, a lock named /mylock is written by two clients competing for it. The actual written keys are: key1="/mylock/UUID1" ,key2="/mylock/UUID2", where UUID represents a globally unique ID to ensure the uniqueness of the two keys. Obviously, the write operation will succeed, but the returned Revision is different, so how to judge who has obtained the lock? Query through the prefix "/mylock", and return a Key-Value list containing two Key-Value pairs, as well as their Revision. Through the Revision size, the client can judge whether it has acquired the lock. If the lock grab fails, it will wait for the lock Release (the corresponding Key is deleted or the lease expires), and then judge whether you can obtain the lock.
- Watch mechanism: that is, the monitoring mechanism. The Watch mechanism supports monitoring a fixed Key, and also supports monitoring a range (prefix mechanism). When the monitored Key or range changes, the client will receive a notification; when implementing a distributed lock , if the lock grab fails, the Key-Value list returned by the Prefix mechanism can be used to obtain the Key with the Revision smaller than itself and the smallest difference (called Pre-Key), and monitor the Pre-Key, because only if it releases the lock, can it be obtained by itself Lock, if you listen to the DELETE event of the Pre-Key, it means that the Pre-Key has been released and you already hold the lock.
2. Etcd implements distributed locks
2.1 Etcd-based distributed lock business process
The following describes the business process of using Etcd to implement distributed locks, assuming that the lock name set for a shared resource is: /anyrtc/mylock.
Step 1: Prepare
the client to connect to Etcd, and create a globally unique Key with /anyrtc/mylock as the prefix. Assume that the first client corresponds to Key="/anyrtc/mylock/UUID1", and the second one is Key="/anyrtc /mylock/UUID2"; the client creates a Lease for its own Key, and the length of the lease is determined according to the time-consuming business, assuming it is 15s.
Step 2: Create a scheduled task as the "heartbeat" of the lease
. While a client holds the lock, other clients can only wait. In order to avoid the lease becoming invalid during the waiting period, the client needs to create a scheduled task as the "heartbeat" to renew the lease. In addition, if the client crashes while holding the lock and the heartbeat stops, the Key will be deleted due to the expiration of the lease, thereby releasing the lock and avoiding deadlock.
Step 3: The client writes its globally unique Key into Etcd
for the Put operation, and writes the Key binding lease created in Step 1 into Etcd. According to the Revision mechanism of Etcd, it is assumed that the Revisions returned by the Put operations of the two clients are respectively 1, 2. The client needs to record the Revision to determine whether it has acquired the lock.
Step 4: The client judges whether the lock is obtained.
The client reads the Key-Value list with the prefix /anyrtc/mylock (with the Revision corresponding to the Key in the Key-Value), and judges whether the Revision of its own Key is the smallest in the current list. If If it is, it is considered to have acquired the lock; otherwise, it will listen to the delete event of the previous Revision Key in the list that is smaller than itself.
Step 5: Execute the business
After obtaining the lock, operate the shared resource and execute the business code.
Step 6: Release the lock
After completing the business process, delete the corresponding Key to release the lock.
2.2 Schematic diagram of Etcd-based distributed lock
According to the business process introduced in the previous section, the schematic diagram of Etcd-based distributed lock is as follows.
For the business flow chart, you can refer to this article "Zookeeper Distributed Lock Implementation Principle" .
2.3 Realize Etcd's distributed lock based on golang
Distributed lock EtcdLocker code
package etcd
import (
"context"
clientV3 "go.etcd.io/etcd/client/v3"
"log"
"os"
"time"
)
// EtcdLocker 分布式锁结构体
type EtcdLocker struct {
client *clientV3.Client // 连接到etcd的客户端实例
lease clientV3.Lease // 在etcd上的租约实例
leaseId clientV3.LeaseID
cancelFunc context.CancelFunc
option Option
}
// Option EtcdClient的配置选项
type Option struct {
ConnectionTimeout time.Duration // 连接到etcd的超时时间，示例：5*time.Second
LeaseTtl int64 // 租约时长，连接异常断开后，未续租的租约会在这个时间之后失效
Prefix string // 锁前缀
Username string // 用户名，可选
Password string // 密码，可选
Debug bool
}
// New 创建一把锁
// etcdEndpoints etcd连接信息，示例：[]string{"localhost:2379"}
// option 连接选项，包clientV3.Config中的配置项很多，我们其实用不到它们那么多，简化一下
func New(etcdEndpoints []string, option Option) (locker *EtcdLocker, err error) {
if option.Prefix == "" {
option.Prefix = "distribution_lock:"
}
if option.ConnectionTimeout <= 0 {
option.ConnectionTimeout = 5 * time.Second
}
if option.LeaseTtl <= 0 {
option.LeaseTtl = 5
}
config := clientV3.Config{
Endpoints: etcdEndpoints,
DialTimeout: option.ConnectionTimeout,
Username: option.Username,
Password: option.Password,
}
locker = &EtcdLocker{
option: option,
}
if locker.client, err = clientV3.New(config); err != nil {
return nil, err
}
var timeoutCtx, cancel = context.Background(), locker.timeoutCancel
timeoutCtx, cancel = context.WithTimeout(context.Background(), option.ConnectionTimeout)
defer cancel()
if _, err := locker.client.Status(timeoutCtx, etcdEndpoints[0]); err != nil {
return nil, err
}
//上锁并创建租约
locker.lease = clientV3.NewLease(locker.client)
var leaseGrantResp *clientV3.LeaseGrantResponse
// 第2个参数TTL，可以用于控制如果当前进程和etcd连接断开了，持有锁的上下文多长时间失效
if leaseGrantResp, err = locker.lease.Grant(context.TODO(), option.LeaseTtl); err != nil {
return nil, err
}
locker.leaseId = leaseGrantResp.ID
var ctx context.Context
// 创建一个可取消的租约，主要是为了退出的时候能够释放
ctx, locker.cancelFunc = context.WithCancel(context.Background())
var keepRespChan <-chan *clientV3.LeaseKeepAliveResponse
if keepRespChan, err = locker.lease.KeepAlive(ctx, locker.leaseId); err != nil {
return nil, err
}
// 续约应答
go func() {
for {
select {
case keepResp := <-keepRespChan:
if keepResp == nil {
if locker.option.Debug {
log.Printf("进程 %+v 的锁 %+v 的租约已经失效了", os.Getpid(), locker.leaseId)
}
return
} else { // 每秒会续租一次, 所以就会收到一次应答
if locker.option.Debug {
log.Printf("进程 %+v 收到自动续租应答 %+v", os.Getpid(), keepResp.ID)
}
}
}
}
}()
return locker, nil
}
func (locker *EtcdLocker) timeoutCancel() {
if locker.option.Debug {
log.Printf("进程 %+v 的锁操作撤销", os.Getpid())
}
}
// GetId 获得当前锁的内部ID
func (locker *EtcdLocker) GetId() int64 {
return int64(locker.leaseId)
}
// Acquire 获得锁
// lockerId 锁ID，推荐使用UUID或雪花算法，确保唯一性，防止复杂业务+大量数据的情况下发生锁冲撞
// 返回值：who 如果获得锁失败，此ID可以标示锁现在在谁手中（这个谁，来自于GetId()的返回值
// 换句话说，A进程获得锁之后，可以通过GetId知道自己的ID是多少，此时B进程获得锁失败，可以通过who返回值知道锁在A手中
func (locker *EtcdLocker) Acquire(lockerId string) (who int64, ok bool) {
var err error
// 在租约时间内去抢锁（etcd 里面的锁就是一个 key）
kv := clientV3.NewKV(locker.client)
// 创建事务
txn := kv.Txn(context.TODO())
// 定义锁的Key
var lockerKey = locker.option.Prefix + lockerId
// If 不存在 key，Then 设置它，Else 抢锁失败
txn.If(clientV3.Compare(clientV3.CreateRevision(lockerKey), "=", 0)).
Then(clientV3.OpPut(lockerKey, lockerId, clientV3.WithLease(locker.leaseId))).
Else(clientV3.OpGet(lockerKey))
var txnResp *clientV3.TxnResponse
if txnResp, err = txn.Commit(); err != nil {
return 0, false
}
if !txnResp.Succeeded {
return txnResp.Responses[0].GetResponseRange().Kvs[0].Lease, false
}
return 0, true
}
// Release 释放锁
func (locker *EtcdLocker) Release() error {
locker.cancelFunc()
if _, err := locker.lease.Revoke(context.TODO(), locker.leaseId); err != nil {
return err
}
return nil
}
EtcdLocker Test method
package etcd
import (
"fmt"
"log"
"os"
"sync"
"sync/atomic"
"testing"
"time"
)
var etcdEndpoint = []string{"192.168.1.111:2379"}
// 一把锁，开调试
func TestEtcdLockerOneAsDebug(t *testing.T) {
option := Option{
ConnectionTimeout: 5 * time.Second,
Prefix: "",
Debug: true,
}
if locker, err := New(etcdEndpoint, option); err != nil {
log.Fatalf("创建锁失败：%+v", err)
} else if who, ok := locker.Acquire("EtcdLockerOneAsDebug"); ok {
// 抢到锁后执行业务逻辑，没有抢到则退出
t.Logf("进程 %+v 持有锁 %+v 正在处理任务中...", os.Getpid(), locker.GetId())
time.Sleep(5 * time.Second) // 这是正在做的事情，假定耗时5秒
t.Logf("进程 %+v 的任务处理完了", os.Getpid())
// 手动释放锁，在后台应用服务中，也可以通过defer释放
if err := locker.Release(); err != nil {
log.Fatalf("释放锁失败：%+v", err)
} else {
time.Sleep(2 * time.Second)
}
} else {
t.Logf("获取锁失败，锁现在在 %+v 手中", who)
}
}
// 一把锁，不开调试带前缀
func TestEtcdLockerOneNoneDebugAndPrefix(t *testing.T) {
option := Option{
ConnectionTimeout: 3 * time.Second,
Prefix: "MyEtcdLocker",
Debug: false,
}
if locker, err := New(etcdEndpoint, option); err != nil {
log.Fatalf("创建锁失败：%+v", err)
} else if who, ok := locker.Acquire("EtcdLockerOneNoneDebugAndPrefix"); ok {
// 抢到锁后执行业务逻辑，没有抢到则退出
t.Logf("进程 %+v 持有锁 %+v 正在处理任务中...", os.Getpid(), locker.GetId())
time.Sleep(5 * time.Second) // 这是正在做的事情，假定耗时5秒
t.Logf("进程 %+v 的任务处理完了", os.Getpid())
// 手动释放锁，在后台应用服务中，也可以通过defer释放
if err := locker.Release(); err != nil {
log.Fatalf("释放锁失败：%+v", err)
} else {
time.Sleep(1 * time.Second)
}
} else {
t.Logf("获取锁失败，锁现在在 %+v 手中", who)
}
}
// 一把锁，多任务（多请求）竞争锁，
// 此测试用例还可以通过命令 go test -run="TestEtcdLockerMultiTask" 开多个进程进行并行竞争测试
// 多进程测试时的结果验证方法，条件：多个测试只要有一个未完成，预期结果是：获取锁失败，successCount的值就是0
func TestEtcdLockerMultiTask(t *testing.T) {
const taskCount = 5
option := Option{
ConnectionTimeout: 3 * time.Second,
Prefix: "MyEtcdLocker",
Debug: false,
}
var successCount int64 = 0
var wg sync.WaitGroup
for i := 0; i < taskCount; i++ {
wg.Add(1)
go func(taskId int) {
defer wg.Done()
if locker, err := New(etcdEndpoint, option); err != nil {
log.Fatalf("[%+v]创建锁失败：%+v", taskId, err)
} else if who, ok := locker.Acquire("EtcdLockerMulti"); ok {
// 抢到锁后执行业务逻辑，没有抢到则退出
t.Logf("[%+v]进程 %+v 持有锁 %+v 正在处理任务中...", taskId, os.Getpid(), locker.GetId())
atomic.AddInt64(&successCount, 1)
time.Sleep(5 * time.Second) // 这是正在做的事情，假定耗时5秒
t.Logf("[%+v]进程 %+v 的任务处理完了", taskId, os.Getpid())
// 手动释放锁，在后台应用服务中，也可以通过defer释放
if err := locker.Release(); err != nil {
log.Fatalf("[%+v]释放锁失败：%+v", taskId, err)
} else {
time.Sleep(1 * time.Second)
}
} else {
t.Logf("[%+v]获取锁失败，锁现在在 %+v 手中", taskId, who)
}
}(i)
}
wg.Wait()
if successCount != 1 {
t.Fatalf("进程 %+v 的分布式锁功能存在BUG", os.Getpid())
}
}
// 多把锁，多任务（多请求），各有各的锁
func TestEtcdLockerMultiBusinessMultiLocker(t *testing.T) {
const taskCount = 5
option := Option{
ConnectionTimeout: 3 * time.Second,
Prefix: "MyEtcdLocker",
Debug: false,
}
var successCount int64 = 0
var wg sync.WaitGroup
for i := 0; i < taskCount; i++ {
wg.Add(1)
go func(taskId int) {
defer wg.Done()
if locker, err := New(etcdEndpoint, option); err != nil {
log.Fatalf("[%+v]创建锁失败：%+v", taskId, err)
} else if who, ok := locker.Acquire(fmt.Sprintf("EtcdLockerMulti_%d", taskId)); ok {
// 抢到锁后执行业务逻辑，没有抢到则退出
t.Logf("[%+v]进程 %+v 持有锁 %+v 正在处理任务中...", taskId, os.Getpid(), locker.GetId())
atomic.AddInt64(&successCount, 1)
time.Sleep(8 * time.Second) // 这是正在做的事情，假定耗时8秒
t.Logf("[%+v]进程 %+v 的任务处理完了", taskId, os.Getpid())
// 手动释放锁，在后台应用服务中，也可以通过defer释放
if err := locker.Release(); err != nil {
log.Fatalf("[%+v]释放锁失败：%+v", taskId, err)
} else {
time.Sleep(1 * time.Second)
}
} else {
t.Logf("[%+v]获取锁失败，锁现在在 %+v 手中", taskId, who)
}
}(i)
}
wg.Wait()
if successCount != taskCount {
t.Fatalf("进程 %+v 的分布式锁功能存在BUG", os.Getpid())
}
}
func TestEtcdLocker_GetId(t *testing.T) {
option := Option{
ConnectionTimeout: 3 * time.Second,
Prefix: "EtcdLocker_GetId",
Debug: true,
}
if locker, err := New(etcdEndpoint, option); err != nil {
log.Fatalf("创建锁失败：%+v", err)
} else if who, ok := locker.Acquire("EtcdLocker_GetId"); ok {
// 抢到锁后执行业务逻辑，没有抢到则退出
t.Logf("进程 %+v 持有锁 %+v 正在处理任务中...", os.Getpid(), locker.GetId())
time.Sleep(2 * time.Second) // 这是正在做的事情，假定耗时2秒
t.Logf("进程 %+v 的任务处理完了", os.Getpid())
// 手动释放锁，在后台应用服务中，也可以通过defer释放
if err := locker.Release(); err != nil {
log.Fatalf("释放锁失败：%+v", err)
} else {
time.Sleep(1 * time.Second)
}
} else {
t.Logf("获取锁失败，锁现在在 %+v 手中", who)
}
}
reference documents
Etcd official website
Etcd
technology article excerpt
Realize distributed lock based on etcd