redis distributed lock, java implementation of setnx+lua script | Jingdong logistics technical team

1 Introduction

In the current work, in order to ensure high availability of services and deal with problems caused by single-machine deployment such as single point of failure and excessive load, multi-machine deployment is often used in the production environment. In order to solve the problem of data inconsistency caused by multi-computer room deployment, we often choose to use distributed locks.

At present, other more common implementation solutions are listed below:

1. Realize distributed lock based on cache (this article mainly uses redis to realize)
2. Realize distributed lock based on database
3. Realize distributed lock based on zookeeper

This article implements distributed locks based on redis cache, which uses the setnx command to lock, the expire command to set the expiration time, and the lua script to ensure transaction consistency. The Java implementation is partly based on the interface provided by JIMDB. JIMDB is a Redis-based distributed cache and high-speed key-value storage service independently developed by JD.com.

2 SETNX

Basic syntax:SETNX KEY VALUE

SETNX means SET if N ot e X ists, that is, the command sets the specified value for the key when the specified key does not exist.

KEY is the name of the key to be set

VALUE is the corresponding value of setting key

If the setting is successful, it returns 1; if the setting fails (key exists), it returns 0.

Therefore, we will choose to use SETNX to realize the distributed lock. When the Key exists, it will return the message that the lock failed.

 

The difference between SET and SETNX:

SET If the key already exists, it will overwrite the original value, regardless of the type

SETNX If the key already exists, it will return 0, indicating that setting the key failed

 



Comparison before and after Redis version 2.6.12:

Before version 2.6.12: Distributed locks cannot be implemented only with SETNX, and the expiration time needs to be set with the EXPIRE command, otherwise, the key will be valid forever. Among them, in order to ensure that SETNX and EXPIRE are in the same transaction, we need to use LUA scripts to complete the transaction implementation. (Because at the time of writing this article, JIMDB has not yet supported the SET key value [EX seconds|PX milliseconds] [NX|XX] [KEEPTTL] syntax, so this article still uses lua transactions)

After version 2.6.12: SET key value [EX seconds|PX milliseconds] [NX|XX] [KEEPTTL] syntax sugar can be used for distributed locks and supports atomic operations, no need to set the expiration time with the EXPIRE command.

 

3 LUA scripts

What is a LUA script?

Lua is a lightweight and compact scripting language written in standard C language and open in the form of source code. Its design purpose is to embed in application programs, thus providing flexible expansion and customization functions for programs.

Why do you need LUA scripts?

The lock implementation in this article is based on two Redis commands - and . To ensure the atomicity of the commands, we write these two commands into LUA scripts and upload them to the Redis server. The Redis server will execute the LUA script in a single thread to ensure that the two commands are not interrupted by other requests during execution.SETNXEXPIRE

Advantages of LUA scripts

• Reduce network overhead. Multiple requests for several commands can be combined into one script for one request
• High reusability. After the script is edited once, the same code logic can be used in multiple places, just pass in different parameters.
• atomicity. If it is expected that the execution of multiple commands will not be interrupted by other requests, or there will be a race condition, it can be implemented with LUA scripts, while ensuring the consistency of transactions.

Realization of Distributed Lock LUA Script

Assuming that only one order can be created at the same time, we can use it as the key value and as the value value. The expiration time is set in seconds.orderIduuid3

The LUA script is as follows, implemented by the eval/evalsha command of Redis:

-- lua加锁脚本
-- KEYS[1],ARGV[1],ARGV[2]分别对应了orderId，uuid，3
-- 如果setnx成功，则继续expire命令逻辑
if redis.call('setnx',KEYS[1],ARGV[1]) == 1 
    then 
      -- 则给同一个key设置过期时间
       redis.call('expire',KEYS[1],ARGV[2]) 
       return 1 
    else 
      -- 如果setnx失败，则返回0
       return 0 
end

-- lua解锁脚本
-- KEYS[1],ARGV[1]分别对应了orderId，uuid
-- 若无法获取orderId缓存，则认为已经解锁
if redis.call('get',KEYS[1]) == false 
    then 
        return 1 
    -- 若获取到orderId，并value值对应了uuid，则执行删除命令
    elseif redis.call('get',KEYS[1]) == ARGV[1] 
    then 
        -- 删除缓存中的key
    	return redis.call('del',KEYS[1]) 
    else 
        -- 若获取到orderId，且value值与存入时不一致，则返回特殊值，方便进行后续逻辑
        return 2 
end

[Note] According to the version of Redis, in the LUA script, when using redis.call('get',key) to determine that the cache key does not exist, you need to pay attention to whether the comparison value is boolean false or null.

 

根据 官方文档 ：Lua Boolean -> RESP3 Boolean reply (note that this is a change compared to the RESP2, in which returning a Boolean Lua true returned the number 1 to the Redis client, and returning a false used to return a null .

In RESP3, when redis cli returns a null value, lua will replace it with boolean type false.

 

Introduction to RESP3

RESP3 is a new feature of Redis6 and a new version of RESP v2. This protocol is used for request-response communication between clients and servers. Since the protocol can be used asymmetrically, that is, the client sends a simple request, and the server can return more complex and expanded related information to the client. The upgraded protocol introduces 13 data types, making it more suitable for database interaction scenarios.

4 Java Distributed Lock Implementation Based on JIMDB

Call class implementation code

SoRedisLock soJimLock = null;
try{
    soJimLock = new SoRedisLock("orderId", jimClient);
    if (!soJimLock.lock(3)) {
        log.error("订单创建加锁失败");
        throw new BPLException("订单创建加锁失败");
    }
} catch(Exception e) {
    throw e;
} finally {
    if (null != soJimLock) {
        soJimLock.unlock();
    }
}

Distributed lock implementation class code

public class SoRedisLock{

    /** 加锁标志 */
    public static final String LOCKED = "TRUE";
    /** 锁的关键词 */
    private String key;
    private Cluster jimClient;
    
    /**
     * lock的构造函数
     * 
     * @param key
     *            key+"_lock" (key使用唯一的业务单号)
     * @param
     *
     */
    public SoRedisLock(String key, Cluster jimClient)
    {
        this.key = key + "_LOCK";
        this.jimClient = jimClient;
    }
    
    /**
     * 加锁
     *
     * @param expire
     *            锁的持续时间（秒），过期删除
     * @return 成功或失败标志
     */
    public boolean lock(int expire)
    {
        try
        {
            log.info("分布式事务加锁，key：{}", this.key);   
            String lua_scripts = "if redis.call('setnx',KEYS[1],ARGV[1]) == 1 then " +
            		"redis.call('expire',KEYS[1],ARGV[2]) return 1 else return 0 end";
            String sha = jimClient.scriptLoad(lua_scripts);
            List<String> keys = new ArrayList<>();
            List<String> values = new ArrayList<>();
            keys.add(this.key);
            values.add(LOCKED);
            values.add(String.valueOf(expire));
            this.locked = jimClient.evalsha(sha, keys, values, false).equals(1L);
            return this.locked;
        } catch (Exception e){
        	throw new RuntimeException("Locking error", e);
        }
    }

    /**
     * 解锁 无论是否加锁成功，都需要调用unlock 建议放在finally 方法块中
     */
    public void unlock()
    {
        if (this.jimClient == null || !this.locked) {
            return ;
        }
        try {
        String luaScript = "if redis.call('get',KEYS[1]) == false then return 1 " +
        		"elseif redis.call('get',KEYS[1]) == ARGV[1] then " +
        		"return redis.call('del',KEYS[1]) else return 2 end";
        String sha = jimClient.scriptLoad(luaScript);
        if(!jimClient.evalsha(sha, Collections.singletonList(this.key), Collections.singletonList(LOCKED), false).equals(1L)){
        	throw new RuntimeException("解锁失败,key:"+this.key);
        }
        } catch (Exception e) {
                log.error("unLocking error, key:{}", this.key, e);
        	throw new RuntimeException("unLocking error, key:"+this.key);
        }
    }
}

Since we just use the key-value to do a locking action, the value is meaningless. Therefore, the value corresponding to the key in this article is given a fixed value. Jimdb provides an API for uploading scripts, and we upload lua scripts to the redis server through the scriptLoad() method. And use the evalsha() method to execute the script. The return value of evalsha() is the return value set in the script.

We pass the parameters into the script through the list, and correspond to the flags in the script. For example in the code above:

" " corresponds to the script in theorderId_LOCKKEYS[1]

" " corresponds to the script in theTRUEARGV[1]

" " corresponds to the script in the3ARGV[2]

[Note] If there are multiple keys in a script, you need to ensure that the hashtag in redis is enabled, in case the keys caused by fragmentation are not in the same fragmentation, and the "Only support single key or use same hashTag " exception occurs. Of course, you need to be cautious when enabling the hashtag , otherwise the uneven fragmentation will lead to the concentration of traffic and cause excessive pressure on the server.

Screenshot of the log in actual use



5 summary

Through the above introduction, we learned how to ensure the atomicity of multiple Redis commands. Of course, Redis transaction consistency can also be achieved by choosing Redis transaction (Transaction) operation. Jimdb also has an API that supports multi, discard, exec, watch, and unwatch commands for transactions. The reason why this article chooses to use LUA scripts for implementation is mainly because when Jimdb executes transactions, the traffic will only hit the main instance, and the load balancing of multiple instances will be invalid. More feasible solutions are waiting for your exploration, see you in the next document.

6 References

Redis distributed lock: https://www.cnblogs.com/niceyoo/p/13711149.html

Use Lua script in Redis: https://zhuanlan.zhihu.com/p/77484377

Redis Eval command: https://www.redis.net.cn/order/3643.html

LUA API： https://redis.io/docs/interact/programmability/lua-api/

Author: JD Logistics Mou Jiayi

Source: Reprinted from Yuanqishuo Tech by JD Cloud developer community, please indicate the source

Microsoft's official announcement: Visual Studio for Mac retired The programming language created by the Chinese developer team: MoonBit (Moon Rabbit) Bjarne Stroustrup, the father of C++, shared life advice Linus also dislikes messy abbreviations, what TM is called "GenPD" Rust 1.72.0 released , the minimum supported version in the future is Windows 10 Wenxin Yiyan opens WordPress to the whole society and launches the "100-year plan" . : Crumb green language V1.0 officially released

{{o.name}}

{{m.name}}