It seems that I haven't written a blog for a long time. Taking advantage of this period of leisure, I would like to summarize the concurrency problems encountered in the business system development process and the solutions. I hope to help you: grin:
Problem recurrence
1. "The strange clone of device A"
Time goes back to a late night a long, long time ago. At that time, the multimedia advertisement playback control system I developed was just put into production and went online. In the first offline fresh food store opened by the company, dozens of large and small multimedia hardware devices were normally connected to the Internet. After that, I was registering one by one and connecting to the already online multimedia advertising broadcast control system.
A brief description of the registration process is as follows:
After each device is registered in the system, a corresponding record will be added to the database device table to store various information of this device.
Everything was going on in an orderly manner, until the registration of Device A broke this tacit tranquility...
After device A was registered, I suddenly discovered that two new records were added to the database device table , and they were two identical records!
I began to think I was dazzled...
A closer look reveals that there are indeed two new additions, and even the unique device identification (crossed out, to be tested later) and creation time are exactly the same!
Looking at the screen, I was lost in thought...
Why are there two?
In my registration logic, before dropping the library, the database will first check whether the device already exists, if it exists, update the existing one, and add it if it does not exist.
So I am puzzled. According to this logic, where did the second identical data come from?
2. Concurrent requests behind the truth
After some investigation and thinking, I found that the problem may lie in the registration request.
When device A sends an http registration request to the cloud, it may send multiple identical requests at the same time.
The cloud server was deployed on multiple Docker containers at the time. By checking the logs, it was found that two containers received the registration request from device A at the same time.
From this, I speculate:
Device A sent two registration requests at the same time. These two requests were sent to different containers in the cloud at the same time. According to my registration logic, after the two containers received the registration request, they simultaneously queried the device table in the database. , At this time, there is no record of device A in the device table, so the two containers have performed the new operation. Because the speed is very fast, the two new records are created in the second accurate to the creation time, and it does not reflect difference.
3. Concurrently added extensions
Since concurrent new operations will cause problems, will there be problems with concurrent update operations?
Solution
Resolve concurrent addition
1. The database unique index (UNIQUE INDEX)
When building a database table, create a unique index for unique fields (such as the above-mentioned unique device identifier), or create a joint unique index for several fields that are unique after being combined.
In this way, during concurrent addition, as long as one addition succeeds, other addition operations will fail due to the exception (java.sql.SQLIntegrityConstraintViolationException) thrown by the database. We only need to deal with the failure of the addition.
注意唯一索引的字段需要非空，因为字段值为空时会导致唯一索引约束失效
2. java distributed lock
By introducing distributed locks into the program, the distributed locks need to be acquired before new operations are performed, and the acquisition is successful to continue, otherwise the addition fails.
This can also solve the problem of data duplication caused by concurrent inserts, but the introduction of distributed locks also increases the complexity of the system. If there are unique fields on the data to be stored in the database, it is recommended to use a unique index method.
In the process of building distributed locks, we need to use Redis. Here we take the distributed locks used during device registration as an example.
A simple question and answer for distributed locks:
Q: What exactly is a lock?
A: The lock is essentially a string that is stored in Redis and is generated based on specific rules (fixed prefix + unique device identifier in the example), which is equivalent to having its own corresponding lock when registering a device, because the lock There is only one. Even if the device has multiple identical registration requests coming at the same time, only the one request that acquired the lock can successfully go on.
Q: What is acquiring a lock?
A: For the same device, the string generated based on the same rules (the string will be referred to as Key in the following text) is always the same. Before performing the new operation, go to Redis to check whether the Key exists. If it already exists, It means that the lock acquisition failed; if it does not exist, the Key will be stored in Redis. If the storage succeeds, it means the lock acquisition is successful, and if the storage fails, it still means the lock acquisition failed.
Q: How does the lock work?
A: As mentioned earlier, the string (Key) generated based on the same rules for the same device is always the same. Before the current thread performs a new operation, first check whether the Key exists in Redis. If it already exists, it means At this point, another thread has successfully acquired the lock and is doing the new operation that the current thread wants to do, so the current thread does not need to perform subsequent operations (yes, you are redundant)
When this Key does not exist, it means that no other thread has acquired the lock, and the current thread can proceed to the next step-save the Key in Redis quickly. When the key storage fails, it means that another thread preempts. The key is saved and the lock is successfully obtained, the current thread is one step late, and the work that you want to do is preempted by others (the current thread can be retired)
If and only if saving this Key in Redis is also successful, it means that the current thread has finally obtained the lock successfully, and you can safely perform the following new operations. During the period, other threads that want to do the same new operation cannot obtain the lock. You can only leave the scene and bye bye: wave:, remember to release the lock after the current thread is executed (delete the Key from Redis).
The distributed lock code used during registration is as follows:
public class LockUtil {
// 对redis底层set/get方法进行了简单封装的工具类
@Autowired
private RedisService redisService;
// 生成锁的固定前缀，从配置文件读取值
@Value("${redis.register.prefix}")
private String REDIS_REGISTER_KEY_PREFIX;
// 锁过期时间：即获取锁后线程能进行操作的最长时间，超过该时间后锁自动被释放（失效），别人可以重新开始获取锁进行对应操作
// 设定锁过期时间是为了防止某线程成功获取锁后在执行任务过程中发生意外挂掉了造成锁永远无法被释放
@Value("${redis.register.timeout}")
private Long REDIS_REGISTER_TIMEOUT;
/**
* 获取设备注册时的分布式锁
* @param deviceMacAddress 设备的Mac地址
* @return
*/
public boolean getRegisterLock(String deviceMacAddress) {
if (StringUtils.isEmpty(deviceMacAddress)) {
return false;
}
// 获取设备对应锁的字符串（Key）
String redisKey = getRegisterLockKey(deviceMacAddress);
// 开始尝试获取锁
// 如果当前任务锁key已存在，则表示当前时间内有其他线程正在对该设备执行任务，当前线程可以退下了
if (redisService.exists(redisKey)){
return false;
}
// 开始尝试加锁，注意此处需使用SETNX指令（因为可能存在多个线程同时到达这一步开始加锁，使用SETNX来确保有且仅有一个设置成功返回）
boolean setLock = redisService.setNX(redisKey, null);
// 开始尝试设置锁过期时间，到了过期时间线程还没有释放锁的话，由保存锁的Redis来确保锁最终被释放，以免出现死锁
// 锁过期时间的设置上，可以评估线程执行任务的正常用时，在正常用时的基础上稍微再大一点
boolean setExpire = redisService.expire(redisKey, REDIS_REGISTER_TIMEOUT);
// 设置锁和设置过期时间均成功时才认为当前线程获取锁成功，否则认为获取锁失败
if (setLock && setExpire) {
return true;
}
// 当发生设置锁成功，但设置过期时间失败的情况时，手动清除刚刚设置的锁Key
redisService.del(redisKey);
return false;
}
/**
* 删除设备注册时的分布式锁
* @param deviceMacAddress 设备的Mac地址
*/
public void delRegisterLock(String deviceMacAddress) {
redisService.del(getRegisterLockKey(deviceMacAddress));
}
/**
* 获取设备注册时分布式锁的key
* @param deviceMacAddress 设备mac地址（每个设备的mac地址都是唯一的）
* @return
*/
private String getRegisterLockKey(String deviceMacAddress) {
return REDIS_REGISTER_KEY_PREFIX + "_" + deviceMacAddress;
}
}An example of using locks in normal registration logic is as follows:
public ReturnObj registry(@RequestBody String device){
Devices deviceInfo = JSON.parseObject(device, Devices.class);
// 开始注册前加锁
boolean registerLock = lockUtil.getRegisterLock(deviceInfo.getMacAddress());
if (!registerLock) {
log.info("获取设备注册锁失败，当前注册请求失败！");
return ReturnObj.createBussinessErrorResult();
}
// 加锁成功，开始注册设备
ReturnObj result = registerDevice(deviceInfo);
// 注册设备完成，删除锁
lockUtil.delRegisterLock(deviceInfo.getMacAddress());
return result;
}Resolve concurrent updates
1. Does concurrent update really cause problems?
When the simultaneous update or one-by-one update has no impact on the business, there is no need to do any processing, so as not to increase the complexity of the system in vain.
2. Optimistic Locking
Repetitive updates can be avoided through optimistic locking, that is: add a "version number" (version) field in the database table, query the record before doing the update operation, write down the query version number, and then perform the actual update operation When determining whether the previously queried version number is consistent with the version number of the record in the current database, if it is consistent, it means that no other thread has updated the record during the period from query to update in the current thread; if it is inconsistent, it means During this period, other threads have changed this record, and the update operation of the current thread is no longer safe and can only be abandoned.
Judgment SQL example:
update a_table set name=test1, age=12, version=version+1 where id = 3 and version = 1Optimistic locking uses the version number to determine at the moment of the last update whether the data read from the database has been modified by others. Its efficiency is higher than pessimistic locking because it is in the period between the current thread query and the last update. , Other threads can read the same record as usual, and can preemptively update.
Pessimistic lock
Pessimistic locking is the opposite of optimistic locking. When the current thread queries this piece of data to be updated, it locks this piece of data and does not allow other threads to modify the data before the update is completed.
Use it select … for update to tell the database "I am going to update this piece of data and lock it for me".
Note: FOR UPDATE is only applicable to InnoDB and must be valid in a transaction. When the query condition has a clear primary key and there is this record, it is row lock (row lock, which only locks the row of data located according to the query condition), query condition When there is no primary key or the primary key is not clear, it is table lock (table lock, locking the whole table, will cause the data of the whole table to be unable to be changed during the lock period), so when using pessimistic locking, the query conditions should be clearly located to a certain row or several OK, don't cause a full table lock
Finally send benefits :
I will share with you a wave of materials. These materials are all Java e-books, study notes, latest learning routes, written test questions, interview questions, development tools, PDF document book tutorials, zero basic to proficient video courses compiled by me over the past few years. , Java job application resume templates, Java programmers face and other learning materials, free to share with you, all the materials are in my Java technology qq exchange group: 127522921, there is no routine, please download it by yourself! I bought many of them with money. Welcome everyone to join the group, you can also discuss the technology, welcome to join!