Scene reconstruction
Recently, a project of a colleague had a strange problem in the test environment. It said that the project was stuck as soon as it was logged in. The interface did not respond and no error was reported. The formal environment did not have this problem. If there is no problem, let me help you find out what is the reason. After a whole morning of searching, I finally found the reason. The code logic is actually very simple. The approximate logic is as follows to simulate the code:
@Cacheable(cacheNames = {
"user","user"},key = "#id",sync =true)
public String getById(String id) {
System.out.println("get from db");
return "user:" + id;
}
important clue
He said that as long as @Cacheable is removed, it will not be stuck, and if it is added, it will be stuck . This is a very important clue. It must be caused by this. First of all, it can be determined that it is not a database query problem.
Troubleshooting steps
-
Check if there is any data in the cache.
I think it must be the cache. Then open the redis client to see if there is such a cache. I found that there is a key of user~lock whose value is locked. It seems that the Redis thread is deadlocked. , If he looks at the Redis database, he will probably find the reason.
-
Find that CacheAspectSupport
has an execute() method, and there is an if (contexts.isSynchronized()) judgment. If sync=true is set, it will be locked when setting the Redis cache
-
Find the deadlock code RedisCache waitForLock() method
here is an infinite loop, go to Redis every 300 milliseconds to check if there is a user~lock key, and the validity period of this key is permanent, so it is stuck here and cannot get out of the loop up.
Solution
Delete the lock key in redis and it will be OK
The cause of the problem
In the doInRedis of RedisWriteThroughCallback, one will be locked first, and finally will be unlocked (connection);
I guess it should be that the test environment deactivated the service and used kill -9, which caused the unlock method in finally not to be executed, and the key did not set an expiration time. So it's a permanent infinite loop, and if you don't add sync =true, it won't be locked.
static class RedisWriteThroughCallback extends AbstractRedisCacheCallback<byte[]> {
public RedisWriteThroughCallback(BinaryRedisCacheElement element, RedisCacheMetadata metadata) {
super(element, metadata);
}
@Override
public byte[] doInRedis(BinaryRedisCacheElement element, RedisConnection connection) throws DataAccessException {
try {
lock(connection);
try {
byte[] value = connection.get(element.getKeyBytes());
if (value != null) {
return value;
}
.........
return value;
} catch (RuntimeException e) {
if (!isClusterConnection(connection)) {
connection.discard();
}
throw e;
}
} finally {
unlock(connection);
}
}
};