Students who know Java may have seen an exception java.util.ConcurrentModificationException, which is usually caused by modifying the elements in the data structure when the iterator accesses the data structure such as Collection and Map.
Some time ago, ShardingSphere encountered an occasional error problem: java.lang.IllegalStateException: Recursive update. This problem is actually similar to the concurrent modification problem. This article mainly records the troubleshooting and analysis process of this problem.
Article directory
problem background
相关 issue java.lang.IllegalStateException: Recursive update caused by #24251
Apache ShardingSphere frequently reported sporadic errors after the Proxy module was refactored java.lang.IllegalStateException: Recursive update.
Refactored PR that caused this issue: https://github.com/apache/shardingsphere/pull/24251
java.util.ServiceConfigurationError: org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLSessionVariableHandler: Provider org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLDefaultSessionVariableHandler could not be instantiated
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:586)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:813)
at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:729)
at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1403)
at org.apache.shardingsphere.infra.util.spi.ShardingSphereServiceLoader.load(ShardingSphereServiceLoader.java:56)
at org.apache.shardingsphere.infra.util.spi.ShardingSphereServiceLoader.<init>(ShardingSphereServiceLoader.java:46)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
at org.apache.shardingsphere.infra.util.spi.ShardingSphereServiceLoader.getServiceInstances(ShardingSphereServiceLoader.java:73)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:71)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:126)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:113)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLSetVariableAdminExecutor.lambda$execute$0(MySQLSetVariableAdminExecutor.java:56)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLSetVariableAdminExecutor.execute(MySQLSetVariableAdminExecutor.java:56)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.executor.MySQLSetVariableAdminExecutorTest.assertExecute(MySQLSetVariableAdminExecutorTest.java:66)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
... 省略 JUnit 调用栈
Caused by: java.lang.IllegalStateException: Recursive update
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1763)
at org.apache.shardingsphere.infra.util.spi.ShardingSphereServiceLoader.getServiceInstances(ShardingSphereServiceLoader.java:73)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:71)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:55)
at org.apache.shardingsphere.proxy.backend.handler.admin.executor.DefaultSessionVariableHandler.<init>(DefaultSessionVariableHandler.java:36)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLDefaultSessionVariableHandler.<init>(MySQLDefaultSessionVariableHandler.java:28)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:484)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:789)
... 88 more
Throwing exception is ConcurrentHashMap instance is following LOADERS.
public final class ShardingSphereServiceLoader<T> {
private static final Map<Class<?>, ShardingSphereServiceLoader<?>> LOADERS = new ConcurrentHashMap<>();
private final Class<T> serviceInterface;
The code can be found in ShardingSphere
https://github.com/apache/shardingsphere/blob/5.3.1/infra/util/src/main/java/org/apache/shardingsphere/infra/util/spi/ShardingSphereServiceLoader.java#L36
Troubleshooting process
Read the ConcurrentHashMap Javadoc
There must be a reason for ConcurrentHashMap to report such an error. First search the documentation and find that there is indeed an explanation about the error.
Java 8
The documentation of Java 8 has stated that computeIfAbsentthe logic of the object should be short and the content of the map should not be updated.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ConcurrentHashMap.html#computeIfAbsent-K-java.util.function.Function-
Java 17
In the Java 17 documentation, the statement that updating the map content is not allowed is highlighted separately.
https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/ConcurrentHashMap.html#computeIfAbsent(K,java.util.function.Function)
ConcurrentHashMap source code analysis
The JDK used in this article is Oracle OpenJDK 19.0.1.
Search for related exceptions and find that there are 9 codes in ConcurrentHashMap that will throw this exception.
Among them, there are 7 codes that judge that if the node is ReservationNodean instance of , an exception is thrown:
else if (f instanceof ReservationNode)
throw new IllegalStateException("Recursive update");
The following is ReservationNodethe definition of , which can be seen from the javadoc and is a dedicated placeholder for the computeIfAbsentand methods.compute
/**
* A place-holder node used in computeIfAbsent and compute.
*/
static final class ReservationNode<K,V> extends Node<K,V> {
ReservationNode() {
super(RESERVED, null, null);
}
Node<K,V> find(int h, Object k) {
return null;
}
}
When a hash bucket of ConcurrentHashMap already exists ReservationNode, it means that the method of this instance is being called computeIfAbsent.
Is it possible that this call occurs in multiple threads at the same time?
When the key is mapped to the specified hash bucket, the subsequent writing logic will be executed at this point as a synchronizedcode block. If different threads call the computeIfAbsentmethod, only the first calling thread can execute the writing logic, and other threads only can wait to enter a critical section.
Therefore, ReservationNodethe thread that can encounter must also be ReservationNodethe thread that writes.
The other two codes roughly mean that if pred.nextis not empty, an exception is thrown:
Node<K,V> pred = e;
if ((e = e.next) == null) {
if ((val = mappingFunction.apply(key)) != null) {
if (pred.next != null)
throw new IllegalStateException("Recursive update");
added = true;
pred.next = new Node<K,V>(h, key, val);
}
break;
}
Among them pred, and eare actually the positions where the current KV is ready to be written. If after the KV is calculated, it is found that the location to be written is not empty, indicating that the content of the current hash bucket has been modified during the calculation of the Value.
Similar to ReservationNodethe case of , writing to the hash bucket will be locked, so this modification can only happen in the current thread.
Preliminary conclusion | Why is it occasional instead of inevitable?
After the source code analysis just now, it can be judged that:
- If there is no hash collision for the recursively updated Key, and the writing locations are on different hash buckets, which do not affect each other, the recursive update problem will not occur;
- If there is a hash collision in the recursively updated Key, and the writing location is on the same hash bucket, a recursive update exception will be thrown;
think further
What would happen if recursive updates were allowed?
Suppose there is a hash collision between Key1 and Key2.
- Call
computethe method for the first time, find the writing location for Key1, and start calculating Value; - As a result, in the logic of calculating Value, Key2 is written. Due to the hash collision, Key2 takes the first step to find a good writing location for Key1;
- After the value of Key1 is calculated, what should I do at this time because the location where I originally found and written it has been occupied? Recalculate write position?
If this recursive call is not only two layers, but recurses many layers, computethe logic inside the series method may become complex and inefficient.
Therefore, directly prohibiting recursive updates may be a way to keep the logic clear and efficient.
What happens when you switch to HashMap?
ConcurrentHashMap does not allow recursive compute to change the existing mapping relationship, so what about HashMap?
public final class ShardingSphereServiceLoader<T> {
- private static final Map<Class<?>, ShardingSphereServiceLoader<?>> LOADERS = new ConcurrentHashMap<>();
+ private static final Map<Class<?>, ShardingSphereServiceLoader<?>> LOADERS = new HashMap<>();
private final Class<T> serviceInterface;
Error messages became standardjava.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1229)
at org.apache.shardingsphere.infra.util.spi.ShardingSphereServiceLoader.getServiceInstances(ShardingSphereServiceLoader.java:73)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:71)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:126)
at org.apache.shardingsphere.infra.util.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:113)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLSetVariableAdminExecutor.lambda$execute$0(MySQLSetVariableAdminExecutor.java:56)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.MySQLSetVariableAdminExecutor.execute(MySQLSetVariableAdminExecutor.java:56)
at org.apache.shardingsphere.proxy.backend.mysql.handler.admin.executor.MySQLSetVariableAdminExecutorTest.assertExecute(MySQLSetVariableAdminExecutorTest.java:66)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:578)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
... 省略 JUnit 调用栈
Therefore, ConcurrentHashMap is Recursive updateessentially ConcurrentModificationExceptionsimilar to ConcurrentHashMap, except that ConcurrentHashMap allows concurrent modification to a certain extent (eg, hash does not collide).
repair method
computeIfAbsentAvoid the recursive call of ConcurrentHashMap's series of methods.
ShardingSphere repair case:
Avoid ConcurrentHashMap Recursive update #24416