Lock wait timeout exceeded? The code is optimized

background

In a recent investigation found problems occasionally occur on the database lock timeout phenomenon, the error message like the following will happen:

Exception in thread "pool-3-thread-1" org.springframework.dao.CannotAcquireLockException: 
### Error updating database.  Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
### The error may involve com.zr.center.mybatis.auto.mapper.UserMapper.updateByExampleSelective-Inline
### The error occurred while setting parameters
### SQL: update user      SET user_name = ?                          WHERE (  user_id = ? )
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
; ]; Lock wait timeout exceeded; try restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:262)
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
    at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:75)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:447)
    at com.sun.proxy.$Proxy101.update(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:295)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:59)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53)
    at com.sun.proxy.$Proxy103.updateByExampleSelective(Unknown Source)
    at com.zr.center.framework.web.service.BaseService.updateByExampleSelective(BaseService.java:97)
    at com.zr.center.api.test.service.TestService.updateUserName(TestService.java:34)
    at com.zr.center.api.test.service.TestService$$FastClassBySpringCGLIB$$bd3aa32.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
    at com.zr.center.api.test.service.TestService$$EnhancerBySpringCGLIB$$59b19302.updateUserName(<generated>)
    at com.zr.center.ApplicationTests.lambda$testTxLockWaiting$0(ApplicationTests.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Lock wait timeout exceeded; try restarting transaction
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
    at com.mysql.jdbc.Util.getInstance(Util.java:408)
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:952)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3912)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2486)

Investigation

After investigation, the log DBA has not given a deadlock because of a deadlock exclusion, inquiry service logs found in times of high concurrent sometimes repeated requests over, there is a certain logic in dealing with service will send a mq news, while at the same time will be spending this message, this time can lead to lock timeout. The reason is because too much time out logic processing of a transaction, it has been argued external services (time-out), there are other multiple tables update operation, which would later lead to a transaction request timeout, reported above error.

Steps to Reproduce

• Database Configuration

  View information about the configuration, you can see the transaction isolation is RR, the transaction lock wait long for the default 50s
  
  

• Transaction Code

/**
 * @description: 用户服务超时测试
 * @author: chong guo
 * @create: 2018-12-10 14:44
 */
@Service
@Slf4j
public class TestService extends BaseService<User, UserExample, Long> {

    /**
     * 更新用户名称
     *
     * @param userId
     * @param name
     */
    @Transactional(rollbackFor = Exception.class)
    public void updateUserName(Long userId, String name) throws InterruptedException {
        log.info("开始更新用户名【{}】，用户ID为【{}】", name, userId);
        UserExample userExample = new UserExample();
        userExample.createCriteria().andUserIdEqualTo(userId);

        User user = new User();
        user.setUserName(name);

        super.updateByExampleSelective(user, userExample);

        // 模拟业务超时，有可能调用外部远程服务超时，也有可能处理其它逻辑
        Thread.sleep(55000);
        log.info("结束更新用户名【{}】，用户ID为【{}】", name, userId);

    }

}

• Concurrent simulation

/**
 * @author  Chong Guo
 */
@RunWith(SpringRunner.class)
@SpringBootTest
@Slf4j
public class ApplicationTests {
    @Resource
    TestService testService;

    private final int threadCount = 5;

    @Test
    public void testTxLockWaiting() throws InterruptedException {
        CountDownLatch countDownLatch = new CountDownLatch(threadCount);

        ExecutorService threadPool = Executors.newFixedThreadPool(300);
        for (int i = 0; i < threadCount; i++) {
            threadPool.execute(() -> {
                try {
                    testService.updateUserName(611526166943105024L, "chongguo");
                } catch (InterruptedException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                } finally {
                    countDownLatch.countDown();
                }

            });
        }
        countDownLatch.await();
        threadPool.shutdown();
        log.info("Test tx lock is over");
        Thread.sleep(100000);
    }
}

After three failed will now run the code, the second success, failure is the cause of lock out

to sum up

1. Code optimization, large transaction logic ripped out independent, asynchronous processing
2. Service provider interface to external calls related to the need to set a long timeout shorter
3. If the caller has a retry mechanism, can be removed for service retry the connetion time set a large number of