Optimizing the upper limit of MySQL in parameter value: exception "IN items count 2723 exceeds the threshold 2000 at com.mysql.cj.jdbc.exceptions"

Brief description: In today's project, a problem was found that the maximum value of mysql in exceeded the upper limit. A related optimization plan was proposed to address this problem for your reference. The exception code is as follows:

**IN items count 2723 exceeds the threshold 2000
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) ~[mysql-connector-j-8.0.31.jar:8.0.31]
at** 

Currently, there are four ways to learn:

1. Modifying the mysql in parameter configuration size value will put greater pressure on performance;
2. Change to batch query, slice user IDs, and finally merge the data. In this case, you need to solve the problems caused by paging and the performance consumption caused by data differences;
3. Change it to a subquery. The current business design cross-database query may need to be migrated to adb for processing. It requires more code changes, and the performance is better than the above two;
4. The code slice exceeds the in parameter ID value, List<List> list parameter, and finally the input parameters are spliced ​​into the mysql execution statement, similar to: select * from table where a or in ( ) ro a in ( ) mode . This method is somewhat different from the second one. Similar, except that the cutting method and splicing parameters are used as input parameters to control the size of the in value. This method also has disadvantages for performance. If the list is too large, it will cause the SQL to be too long or a performance problem. The larger the list, the worse the performance.

There are better optimization methods, you can leave a message to discuss~!

Main topic:

This optimization is to optimize the evaluation plan made for the 4th plan.

Why choose this option:

1. It is based on business design. It cannot be passed due to historical reasons, cross-instance databases, same account permission assignment, etc. ** from aY left jon bU on a.id =b.id is used to create a relational data query result set.

2. The evaluation code has undergone major changes. The project has been running for six or seven years, and UId is implemented in the entire system. If the adjustment area of ​​​​the associated query changes is large, more evaluation and implementation plan research are needed.

3. The modification area is small, the in value is controllable through parameter configuration, and the overall performance is within the control range.

Code reference:

ids fragmentation method:

	public static <F> List<List<F>> idSplitList(List<F> list) {
    
    
		if (CollectionUtil.isEmpty(list)) {
    
    
			return null;
		}
	 
		int groupSize = Integer.valueOf(20);
		int length = list.size();
		// 计算可以分成多少组
		int num = (length + groupSize - 1) / groupSize;
		List<List<F>> newList = new ArrayList<>(num);
		for (int i = 0; i < num; i++) {
    
    
			// 开始位置
			int fromIndex = i * groupSize;
			// 结束位置
			int toIndex = Math.min((i + 1) * groupSize, length);
			newList.add(new ArrayList<F>(list.subList(fromIndex, toIndex)));
		}
		return newList;
	}

mybaits corresponds to the mapper.xml file code

<if test ="idsSlice != null and idsSlice .size >0">
				and (
				<foreach collection="idsSlice " item="idsItem" separator=" " index="index" >
					<if test="index !=0">or</if> id in
					<foreach collection="idsItem" item="item" separator="," close=")" open="(">
						#{item}
					</foreach>
				</foreach>
				)
</if>

Code explanation:

1. The idSplitList method is to slice and divide List ids into 20 pieces at a time.

2.xml explanation, the syntax of this code is written with reference to the syntax of mybaitis mapper.xml, dynamic splicing sql, and those who know basic syntax can probably understand it. The index value must be judged by the first subscript without using or splicing sql, and other The tags need to be spliced ​​into the format : "where a in ( ) or a in ( ) or a in (*) "

The general idea is introduced here. For details, you can refer to this link:
mybatis-plus in error reporting when query conditions exceed 1,000

Note: Which solution to choose, you need to evaluate and choose the optimization solution that suits you based on your own system conditions. If it is an ADB library or a read-only library, it is better to use the left and right link mode, but if it is in a business library and there is no way to dismantle it. Separation can only choose a compromise mode, and the final solution is to solve the system bottleneck problem. I hope this article has inspired and helped you!

Everyone is welcome to comment in the comment area to discuss and propose better solutions!