MyBatis inserts large quantities of data in batches, too much data problem

MyBatis inserts large quantities of data in batches
Scenario: Import excel table, read data, and then batch insert database
service The key code is as follows.
Pass in List collection and insert data directly

bindCardInsertLogMapper.batchInsertHzjjBindCodeLog(list);

XML

    <insert id="batchInsertHzjjBindCodeLog">
        INSERT INTO hzjj_batch_bind_code_log  (id_card,yt_no,redeem_code,card_no,password,name,batch_id,ip,course_id) VALUES
        <foreach collection="list" item="log" separator=",">
            (#{log.idCard},#{log.ytNo},#{log.redeemCode},#{log.cardNo},#{log.password},#{log.name},#{log.batchId},#{log.ip},#{log.courseId})
        </foreach>
    </insert>

At first, there was no problem with calling the interface and everything was normal. Later, when importing 300 pieces of data at a time, an error was reported directly: the
incoming tabular data stream (TDS) remote procedure call (RPC) protocol flow was incorrect. Too many parameters were provided in this RPC request. It should be up to 2100 to
check the data to find out that when Mybatis calls SQL SERVER, no matter insert or query modification and other operations, the parameters are limited**, when the parameter length exceeds 2100, this error will be reported. **
Solution:
multiple threads insert data in batches, originally 1000 data, start 10 threads, and insert 100 per thread is OK.
After the solution, the code is as follows:

					//因为我这边数据不会很多,所以线程池就直接使用CachedThreadPool,没有自定义一个线程池。
					ExecutorService executorService = Executors.newCachedThreadPool();
					//num 数据被分为多少批可以发送完
                    //100 每个线程批量插入100个
                    int num=list.size()/100;
                    if(list.size()%100>=0){
    
    
                        num=num+1;
                    }
                    // lastIndex 截取结束坐标
                    int lastIndex=0;
                    for (int i=0;i<num;i++){
    
    
                        if(100*(i+1)>list.size()){
    
    
                            lastIndex=list.size();
                        }else{
    
    
                            lastIndex= 100*(i+1);
                        }
                        //多线程批量添加
                        executorService.submit(new BatchInsertHzjjCodeThread(list.subList(i*100,lastIndex),bindCardInsertLogMapper));
                    }
                    //关闭线程池,拒绝接收任务
                    executorService.shutdown();
                    while (true){
    
    
                        //任务是否结束 进行下一步操作
                        if(executorService.isTerminated()){
    
    
                            bindCardInsertLogMapper.batchBindHzjjCode(batchId);
                            break;
                        }
                    }

Then import the table again and you can add it normally

Guess you like

Origin blog.csdn.net/ChenLong_0317/article/details/109813322