[24] springboot uses EasyExcel and thread pool to realize multi-threaded import of Excel data

The overall column of the springboot chapter:

During the development of the company, I encountered a very common import function requirement, which required importing Excel files . From this, I thought of Alibaba’s EasyExcel , a convenient tool. When the customer told me that it needs to support the import of large amounts of data, I thought of it. Use the thread pool to multi-thread the operation of importing into the database. Therefore, this chapter records this operation.

Qq exchange group navigation ——> 231378628

First of all, the overall approximate process looks like this:

Effect:

The real project is the RPC framework packaged by the company. This demo is written directly using the previous demo that integrates mybatis-plus .

First look at the directory structure :

The framed part in the picture is the file that needs to be modified or newly created in this chapter , which will be interpreted one by one later. The following description can be used to mark the class name in this picture.

Then introduce the preparation work that needs to be done in this chapter:

Database Table
Modify the application configuration file, and modify the maximum file upload limit of tomcat (otherwise the excel file is too large, and the upload will report an error)
Enable the batch insert function of mybatis-plus, mybatis-plus only has the single insert function of insert by default (if your own project does not use this, you don’t need it, this is just that there is no batch insert method on my demo)
Create various classes required for excel multi-threaded import interface

Table of contents

1. Prepare the database and Excel file

2. Introduce the required dependencies

3. Modify the configuration file and modify the default file size limit

4. Open the batch save function of mybatis-plus

5. Create the required tool classes

6. Create codes for each layer of business

7. Create an easyExcel event listener

Eight, create a custom thread class

Nine, test the efficiency of single-threaded and multi-threaded processing

1. Prepare the database and Excel file

CREATE TABLE `deadman` (
  `uid` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT '主键',
  `idCard` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT '身份证号',
  `userName` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT '死者姓名',
  `sex` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT '死者性别',
  `age` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT '死者年龄',
  `reason` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT '死因',
  `house` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL COMMENT '安排地狱层数',
  PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin;

2. Introduce the required dependencies

Mainly the dependencies involved in importing easyExcel are as follows:

3. Modify the configuration file and modify the default file size limit

This must be modified , otherwise the subsequent interfaces will not be able to enter. The default tomcat server limits the size of uploaded files.

spring:
  servlet:
    multipart:
      max-file-size: 100MB
      max-request-size: 100MB

server:
  tomcat:
    max-swallow-size: 100MB

4. Open the batch save function of mybatis-plus

The rpc framework integrated in this demo is Mybatis-plus, so I found a way to realize batch saving , as follows:

1. Create a new SpiceSqlInjector class

PS: Pay attention to the box selection part of the class name.

2. Create the SpiceBaseMapper interface class

PS: Originally, the interface layer of the business inherits BaseMapper, and here it inherits BaseMapper, and then adds the method as shown in the figure above. Note that the name must be this, otherwise an error will be reported when calling this method later.

3. Create the Mapper layer required by the business

Inherit the interface class you just customized.

From this, batch insertion can be realized by injecting DeadManMapper and calling the above method.

as follows:

5. Create the required tool classes

When importing through EasyExcel, it will involve some data types that do not meet the input parameters. This demo involves the following tool classes.

1、MultipartFileToFileUtils

Convert the incoming MultipartFile type to File type, the Controller receives the MultipartFile type, and the EasyExcel.read method needs the File type.

2、SpringJobBeanFactory

When the listener class is injected into the mappe, a null pointer will be reported, and the following tool class is used to inherit ApplicationContextAware to obtain the bean object.

6. Create codes for each layer of business

1. The controller layer

2. Mapping the entity class of the database

For comments related to mybatis-plus, you can read the previous article.

3. The entity class of easyExcel

index refers to the row number above the excel sheet, for example

4. service layer

5. The implementation class layer of service

The above method is a multi-threaded event listener, the following is a single-threaded event listener, and then compare the two methods.

7. Create an easyExcel event listener

1. Single-threaded event listener

Analysis: The listener inherits from the AnalysisEventListener class, and the generic type is specified as the entity class object class specified by easyExcel above. Override two methods of this class: invoke and doAfterAllAnalysed.

invoke(): This method will start reading data from the second row of the excel sheet.

doAfterAllAnalysed(): This method will be executed after invoke has analyzed all the excel data, so it is enough to store data in this class.

At this point, the single-threaded import has been completed. Finally, we will compare and test the efficiency of the two listeners, and now create a multi-threaded event listener.

2. Multi-threaded event listener

Analysis: It is also to rewrite the two methods mentioned above. The processing of the invoke method remains unchanged, and the processing of the doAfterAllAnalysed method is modified. In this method, the created thread task is submitted to the thread pool through the method of creating a thread pool, and the thread pool is allowed to perform multi-thread task execution, thereby realizing multi-thread execution import operation .

Process: Create a thread pool——"Calculate the data that each thread needs to process——"Create a CountDownLatch object (to ensure that each thread will return to the main thread at the end)——"Circulate the number of threads and submit thread tasks to the thread pool—" Execute the await method of the CountDownLatch object, so that the current thread is in a waiting state, and the current thread will be awakened after the CountDownLatch is reduced to 1——"Each thread processes its own data and executes the countDown method of the CountDownLatch object after processing, so that the CountDownLatch object The value minus one——"When the value of CountDownLatch is 0 (indicating that the thread task of the thread pool class and the execution are completed), execute the code of the main thread—"close the thread pool

Eight, create a custom thread class

This step creates the threaded task class described above.

Implement Callable or Runable or inherit Thread. Here implement Runable and rewrite the run method.

According to the incoming data interval assigned to him, after the data in the interval is taken out through the subList method, the batch insertion method implemented above is executed to store the data. In order to prevent code problems from not executing the coutDown method, write this code into finally.

PS: The two methods of CountDownLatch (countDown, await) are used together to ensure that when the code of each thread is executed, the main thread enters the wait, and then returns to the main thread for execution after each thread task is executed.

Nine, test the efficiency of single-threaded and multi-threaded processing

The preparation of single-threaded and multi-threaded event listeners has been completed above, and the efficiency of the two will be tested below.

1. Use a single-threaded event listener

Result: 100033 pieces of data, the data is correct.

2. Use multi-threaded event listeners

Result: 100033 pieces of data, the data is correct, it is indeed much faster.

I think that for simple multi-threaded business scenarios, you can use this demo to modify it. If there is any problem, thank you for pointing it out. Resby.

Qq exchange group navigation ——> 231378628