Use EasyExcel to read local excel data and import it into mysql database

This article only records the reading (read) of EasyExcel

1. What does EasyExcel do?

First look at the explanation of EasyExcel:
EasyExcel is an excel processing framework open sourced by Alibaba. It is a simple, memory-saving open source project for reading and writing Excel based on Java. The main reason why EasyExcel can greatly reduce the memory usage is that it does not load all the file data into the memory at one time when parsing Excel, but reads the data line by line from the disk and parses them one by one.

Open source address:
github address: https://github.com/alibaba/easyexcel
(I’m still a little confused just looking at the code, I searched for related videos on Xiaopo Station, and a video of Crazy God explained it clearly. Almost directly Just do as he said, of course, you need to make appropriate changes according to your actual situation.)

2. Project Background

A small project I made needs to import the data in the excel table into the database, but some data columns also contain many array objects [{…},{…},{…},{…}], and here The content is to be stored in the database as fields one by one, because manual work or excel is not realistic. So I went to understand the use of this framework.

Excel is as follows:

Note: Excel does not have any headers, because it starts reading directly from the parameter column corresponding to the entity class

3. Specific use process

1. The pom.xml configuration needs to be added:

		 <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>easyexcel</artifactId>
            <version>2.2.6</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.75</version>
        </dependency>

2. There are three steps in the code:

Create the entity object Dao class corresponding to excel;
Write listener: the default is to read excel line by line, so you need to create a callback listener for excel line by line. The listener mainly realizes reading and storage;
Call the listener: configure the location of excel, call the listener to read the table, and then store it in the database

Step 1: Create the entity class corresponding to excel

ps: Use the @ExcelProperty annotation to illustrate
insert image description here
the entity class corresponding to the corresponding excel column "resistance test data"
(it needs to be explained here, I replaced all the four parameter names with the corresponding English names on excel at the time)

If you want to The table structure stored in the database is different from the entity class corresponding to excel (for example, if a field read is an array object structure such as the "resistance test data" in excel in the above figure, then it needs to be processed and split into multiple fields can be stored in the database separately), then it is necessary to create an additional class corresponding to the table of the database (slightly) to receive all the data and store it in the database.

Tables created by the database: insert image description here

Step 2: Write the listener

Listener implementation

/**
 * Date:2021/10/20
 * Description: excel一行一行的回调监听器
 *
 * @author ivyhu
 */
@Component
public class DemoDataListener extends AnalysisEventListener<ExcelModel> {
    
    

    @Resource
    ChargingStationMapper charingStationResistanceReportMapper;
    @Resource
    CharingStationServiceImpl charingStationServiceImpl;

    private static final Logger LOGGER = LoggerFactory.getLogger(DemoDataListener.class);

    /**
     * 有参构造
     * 如果使用了spring,请使用这个构造方法。每次创建Listener的时候需要把spring管理的类传进来
     * @param serviceImpl
     */
    public DemoDataListener(CharingStationServiceImpl serviceImpl) {
    
    
        this.charingStationServiceImpl = serviceImpl;
    }
    //无参构造
    public DemoDataListener() {
    
    
    }
    /**
     * 读取数据或执行这个invoke()方法
     * @param excelModel 类型
     * @param context    分析器
     */
    @Override
    public void invoke(ExcelModel excelModel, AnalysisContext context) {
    
    
        LOGGER.info("解析到一条数据:{}", JSON.toJSONString(excelModel));
        //最终要保存到数据库的数据
        ChargingStationModel reportModelData = new ChargingStationModel();

        String productModelNum = "";
        String itemNo = "";
        Double groundResistance = 0.0;
        Double insulationResistance = 0.0;
        Byte type = 0;
        Integer singlePilePower = 0;
        /**
         * 获取excel表中充电设施产品信息
         */
        String str1 = excelModel.getInfoData();
        JSONArray array1 = JSON.parseArray(str1);
        //如果不为空，就解析[{},{},{},{}]类型的数组对象并获取值
        if (!ObjectUtils.isEmpty(array1)) {
    
    
            for (Object obj : array1) {
    
    
                JSONObject jsonObject = (JSONObject) obj;
                ChargingStationInfoDataExcelModel infoExcelModel = JSONObject.parseObject(String.valueOf((JSONObject) obj), ChargingStationInfoDataExcelModel.class);
                type = infoExcelModel.getType();
                singlePilePower = infoExcelModel.getSinglePilePower();
            }
        }
        /**
         * 获取excel表中电阻测试数据
         */
        String str = excelModel.getResistanceData();
        JSONArray array = JSON.parseArray(str);
        //电阻测试数据不为空，就解析[{},{},{},{}]类型的数组对象并获取值
        if (!ObjectUtils.isEmpty(array)) {
    
    
            for (Object obj : array) {
    
    
                JSONObject jsonObject = (JSONObject) obj;
                ChargingStationResistanceDataExcelModel reportModel = JSONObject.parseObject(String.valueOf((JSONObject) obj), ChargingStationResistanceDataExcelModel.class);
                productModelNum = reportModel.getProductModelNum();
                itemNo = reportModel.getItemNo();
                groundResistance = reportModel.getGroundResistance();
                insulationResistance = reportModel.getInsulationResistance();
                //插入数据到数据库
                reportModelData = new ChargingStationModel(excelModel.getReportNo(), excelModel.getModelNum(), productModelNum, itemNo, groundResistance, insulationResistance, type, singlePilePower,
                        excelModel.getOutputVoltageControlErrorOfCharger(), excelModel.getOutputCurrentControlErrorOfCharger(), excelModel.getActiveStopChargeTestOutputCurrentStopRate(), excelModel.getPassiveStopChargeTestOutputCurrentStopRate(),
                        excelModel.getChargerOutputCurrentAdjustmentTimeAbove20A(), excelModel.getChargerOutputCurrentAdjustmentTimeUnder20A(), excelModel.getImpulseCurrent());
                saveData(reportModelData);
            }
        }
    }

    /**
     * 所有数据解析完成了 都会来调用
     *
     * @param context
     */
    @Override
    public void doAfterAllAnalysed(AnalysisContext context) {
    
    

        // 这里也要保存数据，确保最后遗留的数据也存储到数据库
        /*saveData();*/
        LOGGER.info("所有数据解析完成！");
    }

    /**
     * 加上存储数据库
     */
    private void saveData(ChargingStationModel resistanceReportModel) {
    
    
        //调用mapper插入数据库
        charingStationServiceImpl.save(resistanceReportModel);
    }


}

Step 3: Call the listener to read data
1. Controller layer

@RestController
@Controller
@RequestMapping("/product/charging_station")
public class ChargingStationController {
    
    

    @Resource
    CharingStationService charingStationService;
    /**
     * 提取excel表格数据的接口
     */
    @PostMapping()
    public void saveExcel(){
    
    
        charingStationService.saveData();
    	}
    }

2. Service layer

@Service
public class CharingStationServiceImpl implements CharingStationService {
    
    

    @Resource
    ChargingStationMapper chargingStationMapper;
    @Resource
    ExcelUtil excelUtil;
    @Resource
    ChargingStationAcceptanceValueMapper chargingStationAcceptanceValueMapper;

    /**
     * 调用读取excel数据的方法
     */
    @Override
    public void saveData() {
    
    
        excelUtil.excelRead();
    }

3. Listener entry
Set the file address, file name, and entity class corresponding to excel

@Component
public class ExcelUtil {
    
    

    @Autowired
    private CharingStationServiceImpl charingStationServiceImpl;
    /**
     * 最简单的读
     * 1. 创建excel对应的实体对象 ExcelModel
     * 2. 由于默认一行行的读取excel，所以需要创建excel一行一行的回调监听器，参照{@link DemoDataListener}
     * 3. 直接读即可
     */
    private static final Logger LOGGER = LoggerFactory.getLogger(ExcelUtil.class);
    
    //文件位置
    String PATH = "D:\\excelTest\\";


    public void excelRead() {
    
    

        // 有个很重要的点 DemoDataListener 不能被spring管理，要每次读取excel都要new,然后里面用到spring可以构造方法传进去
        String fileName = PATH + "充电桩验收测试数据.xlsx";
        // 这里 需要指定读用哪个class去读，然后读取第一个sheet 文件流会自动关闭
        EasyExcel.read(fileName, ExcelModel.class, new DemoDataListener(charingStationServiceImpl)).sheet().doRead();

    }
}

Knowledge points in this article:

Convert Json, parse [{}, {}, {}, {}] type array object to get a single content inside

JSONObject jsonObject = (JSONObject) obj;
ChargingStationResistanceDataExcelModel reportModel = JSONObject.parseObject(String.valueOf((JSONObject) obj);

Use the annotation @Component to hand over the DemoDataListener and listener entries to Spring to manage

@Component
public class DemoDataListener extends AnalysisEventListener<ExcelModel> {
    
    
...
}

Configure the file location to be read

//文件位置
    String PATH = "D:\\excelTest\\";

    public void excelRead() {
    
    

        // 有个很重要的点 DemoDataListener 不能被spring管理，要每次读取excel都要new,然后里面用到spring可以构造方法传进去
        String fileName = PATH + "充电桩验收测试数据.xlsx";

        // 这里 需要指定读用哪个class去读，然后读取第一个sheet 文件流会自动关闭
        EasyExcel.read(fileName, ExcelModel.class, new DemoDataListener(charingStationServiceImpl)).sheet().doRead();

    }

ps: A few errors were reported at the beginning of the operation:

The mapper reports a null pointer. Two possibilities, I checked and found that one is because the DemoDataListener and the listener entry class are not injected into Spring, so the bean cannot be found, and @Component needs to be used. The second is that the model call of Spring mvc is not passed, but the unit test @Test in the open source framework is applied to run.
Reading the file failed because the header column was not deleted.

This article is just a little record of the little rookie, if there is a mistake, please correct me!