SpringBoot combined with POI million-level data report operation

POI report advanced operation

The previous article has introduced that Excel can be divided into two versions: Excel2003 and Excel2007. Excel2003 uses HSSF objects in POI. A sheet allows up to 65536 data items. It can be used when processing less data, but Excel2003 certainly cannot accommodate when processing millions of data. ; Excel2007 uses XSSF objects in POI, allowing a sheet to store 1048576 pieces of data at most, indicating that it can support millions of data, but there may be problems in actual operation. The reason is that the objects generated by POI reports, cell objects, Font objects will not be destroyed, leading to the risk of OutOfMemoryError (OOM) memory overflow.

Million data report export overview

For the export of millions of data Excel, basically only discuss the use of Excel2007 version. ApachePOI provides three ways to solve the import and export of large amounts of data:

• User mode: User mode has many encapsulated methods, which are easy to operate, but create too many objects and consume huge memory
• Event mode: Parsing XML based on SAX (Simple API for XML) is an interface and a software package. It is an alternative to XML parsing. The difference from DOM parsing is that it parses line by line and does not load all the data at once. Memory is equivalent to analyzing while scanning.
• SXSSF object: is used to generate a large number of Excel files, mainly with temporary storage space to generate Excel

  Export of millions of data

demand analysis

In the Internet era, millions of data are often generated, and there are various reasons for data export

Solution

• 1. Thinking analysis:

Using POI's XSSFWORK object to export Excel reports is to transfer all the data to the cell object at one time and save it in the memory. When all the cells are created, it will be written to Excel for export at one time. When reaching the millions of data export, with the continuous creation of cell objects, more and more data in the memory, until OOM. The SXSSFWORK object of POI is specially exported from Excel for handling large amounts of data.

2. Principle analysis:

When instantiating the SXSSFWork object, you can specify the number of POI objects generated in the memory (default 100). Once the number of memory objects reaches the specified number, the data in the memory is written to the disk, and the data in the memory can be destroyed To loop until the Excel export is complete

Code

Take user data as an example

Add in FileUtil

Made some minor changes to the previous one

package com.cn.greemes.common.util;

import cn.hutool.core.collection.CollUtil;
import cn.hutool.core.io.IoUtil;
import cn.hutool.core.util.IdUtil;
import cn.hutool.poi.excel.BigExcelWriter;
import cn.hutool.poi.excel.ExcelUtil;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.xssf.streaming.SXSSFSheet;
import org.apache.poi.xssf.streaming.SXSSFWorkbook;

import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletResponse;
import java.io.File;
import java.io.IOException;
import java.net.URLEncoder;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * 文件操作
 */
public class FileUtil {

    public static final String SYS_TEM_DIR =System.getProperty("java.io.tmpdir")+ File.separator;

    public  void downloadExcel(List<Map<String, String>> list, HttpServletResponse response) throws IOException {
        String tempPath = SYS_TEM_DIR + IdUtil.fastSimpleUUID() + ".xlsx";
        File file = new File(tempPath);
        BigExcelWriter writer = ExcelUtil.getBigWriter(file);
        // 一次性写出内容，使用默认样式，强制输出标题
        writer.write(list, true);
        SXSSFSheet sheet = (SXSSFSheet)writer.getSheet();
        //上面需要强转SXSSFSheet  不然没有trackAllColumnsForAutoSizing方法
        sheet.trackAllColumnsForAutoSizing();
        //列宽自适应
       // writer.autoSizeColumnAll();
        //response为HttpServletResponse对象
        response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=utf-8");
        //test.xls是弹出下载对话框的文件名，不能为中文，中文请自行编码
        response.setHeader("Content-Disposition", "attachment;filename=file.xlsx");
        ServletOutputStream out = response.getOutputStream();
        // 终止后删除临时文件
        file.deleteOnExit();
        writer.flush(out, true);
        //此处记得关闭输出Servlet流
        IoUtil.close(out);
    }

    public  void downloadExcelBySXSSF(List<Map<String, String>> list, HttpServletResponse response) throws IOException {
        String tempPath = SYS_TEM_DIR + IdUtil.fastSimpleUUID() + ".xlsx";
        File file = new File(tempPath);
        //2.创建工作簿
         SXSSFWorkbook workbook = new SXSSFWorkbook();
        //3.构造sheet
        Sheet sheet =workbook.createSheet();
        //创建表头
        Row row = sheet.createRow(0);
        Map<String,String> mapfirst = list.get(0);

        String listHead = null;
        AtomicInteger headersAi = new AtomicInteger();

        for (String key : mapfirst.keySet()) {
           Cell cell = row.createCell(headersAi.getAndIncrement());

           cell.setCellValue(key);
        }
        AtomicInteger datasAi = new AtomicInteger(1);
        Cell cell =null;
        for(Map<String, String> map : list){
            Row dataRow = sheet.createRow(datasAi.getAndIncrement());
            int i=0;
            for (String key : map.keySet()) {
                Cell cell1 = dataRow.createCell(datasAi.getAndIncrement());
                String value= (String)map.get(key);
                cell = dataRow.createCell(i);
                cell1.setCellValue(value);
                i++;
            }
        }
        String fileName = URLEncoder.encode("用户信息.xlsx", "UTF-8");
        response.setContentType("application/octet-stream");
        response.setHeader("content-disposition", "attachment;filename=" + new
                String(fileName.getBytes("ISO8859-1")));
        response.setHeader("filename", fileName);
        workbook.write(response.getOutputStream());

    }
}

Configure in the controller

 @ApiOperation("导出用户数据")
    @RequestMapping(value = "/export2", method = RequestMethod.GET)
    @ResponseBody
    public void export2(HttpServletResponse response, @RequestParam(value = "keyword", required = false) String keyword,
                       @RequestParam(value = "pageSize", defaultValue = "5") Integer pageSize,
                       @RequestParam(value = "pageNum", defaultValue = "1") Integer pageNum) throws UnsupportedEncodingException, IOException {
        Page<MesAdmin> adminList = adminService.list(keyword, pageSize, pageNum);

        List<Map<String,String>> list = new ArrayList();
        //因为只有七条数据,所以做了多次循环添加数据
        for(int i=0;i<149795;i++) {
            for (MesAdmin umsAdmin : adminList.getRecords()) {
                Map<String, String> map = new HashMap<>(6);
                DateFormat d1 = DateFormat.getDateInstance();
                map.put("姓名", umsAdmin.getUsername());
                map.put("邮箱", umsAdmin.getEmail());
                map.put("昵称", umsAdmin.getNickName());
                map.put("备注信息", umsAdmin.getNote());
                map.put("创建时间", d1.format( umsAdmin.getCreateTime()));
                String loginTime ="";
                if(umsAdmin.getLoginTime()!=null){
                    loginTime=d1.format( umsAdmin.getLoginTime());
                }
                map.put("最后登录时间",loginTime );
                list.add(map);
            }
        }

        fileUtil.downloadExcelBySXSSF(list,response);
    }

Conclusion:

Originally I wanted to introduce the export of data, but I found that the export of millions of data also needs to be introduced. I will introduce the export of data tomorrow.

Github address:

github address: https://github.com/bangbangzhou/greemes/tree/master

the public