Java, excel massive millions of data export optimization measures, SXSSFWorkbook streaming, batch export example

When exporting millions of data, if you do not take appropriate optimization measures, it may indeed cause problems such as crashes and memory crashes.

To avoid these problems, the following optimization measures can be adopted:

1. Read data in batches: Divide the data to be exported into multiple batches for reading and writing, read part of the data each time, and clear the memory immediately after writing to Excel. This can avoid the problem of excessive memory usage caused by loading all data at once.
2. Use the streaming writing method: While traversing the result set, use the streaming writing method (such as SXSSF) to write the data into the Excel table, avoiding writing all the data into the memory at one time.
3. Close unnecessary connections: When writing code, you should close ResultSet, Statement and Connection connections in time to release resources and reduce the burden on the system.
4. Use appropriate server hardware configuration: In the case of a large amount of exported data, it is recommended to use a high-performance server, such as equipped with more CPU and memory, to improve system processing capacity and speed.
5. Reasonably set JVM memory allocation: If the JVM memory allocation is unreasonable, it may cause problems such as memory overflow. The memory allocation of the JVM can be reasonably set by adjusting parameters such as -Xmx and -Xms.

Add the following dependencies to the pom.xml file:

<dependencies>
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>4.1.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>4.1.2</version>
    </dependency>
</dependencies>

public void exportExcel() throws Exception{
    // 加载驱动程序
    Class.forName("com.mysql.cj.jdbc.Driver");

    // 连接数据库
    Connection conn = DriverManager.getConnection(url, username, password);

    // 查询语句
    String sql = "SELECT * FROM user";
    Statement stmt = conn.createStatement();
    ResultSet rs = null;

    // 创建一个 Excel 文档
    SXSSFWorkbook workbook = new SXSSFWorkbook(100);

    try {
        // 设置表格格式
        Sheet sheet = workbook.createSheet("数据导出");
        Row headerRow = sheet.createRow(0);
        String[] headers = {
     
     "ID", "姓名", "年龄"};
        for (int i = 0; i < headers.length; i++) {
            Cell cell = headerRow.createCell(i);
            cell.setCellValue(headers[i]);
        }
        
        int rownum = 1;
        while (true) {
            rs = stmt.executeQuery(sql + " LIMIT " + rownum + ", 10000"); // 每次查询10000条数据

            if (!rs.next()) { // 数据已经读取完
                rs.close();
                break;
            }

            do {
                Row row = sheet.createRow(rownum++);
                String id = rs.getString("id");
                String name = rs.getString("name");
                String age = rs.getString("age");

                row.createCell(0).setCellValue(id);
                row.createCell(1).setCellValue(name);
                row.createCell(2).setCellValue(age);
            } while (rs.next());

            ((SXSSFSheet) sheet).flushRows();
            ((SXSSFSheet) sheet).clearRowBreaks();
        }

        // 保存 Excel 文件
        FileOutputStream out = new FileOutputStream("数据导出.xlsx");
        workbook.write(out);
        out.close();

    } finally {
        // 关闭连接
        if (rs != null) { rs.close(); }
        stmt.close();
        conn.close();

        // 释放资源
        workbook.dispose();
    }

}

Optimization measures such as reading data in batches, using streaming writing, setting a reasonable cache size, and closing unnecessary connections have greatly improved code efficiency and stability. At the same time, it is also recommended to try to avoid exporting all the data at one time when exporting. For millions of data, you can export it in pages similar to the above example.