Cause analysis and solutions for the slow data export and sometimes flashback phenomenon of java web online projects

1. Symptoms

Recently, some online projects reported that when exporting data (excel file format), after waiting for a period of time (about 60 seconds), there was a flashback phenomenon, and it directly jumped to the login page.

2. Cause Analysis

After comprehensive analysis, there are mainly the following reasons:

1. The timeout setting of Alibaba Cloud load balancing is too small (60 seconds), and there is no problem with the normal page, but the task with a long time is easy to time out, and the page will jump to the login page after timeout;
2. The database query time is long, and the time for writing to Excel is long, resulting in a timeout;
3. Query the organization name once for each piece of data, resulting in many database visits and a long time;
4. The maximum memory setting of the java process is set to 1GB;
5. The timeout period of load balancing is 60 seconds, which is too short to meet the demand;

In general, as the amount of data continues to increase, the time required to query data becomes longer (SQL also needs to be further optimized to improve query efficiency). There is no limit to the amount of data exported at a time, and it is possible to export 300,000 to 500,000 data at a time, which increases the time for database query and the time for generating excel files.

3. Solutions

The solution ranges from simple to complex, and provides three solutions.

1. The most direct way: modify the timeout period of the load, and change the timeout period to 180 seconds; this method can greatly reduce the problem of flashback, but it still cannot solve the problem of slowness;
2. The database is re-optimized to provide a joint index for a specific combination of query conditions; a joint index for common query key field combinations is established to improve query speed;
3. Use an asynchronous method (such as a scheduled task) to produce and export an Excel file, and the user will download it after it is generated. There are two types of processing for the amount of data that needs to be exported. If the amount of data to be exported is small, such as less than 50,000, it can be directly exported on the page.

After the improvements in the above three aspects, the problems of slow data export and flashbacks have been completely solved.