Vue2 Excel exports too much data to cause OOM problems and solutions

Project scenario:

Test server situation: our server has 8g memory, Admin and API projects take up 3g, mysql takes up about 1g, and Mycat takes up 2g.

The equipment module involves multi-table operations, and then transfers to VO to return to the front end, and displays the data after modifying the data in the previous section. Because it involves multi-table query, the query efficiency is not bad.

Problem Description

When you click Export Excel, the data will be exported according to the query conditions. For example, the query time will increase with the increase of the time interval, and the amount of data will also increase. Exported is all. If the amount of data is relatively small, it will cause freezes or Request timeout, if the amount of data is very large, it will cause request timeout or server downtime (stuck).

solution:

There are many reasons for the analysis of the problem, but the main problem is the problem of the derived condition, which is determined by the demand and cannot be changed.
Solution: We query the query conditions in segments. For example, the current query data has more than 20,000 pieces of data. According to the segmented query, for example, 100/500/1000 rows of data query per page, this will reduce the pressure on the server from the request, but The volume of requests will vary.

Vue code:

async findHospitalReportExport (params) {
    
    
    // 获取导出数据的标题头
    let exlTemplate = await postRequest('/oauth2_api/doctor_report/find_hospital_report_export.json', params)
    // 配置分段查询语条件
    let pageableExcel = {
    
    
      pageNumber: 1,
      pageSize: 100,
      totalPages: 0,
      total: 0,
      filter: {
    
    
        searchType: 'hospitalName',
        searchValue: '',
        area_saleChanelId: null,
        myStartTime: null,
        myEndTime: null
      }
    }
    pageableExcel.pageNumber = 1
    pageableExcel.pageSize = 100
    pageableExcel.filter = params.filter
    let tableData = []
    // 获取导出文件
    if (emptyUtil.isNotBlank(exlTemplate) && exlTemplate.code === '1' && emptyUtil.isNotBlank(exlTemplate.object)) {
    
    
      // 获取导出数据
      do {
    
    
        // 分段查询数据请求
        let doctorReportData = await this.findHospitalReportPage(pageableExcel)
        if (emptyUtil.isNotBlank(doctorReportData) && doctorReportData.code === '1' && emptyUtil.isNotBlank(doctorReportData.object)) {
    
    
          // 数据回填
          pageableExcel.pageNumber = doctorReportData.object.pageNumber + 1
          pageableExcel.totalPages = doctorReportData.object.totalPages
          pageableExcel.pageSize = doctorReportData.object.pageSize
          let doctorReportList = doctorReportData.object.content
          if (emptyUtil.arrayIsNotBlank(doctorReportList)) {
    
    
            doctorReportList.forEach(doctorReport => {
    
    
              // 数据添加数据
              tableData.push(doctorReport)
            })
          }
        }
        // 当前页数是否小于分页查询的总页数
      } while (pageableExcel.pageNumber <= pageableExcel.totalPages)
    }
    exlTemplate.object.tableData = tableData
    // 返回数据
    return exlTemplate
  }

Summarize

Generally speaking, the problem can be solved, but I feel that the solution is not very good. If you have a better solution, you are welcome to put it in the comment area.