Use webworker to export excel

Exporting Excel files is a very common function during project development. According to the way of generating Excel files, it is usually divided into back-end generation and front-end generation.

There are usually two ways for the front-end to download the Excel generated by the back-end. One is that the back-end returns the temporary download address of the Excel file and the front-end directly downloads it. The other is that the back-end returns ArrayBuffer binary data, which is downloaded after the front-end processes.

It is much simpler for the front-end to generate Excel. The back-end only needs to return JSON data in a specified format. The step of generating Excel is completed by the front-end browser, which can greatly reduce the pressure on the server and save server resources. However, when the front end exports a large amount of data in the main thread, excelit will inevitably block the main thread, causing the page to freeze and affecting the user experience ;

SheetJS

SheetJS, also known as XLSXJS, is called SheetJS on the official website. It supports browsers, nodejs, deno, and react-native, and browsers are compatible with ie10+.

SheetJS Community Edition provides a battle-tested open source solution for extracting useful data from almost any complex spreadsheet and generating new spreadsheets that can be used with legacy and modern software. --Excerpt from the official website

Here is 十万行 20列 an example of an exported data:

const cloLen = 20;
const rowLen = 100000;
const row = Array(cloLen).fill().reduce((t, e, i) => ({ ...t, [`字段${i + 1}`]: `字段${i + 1}的值` }), {});
const list = Array(rowLen).fill({ ...row });

const wb = XLSX.utils.book_new();
const ws = XLSX.utils.json_to_sheet(list, { dense: true });
XLSX.utils.book_append_sheet(wb, ws);
XLSX.writeFile(wb, `${new Date().toLocaleTimeString()}.xlsx`, {
  bookSST: true,
});

Because of the single-threaded nature of JavaScript, a large number of JS operations will cause the browser rendering process to be blocked, resulting in various degrees of browser suspended animation.

Taking my mac pro as an example, it takes 十万行 20列about 1700-1800ms to generate an Excel file, and the browser will also freeze for such a long time. This time increases linearly as the number of rows and columns increases.

At this time, you need to use WebWorker to avoid browser rendering blocking.

const cloLen = 20;
const rowLen = 100000;
const row = Array(cloLen).fill().reduce((t, e, i) => ({ ...t, [`字段${i + 1}`]: `字段${i + 1}的值` }), {});
const list = Array(rowLen).fill({ ...row });

const SheetJSWebWorker = new Worker(
  URL.createObjectURL(
    new Blob([
      `
      importScripts("https://cdn.sheetjs.com/xlsx-latest/package/dist/xlsx.full.min.js");
      onmessage = ({ data }) => {
        const wb = XLSX.utils.book_new();
        const ws = XLSX.utils.json_to_sheet(data, { dense: true });
        XLSX.utils.book_append_sheet(wb, ws);
        postMessage(XLSX.write(wb, { type: "array", bookType: "xlsx", bookSST: true, compression: true }))
      };
    `,
    ])
  )
);

SheetJSWebWorker.postMessage(list);
SheetJSWebWorker.onmessage = ({ data }) => {
  const a = document.createElement("a");
  a.download = `${new Date().toLocaleTimeString()}.xlsx`;
  a.href = URL.createObjectURL(
    new Blob([data], { type: "application/octet-stream" })
  );
  a.click();
};

At this time, the execution time to generate Excel is about 2400 ~ 2500ms. But the page renders without any blocking.

Note that SheetJS is a commercial project. We are using the completely free community edition. The community edition only provides very limited functions such as generating Excel and merging cells. If we want to customize the style of generating Excel, we need to use some open source peripherals of SheetJS, such as xlsx-style and xlsx-populate . I won't expand here, and everyone who is interested can do their own research.

If you want to export Excel with custom functions, you need to use ExcelJS to achieve it.

ExcelJS

ExcelJS provides a more powerful custom export Excel function, but the performance is only a quarter of that of SheetJS.

Single thread example:

const cloLen = 20;
const rowLen = 100000;
const row = Array(cloLen).fill().reduce((t, e, i) => ({ ...t, [`字段${i + 1}`]: `字段${i + 1}的值` }), {});
const list = Array(rowLen).fill({ ...row });

const workbook = new ExcelJS.Workbook();
const worksheet = workbook.addWorksheet();
worksheet.columns = Object.keys(list[0]).map((e) => ({ header: e, key: e, width: 20 }));
worksheet.addRows(list);

const a = document.createElement("a");
a.download = `${new Date().toLocaleTimeString()}.xlsx`;
a.href = window.URL.createObjectURL(
  new Blob([await workbook.xlsx.writeBuffer()], {
    type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8",
  })
);
a.click();

Multithreaded example:

const cloLen = 20;
const rowLen = 100000;
const row = Array(cloLen).fill().reduce((t, e, i) => ({ ...t, [`字段${i + 1}`]: `字段${i + 1}的值` }), {});
const list = Array(rowLen).fill({ ...row });

const ExcelJSWebWorker = new Worker(
  URL.createObjectURL(
    new Blob([
      `
      importScripts("https://cdn.bootcdn.net/ajax/libs/exceljs/4.3.0/exceljs.js");
      onmessage = async ({ data }) => {
        const workbook = new ExcelJS.Workbook();
        const worksheet = workbook.addWorksheet();
        worksheet.columns = Object.keys(data[0]).map(e => ({ header: e, key: e, width: 20 }));
        worksheet.addRows(data);
        postMessage(await workbook.xlsx.writeBuffer())
      };
    `,
    ])
  )
);

ExcelJSWebWorker.postMessage(list);
ExcelJSWebWorker.onmessage = ({ data }) => {
  const a = document.createElement("a");
  a.download = `${new Date().toLocaleTimeString()}.xlsx`;
  a.href = window.URL.createObjectURL(
    new Blob([data], {
      type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8",
    })
  );
  a.click();
};

CSV

Most of the time, the Excel files we export can also be generated in CSV format.

From the name, one is to export CSV, and the other is to export Excel.

So what's the difference between the two?

Excel is a spreadsheet and saves files in its own proprietary format, xls or xlsx, which holds information about all the worksheets in the workbook.

CSV stands for Comma Separated Values, which is a plain text format that comma-separates a series of values, but contains no formatting, formulas, macros, etc.

To sum up, Excel can not only store data, but also store the operation results on data. CSV file is just a text file, it only stores data, so it is very easy to generate.

At the same time, the CSV document is displayed in the form of Excel by default after being opened by word, and it can be converted into Excel only by simple processing.

Single thread example:

const cloLen = 20;
const rowLen = 100000;
const row = Array(cloLen).fill().reduce((t, e, i) => ({ ...t, [`字段${i + 1}`]: `字段${i + 1}的值` }), {});
const list = Array(rowLen).fill({ ...row });

let str = Object.keys(list[0]).join() + "\n";
for (let i = 0; i < list.length; i++) {
  for (const key in list[i]) {
    str += `${list[i][key] + "\t"},`;
  }
  str += "\n";
}
const a = document.createElement("a");
a.href = "data:text/csv;charset=utf-8,\ufeff" + encodeURIComponent(str);
a.download = `${new Date().toLocaleTimeString()}.csv`;
a.click();

Multithreaded example:

const cloLen = 20;
const rowLen = 100000;
const row = Array(cloLen).fill().reduce((t, e, i) => ({ ...t, [`字段${i + 1}`]: `字段${i + 1}的值` }), {});
const list = Array(rowLen).fill({ ...row });

const VanillaJSWebWorker = new Worker(
  URL.createObjectURL(
    new Blob([
      `
      importScripts("https://cdn.bootcdn.net/ajax/libs/exceljs/4.3.0/exceljs.js");
      onmessage = async ({ data }) => {
        let str = Object.keys(data[0]).join() + String.fromCharCode(10)
        for (let i = 0; i < data.length; i++) {
          for (const key in data[i]) {
            str += data[i][key] + '\t,';
          }
          str += String.fromCharCode(10);
        }
        postMessage('data:text/csv;charset=utf-8,\ufeff' + encodeURIComponent(str))
      };
    `,
    ])
  )
);

VanillaJSWebWorker.postMessage(list);
VanillaJSWebWorker.onmessage = ({ data }) => {
  const a = document.createElement("a");
  a.download = `${new Date().toLocaleTimeString()}.csv`;
  a.href = data;
  a.click();
};

Summarize

The biggest advantage of generating Excel at the front end is that it can reduce the consumption of server resources, make full use of the computing power resources of the client, and transfer the computing pressure from the server to the browser. The second is that the amount of transmitted data is smaller and relatively faster.

At the same time, the disadvantages are also obvious. First of all, JavaScript is not a programming language with high computing performance, and its relative computing performance is not as good as server computing. Secondly, the most important factor affecting the speed of generating the export file depends on the user's hardware, and computers with different performance may have a large difference in user experience.

The following are general applicable scenarios of the above solutions, please refer to:

Millions of data use CSV.
Data-heavy unstyled features require SheetJS.
A small amount of data is fully functional using ExcelJS.
Large amounts of data require functionality using the commercial version of SheetJS.