Reprinted: https://blog.csdn.net/tim_phper/article/details/77581071
Summary:
Recently the company project requirements in addition to the data page output also want to have the export function, export feature, although a few have done before, but this time the amount of data is relatively large, almost a day data is 20W, I claim a guide 7 days or 30 days , then the amount of data is easily broken million to even break 10 million, so the development process found some great pit data export, share this with you, learn from each other.
Preparation:
1, PHP settings pit:
set_time_limit - set the script maximum execution time:
this configuration is generally PHP default is 30 seconds, if you are a small data, may not find this setting, but if you reach millions of data export grade, often 30 seconds is not enough, so you need to add set_time_limit (0) in your script, the script does not allow the execution time is now
memory_limit - PHP memory limit:
This is the default configuration is generally php 128M, if done little data before friends may also move through this configuration can solve many problems, some may think that you have this large data transfer is not big on the line Yet? It really is too young too native, you can set up local 1G or unlimited maybe no problem, but the official field, you do so sooner or later you will have trouble, a PHP program accounts for so much of the memory space, if you call operation and maintenance company to help you tune about the configuration, operation and maintenance is estimated some reluctantly, server hardware so engaging is asking too much. So, we should try to avoid turn up the setting.
2, excel pit:
Since it is exporting data, of course, immediately thought of you guys excel format, and more convenient way to view the data, but never expected to excel also have temper ah!
The data in Table Limit:
2003 and the following versions of Excel. A table maximum support 65536 rows by 256 columns.
Excel 2007-2010 version. A table maximum support line 1048576, 16384.
1
2
That you want several million EXCEL easy one-time import a table does not work, you at least need to split the data to ensure that data can not exceed 104W a table.
PHPexcel Memory Overflow:
Now that the data is limited to 104W, then the data partitioning on data partition chant, so you try 50W once import table, but the internal PHPexcel have function reported memory overflow error, then you will continue to adjust the amount of small data until 5W once import you will find there is a memory overflow error. This is why, although you split the data to import multiple tables, but inside the last PHPexcel or one-time all the data into a table variable to create the file ...... forehead, which millions of data stored in a variable, you want memory does not overflow, really a little difficult.
(After reading some articles I have also found PHPExcel solutions, PHPExcel_Settings :: setCacheStorageMethod way to change the way a buffer to reduce memory usage)
3, csv pit:
EXCEL too much trouble, I do not have enough yet? I use the csv file storage, not to limit the number, but also directly with EXCEL to view, but after the files into a database, in one fell swoop a few too would not Miya? Hey, Shaoxia good idea! But there are pit CSV Oh!
Excessive output buffer:
When you () in fact use a native PHP function putcsv to the output cache buffer, if you put millions of data has been using this function outputs will lead to the output cache too big error, so we have every certain amount of time, the contents must be taken out of the output buffer, the output is set to wait state. Concrete action is:
ob_flush ();
the flush ();
. 1
2
DETAILED DESCRIPTION Introduction: The difference PHP flush () and ob_flush () of the Detailed
EXCEL CSV file to view the number of restrictions:
Most people see the direct use of EXCEL csv files are opened. Amount, is not that EXCEL back to pit yet? EXCEL Data show that restrictions Yeah, you just give you a few million data look 104W only. what? Whether you? That is their open it the wrong way? Bad bad, we is not difficult to solve it, we split the data look like, and then separated from csv file is saved, anyway, you will not split the variable data memory overflow.
4, summarized practice
complete analysis of those above the pit, then our solutions, and assume that the amount of data is several million dollars.
1, we read from the database, the amount of data to be read in batches to prevent memory overflow variable,
2, we select the data to save the file format is csv file, to facilitate the reading after the export, import and other database operations.
3, to prevent inconvenient to read excel csv file, we need to have the data before 104W division multiple csv file is saved
4, a plurality of output to the user to download the csv file is unfriendly, we also need more csv file compression, and finally to provide a compressed ZIP format packet users to download just fine.
Code:
// Export Description: Because EXCEL 104W single table can only display data while using PHPEXCEL easy because the amount of data is too large lead to take up too much memory,
// therefore, output format output csv file data, but csv file reading EXCEL software will also show the presence of only 104W, so dividing the data stored in the plurality csv file, and finally compressed into a zip file download
function putCsv (array $ head, $ data, $ mark = 'attack_ip_info ', $ fileName = "test.csv")
{
set_time_limit (0);
$ = $ DATA- the SQLCOUNT> COUNT ();
// output Excel file header, the user.csv be replaced by the file name you want
header (' the Type-the Content: file application / vnd.ms-Excel; charset = UTF-. 8 ');
header (' the Content-Disposition: Attachment; filename = " '$ fileName..'" ');
header (' the Cache-Control: max -age = 0 ');
$ sqlLimit = 100000; // each taking only 100,000 from the database cache to prevent too variable
// $ limit every row, refresh the output buffer, not too big, nor too small,
$ limit = 100000;
// Buffer counter
$ CNT = 0;
$ fileNameArr = Array ();
// retrieve data row, do not waste memory
for ($ I = 0; $ I <ceil (SQLCOUNT $ / $ sqlLimit); $ I ++) {
$ FP = the fopen (.. $ mark '_' $ i '.csv', 'w'.); // creates a temporary file
// chmod ( 'attack_ip_info_' $ i '.csv', 777..); // modify executable permissions
$ fileNameArr [] = $ Mark '_' I $ '.csv';...
// the data is written by the file handle fputcsv
fputcsv (FP $, $ head);
$ = $ dataArr DATA-> offset ($ * $ sqlLimit I) -> limit ($ sqlLimit) -> GET () -> toArray ();
the foreach ($ A $ dataArr AS) {
$ CNT ++;
IF ($ limit == $ cnt) {
// refresh output buffer, to prevent problems caused by too much data
ob_flush ();
flush ();
CNT = 0 $;
}
fputcsv (FP $, $ A);
}
fclose ($ FP); // Close each generate a file
}
// file compression plurality
$ ZIP the ZipArchive new new = ();
$ filename = $ Mark . ".zip";
$ zip-> open ($ filename, the ZipArchive :: the CREATE); // open compressed
the foreach ($ $ File fileNameArr AS) {
$ zip-> the addFile ($ File, the basename ($ File)) ; // add to the archive file
}
$ zip-> Close (); // Close compressed
the foreach ($ $ file fileNameArr AS) {
the unlink ($ file); // delete temporary files csv
}
// output archive download
header ( "the Cache-Control: max-Age = 0");
header ( "the Content-the Description: File Transfer");
header (. 'the Content-Disposition: Attachment; filename =' the basename ($ filename)); / / file name
header ( "Content-Type: application / zip"); // zip format
header ( "the Content-Transfer-Encoding: binary"); //
header ( 'the Content-the Length:'. filesize ($ filename)); / /
@readfile ($ filename); // output file;
unlink ($ filename); // delete temporary files archive
}
summary:
in fact, the above code still has room for optimization, for example with abnormal catch to prevent because some error which led to generate some temporary files and no normal delete, and PHPexcel cache settings may be able to solve the memory leak problem, EXCEL file can be generated in the form of a multiple worksheets, so for file viewers, the more friendly.
The above is my big export data PHP insights, I hope to help you to have, at the same time the local shortage of a lot of advice!
----------------
Disclaimer: This article is CSDN blogger "Tim_PHPer 'original article, follow the CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement. .
Original link: https: //blog.csdn.net/tim_phper/article/details/77581071