Recently, I encountered a need to upload very large files. I investigated the slice and multipart upload functions of Qiniu and Tencent Cloud, so here is the implementation of the front-end large file upload related functions.
In some businesses, uploading large files is a more important interactive scenario, such as importing large Excel table data in the library, uploading video files, etc. If the file size is relatively large, or the network conditions are not good, the upload time will be longer (more messages need to be transmitted, and the probability of packet loss and retransmission is greater), the user cannot refresh the page, and can only wait patiently for the request to complete .
The following starts with the file upload method, organizes the ideas for uploading large files, and gives relevant example codes. Since PHP has built-in convenient file splitting and splicing methods, the server-side code uses PHP for sample writing.
The relevant sample code of this article is located on github, the main reference
Talk about uploading large files
Large file cutting upload
Several ways to upload files
First, let's look at several ways of uploading files.
Normal form upload
Using PHP to display regular form uploads is a good choice. First construct the file upload form, and specify the submitted content type of the form as enctype="multipart/form-data", indicating that the form needs to upload binary data.
Then write the index.php upload file receiving code, use the move_uploaded_file method (php Dafa is good...)
When uploading large files in the form form, it is easy to encounter the server timeout problem. Through xhr, the front end can also perform asynchronous file upload operations, generally there are two ideas.
File encoding upload
The first idea is to encode the file, and then decode it on the server. I wrote a blog that realized image compression and upload on the front end. The main implementation principle is to convert the image to base64 for transmission.
varimgURL = URL.createObjectURL(file);
ctx.drawImage(imgURL, 0, 0);
// Get the encoding of the picture, and then pass the picture as a long string
vardata= canvas.toDataURL( "image/jpeg", 0.5);
What needs to be done on the server side is also relatively simple, first decode base64, and then save the picture.
$imgData = $_REQUEST[ 'imgData'];
$base64 = explode( ',', $imgData)[ 1];
$img = base64_decode($base64);
$url = './test.jpg';
if(file_put_contents($url, $img)) {
exit(json_encode( array(
url => $ url
)));
}
The disadvantage of base64 encoding is that its volume is larger than the original picture (because Base64 converts three bytes into four bytes, so the encoded text will be about one-third larger than the original text), which is very large For the files, the time for uploading and parsing will increase significantly.
For more knowledge about base64, you can refer to Base64 notes.
In addition to base64 encoding, you can also read the file content directly at the front end and upload it in binary format
// read binary file
functionreadBinary(text){
vardata = newArrayBuffer(text.length);
varui8a = newUint8Array (data, 0);
for( vari = 0; i < text.length; i++){
ui8a[i] = (text.charCodeAt(i) & 0xff);
}
console.log(ui8a)
}
varreader = newFileReader;
reader. = function{
readBinary( this.result) // read result or upload directly
}
// Put the content of the file read from input into the result field of fileReader
reader.readAsBinaryString(file);
formData asynchronous upload
The FormData object is mainly used to assemble a set of key/value pairs for sending requests, which can send Ajax requests more flexibly. You can use FormData to simulate form submission.
letfiles = e.target.files // Get the input file object
letformData = newFormData;
formData.append( 'file', file);
axios.post (url, formData);
The server processing method is basically the same as the direct form request.
iframe does not refresh the page
On low-version browsers (such as IE), xhr does not support uploading formdata directly, so you can only use form to upload files, and form submission itself will jump to the page, which is caused by the target attribute of the form form , Its values are
_self, the default value, opens the response page in the same window
_blank, open in a new window
_parent, open in the parent window
_top, open in the topmost window
framename, open in the iframe with the specified name
If you need to let users experience the feeling of uploading files asynchronously, you can specify iframe by framename. Set the target attribute of the form to an invisible iframe, then the returned data will be accepted by this iframe, so only the iframe will be refreshed. As for the returned result, it can also be obtained by parsing the text in this iframe.
functionupload {
varnow = + newDate
varid = 'frame'+ now
$( "body").append( `<iframe style="display:none;" name="${id}" id="${id}" />`);
var$form = $( "#myForm")
$form.attr({
"action": '/index.php',
"method": "post",
"enctype": "multipart/form-data",
"encoding": "multipart/form-data",
"target": id
}).submit
$( "#"+id).on( "load", function{
varcontent = $( this).contents.find( "body").text
try{
vardata = JSON.parse(content)
} catch(e){
console.log(e)
}
})
}
Large file upload
Now let’s take a look at the timeout problems encountered in uploading large files in the several uploading methods mentioned above.
Form upload and iframe upload without refresh page are actually uploading files through the form tag. In this way, the entire request is completely handed over to the browser for processing. When uploading a large file, the request may time out.
Through fromData, it actually encapsulates a set of request parameters in xhr to simulate form requests, which cannot avoid the problem of large file upload timeout
Encoding upload, we can control the uploaded content more flexibly
The main problem with uploading large files is that a large amount of data needs to be uploaded in the same request, which leads to a relatively long process, and the upload must be restarted after failure. Just imagine, if we split this request into multiple requests, the time of each request will be shortened, and if a request fails, we only need to resend this request without starting from the beginning. This can solve the large file What about the upload problem?
Based on the above problems, it seems that large file uploads need to meet the following requirements
Support split upload request (ie slice)
Support breakpoint resume
Support display upload progress and pause upload
Next, let us implement these functions in turn, it seems that the most important function should be slice.
File slice
Reference: Large file cutting and uploading
In encoding upload, we only need to get the binary content of the file at the front end, then split the content, and finally upload each slice to the server.
In Java, the file FIle object is a subclass of the Blob object. The Blob object contains an important method slice. Through this method, we can split the binary file.
The following is an example of splitting files. For up6, developers do not need to care about the details of splitting, but the control helps to achieve it. Developers only need to care about business logic.
When the control is uploaded, relevant information will be added to each file block data, and the developer can process it after receiving the data on the server.
After the server receives these slices, they can be spliced together. Below is the sample code of PHP splicing slices
For up6, developers do not need to splicing, up6 has provided sample code and has implemented this logic.
To ensure uniqueness, the control will add information to each file block, such as block index, block MD5, file MD5
http
Up6 has its own resuming function. Up6 has saved file information on the server side and file progress information on the client side. The control will automatically load the file progress information during upload, and the developer does not need to care about these details. In the processing logic of the file block, it only needs to be identified according to the file block index.
At this time, refresh the page or close the browser when uploading, and upload the same file again, the previously uploaded slices will not be uploaded again.
The logic for the server to realize the resumable upload is basically similar. As long as the query interface of the server is called inside getUploadSliceRecord to obtain the record of the uploaded slice, it will not be expanded here.
In addition, resumable transfers need to consider the expiration of slices: if the mkfile interface is called, the slice content on the disk can be cleared. If the client never calls the mkfile interface, leaving these slices on the disk forever is obviously Unreliable. Under normal circumstances, slice uploads have a period of validity. After the validity period, they will be cleared. For the above reasons, the resumable transmission must also synchronize the implementation logic of slice expiration.
Resume effect
Upload progress and pause
The progress method in xhr.upload can be used to monitor the upload progress of each slice.
The realization of upload pause is also relatively simple. Through xhr.abort, you can cancel the upload of the currently unfinished uploaded slices to achieve the effect of upload pause. Resuming the upload is similar to resuming the upload. Uploaded slices.
Due to space, the upload progress and pause functions will not be implemented here.
Realization effect:
summary
At present, there are some mature large file upload solutions in the community, such as Qiniu SDK, Tencent Cloud SDK, etc. It may not be necessary for us to manually implement a simple large file upload library, but it is still necessary to understand its principles.
This article first sorts out several ways of uploading front-end files, and then discusses several scenarios for uploading large files, and several functions that need to be implemented for uploading large files
Split the file into slices through the slice method of the Blob object
Sorted out the conditions and parameters required by the server to restore files, and demonstrated that PHP restores slices to files
Resumable upload by saving the record of uploaded slices
There are still some problems left, such as: avoiding memory overflow when merging files, slicing invalidation strategy, upload progress pause and other functions, which have not been implemented in depth or one by one, continue to learn
Most of the back-end code logic is the same, and it currently supports MySQL, Oracle, and SQL. You need to configure the database before use, you can refer to this article I wrote: http://blog.ncmem.com/wordpress/2019/08/12/java-http%E5%A4%A7%E6%96%87 %E4%BB%B6%E6%96%AD%E7%82%B9%E7%BB%AD%E4%BC%A0%E4%B8%8A%E4%BC%A0/
Welcome to join the group to discuss together: 374992201