foreword
For the upload processing of large files, it is impossible for us to simply upload the entire file through one request, which is very inefficient and the upload speed is very slow. So at this time, the front end needs to split the uploaded file, divide the file into small pieces, and then initiate multiple requests to upload these pieces at the same time. After the file is uploaded, finally initiate a merge request. These fragments are merged on the server to form the entire file and saved on the server, which completes the process of fast uploading of the entire large file.
For operations such as file segmentation and processing, HTML5 has provided us with a series of files API. But here we will not talk about how to use these APIs to split and upload files, but introduce the simple-uploader plug-in that has been packaged based on these APIs, a plug-in that can help us quickly develop the split upload function.
Plug-in introduction
Documentation:
Excerpted from the first sentence of the document: simple-uploader.js (also known as Uploader) is an upload library that supports multiple concurrent uploads, folders, drag and drop, pausing and continuing, uploading in seconds, uploading in chunks, automatic retransmission when errors occur, Manual retransmission, progress, remaining time, upload speed and other features; the upload library relies on HTML5 File API. It can be seen that the plug-in has already packaged many functions for us, just use it directly.
But the process I will elaborate on next is mainly based on the vue-simple-uploader plug-in, because the vue-simple-uploader plug-in is encapsulated based on simple-uploader, so its API is exactly the same as simple-uploader, except It is packaged into a form that conforms to the vue component for us to use.
process thinking
I will describe the general process of front-end and back-end in the entire multipart upload, but the specific code is only for the front-end.
front end
- Upload the file on the interface to get the content of the file
- The purpose of MD5 processing the file is to generate a unique key mark file according to the content, and each fragmented file fragment will have this mark
- Initiate a test (test) request, the server judges which fragments have been uploaded according to the key tag search, and returns the existing fragments in the file location, you can skip it directly
- The file will be divided in the component, and each divided segment will execute the checkChunkUploadedByResponse
function. The parameters of this function also include the response of the check (test) request, which is judged according to the array of uploaded segment positions in the response, for example: uploaded[ 2,3], the file can be divided into four pieces in total, 1 and 4 are not uploaded. The function returns
true to indicate that it has been uploaded, and returns false to indicate that it has not been uploaded. For fragments that have not been uploaded, the plugin will re-initiate the upload request. - After all the fragments are successfully uploaded, the fileSuccess
function will be executed. In the callback of this function, the request response result of the last successful upload can be obtained. At this time, the backend can return a field indicating that it can be merged and initiate a merge request. At this point, the front-end work has been done. - Pause the upload, the plug-in already provides the pause function, it will cancel the request that has been initiated but is in the pending
state (the request that has not yet been responded), so as to achieve the pause effect, and there is no need for backend operations here. - Cancel the upload, initiate a request to cancel the upload, tell the backend that the file will not be uploaded, and the backend can clean up the uploaded file fragments on the server.
rear end
- Detect (test) files, according to a key tag field, detect the upload status of the file on the server, and return the uploaded part location to the front end
- Save the file fragments, the front end uploads the file fragments successfully, gets the file fragments and saves its information, divides the classes of these file fragments according to the key, and the fragments with the same key are the same file.
- To merge fragments, both the front and rear ends confirm that the file upload is successful, and after receiving the merge request initiated by the front end, the
file fragments marked with the same key are merged in order according to their block location information to obtain a complete file. - Cancel the upload. If the front end cancels the upload halfway through the upload, the uploaded fragments need to be cleaned up.
- Clean up regularly, because during the upload process of some files, there may be other accidents that cause the file upload to fail or stop, and the front end has not initiated a request to cancel the upload. You can set a certain period of time (for example, three days) and continue to save file fragments within this period of time. , after this time, these file fragments will also be cleaned up.
the code
The project I am developing is vue2.0, the following code is for reference only.
1. First install the plug-in
Install vue-simple-uploader, it will install simple-uploader together.
npm install vue-simple-uploader --save
Because the spark-md5 plug-in is also needed to calculate the file MD5, this plug-in is also installed
npm install spark-md5 --save
2. Introduce plug-ins
import uploader from 'vue-simple-uploader'
Vue.use(uploader)
3. Use
After introducing vue-simple-uploader, global components such as uploader, uploader-unsupport, uploader-btn, etc. will be registered for us globally. This is the development idea of vue, a ready-made component wheel.
template section:
<uploader
ref="myUploader"
:fileStatusText="fileStatusText"
:options="options"
:autoStart="false"
@file-added="onFileAdded"
@file-success="onFileSuccess"
@file-error="onFileError"
class="uploader-app">
<uploader-unsupport></uploader-unsupport>
<!-- 这里我把选择上传文件按钮和拖拽组件结合在一起使用了-->
<uploader-btn ref="uploadBtn">
<uploader-drop @click.native="()=>{$refs.uploadBtn.click}">
<p>请点击虚线框选择要上传的文件或拖拽文件到虚线框内</p>
</uploader-drop>
</uploader-btn>
<uploader-list>
<div class="file-panel" slot-scope="props">
<ul class="file-list">
<li v-for="file in props.fileList" :key="file.id">
<uploader-file :class="'file_' + file.id" ref="files" :file="file" :list="true"></uploader-file>
</li>
<div class="no-file" v-if="!props.fileList.length">暂无待上传文件</div>
</ul>
</div>
</uploader-list>
</uploader>
js part:
export default {
data() {
const _this = this
return {
// 用于替换组件原来的状态显示文字
filsStatusText: {
success:"成功",
error:"失败",
uploading:"上传中",
paused:"暂停",
waiting:"等待中"
},
// uploader 的主要配置
options: {
chunkSize: 2.5 * 1024 * 1024, // 允许片段最大为5M，因为最后一块默认是可以大于等于这个值，但必须小于它的两倍。
simultaneousUploads: 5, // 最大并发上传数
target:"xxx", // 目标上传 URL，若测试和上传接口不是同个路径，可以用函数模式
permanentErrors:[404,415,500,501,409],// 原来默认是没有409的，但我这接口有409报错，需进入错误回调函数中提示错误信息
// 每次发起测试校验，所有分片都会进入这个回调
checkChunkUploadedByResponse:function (chunk,message) {
// 每次校验，chunk（片段）会不一样，但message（只有一个测试请求）一样
let objMessage = JSON.parse(message);
// 后端认为这个文件上传过，则直接跳过
if(objMessage.skipUpload) return true;
// 若文件检测出现异常，则提示并返回false
if(objMessage.error) {
_this.$message.error(objMessage.message);
return false
}
// 一些校验信息，需要用到的，可以绑定在chunk上
chunk.xxx = objMessage.xxx; // 可写可不写，看自己情况
// chunk的offset就是分块后，该块片段在文件所有模块的位置
return (objMessage.uploadedList||[]).indexOf(chunk.offset+1)>=0
},
// 处理所有请求的参数
processParams:(params,file,chunk) => {
// 这里需要根据后端的要求，处理一些请求参数
params.xxx = chunk.xxx // 比如一些需要在上传时，带上测试校验返回的一些信息字段
return params;
}
}
}
},
methods:{
// 导入文件时
onFileAdded(file) {
// 计算文件 MD5 并标记文件
this.computeMD5(file);
},
// 上传失败
onFileError(rootFile,file,res) {
this.$message.error(JSON.parse(res).message);
},
// 取消文件上传
onFileRemoved(file) {
// 发起取消上传请求给后端
this.$axios.cancelUploadFile({
filename:file.name,
identifier:file.uniqueIdentifer // 文件标记
})
},
// 所有片段上传成功后，进入文件上传成功回调
onFileSuccess(rootFile,file,res) {
res = JSON.parse(res);
// 后端返回成功的字段，插件认为只要所有片段的上传请求都成功了就是上传成功了，而对于其他的错误它是无法处理的
if (!res.result) {
this.$message.error(res.message);
return
}
// 如果后端返回可以合并的字段则发起合并请求
if(res.needMerge) {
// 获取组件的成功状态显示dom节点
const metaDom = document.querySelector(`.uploader-file.file_${
file.id} .uploader-file-status`);
// 因为插件在所有片段请求成功后就显示上传成功的状态
// 可合并是否成功却不管了，而插件并未提供处理方式
// 只能通过节点操作修改状态来处理了
metaDom.innerText = "合并中...";
this.$axios.mergeChunk({
...}); // 发起合并请求
} else {
// 分片上传成功，但整个文件上传并未结束，不需要合并
console.log("上传成功")
}
},
// 根据文件内容计算 MD5 并标记文件 file
computeMD5(file) {
let fileReader = new FileReader();
let time = new Date().getTime();
let blobSlice = File.prototype.slice || File.prototype.mozSlice || File.prototype.webkitSlice;
let currentChunk = 0;
const chunkSize = this.options.chunkSize;
let chunks = Math.ceil(file.size / chunkSize); // 总模块数
let spark = new SparkMD5.ArrayBuffer();
// 文件状态设为"计算MD5"
this.statusSet(file.id, 'md5');
file.pause();// 先暂停文件的上传过程
loadNext(); // 开始读取文件内容
// FileReader 加载完成
fileReader.onload = (e => {
// 插入读取的片段内容到 SparkMD5 中
spark.append(e.target.result);
// 按分片顺序读取，小于最后模块位置的就继续读取
if (currentChunk < chunks) {
currentChunk++;
// 实时展示MD5的计算进度
// 对于大型文件读取内容还是会花不少时间
// 所以需要显示读取进度在界面
let dom = document.querySelector(`.uploader-file.file_${
file.id} .uploader-file-meta`);
let md5Progress = ((currentChunk/chunks)*100).toFixed(0);
if (md5Progress < 100) {
dom.innerText = "MD5校验："+md5Progress+"%";
} else if(md5Progress === 100) {
dom.innerText = "MD5校验完成"
}
loadNext();
} else {
// 所有文件分片内容读取完成，生成MD5
let md5 = spark.end();
// 在computeMD5Success中标记文件
this.computeMD5Success(md5, file);
console.log(`MD5计算完毕：${
file.name} \nMD5：${
md5} \n分片：${
chunks} 大小:${
file.size} 用时：${
new Date().getTime() - time} ms`);
}
});
// FileReader 加载失败
fileReader.onerror = function () {
this.error(`文件${
file.name}读取出错，请检查该文件`)
file.cancel();
};
// 分片读取文件内容函数
function loadNext() {
let start = currentChunk * chunkSize;
let end = ((start + chunkSize) >= file.size) ? file.size : start + chunkSize;
// fileReader 读取文件
fileReader.readAsArrayBuffer(blobSlice.call(file.file, start, end));
}
},
computeMD5Success(md5,file) {
file.uniqueIdentifer = md5;// 标记文件file
file.resume();// 继续上传
}
}
}
Analysis and Summary of Issues
analyze
1. Pause function
The pause function is nothing more than canceling the request that has been initiated but still in the pending state by the front end and canceling the continued upload of the remaining files (if we want to write it ourselves, we have to write a lot. Fortunately, there are plug-ins to help We have achieved it)
2.
Pause the resuming function, refresh the page or continue uploading on other pages. The resuming function mainly depends on the checkChunkUploadedByResponse function. In fact, before each reupload, file verification will be performed, and the backend will return the received The position of the file fragments, skip these uploaded file fragments to achieve the function of resume uploading.
3. Instant transmission
For small files, the instant transmission function is mainly because the files are uploaded in pieces. Compared with the previous one upload file to one interface, and the current upload file fragmentation to multiple interfaces, the upload speed is naturally different. Multiple requests are much faster than one request to upload.
4. Upload in chunks
The core of chunking (fragmentation) is that the file generates a unique MD5 mark according to the content, and divides the uploaded file according to this mark, and the offset after fragmentation is the position, and divides the fragments of each file ( block), combined with the MD5 mark and its position in the file, each fragment can be unique and identifiable.
5. Cancel the upload
This is almost unchanged. Cancel the upload to clear the file and request the cleanup interface of the file to clean up the uploaded content of the file.
question
1. The merged fields are not returned in the last uploaded fragment.
This actually belongs to the backend problem, because uploading files is a concurrent request. Logically speaking, the last successfully uploaded fragment, that is, the last upload request that responds, should have the required The merged fields are right. But the problem is that the backend returns the merged fields in the penultimate segment, but the plug-in does not enter the upload success function. When the last one is also uploaded successfully, it enters the success callback, but there is no merged field. This prevents the merge request from being sent out.
Some people will say, since the successful callback of the plug-in is entered, can't it be merged directly? As mentioned above, the plug-in determines that the upload is successful based on the successful response of all upload requests of the fragments. But if there are other errors, such as the status code of the response is indeed a successful request, but the backend does not receive it, that is, the upload result returned by the interface is false, then it is obvious that the upload failed, and the merge should not be requested.
What about the front end operating in the processResponse function of the request response? It means that the fields that can be merged are obtained in the request response of the penultimate uploaded fragment, so the backend thinks that it can be merged, so why not directly initiate a merge request? I also tried this, but it happened that although the fields that could be merged appeared in the penultimate fragment upload request response, the last fragment failed to upload.
No matter how the front end is changed, it is impossible to achieve the situation where the front and back ends can be merged. So the front end must still make the merge in the successful callback instead of writing it elsewhere. What needs to be changed is the backend. The backend needs to ensure that the last fragment in the file upload request has fields that can be merged according to the maximum number of concurrent requests.