Cause : Recently, I received a request to upload and download a large file at work. The file is required to be uploaded to a share disk. When downloading, single or multiple files are packaged according to different conditions transmitted by the front end and a directory is set for downloading.
At the beginning, I thought that I would just use the old method file.transferTo(newFile)and just count it as a large file. I just need to wait and it will be uploaded.
(Forgive my ignorance...) After trying it later, I found that it was really whimsical. If you directly use the ordinary upload method, you will basically encounter the following 4 problems:
- File upload timeout : The reason is that the front-end request framework limits the maximum request duration, the back-end sets the timeout for interface access, or nginx (or other proxy/gateway) limits the maximum request duration.
- File size exceeds limit : The reason is that the backend limits the size of a single request. Generally, nginx and server will impose this limit.
- Uploading takes too long (think 10 gigabytes of file upload, this should take several hours)
- The upload failed due to various network reasons, and you need to start from the beginning after failure.
So I can only seek help with slice upload.
the whole idea
The front end cuts the uploaded file into several small files according to the fragment size set in the code , and uploads them sequentially in multiple requests . The back end then splices the file fragments into a complete file. Even if a certain fragment fails to be uploaded, it will not be uploaded. It will affect other file fragments, so you only need to re-upload the failed part. Moreover, multiple requests are sent together to increase the upper limit of transmission speed. (The core of front-end slicing is to use the method, which is similar to the array method. The file method can return a certain slice of the original file)Blob.prototype.slicesliceslice
The next step is to code!
front-end code
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<!-- 引入 Vue -->
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/vue.min.js"></script>
<!-- 引入样式 -->
<link rel="stylesheet" href="https://unpkg.com/element-ui/lib/theme-chalk/index.css">
<!-- 引入组件库 -->
<script src="https://unpkg.com/element-ui/lib/index.js"></script>
<title>分片上传测试</title>
</head>
<body>
<div id="app">
<template>
<div>
<input type="file" @change="handleFileChange" />
<el-button @click="handleUpload">上传</el-button>
</div>
</template>
</div>
</body>
</html>
<script>
// 切片大小
// the chunk size
const SIZE = 50 * 1024 * 1024;
var app = new Vue({
el: '#app',
data: {
container: {
file: null
},
data: [],
fileListLong: '',
fileSize:''
},
methods: {
handleFileChange(e) {
const [file] = e.target.files;
if (!file) return;
this.fileSize = file.size;
Object.assign(this.$data, this.$options.data());
this.container.file = file;
},
async handleUpload() {
},
// 生成文件切片
createFileChunk(file, size = SIZE) {
const fileChunkList = [];
let cur = 0;
while (cur < file.size) {
fileChunkList.push({
file: file.slice(cur, cur + size) });
cur += size;
}
return fileChunkList;
},
// 上传切片
async uploadChunks() {
const requestList = this.data
.map(({
chunk, hash }) => {
const formData = new FormData();
formData.append("file", chunk);
formData.append("hash", hash);
formData.append("filename", this.container.file.name);
return {
formData };
})
.map(({
formData }) =>
this.request({
url: "http://localhost:8080/file/upload",
data: formData
})
);
// 并发请求
await Promise.all(requestList);
console.log(requestList.size);
this.fileListLong = requestList.length;
// 合并切片
await this.mergeRequest();
},
async mergeRequest() {
await this.request({
url: "http://localhost:8080/file/merge",
headers: {
"content-type": "application/json"
},
data: JSON.stringify({
fileSize: this.fileSize,
fileNum: this.fileListLong,
filename: this.container.file.name
})
});
},
async handleUpload() {
if (!this.container.file) return;
const fileChunkList = this.createFileChunk(this.container.file);
this.data = fileChunkList.map(({
file }, index) => ({
chunk: file,
// 文件名 + 数组下标
hash: this.container.file.name + "-" + index
}));
await this.uploadChunks();
},
request({
url,
method = "post",
data,
headers = {
},
requestList
}) {
return new Promise(resolve => {
const xhr = new XMLHttpRequest();
xhr.open(method, url);
Object.keys(headers).forEach(key =>
xhr.setRequestHeader(key, headers[key])
);
xhr.send(data);
xhr.onload = e => {
resolve({
data: e.target.response
});
};
});
}
}
});
</script>
Considering convenience and versatility, we do not use a third-party request library here, but use native XMLHttpRequest to make a simple encapsulation to send requests.
When the upload button is clicked, createFileChunk will be called to slice the file. The number of slices is controlled by the file size. Here, 50MB is set, which means that a 100 MB file will be divided into two 50MB slices.
CreateFileChunk uses the while loop and the slice method to put slices into the fileChunkList array and return
When generating file slices, you need to give each slice an identifier as a hash. Here, the file name + subscript is temporarily used, so that the backend can know which slice the current slice is for subsequent merged slices.
Then call uploadChunks to upload all file slices, put the file slices, slice hash, and file name into formData, then call the request function in the previous step to return a promise, and finally call Promise.all to upload all slices concurrently
backend code
Entity class
@Data
public class FileUploadReq implements Serializable {
private static final long serialVersionUID = 4248002065970982984L;
//切片的文件
private MultipartFile file;
//切片的文件名称
private String hash;
//原文件名称
private String filename;
}
@Data
public class FileMergeReq implements Serializable {
private static final long serialVersionUID = 3667667671957596931L;
//文件名
private String filename;
//切片数量
private int fileNum;
//文件大小
private String fileSize;
}
@Slf4j
@CrossOrigin
@RestController
@RequestMapping("/file")
public class FileController {
final String folderPath = System.getProperty("user.dir") + "/src/main/resources/static/file";
@RequestMapping(value = "upload", method = RequestMethod.POST)
public Object upload(FileUploadReq fileUploadEntity) {
File temporaryFolder = new File(folderPath);
File temporaryFile = new File(folderPath + "/" + fileUploadEntity.getHash());
//如果文件夹不存在则创建
if (!temporaryFolder.exists()) {
temporaryFolder.mkdirs();
}
//如果文件存在则删除
if (temporaryFile.exists()) {
temporaryFile.delete();
}
MultipartFile file = fileUploadEntity.getFile();
try {
file.transferTo(temporaryFile);
} catch (IOException e) {
log.error(e.getMessage());
e.printStackTrace();
}
return "success";
}
@RequestMapping(value = "/merge", method = RequestMethod.POST)
public Object merge(@RequestBody FileMergeReq fileMergeEntity) {
String finalFilename = fileMergeEntity.getFilename();
File folder = new File(folderPath);
//获取暂存切片文件的文件夹中的所有文件
File[] files = folder.listFiles();
//合并的文件
File finalFile = new File(folderPath + "/" + finalFilename);
String finalFileMainName = finalFilename.split("\\.")[0];
InputStream inputStream = null;
OutputStream outputStream = null;
try {
outputStream = new FileOutputStream(finalFile, true);
List<File> list = new ArrayList<>();
for (File file : files) {
String filename = FileNameUtil.mainName(file);
//判断是否是所需要的切片文件
if (StringUtils.equals(filename, finalFileMainName)) {
list.add(file);
}
}
//如果服务器上的切片数量和前端给的数量不匹配
if (fileMergeEntity.getFileNum() != list.size()) {
return "文件缺失，请重新上传";
}
//根据切片文件的下标进行排序
List<File> fileListCollect = list.parallelStream().sorted(((file1, file2) -> {
String filename1 = FileNameUtil.extName(file1);
String filename2 = FileNameUtil.extName(file2);
return filename1.compareTo(filename2);
})).collect(Collectors.toList());
//根据排序的顺序依次将文件合并到新的文件中
for (File file : fileListCollect) {
inputStream = new FileInputStream(file);
int temp = 0;
byte[] byt = new byte[2 * 1024 * 1024];
while ((temp = inputStream.read(byt)) != -1) {
outputStream.write(byt, 0, temp);
}
outputStream.flush();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}finally {
try {
if (inputStream != null){
inputStream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
try {
if (outputStream != null){
outputStream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
// 产生的文件大小和前端一开始上传的文件不一致
if (finalFile.length() != Long.parseLong(fileMergeEntity.getFileSize())) {
return "上传文件大小不一致";
}
return "上传成功";
}
}
For the convenience of the diagram, I just returnused strings directly. (Of course, I wrote the encapsulation of the unified result of the method in this demo, so the output is still a restful-style result. For details, you can read my previous article "Spring uses AOP to complete the unified result." Package》)
When the front-end calls the upload interface, the back-end will put the file passed by the front-end into a temporary folder.
When the merge interface is called, the backend will think that all the fragmented files have been uploaded and will merge the files.
The backend mainly determines the order of fragmented files based on the hash value returned by the frontend.
end
In fact, multi-part upload may sound like a lot of trouble. In fact, it is not difficult as long as you figure out the idea clearly. It is a relatively simple requirement.
Of course, this is just a relatively simple demo. It just implements a relatively simple multi-part upload function, such as breakpoint upload and upload pause. I haven’t had time to write it into the demo yet. I will open a new article to write about these when I have time. Extra content.
See you in the next article. If you like the blogger, you can follow and like it.