Implementing multipart upload in Java
Article directory
1: Background
The need to upload and download large files on the Web requires uploading files to object storage. Uploading large files has the following pain points:
- File upload timeout: The reason is that the front-end request framework limits the maximum request duration, the back-end sets the timeout for interface access, or nginx (or other proxy/gateway) limits the maximum request duration.
- File size exceeds limit: The reason is that the backend limits the size of a single request. Generally, nginx and server will impose this limit.
- Upload takes too long
- The upload failed due to various network reasons, and you need to start from the beginning after failure.
Two: Solution
1. Overall plan
1. The front-end cuts the uploaded file into several small files according to the fragment size set in the code, and uploads them sequentially in multiple requests. The back-end then splices the file fragments into a complete file, and then uploads it.
2. If a certain fragment fails to be uploaded, it will not affect other file fragments. You only need to re-upload the failed part. You need to design a table to maintain some information related to the uploaded slices.
2. Code examples
@Data
@AllArgsConstructor
@NoArgsConstructor
public class ShardingFileDTO {
//文件名称(包含文件后缀)
private String fileName;
//文件总大小 MB
private String size;
//文件总分片数
private int shardTotal;
//分片文件索引下标
private int shardIndex;
//文件后缀，视频后缀为mp4,图片则为jpg等
private String suffix;
//唯一标识
private String onlyCode;
}
package com.xxy.demotest.controller.ShardingFile;
import cn.hutool.core.util.IdUtil;
import com.alibaba.fastjson.JSON;
import com.xxy.demotest.controller.ShardingFile.model.ShardingFileDTO;
import com.xxy.demotest.haikang.aliyun.ALiYun;
import com.xxy.demotest.result.baseresult.BaseResponse;
import com.xxy.demotest.utils.WorkUtil.FileUtil;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections.CollectionUtils;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import java.io.*;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* @beLongProjecet: demo-test
* @beLongPackage: com.xxy.demotest.controller.ShardingFile
* @author: xxx
* @createTime: 2023/09/01 15:16
* @description: 分片文件上传
* @version: v1.0
*/
@RestController
@RequestMapping("sharding")
@RequiredArgsConstructor
@Slf4j
public class ShardingFileController {
public static final String shardPath="D:\\test\\sharding\\";
public static final String savePath="D:\\test\\save\\";
private static void excuteFile(ShardingFileDTO dto, MultipartFile multipartFile) throws IOException {
log.info("文件分片上传请求开始,请求参数: {}", JSON.toJSONString(dto));
//获取本地文件夹地址
String fileFolderPath = savePath + dto.getOnlyCode();
log.info("本地文件夹地址,fileFolder的值为:{}", fileFolderPath);
//如果目标文件夹不存在，则直接创建一个
FileUtil.createFolder(fileFolderPath);
//本地文件全路径
String fileFullPath =fileFolderPath + File.separator+ dto.getFileName()+"_"+ dto.getShardIndex()+"."+ dto.getSuffix();
log.info("本地文件全路径,fileFullPath的值为:{}", fileFullPath);
//将分片文件保存到指定路径
multipartFile.transferTo(new File(fileFullPath));
//更新到文件上传表中
//判断当前分片索引是否等于分片总数，如果等于分片总数则执行文件合并
if (dto.getShardIndex()==dto.getShardTotal()) {
//文件合并
log.info("文件分片合并开始");
File dirFile = new File(fileFolderPath);
if (!dirFile.exists()) {
throw new RuntimeException("文件不存在");
}
//分片上传的文件已经位于同一个文件夹下，方便寻找和遍历（当文件数大于十的时候记得排序用冒泡排序确保顺序是正确的）
List<String> filePaths = FileUtil.listFiles(fileFolderPath);
if (CollectionUtils.isNotEmpty(filePaths)) {
//将此里面文件按照索引进行排序
log.info("filePaths的值为:{}", filePaths);
// 使用自定义的Comparator来对文件路径进行排序
Collections.sort(filePaths, new FilePathComparator());
//进行合并,顺序按照索引进行合并
String mergedFilePath =fileFolderPath+File.separator+dto.getFileName();
log.info("生成新的文件的路径,mergedFilePath的值为:{}", mergedFilePath);
mergeFiles(filePaths, mergedFilePath);
//合并完成将新文件上传到对象存储中
String upload = ALiYun.upload(FileUtil.fileToMultipartFile(new File(mergedFilePath)));
log.info("文件最终访问地址,upload的值为:{}", upload);
//可以异步
//删除所有临时切片文件
deleteFolderAndSubfolders(fileFolderPath);
//删除所有切片
deleteFolderAndSubfolders(shardPath+dto.getOnlyCode());
}
}
}
public static void main(String[] args) {
String fastUUID = IdUtil.fastSimpleUUID();
//String sourceFilePath = "C:\\Users\\wonder\\Desktop\\ai测试图片\\切片文件上传\\metacosmic_conference.zip"; // 源文件路径
String sourceFilePath = "C:\\Users\\wonder\\Desktop\\ai测试图片\\人像.png"; // 源文件路径
String outputDirectory = shardPath + fastUUID;
FileUtil.createFolder(outputDirectory);
//封装dto参数
ShardingFileDTO shardingFileDTO = new ShardingFileDTO();
shardingFileDTO.setFileName(new File(sourceFilePath).getName());
shardingFileDTO.setSize("10MB");
shardingFileDTO.setSuffix(getFileExtension(sourceFilePath));
shardingFileDTO.setOnlyCode(fastUUID);
long sliceSize = 5 * 1024 * 1024; // 切片大小，这里设置为5MB
try {
File sourceFile = new File(sourceFilePath);
String fileName = sourceFile.getName();
int lastDotIndex = fileName.lastIndexOf('.');
String suffix = fileName.substring(lastDotIndex + 1);
long fileSize = sourceFile.length(); // 获取文件大小
int sliceNumber = (int) Math.ceil((double) fileSize / sliceSize); // 计算切片数量
log.info("共切割成 " + sliceNumber + " 个文件切片");
shardingFileDTO.setShardTotal(sliceNumber);
FileInputStream fis = new FileInputStream(sourceFile);
byte[] buffer = new byte[(int) sliceSize];
int bytesRead;
List<String> sliceFilePaths = new ArrayList<>();
for (int i = 0; i < sliceNumber; i++) {
int num = i + 1;
shardingFileDTO.setShardIndex(num);
String sliceFileName = "slice_" + num;
String sliceFilePath = outputDirectory + File.separator + sliceFileName+"."+suffix;
// 创建切片文件并写入数据
FileOutputStream fos = new FileOutputStream(sliceFilePath);
bytesRead = fis.read(buffer, 0, (int) sliceSize);
fos.write(buffer, 0, bytesRead);
fos.close();
sliceFilePaths.add(sliceFilePath);
File file = new File(sliceFilePath);
excuteFile(shardingFileDTO,FileUtil.fileToMultipartFile(file));
}
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 合并成新的文件
* @param filePaths
* @param mergedFilePath
*/
public static void mergeFiles(List<String> filePaths, String mergedFilePath) {
try (FileOutputStream fos = new FileOutputStream(mergedFilePath);
BufferedOutputStream bos = new BufferedOutputStream(fos)) {
for (String filePath : filePaths) {
try (FileInputStream fis = new FileInputStream(filePath);
BufferedInputStream bis = new BufferedInputStream(fis)) {
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = bis.read(buffer)) != -1) {
bos.write(buffer, 0, bytesRead);
}
}
}
log.info("文件合并完成");
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 获取文件扩展名
* @param fileName
* @return
*/
public static String getFileExtension(String fileName) {
int lastDotIndex = fileName.lastIndexOf('.');
if (lastDotIndex > 0) {
return fileName.substring(lastDotIndex + 1);
}
return ""; // 如果文件名中没有点，返回空字符串
}
/**
* 删除文件夹下所有的文件
* @param folderPath
*/
public static void deleteFilesInFolder(String folderPath) {
File folder = new File(folderPath);
// 检查文件夹是否存在
if (!folder.exists() || !folder.isDirectory()) {
System.out.println("指定的路径不是一个有效的文件夹.");
return;
}
File[] files = folder.listFiles();
if (files != null) {
for (File file : files) {
if (file.isFile()) {
// 删除文件
if (file.delete()) {
System.out.println("已删除文件: " + file.getName());
} else {
System.out.println("无法删除文件: " + file.getName());
}
}
}
}
}
/**
* 删除文件夹
* @param folderPath
*/
public static void deleteFolder(String folderPath) {
File folder = new File(folderPath);
// 删除文件夹
if (folder.exists() && folder.isDirectory()) {
if (folder.delete()) {
System.out.println("已删除文件夹: " + folderPath);
} else {
System.out.println("无法删除文件夹: " + folderPath);
}
}
}
/**
* 删除文件夹中所有文件和子文件夹
* @param folderPath
*/
public static void deleteFolderAndSubfolders(String folderPath) {
File folder = new File(folderPath);
// 检查文件夹是否存在
if (!folder.exists()) {
System.out.println("文件夹不存在.");
return;
}
if (folder.isDirectory()) {
File[] files = folder.listFiles();
if (files != null) {
for (File file : files) {
if (file.isDirectory()) {
// 递归删除子文件夹及其内容
deleteFolderAndSubfolders(file.getAbsolutePath());
} else {
// 删除文件
if (file.delete()) {
System.out.println("已删除文件: " + file.getName());
} else {
System.out.println("无法删除文件: " + file.getName());
}
}
}
}
}
// 删除文件夹本身
if (folder.delete()) {
System.out.println("已删除文件夹: " + folderPath);
} else {
System.out.println("无法删除文件夹: " + folderPath);
}
}
}
class FilePathComparator implements Comparator<String> {
private static final Pattern NUMBER_PATTERN = Pattern.compile("\\d+");
@Override
public int compare(String filePath1, String filePath2) {
int number1 = extractNumber(filePath1);
int number2 = extractNumber(filePath2);
return Integer.compare(number1, number2);
}
private int extractNumber(String filePath) {
Matcher matcher = NUMBER_PATTERN.matcher(filePath);
if (matcher.find()) {
return Integer.parseInt(matcher.group());
}
return 0; // 如果找不到数字，则返回0或其他适当的默认值
}
}
3. Description
It is a bit troublesome for the backend to call requests one by one, so a main method is used for the following explanation. The execution process is:
Program slicing –> Save slices (front-end upload) –> Perform file merging when uploading the last slice (back-end merges based on conditional index) –> Merger completed –> Execute upload object storage –> Delete slice files –> Interface response link
Notice:
In normal interface requests, excuteFile needs to be slightly modified, and the slicing step can be omitted;
4. Methods in FileUtil
/**
* 创建文件夹
*
* @param path
*/
public static void createFolder(String path) {
File folder = new File(path);
if (!folder.exists()) {
folder.mkdirs();
}
}
/**
* 获取当前文件夹下面的文件列表
*
* @param folderPath
* @return
*/
public static List<String> listFiles(String folderPath) {
List<String> objects = new ArrayList<>();
File folder = new File(folderPath);
File[] files = folder.listFiles();
if (files != null) {
for (File file : files) {
if (file.isFile()) {
objects.add(file.getAbsolutePath());
} else if (file.isDirectory()) {
listFiles(file.getAbsolutePath());
}
}
}
return objects;
}
/**
* File转换为MultipartFile
* @param file
* @return
*/
public static MultipartFile fileToMultipartFile(File file) {
FileItem item = new DiskFileItemFactory().createItem("file"
, MediaType.MULTIPART_FORM_DATA_VALUE
, true
, file.getName());
try (InputStream input = new FileInputStream(file);
OutputStream os = item.getOutputStream()) {
// 流转移
IOUtils.copy(input, os);
} catch (Exception e) {
throw new IllegalArgumentException("Invalid file: " + e, e);
}
return new CommonsMultipartFile(item);
}
Reference: Java implements file upload in parts