Foreword: I recently encountered a requirement in an actual project. Customers may upload relatively large files. If the traditional file upload solution is used, there may be some security risks such as high server pressure, resource waste, and even memory overflow. Therefore, in order to solve a series of The problem is that a new technical solution is needed to realize the upload of large files; when I am free, I refer to some related tutorials on the Internet, and finally summarize it myself. In this blog, I will explain step by step how I realize large files. For the three functions of second transmission, breakpoint resume transmission and fragmentation, each piece of code will be explained. At the end of the blog, I will provide the Gitee source code for everyone to download.
Table of contents
1. Why use this technical solution
2. What is Instant Transmission
3. What is resume from breakpoint
6. Build the SpringBoot project
6.1.1. Import pom dependencies
6.1.3, mybatis-config.xml configuration file
6.2.1, HttpStatus common return status code constants
6.3.1, AjaxResult unified result encapsulation
6.4, GlobalCorsConfig global CORS cross-domain configuration
6.5.1, FileChunkMapper.xml file
6.5.3, FileInfoMapper.xml file
6.6. File upload core service class (important)
6.6.1, the logic of file upload check
6.6.2. Realize the core logic of fragment upload
6.6.3. File Fragmentation Processing
6.7, FileController request layer
7.2.1. Using the uploader component
7.2.3, checkChunkUploadedByResponse method (important)
7.2.4, parseTimeRemaining method
7.2.6, onFileAdded method (important)
7.2.7, getFileMD5 method (important)
1. Why use this technical solution
If the front-end uploads a very large file (such as 1G) at one time, and does not adopt technical solutions such as fragmentation/breakpoint resume transmission, it will mainly face the following hidden dangers or problems:
1. The network transmission speed is slow
The upload time is long and the einmal complete upload of the file needs to occupy continuous and stable upstream bandwidth. If the network condition is not good, the upload will be very slow, which will damage the user experience.
2. If it fails in the middle, you need to re-upload
If the upload process is interrupted due to reasons such as the network, the entire transmission will fail. This just requires the user to upload the complete file again, repeating the labor.
3. The server is under heavy pressure
The server needs to occupy a lot of resources to continuously process a large file, which puts a lot of pressure on the server performance and may affect other services.
4. Waste of traffic resources
If a large file is uploaded completely at one time, if the same file already exists, it will repeatedly consume a large amount of network traffic, which is a waste of data.
5. It is difficult to realize the upload progress prompt
The user cannot perceive the progress of the upload, and does not know how much data has been uploaded if the upload fails.
Therefore, in order to solve these problems, it is very important to use technologies such as fragmentation and resumed transmission. It can upload data blocks in batches, avoiding the disadvantage of uploading all at once. At the same time, combined with methods such as verification and recording of uploaded fragments, it can make the entire upload process controllable, recoverable, save traffic, and greatly improve transmission efficiency.
2. What is Instant Transmission
I will use this project to explain the implementation logic of Miaochuan in an easy-to-understand way.
1. When the client Vue uploads a file, it first calculates the MD5 value of the file, and then sends the MD5 value to the SpringBoot server.
2. After the SpringBoot server receives the MD5 value , it uses MyBatis to query the MySQL database to check whether a file with the same MD5 value already exists .
3. If it exists, it means that the file has been uploaded, and the server directly inquires which fragments exist in the file from the database, and returns it to the client.
4. After the client gets the file fragmentation information, it will directly assemble the complete file without uploading the actual file content.
5. If the MD5 value does not exist in the database, it means that the file has not been uploaded, and the server will return that the client needs to upload the entire file.
6. After the client uploads the file, the server will add the corresponding relationship between the file and MD 5 in the MySQL database, and store the file fragmentation information.
7. When the same file is uploaded next time, the MD5 value can be used to upload it in seconds.
So the core is to use the MySQL database to record the MD5 and fragmentation information of each file , query through MD5 when uploading , and MySQL can judge whether to allow instant transmission, so as to avoid repeated uploads of the same file.
3. What is resume from breakpoint
Then I will explain the concept of resuming transmission with this project in an easy-to-understand manner.
1. When the front-end Vue uploads a file, it cuts the file into multiple small pieces and uploads one small piece each time.
2. Every time a small block is uploaded, the backend SpringBoot will record the information of this small block, such as the serial number of the small block, file MD5, content Hash, etc. Can be saved in a MySQL database.
3. If the upload is interrupted, the Vue side can ask SpringBoot which small pieces have been uploaded.
4. SpringBoot queries from the database and returns the uploaded small piece of information to Vue.
5. Vue can continue to upload only the small part that is interrupted in the middle.
6. SpringBoot will reassemble these small blocks into a complete file according to the serial number of the small blocks and the MD5 of the file.
7. If the file is uploaded again in the future, the MD5 value can be used to know that the file already exists, and the upload success will be returned directly without uploading the actual content. In this way, by uploading slices and persistently recording the information of the uploaded slices, it is possible to resume uploading from breakpoints.
The key is that SpringBoot needs to provide an interface to record and obtain the uploaded slice information. The Vue side needs to slice and upload in order. Finally, SpringBoot puts together files. Simply put, the upload is interrupted due to network and other reasons. By recording the amount of transmitted data, in A technical solution for continuing to upload remaining data after an interruption.
4. What is multipart upload
In the end, I will explain the concept of multi-part upload in an easy-to-understand way based on this project.
1. The purpose of uploading in pieces is to cut a large file into multiple small pieces to achieve concurrent uploads to improve transmission speed.
2. Large files can be divided according to the configured fragment size (for example, a fragment of 50M).
3. The Vue project uploads each split to the SpringBoot server in order, and then uploads each piece in order.
4. After the SpringBoot server receives the fragment, it can temporarily store it locally, and record the characteristic information of the fragment, such as the fragment serial number, file MD5, etc., and write it into the database.
5. After uploading all the fragments, SpringBoot reassembles them into the original complete files in sequence.
5. Upload process
This is the front-end slice processing logic I drew:
This is the logic I drew for processing slices in the backend:
This is my logical combing of the overall project execution process:
1. When created, initialize the uploader component, specify the fragment size, upload method and other configurations.
2. In the onFileAdded method, after the MD5 is calculated for the selected file, call file.resume() to start uploading.
3. file.resume() first sends a GET request internally to ask the server for the uploaded fragments of the file.
4. The server returns a JSON containing a list of uploaded fragments.
5. The uploader component calls checkChunkUploadedByResponse to check whether the current chunk is in the uploaded list.
6. For unuploaded segments, file.resume() will continue to trigger a POST request to upload the segment.
7. The POST request will contain information such as the data and offset of a fragment.
8. The server receives the fragmented data, writes it to the specified location of the file and returns a successful response.
9. The uploader component will record that the segment has been uploaded.
10. After uploading all the fragments in sequence, the server side merges all the fragments into a complete file.
11. onFileSuccess is called to notify that the upload is successful.
12. In this way, through the GET request to inquire about the uploaded fragments + POST uploading unfinished fragments + verification, the breakpoint resume / fragment upload is realized.
Here is the swimlane graph I drew:
6. Build the SpringBoot project
Next, I will explain step by step how to build the backend of this project. Here is a screenshot of the complete project.
6.1. Preparations
6.1.1. Import pom dependencies
This is the complete dependency information of the backend.
Full code:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- 常用工具类 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<!-- MySQL依赖 -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.29</version>
</dependency>
<!-- Mybatis依赖 -->
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>2.2.2</version>
</dependency>
<!-- Lombok依赖 -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>6.1.2, yml configuration file
The main configuration backend port is 9090, the single file limit is up to 100MB, the configuration information of MySQL and the configuration information of MyBatis.
Full code:
server:
port: 9090
spring:
servlet:
multipart:
max-request-size: 100MB
max-file-size: 100MB
datasource:
username: 用户名
password: 密码
url: jdbc:mysql://localhost:3306/数据库名?useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=UTC
driver-class-name: com.mysql.cj.jdbc.Driver
# MyBatis配置
mybatis:
# 搜索指定包别名
typeAliasesPackage: com.example.**.domain
# 配置mapper的扫描，找到所有的mapper.xml映射文件
mapperLocations: classpath:mapping/*.xml
# 加载全局的配置文件
configLocation: classpath:mybatis/mybatis-config.xml
6.1.3, mybatis-config.xml configuration file
Create a new mybatis folder under the resource folder to store the configuration file of mybatis.
Full code:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration
PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
<!-- 全局参数 -->
<settings>
<!-- 使全局的映射器启用或禁用缓存 -->
<setting name="cacheEnabled" value="true" />
<!-- 允许JDBC 支持自动生成主键 -->
<setting name="useGeneratedKeys" value="true" />
<!-- 配置默认的执行器.SIMPLE就是普通执行器;REUSE执行器会重用预处理语句(prepared statements);BATCH执行器将重用语句并执行批量更新 -->
<setting name="defaultExecutorType" value="SIMPLE" />
<!-- 指定 MyBatis 所用日志的具体实现 -->
<setting name="logImpl" value="SLF4J" />
<!-- 使用驼峰命名法转换字段 -->
<setting name="mapUnderscoreToCamelCase" value="true"/>
</settings>
</configuration>6.1.4, SQL file
Store file fragmentation table:
CREATE TABLE `file_chunk` (
`id` bigint NOT NULL AUTO_INCREMENT COMMENT '主键',
`identifier` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '文件md5',
`chunk_number` int NULL DEFAULT NULL COMMENT '当前分块序号',
`chunk_size` bigint NULL DEFAULT NULL COMMENT '分块大小',
`current_chunk_size` bigint NULL DEFAULT NULL COMMENT '当前分块大小',
`total_size` bigint NULL DEFAULT NULL COMMENT '文件总大小',
`total_chunks` int NULL DEFAULT NULL COMMENT '分块总数',
`filename` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '文件名',
`create_time` datetime NULL DEFAULT NULL COMMENT '创建时间',
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_general_ci COMMENT = '文件分片表' ROW_FORMAT = DYNAMIC;Store file information table:
CREATE TABLE `file_info` (
`id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '主键',
`origin_file_name` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL COMMENT '文件源名',
`file_name` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '新存放文件名',
`file_path` varchar(1000) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL COMMENT '文件存放路径',
`file_size` bigint NULL DEFAULT NULL COMMENT '文件总大小',
`create_time` datetime NULL DEFAULT NULL COMMENT '创建时间',
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci COMMENT = '文件信息表' ROW_FORMAT = Dynamic;6.2, constant class
6.2.1, HttpStatus common return status code constants
Full code:
package com.example.bigupload.constant;
/**
* 返回状态码
* @author HTT
*/
public class HttpStatus
{
/**
* 操作成功
*/
public static final int SUCCESS = 200;
/**
* 对象创建成功
*/
public static final int CREATED = 201;
/**
* 请求已经被接受
*/
public static final int ACCEPTED = 202;
/**
* 操作已经执行成功，但是没有返回数据
*/
public static final int NO_CONTENT = 204;
/**
* 资源已被移除
*/
public static final int MOVED_PERM = 301;
/**
* 重定向
*/
public static final int SEE_OTHER = 303;
/**
* 资源没有被修改
*/
public static final int NOT_MODIFIED = 304;
/**
* 参数列表错误（缺少，格式不匹配）
*/
public static final int BAD_REQUEST = 400;
/**
* 未授权
*/
public static final int UNAUTHORIZED = 401;
/**
* 访问受限，授权过期
*/
public static final int FORBIDDEN = 403;
/**
* 资源，服务未找到
*/
public static final int NOT_FOUND = 404;
/**
* 不允许的http方法
*/
public static final int BAD_METHOD = 405;
/**
* 资源冲突，或者资源被锁
*/
public static final int CONFLICT = 409;
/**
* 不支持的数据，媒体类型
*/
public static final int UNSUPPORTED_TYPE = 415;
/**
* 系统内部错误
*/
public static final int ERROR = 500;
/**
* 接口未实现
*/
public static final int NOT_IMPLEMENTED = 501;
}
6.3, Entity class
6.3.1, AjaxResult unified result encapsulation
Full code:
package com.example.bigupload.domain;
import com.example.bigupload.constant.HttpStatus;
import org.apache.commons.lang3.ObjectUtils;
import java.util.HashMap;
/**
* 操作消息提醒
*
* @author HTT
*/
public class AjaxResult extends HashMap<String, Object>
{
private static final long serialVersionUID = 1L;
/** 状态码 */
public static final String CODE_TAG = "code";
/** 返回内容 */
public static final String MSG_TAG = "msg";
/** 数据对象 */
public static final String DATA_TAG = "data";
/**
* 初始化一个新创建的 AjaxResult 对象，使其表示一个空消息。
*/
public AjaxResult()
{
}
/**
* 初始化一个新创建的 AjaxResult 对象
*
* @param code 状态码
* @param msg 返回内容
*/
public AjaxResult(int code, String msg)
{
super.put(CODE_TAG, code);
super.put(MSG_TAG, msg);
}
/**
* 初始化一个新创建的 AjaxResult 对象
*
* @param code 状态码
* @param msg 返回内容
* @param data 数据对象
*/
public AjaxResult(int code, String msg, Object data)
{
super.put(CODE_TAG, code);
super.put(MSG_TAG, msg);
if (ObjectUtils.isNotEmpty(data))
{
super.put(DATA_TAG, data);
}
}
/**
* 返回成功消息
*
* @return 成功消息
*/
public static AjaxResult success()
{
return AjaxResult.success("操作成功");
}
/**
* 返回成功数据
*
* @return 成功消息
*/
public static AjaxResult success(Object data)
{
return AjaxResult.success("操作成功", data);
}
/**
* 返回成功消息
*
* @param msg 返回内容
* @return 成功消息
*/
public static AjaxResult success(String msg)
{
return AjaxResult.success(msg, null);
}
/**
* 返回成功消息
*
* @param msg 返回内容
* @param data 数据对象
* @return 成功消息
*/
public static AjaxResult success(String msg, Object data)
{
return new AjaxResult(HttpStatus.SUCCESS, msg, data);
}
/**
* 返回错误消息
*
* @return
*/
public static AjaxResult error()
{
return AjaxResult.error("操作失败");
}
/**
* 返回错误消息
*
* @param msg 返回内容
* @return 警告消息
*/
public static AjaxResult error(String msg)
{
return AjaxResult.error(msg, null);
}
/**
* 返回错误消息
*
* @param msg 返回内容
* @param data 数据对象
* @return 警告消息
*/
public static AjaxResult error(String msg, Object data)
{
return new AjaxResult(HttpStatus.ERROR, msg, data);
}
/**
* 返回错误消息
*
* @param code 状态码
* @param msg 返回内容
* @return 警告消息
*/
public static AjaxResult error(int code, String msg)
{
return new AjaxResult(code, msg, null);
}
/**
* 方便链式调用
*
* @param key 键
* @param value 值
* @return 数据对象
*/
@Override
public AjaxResult put(String key, Object value)
{
super.put(key, value);
return this;
}
}
6.3.2, file slice class
Full code:
package com.example.bigupload.domain;
import lombok.Data;
import org.springframework.web.multipart.MultipartFile;
import java.io.Serializable;
import java.util.Date;
@Data
public class FileChunk implements Serializable {
/**
* 主键
*/
private Long id;
/**
* 文件 md5
*/
private String identifier;
/**
* 当前分块序号
*/
private Integer chunkNumber;
/**
* 分块大小
*/
private Long chunkSize;
/**
* 当前分块大小
*/
private Long currentChunkSize;
/**
* 文件总大小
*/
private Long totalSize;
/**
* 分块总数
*/
private Integer totalChunks;
/**
* 文件名
*/
private String filename;
/**
* 创建时间
*/
private Date createTime;
/**
* 文件分片数据
*/
private MultipartFile file;
}
6.3.3, file information class
Full code:
package com.example.bigupload.domain;
import lombok.Data;
import java.util.Date;
@Data
public class FileInfo {
/**
* 主键
*/
private String id;
/**
* 文件源名
*/
private String originFileName;
/**
* 新存放文件名
*/
private String fileName;
/**
* 文件存放路径
*/
private String filePath;
/**
* 文件总大小
*/
private Long fileSize;
/**
* 创建时间
*/
private Date createTime;
}
6.4, GlobalCorsConfig global CORS cross-domain configuration
1. The @Configuration annotation indicates that this is a configuration class.
2. The WebMvcConfigurer interface is used to customize the configuration of SpringMVC.
3. The addCorsMappings method is used to define cross-domain access policies.
4. addMapping("/**") means to intercept all request paths.
5. allowedOriginPatterns("*") indicates that all domain name requests are allowed.
6. allowCredentials(true) means that cookies are allowed.
7. allowedHeaders indicates the allowed request headers.
8. allowedMethods indicates the allowed request methods.
9. maxAge(3600) indicates that the validity period of the preflight request is 3600 seconds.
In this way, a global CORS cross-domain configuration is realized, allowing requests from all domain names to access this service, and the request header and method can be set freely, and the maximum validity period is 1 hour.
Full code:
package com.example.bigupload.config;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.CorsRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
@Configuration
public class GlobalCorsConfig implements WebMvcConfigurer {
@Override
public void addCorsMappings(CorsRegistry registry) {
registry.addMapping("/**")
.allowedOriginPatterns("*")
.allowCredentials(true)
.allowedHeaders("GET", "POST", "PUT", "DELETE", "OPTIONS")
.allowedHeaders("Authorization", "Cache-Control", "Content-Type")
.maxAge(3600);
}
}
6.5, persistence layer
6.5.1, FileChunkMapper.xml file
Mainly write two SQL, one is to query all the fragmentation information of the database according to the md5 encrypted information, and the other is to record the file information of each successful fragment upload.
Full code:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper
PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.example.bigupload.mapper.FileChunkMapper">
<select id="findFileChunkParamByMd5" resultType="com.example.bigupload.domain.FileChunk">
SELECT * FROM file_chunk where identifier =#{identifier}
</select>
<select id="findCountByMd5" resultType="Integer">
SELECT COUNT(*) FROM file_chunk where identifier =#{identifier}
</select>
<insert id="insertFileChunk" parameterType="com.example.bigupload.domain.FileChunk">
INSERT INTO file_chunk
<trim prefix="(" suffix=")" suffixOverrides=",">
<if test="identifier != null">identifier,</if>
<if test="chunkNumber != null">chunk_number,</if>
<if test="chunkSize != null">chunk_size,</if>
<if test="currentChunkSize != null">current_chunk_size,</if>
<if test="totalSize != null">total_size,</if>
<if test="totalChunks != null">total_chunks,</if>
<if test="filename != null">filename,</if>
<if test="createTime != null">create_time,</if>
</trim>
<trim prefix="VALUES (" suffix=")" suffixOverrides=",">
<if test="identifier != null">#{identifier},</if>
<if test="chunkNumber != null">#{chunkNumber},</if>
<if test="chunkSize != null">#{chunkSize},</if>
<if test="currentChunkSize != null">#{currentChunkSize},</if>
<if test="totalSize != null">#{totalSize},</if>
<if test="totalChunks != null">#{totalChunks},</if>
<if test="filename != null">#{filename},</if>
<if test="createTime != null">#{createTime},</if>
</trim>
</insert>
</mapper>
6.5.2, FileChunkMapper file
Full code:
package com.example.bigupload.mapper;
import com.example.bigupload.domain.FileChunk;
import org.apache.ibatis.annotations.Mapper;
import java.util.List;
/**
* @author HTT
*/
@Mapper
public interface FileChunkMapper {
public List<FileChunk> findFileChunkParamByMd5(String identifier);
public Integer findCountByMd5(String identifier);
public int insertFileChunk(FileChunk fileChunk);
}
6.5.3, FileInfoMapper.xml file
Full code:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper
PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.example.bigupload.mapper.FileInfoMapper">
<insert id="insert">
INSERT INTO file_info
(id, origin_file_name, file_name, file_path, file_size, create_time)
VALUES
(#{id}, #{originFileName}, #{fileName}, #{filePath}, #{fileSize}, #{createTime});
</insert>
</mapper>
6.5.4, FileInfoMapper file
package com.example.bigupload.mapper;
import com.example.bigupload.domain.FileInfo;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface FileInfoMapper {
int insert(FileInfo fileInfo);
}
6.6. File upload core service class (important)
6.6.1, the logic of file upload check
1. According to the file identifier (md5), query whether the file already exists from the database.
List<FileChunk> list = fileChunkMapper.findFileChunkParamByMd5(fileChunk.getIdentifier());
Map<String, Object> data = new HashMap<>(1);2. If it cannot be found, it means that the file has not been uploaded, and uploaded=false is returned.
if (list == null || list.size() == 0) {
data.put("uploaded", false);
return AjaxResult.success("文件上传成功",data);
}3. If only one piece is found in the query, it means that the entire file has been uploaded, and uploaded=true is returned.
if (list.get(0).getTotalChunks() == 1) {
data.put("uploaded", true);
data.put("url", "");
return AjaxResult.success("文件上传成功",data);
}4. If multiple pieces of data are queried, it means that this is a large file uploaded in pieces.
5. Traverse the fragmented data, obtain the number of each file block and save it in the uploadedFiles array.
6. Finally, return uploadedChunks to the front end.
// 处理分片
int[] uploadedFiles = new int[list.size()];
int index = 0;
for (FileChunk fileChunkItem : list) {
uploadedFiles[index] = fileChunkItem.getChunkNumber();
index++;
}
data.put("uploadedChunks", uploadedFiles);
return AjaxResult.success("文件上传成功",data);7. After the front-end gets the data, it knows which fragments have been uploaded to the large file and which fragments are left to be uploaded.
8. Then continue to upload the remaining fragments at the breakpoint to achieve the effect of resume at the breakpoint. So this code mainly checks the file database to determine whether all or part of the file already exists, so as to determine whether it is necessary to upload the complete file or continue the breakpoint upload.
In this way, repeated uploads can be avoided and transmission efficiency can be improved, which is the key logic to realize instant transmission and resumable transmission.
key code:
public AjaxResult checkUpload(FileChunk fileChunk) {
List<FileChunk> list = fileChunkMapper.findFileChunkParamByMd5(fileChunk.getIdentifier());
Map<String, Object> data = new HashMap<>(1);
// 判断文件存不存在
if (list == null || list.size() == 0) {
data.put("uploaded", false);
return AjaxResult.success("文件上传成功",data);
}
// 处理单文件
if (list.get(0).getTotalChunks() == 1) {
data.put("uploaded", true);
data.put("url", "");
return AjaxResult.success("文件上传成功",data);
}
// 处理分片
int[] uploadedFiles = new int[list.size()];
int index = 0;
for (FileChunk fileChunkItem : list) {
uploadedFiles[index] = fileChunkItem.getChunkNumber();
index++;
}
data.put("uploadedChunks", uploadedFiles);
return AjaxResult.success("文件上传成功",data);
}6.6.2. Realize the core logic of fragment upload
This code is to use RandomAccessFile to realize the logic of fragment upload and write.
Specific logic:
1. Create a RandomAccessFile object, store it according to the file path, open the file in the mode of reading and writing, and use it to perform random access reading and writing to the file.
RandomAccessFile randomAccessFile = new RandomAccessFile(filePath, "rw");2. Calculate the chunkSize of each fragment. If the front end does not pass it in, use the default value (50MB, which is a constant I defined).
public static final long DEFAULT_CHUNK_SIZE = 50 * 1024 * 1024;long chunkSize = fileChunk.getChunkSize() == 0L ? DEFAULT_CHUNK_SIZE : fileChunk.getChunkSize().longValue();3. Calculate the offset of the current fragment, which is calculated by (fragment number - 1) * fragment size.
long offset = chunkSize * (fileChunk.getChunkNumber() - 1);4. Use the seek() method to locate the offset position of the fragment.
randomAccessFile.seek(offset);5. Use the write() method to write the bytes of the file content of the current fragment.
randomAccessFile.write(fileChunk.getFile().getBytes());6. Repeat this process until all shards are written.
7. Finally close the file.
randomAccessFile.close();RandomAccessFile can read and write files at any position, so it can write to the specified position in the order of fragments to achieve the effect of fragment upload. This method can make full use of the file cache of the operating system, which is more efficient. Each shard is only written once, and there is no need to read and modify files, which saves IO operations. This is the common way to implement fragment upload through RandomAccessFile, splicing each fragment into a complete file.
Let me explain it in an easy-to-understand manner by introducing a simple practical example:
Let me first introduce what is the offset: it is used to determine the position of each fragment in the entire file.
According to this key code:
// 计算每个分片大小
long chunkSize = fileChunk.getChunkSize() == 0L ? DEFAULT_CHUNK_SIZE : fileChunk.getChunkSize().longValue();
// 计算分片的偏移量
long offset = chunkSize * (fileChunk.getChunkNumber() - 1); Offset = fragment size * (shard sequence number - 1)
Take a 230MB file as an example:
1. The chunk size is 50MB.
2. The first fragment number fileChunk, getChunkNumber() is 1.
3. Then the offset of the first slice = 50MB * (1 - 1) = 0.
The fragment number of the second slice is 2, then the offset of the second slice = 50MB * (2 - 1) = 50MB. By analogy, the offset of each slice can be calculated.
The offset formula of the nth slice is: offset = slice size * (segment number of the nth slice - 1).
So the complete execution process is like this:
The 230MB file will be divided into 5 fragments:
1. The first fragment: 50MB, offset is 0.
2. The second fragment: 50MB, with an offset of 50MB.
3. The third fragment: 50MB, with an offset of 100MB.
4. The fourth fragment: 50MB, with an offset of 150MB.
5. The fifth fragment: 30MB, with an offset of 200MB.
The client can upload these 5 fragments concurrently:
1. Upload the first fragment and write to offset 0.
2. Upload the second fragment and write to offset 50MB.
3. Upload the third fragment and write to the offset 100MB.
4. Upload the fourth fragment and write it at offset 150MB.
5. Upload the fifth fragment and write to the offset 200MB.
The server side can directly write the content of each shard according to the offset through RandomAccessFile. In this way, the large file of 230MB can be quickly uploaded through five 50MB fragments, realizing the effect of fragment upload and improving the transmission speed.
key code:
private void uploadFileByRandomAccessFile(String filePath, FileChunk fileChunk) throws IOException {
RandomAccessFile randomAccessFile = new RandomAccessFile(filePath, "rw");
// 分片大小必须和前端匹配，否则上传会导致文件损坏
long chunkSize = fileChunk.getChunkSize() == 0L ? DEFAULT_CHUNK_SIZE : fileChunk.getChunkSize().longValue();
// 偏移量
long offset = chunkSize * (fileChunk.getChunkNumber() - 1);
// 定位到该分片的偏移量
randomAccessFile.seek(offset);
// 写入
randomAccessFile.write(fileChunk.getFile().getBytes());
randomAccessFile.close();
}6.6.3. File Fragmentation Processing
The most important thing here is the uploadFileByRandomAccessFile method, which implements random reading and writing of file fragments, and can write the uploaded file fragment data to the correct offset position.
Query the total number of fragments in the database according to the unique identifier (MD5) of the file. If it is equal to the total number of front-end slices, it means that the large file has been uploaded, and the complete file information uploaded this time will be stored in the database.
key code:
public AjaxResult uploadChunkFile(FileChunk fileChunk) throws IOException {
String newFileName = fileChunk.getIdentifier() + fileChunk.getFilename();
String filePath = "D:\\大文件分片存放\\" + newFileName;
uploadFileByRandomAccessFile(filePath, fileChunk);
fileChunk.setCreateTime(new Date());
fileChunkMapper.insertFileChunk(fileChunk);
//数据库中已上传的分片总数
Integer count = fileChunkMapper.findCountByMd5(fileChunk.getIdentifier());
if(fileChunk.getTotalChunks().equals(count)){
FileInfo fileInfo = new FileInfo();
String originalFilename = fileChunk.getFile().getOriginalFilename();
fileInfo.setId(UUID.randomUUID().toString());
fileInfo.setOriginFileName(originalFilename);
fileInfo.setFileName(newFileName);
fileInfo.setFilePath(filePath);
fileInfo.setFileSize(fileChunk.getTotalSize());
fileInfo.setCreateTime(new Date());
fileInfoMapper.insert(fileInfo);
}
return AjaxResult.success("文件上传成功");
}6.6.4, complete code
package com.example.bigupload.service;
import com.example.bigupload.domain.AjaxResult;
import com.example.bigupload.domain.FileChunk;
import com.example.bigupload.domain.FileInfo;
import com.example.bigupload.mapper.FileChunkMapper;
import com.example.bigupload.mapper.FileInfoMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import javax.annotation.Resource;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.*;
/**
* @author HTT
*/
@Slf4j
@Service
public class UploadService {
/**
* 默认的分片大小：50MB
*/
public static final long DEFAULT_CHUNK_SIZE = 50 * 1024 * 1024;
@Resource
private FileChunkMapper fileChunkMapper;
@Resource
private FileInfoMapper fileInfoMapper;
public AjaxResult checkUpload(FileChunk fileChunk) {
List<FileChunk> list = fileChunkMapper.findFileChunkParamByMd5(fileChunk.getIdentifier());
Map<String, Object> data = new HashMap<>(1);
// 判断文件存不存在
if (list == null || list.size() == 0) {
data.put("uploaded", false);
return AjaxResult.success("文件上传成功",data);
}
// 处理单文件
if (list.get(0).getTotalChunks() == 1) {
data.put("uploaded", true);
data.put("url", "");
return AjaxResult.success("文件上传成功",data);
}
// 处理分片
int[] uploadedFiles = new int[list.size()];
int index = 0;
for (FileChunk fileChunkItem : list) {
uploadedFiles[index] = fileChunkItem.getChunkNumber();
index++;
}
data.put("uploadedChunks", uploadedFiles);
return AjaxResult.success("文件上传成功",data);
}
/**
* 上传分片文件
* @param fileChunk
* @return
* @throws Exception
*/
public AjaxResult uploadChunkFile(FileChunk fileChunk) throws IOException {
String newFileName = fileChunk.getIdentifier() + fileChunk.getFilename();
String filePath = "D:\\大文件分片存放\\" + newFileName;
uploadFileByRandomAccessFile(filePath, fileChunk);
fileChunk.setCreateTime(new Date());
fileChunkMapper.insertFileChunk(fileChunk);
//数据库中已上传的分片总数
Integer count = fileChunkMapper.findCountByMd5(fileChunk.getIdentifier());
if(fileChunk.getTotalChunks().equals(count)){
FileInfo fileInfo = new FileInfo();
String originalFilename = fileChunk.getFile().getOriginalFilename();
fileInfo.setId(UUID.randomUUID().toString());
fileInfo.setOriginFileName(originalFilename);
fileInfo.setFileName(newFileName);
fileInfo.setFilePath(filePath);
fileInfo.setFileSize(fileChunk.getTotalSize());
fileInfo.setCreateTime(new Date());
fileInfoMapper.insert(fileInfo);
}
return AjaxResult.success("文件上传成功");
}
private void uploadFileByRandomAccessFile(String filePath, FileChunk fileChunk) throws IOException {
RandomAccessFile randomAccessFile = new RandomAccessFile(filePath, "rw");
// 分片大小必须和前端匹配，否则上传会导致文件损坏
long chunkSize = fileChunk.getChunkSize() == 0L ? DEFAULT_CHUNK_SIZE : fileChunk.getChunkSize().longValue();
// 偏移量
long offset = chunkSize * (fileChunk.getChunkNumber() - 1);
// 定位到该分片的偏移量
randomAccessFile.seek(offset);
// 写入
randomAccessFile.write(fileChunk.getFile().getBytes());
randomAccessFile.close();
}
}
6.7, FileController request layer
The writing is very concise, the reason why they are all the same /upload interface, the request method is get and post respectively, because the uploader component first calls the get request of /upload when checking, and then uploads the fragment and calls the post request of /upload .
1. First use the GET request to verify whether the file exists, and decide whether to implement instant transmission, breakpoint resume transmission or multi-part upload.
2. Then the client uses POST requests to upload each fragment in order, and the server writes the files after receiving the fragments.
Full code:
package com.example.bigupload.controller;
import com.example.bigupload.domain.AjaxResult;
import com.example.bigupload.domain.FileChunk;
import com.example.bigupload.service.UploadService;
import org.springframework.web.bind.annotation.*;
import javax.annotation.Resource;
@RestController
@RequestMapping("/file")
public class FileController {
@Resource
private UploadService uploadService;
@GetMapping("/upload")
public AjaxResult checkUpload(FileChunk fileChunk){
return uploadService.checkUpload(fileChunk);
}
@PostMapping("/upload")
public AjaxResult uploadChunkFile(FileChunk fileChunk) throws Exception {
return uploadService.uploadChunkFile(fileChunk);
}
}
7. Build a Vue project
Here I am using Vue2.0, this is the complete screenshot of the project
7.1. Preparations
1. Install uploader
npm install --save vue-simple-uploader2. According to spark-md5
npm install --save spark-md53. Introduce components in main.js
import uploader from 'vue-simple-uploader'
Vue.use(uploader)7.2, HomeView page
In fact, the main logic here is to first slice and generate the MD5 check code of this file through the getFileMD5 method, and then use the file. All the uploaded fragment information of the current file, and then use the logic in the checkChunkUploadedByResponse method to judge which fragments still need to be uploaded.
7.2.1. Using the uploader component
This component uses the third-party library vue-uploader, which is an upload component.
Parameter explanation:
1. ref: Give the component a reference name, here is uploader.
2. options: upload options, such as upload address, accepted file types, etc.
3. autoStart: Whether to start uploading automatically, the default is true, here it is set to false.
4. fileStatusText: Custom upload status prompt text.
5. @file-added: The hook when the file is added.
6. @file-success: Hook when the file is uploaded successfully.
7. @file-error: Hook when file upload fails.
8. @file-progress: Hook for file upload progress.
Contains three subcomponents:
1. uploader-unsupport: If the browser does not support it, display this component.
2. uploader-drop: Drag and drop the upload area component to customize the content.
3. uploader-btn: upload button component.
4. uploader-files: A list component that has selected files to be uploaded.
Therefore, this component implements complete file upload functions such as selecting files, dragging and dropping uploads, displaying upload progress, and calling back upload results. We can configure upload parameters through options, process upload results through various hooks, and customize the style of the upload area to implement a complete upload component.
Full code:
<template>
<div class="home">
<uploader
ref="uploader"
:options="options"
:autoStart="false"
:file-status-text="fileStatusText"
@file-added="onFileAdded"
@file-success="onFileSuccess"
@file-error="onFileError"
@file-progress="onFileProgress"
class="uploader-example"
>
<uploader-unsupport></uploader-unsupport>
<uploader-drop>
<p>将文件拖放到此处以上传</p>
<uploader-btn>选择文件</uploader-btn>
</uploader-drop>
<uploader-files> </uploader-files>
</uploader>
<br />
</div>
</template>7.2.2. Initialize data data
First introduce the Md5 tool and initialize the fragment size in advance.
import SparkMD5 from "spark-md5";
const CHUNK_SIZE = 50 * 1024 * 1024;options parameter:
1. target: the uploaded interface address
2. testChunks: whether to enable multipart upload verification. Partial upload is to divide the file into small pieces and upload them concurrently, which can improve the efficiency of uploading large files.
3. uploadMethod: upload method, the default is POST.
4. chunkSize: Fragment size, the default is 1MB. This can be adjusted according to the actual situation. If the fragment is too small, the number of requests will be increased, and if the fragment is too large, it will retransmit a lot if the upload fails.
5. simultaneousUploads: The number of concurrent upload blocks, the default is 3, which can be adjusted as needed.
7.2.3, checkChunkUploadedByResponse method (important)
The function of this method is to verify whether the currently uploaded fragment has been successfully uploaded.
This method will only call the get request provided by the upload interface once to get all the fragment information of this file from the backend. It has two parameters:
chunk: Information about the current fragment, including data such as offset.
message: call all fragmentation information returned by the get request provided by the upload interface.
If this method returns true, it means that the current segment has been successfully uploaded, then there is no need to call the upload method to send a post upload request.
Inside the function, the response content message returned by the server is first parsed into a JSON object.
let messageObj = JSON.parse(message);Then take out the data field as a data object.
let dataObj = messageObj.data;Then judge whether the uploaded field is included in the data object. If there is a value of uploaded directly returned, the uploaded field is usually a sign of the server indicating whether the upload of the fragment is completed.
if (dataObj.uploaded !== undefined) {
return dataObj.uploaded;
}If the uploaded part list is: [1, 2], that is, part 1 and part 2 have been uploaded successfully, and part 3 and part 4 have not been uploaded yet.
The current block to be verified is block 2, offset=1.
Then the execution process is:
1. The value of dataObj.uploadedChunks is [1, 2].
2. The value of chunk.offset is still 1.
3. Chunk.offset+1 calculates that the sequence number of block 2 is 2.
4. indexOf(2) looks up the value 2 in the array [1, 2].
5. The returned index value is 1, that is, found.
So for part 2, return true, indicating that the part has been uploaded successfully.
return (dataObj.uploadedChunks || []).indexOf(chunk.offset + 1) >= 0;The purpose of return is to check whether the current block has been successfully uploaded by looking at the list of uploaded blocks returned by the server, otherwise continue to call the post request provided by upload to upload.
7.2.4, parseTimeRemaining method
Method to format the remaining upload time. The original time will be formatted as xx days xx hours xx minutes xx seconds.
key code:
parseTimeRemaining: function (timeRemaining, parsedTimeRemaining) {
return parsedTimeRemaining
.replace(/\syears?/, "年")
.replace(/\days?/, "天")
.replace(/\shours?/, "小时")
.replace(/\sminutes?/, "分钟")
.replace(/\sseconds?/, "秒");
},Therefore, these configurations are mainly for uploading in pieces, and improve the efficiency and experience of uploading large files through mechanisms such as chunking, concurrency, and verification.
key code:
options: {
target: 'http://127.0.0.1:9090/file/upload',
testChunks: true,
uploadMethod: "post",
chunkSize: CHUNK_SIZE,
// 并发上传数，默认为 3
simultaneousUploads: 3,
checkChunkUploadedByResponse: (chunk, message) => {
let messageObj = JSON.parse(message);
let dataObj = messageObj.data;
if (dataObj.uploaded !== undefined) {
return dataObj.uploaded;
}
return (dataObj.uploadedChunks || []).indexOf(chunk.offset + 1) >= 0;
},
parseTimeRemaining: function (timeRemaining, parsedTimeRemaining) {
return parsedTimeRemaining
.replace(/\syears?/, "年")
.replace(/\days?/, "天")
.replace(/\shours?/, "小时")
.replace(/\sminutes?/, "分钟")
.replace(/\sseconds?/, "秒");
},
},7.2.5. Other parameters
Parameter role:
1. fileStatus: Define the text description of the upload status, which will be used in the component that displays the upload status.
2. uploadFileList: the list of files that have been successfully uploaded, which can be used to save the information of the successfully uploaded files.
key code:
fileStatus: {
success: "上传成功",
error: "上传错误",
uploading: "正在上传",
paused: "停止上传",
waiting: "等待中",
},
uploadFileList: [],7.2.6, onFileAdded method (important)
Specific logic:
1. Add the added file to uploadFileList.
2. After calculating MD5, call file.resume().
3. file.resume() will construct a GET request internally, including the MD5 information of the file, and send it to the server.
4. After receiving the request, the server checks whether the file exists according to MD5.
5. If it exists, return the uploaded block information.
6. After the front end receives the response, it calls the checkChunkUploadedByResponse logic.
7. checkChunkUploadedByResponse checks the upload status of each chunk according to the response data, and judges which chunks have been uploaded and which have not been uploaded according to the response.
8. Upload the unuploaded chunks. For unuploaded pieces, file.resume() will continue to trigger the request (POST) to upload the piece until all the pieces are verified and uploaded.
So the file.resume() method includes not only the verification logic of checkChunkUploadedByResponse, but also the logic of triggering the actual part upload.
key code:
onFileAdded(file) {
this.uploadFileList.push(file);
// 2. 计算文件 MD5 并请求后台判断是否已上传，是则取消上传
this.getFileMD5(file, (md5) => {
if (md5 != "") {
// 修改文件唯一标识
file.uniqueIdentifier = md5;
// 恢复上传
file.resume();
}
});
},7.2.7, getFileMD5 method (important)
Calculate the MD5 value of the file before uploading, and perform MD5 accumulation calculation while slicing. The general logic is as follows:
First initialize the SparkMD5 object spark to save the cumulative value of MD5, and then read the file content in blocks through FileReader, each time reading a chunk-sized slice, when reading each slice, append the slice content to the spark object, Perform MD5 accumulation, and then repeat the process until all slices are read. Finally, call spark.end() to get the cumulatively calculated overall MD5 value of the file.
Specific logic:
1. Create the SparkMD5 object spark to calculate the MD5 value.
let spark = new SparkMD5.ArrayBuffer();2. Create a FileReader object fileReader for reading file content.
let fileReader = new FileReader();3. Obtain the compatible implementation of the slice method of the File object, responsible for obtaining the slice method of the file.
let blobSlice =
File.prototype.slice ||
File.prototype.mozSlice ||
File.prototype.webkitSlice;4. Calculate the total number of chunks, and the size of each chunk is CHUNK_SIZE.
let currentChunk = 0;
let chunks = Math.ceil(file.size / CHUNK_SIZE);5. Record start time startTime.
let startTime = new Date().getTime();6. Pause uploading file file.pause().
file.pause();7. The loadNext function is used to load file fragments.
Specific logic:
The first step is to calculate the starting position start of the current fragment, which is the current fragment serial number currentChunk multiplied by the fragment size CHUNK_SIZE.
The second step is to calculate the end position of the current fragment: if start + fragment size is greater than or equal to the total size file.size, then end is the total size file.size, otherwise end is start + fragment size CHUNK_SIZE.
The third step is to obtain a slice Blob object of a file through the blobSlice method, which intercepts the part from start to end of the file. In fact, it is to call the slicing method, pass in the file object and range, and perform the actual file slicing operation.
Step 4: Call the readAsArrayBuffer method of FileReader to read the content of the Blob fragment object.
Step 5: readAsArrayBuffer will trigger the onload event of FileReader , and the fragment content can be obtained and processed later in the event callback.
So the function of loadNext is to obtain a fragment Blob of the specified file according to the fragment size and serial number, and then use FileReader to read the ArrayBuffer content of this fragment.
key code:
function loadNext() {
const start = currentChunk * CHUNK_SIZE;
const end =
start + CHUNK_SIZE >= file.size ? file.size : start + CHUNK_SIZE;
fileReader.readAsArrayBuffer(blobSlice.call(file.file, start, end));
}8. This code is the callback function of the onload event of FileReader, which is used to process the MD5 accumulation calculation after reading the fragment content.
Specific logic:
The first step is to append the read fragmented ArrayBuffer content to spark through spark.append(e.target.result), and perform MD5 accumulation.
The second step is to judge that if the currentChunk of the fragment number is still smaller than the total number of chunks, it means that there are still unread fragments: currentChunk is incremented automatically, and loadNext() is called to load the next fragment.
The third step, otherwise, it means that all fragments have been read: calculate the final MD5 value md5 through spark and end(), print and calculate time-consuming, and return the md5 value through callback.
The fourth step, the callback function will receive md5 from the outside, judge whether the file is duplicated and determine the follow-up logic.
So the function of this code is to recursively read the file fragments, accumulate and calculate MD5, and finally return the MD5 result, completing the MD5 verification of the entire file. By loading in pieces, performance problems caused by loading the entire large file at one time are avoided, and efficient MD5 calculation is realized.
key code:
fileReader.onload = function (e) {
spark.append(e.target.result);
if (currentChunk < chunks) {
currentChunk++;
loadNext();
} else {
let md5 = spark.end();
console.log(
`MD5计算完毕：${md5}，耗时：${new Date().getTime() - startTime} ms.`
);
callback(md5);
}
};
fileReader.onerror = function () {
this.$message.error("文件读取错误");
file.cancel();
};7.2.8, complete code
<template>
<div class="home">
<uploader
ref="uploader"
:options="options"
:autoStart="false"
:file-status-text="fileStatusText"
@file-added="onFileAdded"
@file-success="onFileSuccess"
@file-error="onFileError"
@file-progress="onFileProgress"
class="uploader-example"
>
<uploader-drop>
<p>将文件拖放到此处以上传</p>
<uploader-btn>选择文件</uploader-btn>
</uploader-drop>
<uploader-files> </uploader-files>
</uploader>
<br />
</div>
</template>
<script>
import HelloWorld from '@/components/HelloWorld.vue'
import SparkMD5 from "spark-md5";
const CHUNK_SIZE = 50 * 1024 * 1024;
export default {
name: 'HomeView',
components: {
HelloWorld
},
data() {
return {
options: {
target: 'http://127.0.0.1:9090/file/upload',
testChunks: true,
uploadMethod: "post",
chunkSize: CHUNK_SIZE,
simultaneousUploads: 3,
checkChunkUploadedByResponse: (chunk, message) => {
let messageObj = JSON.parse(message);
let dataObj = messageObj.data;
if (dataObj.uploaded !== undefined) {
return dataObj.uploaded;
}
return (dataObj.uploadedChunks || []).indexOf(chunk.offset + 1) >= 0;
},
parseTimeRemaining: function (timeRemaining, parsedTimeRemaining) {
return parsedTimeRemaining
.replace(/\syears?/, "年")
.replace(/\days?/, "天")
.replace(/\shours?/, "小时")
.replace(/\sminutes?/, "分钟")
.replace(/\sseconds?/, "秒");
},
},
fileStatus: {
success: "上传成功",
error: "上传错误",
uploading: "正在上传",
paused: "停止上传",
waiting: "等待中",
},
uploadFileList: [],
};
},
methods: {
onFileAdded(file) {
this.uploadFileList.push(file);
// 2. 计算文件 MD5 并请求后台判断是否已上传，是则取消上传
this.getFileMD5(file, (md5) => {
if (md5 != "") {
// 修改文件唯一标识
file.uniqueIdentifier = md5;
// 恢复上传
file.resume();
}
});
},
onFileSuccess(rootFile, file, response, chunk) {
console.log("上传成功");
},
onFileError(rootFile, file, message, chunk) {
console.log("上传出错：" + message);
},
onFileProgress(rootFile, file, chunk) {
console.log(`当前进度：${Math.ceil(file._prevProgress * 100)}%`);
},
getFileMD5(file, callback) {
let spark = new SparkMD5.ArrayBuffer();
let fileReader = new FileReader();
let blobSlice =
File.prototype.slice ||
File.prototype.mozSlice ||
File.prototype.webkitSlice;
let currentChunk = 0;
let chunks = Math.ceil(file.size / CHUNK_SIZE);
let startTime = new Date().getTime();
file.pause();
loadNext();
fileReader.onload = function (e) {
spark.append(e.target.result);
if (currentChunk < chunks) {
currentChunk++;
loadNext();
} else {
let md5 = spark.end();
console.log(
`MD5计算完毕：${md5}，耗时：${new Date().getTime() - startTime} ms.`
);
callback(md5);
}
};
fileReader.onerror = function () {
this.$message.error("文件读取错误");
file.cancel();
};
function loadNext() {
const start = currentChunk * CHUNK_SIZE;
const end =
start + CHUNK_SIZE >= file.size ? file.size : start + CHUNK_SIZE;
fileReader.readAsArrayBuffer(blobSlice.call(file.file, start, end));
}
},
fileStatusText(status) {
console.log(11111)
console.log(status)
if (status === "md5") {
return "校验MD5";
}
return this.fileStatus[status];
},
},
}
</script>
Eight, run the project
Next, let me actually run this project and show you the effect.
8.1. Upload in pieces
I prepared a file with a size of 141.1MB for upload in advance, and it was cut into 2 pieces by default for upload.
It is the information written in the database:
Fragment information:
File information:
Are you wondering why the 141M file is not cut into 3 pieces, but only 2 pieces, and the size of the second piece is 91MB. Let me explain the reason in detail below:
This is because theoretically speaking, the 141M file will be divided into three fragmented files of 50MB, 50MB and 41MB, but the last 41MB does not meet 50MB. The 3 blocks are merged to reduce the number of splits, so the final slices are 50MB and 91MB.
8.2. Breakpoint resume
Sometimes, there may be some network reasons or the customer wants to save the rest and upload it tomorrow.
Next time, the last upload progress will be kept until the upload is successful!
This is the information written to the database:
Fragment information:
Because only part of the shard information is uploaded, but it will be recorded by the database.
File information:
Because the file has not been uploaded completely, the file information table will not store the file information.
Next time, we will continue to upload according to the progress of last time.
This is the information written to the database:
Fragment information:
File information:
8.3. Second pass
It is very simple to upload in seconds. I choose to upload the large file just now.
Obviously, no slicing was performed, only the verification interface was called once, and all the shard information of this file was queried from the database based on MD5 and returned to the front end, compared with the shard information of the front end, it was checked that the file had been completely uploaded , so the upload is completed within 1s!
9. Gitee source code address
Here I open source the complete code of the project, so you can learn by yourself!
Front end: Vue implements large file transfer in seconds + breakpoint resume transfer + fragment upload
10. Summary
The above is my personal explanation of the complete process of Vue+SpringBoot's realization of large file transfer in seconds, breakpoint resume transfer, and fragment upload. Almost every place is in place. If you have any questions, welcome to discuss in the comment area!