Demonstration of Netty file upload breakpoint resume
1. Theory and protocol specifications and tools, etc.
1. Implementation principle:
Netty file upload is implemented using a custom protocol. The resumable upload is mainly based on the random read and write capabilities of the RandomAccessFile class. The main process is that the client initiates a request. It will need to upload the file name, path, read file data, and read Take information such as the starting position of the file, and cache it in the server (using the file path as the key and the custom protocol object as the value). The server will write the file after receiving the above data sent by the client, and finish writing The file will also record information such as the location of the data that has been written, and send the information again to the data that the client needs to read next time.
If the client disconnects the link during this process, at this time, because the server caches the location of the file that has been written, it will only read the file from the location where the file has been written, and then transfer the file to the server for writing and other cyclic operations. So as to achieve the effect of resuming the tea transmission.
2. Basic use of RandomAccessFile
* Basic api of RandomAccessFile
* .getFilePointer: Get the position of the current operation
* .seek(index): Set the operation position to index
* .read(byte): Read the file into the byte array, return the length of the read file
*
* Structure Function
* rf = new RandomAccessFile(new File(filePath),"r"); // Parameter 2 is the mode
*
* Mode details:
* "r" opens the specified folder in read-only mode. If you try to execute the write method on this RandomAccessFile, an IOException will be thrown.
* "Rw" opens the specified file in read or write mode. If the file does not exist yet, try to create the file.
* "Rws" opens the specified file in read and write mode. Compared with the "rw" mode, each update to the file content or metadata is also required to be synchronously written to the underlying device.
* "Rwd" opens the specified file in read and write mode. Compared with the "rw" mode, it also requires that every update of the file content is synchronously written to the underlying device.
* 测试1：下面演示基本的读文件操作。
*/
@Test
public void test1() throws IOException {
String filePath = "src/d00_file/a.txt";
RandomAccessFile rf = null;
try {
rf = new RandomAccessFile(new File(filePath), "r");
System.out.println("输入内容：" + rf.getFilePointer());
// 从10位置操作
rf.seek(10);
byte[] b = new byte[1024];
int len = 0;
// 循环读写 (len是读取的长度)
while ((len = rf.read(b)) > 0) {
System.out.print(new String(b, 0, len));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
rf.close();
}
}
/**
* 测试2： 向文件追加内容
*/
@Test
public void test2() throws IOException{
String filePath="src/d00_file/a.txt";
RandomAccessFile rf=null;
try {
rf = new RandomAccessFile(new File(filePath), "rw");
// 将操作指针移动到文件的最后
rf.seek(rf.length());
// 向文件写出数据
rf.write("这是追加的内容。。".getBytes());
}catch (IOException e){
e.printStackTrace();
}finally {
rf.close();
}
}
/**
* 测试3：修改文件内容
*/
@Test
public void test3() {
RandomAccessFile rf = null;
String oldStr = "www.www.www";
String newStr = "hahahaha";
try {
rf = new RandomAccessFile("src/d00_file/a.txt", "rw");
String line = null;
// 记录上次操作点
long lastpoint = 0;
while ((line = rf.readLine()) != null) {
long ponit = rf.getFilePointer();
// 如果包含替换字符串
if (line.contains(oldStr)) {
// 替换字符串
String str = line.replace(oldStr, newStr);
// 恢复到上次读取位置 （readLine前的位置）
rf.seek(lastpoint);
// 重写行数据
if (oldStr.length() != newStr.length()) {
byte[] newb = Arrays.copyOf(str.getBytes(), line.getBytes().length);
rf.write(newb);
// lastpoint = lastpoint + newb.length;
lastpoint = rf.getFilePointer();
continue;
} else {
rf.write(str.getBytes());
}
}
lastpoint = ponit;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
rf.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}3. Design of transmission object and protocol object
There are three types of transmission objects, one is the initial transmission information type sent by the client to the server, the other is the fragment file data type, and the other is the fragment file instruction type returned by the server to the client.
The protocol object contains two fields, one is the type (the type of the above three objects), and the other is the obejct field, which represents the transmission object.
constant:
/**
* 文件传输常量
*/
public class Constants {
/**
* 文件传输状态的标示
*/
public static class FileStatus{
public static int BEGIN = 0; //开始
public static int CENTER = 1; //中间
public static int END = 2; //结尾
public static int COMPLETE = 3; //完成
}
/**
* 协议对象的传输对象类型
*/
public static class TransferType{
public static int REQUEST = 0; //文件信息类型
public static int INSTRUCT = 1; //文件指令类型
public static int DATA = 2; //文件分片类型
}
}
Agreement object:
public class FileTransferProtocol {
private Integer transferType; // 类型
private Object transferObj; // 数据对象；(0)FileDescInfo、(1)FileBurstInstruct、(2)FileBurstData
public Integer getTransferType() {
return transferType;
}
public void setTransferType(Integer transferType) {
this.transferType = transferType;
}
public Object getTransferObj() {
return transferObj;
}
public void setTransferObj(Object transferObj) {
this.transferObj = transferObj;
}
}File transfer related objects:
/**
* 文件描述信息
*/
public class FileDescInfo {
private String fileUrl; // 文件url
private String fileName; // 文件名称
private Long fileSize; // 文件size
public String getFileUrl() {
return fileUrl;
}
public void setFileUrl(String fileUrl) {
this.fileUrl = fileUrl;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public Long getFileSize() {
return fileSize;
}
public void setFileSize(Long fileSize) {
this.fileSize = fileSize;
}
}
/**
* 文件分片指令
*/
public class FileBurstInstruct {
private Integer status; // Constants.FileStatus ｛0开始、1中间、2结尾、3完成｝
private String clientFileUrl; // 客户端文件URL
private Integer readPosition; // 读取位置
public FileBurstInstruct() {
}
public FileBurstInstruct(Integer status) {
this.status = status;
}
public Integer getStatus() {
return status;
}
public void setStatus(Integer status) {
this.status = status;
}
public String getClientFileUrl() {
return clientFileUrl;
}
public void setClientFileUrl(String clientFileUrl) {
this.clientFileUrl = clientFileUrl;
}
public Integer getReadPosition() {
return readPosition;
}
public void setReadPosition(Integer readPosition) {
this.readPosition = readPosition;
}
}/**
* 文件分片数据
*/
public class FileBurstData {
private String fileUrl; //客户端文件地址
private String fileName; //文件名称
private Integer beginPos; //开始位置
private Integer endPos; //结束位置
private byte[] bytes; //文件字节；再实际应用中可以使用非对称加密，以保证传输信息安全
private Integer status; //Constants.FileStatus ｛0开始、1中间、2结尾、3完成｝
public FileBurstData(){
}
public String getFileUrl() {
return fileUrl;
}
public void setFileUrl(String fileUrl) {
this.fileUrl = fileUrl;
}
public FileBurstData(Integer status){
this.status = status;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public Integer getBeginPos() {
return beginPos;
}
public void setBeginPos(Integer beginPos) {
this.beginPos = beginPos;
}
public Integer getEndPos() {
return endPos;
}
public void setEndPos(Integer endPos) {
this.endPos = endPos;
}
public byte[] getBytes() {
return bytes;
}
public void setBytes(byte[] bytes) {
this.bytes = bytes;
}
public Integer getStatus() {
return status;
}
public void setStatus(Integer status) {
this.status = status;
}
}4. Codec
From the design of the protocol object, we know that this file upload will use the object transmission method, so you must customize the codec. The custom codec is implemented based on protostuff. When encoding, add an int type When decoding the length field and the serialized protocol object, after decoding the length field, read the data, and then deserialize the protocol object.
1) Serialization tool
/**
* protostuff序列化工具类
*/
public class SerializingUtil {
/**
* 将目标类序列化为byte数组
*/
public static <T> byte[] serialize(T source) {
Schema<T> schema = RuntimeSchema.getSchema((Class<T>) source.getClass());
LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
final byte[] result;
try {
result = ProtobufIOUtil.toByteArray(source, schema, buffer);
} finally {
buffer.clear();
}
return result;
}
/**
* 将byte数组序列化为目标类
*/
public static <T> T deserialize(byte[] source, Class<T> clazz) {
Schema<T> schema = RuntimeSchema.getSchema(clazz);
T t = schema.newMessage();
ProtobufIOUtil.mergeFrom(source, t, schema);
return t;
}
}2) Object decoder (input to processing-placed first in the processor chain-subsequent processors can directly obtain Clazz type objects)
/**
* 对象解码器
*/
public class ObjectDecode extends ByteToMessageDecoder{
private Class<?> clazz ;
public ObjectDecode(Class<?> clazz) {
super();
this.clazz = clazz;
}
@Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
if (in.readableBytes() < 4) {
return;
}
// 标记包头位置
in.markReaderIndex();
// 读取长度
int len = in.readInt();
// 可读数少于长度
if(in.readableBytes() < len) {
// 重置到包头位置
in.resetReaderIndex();
return;
}
// 读取数据并序列化
byte[] bytes = new byte[len];
in.readBytes(bytes);
Object object = SerializingUtil.deserialize(bytes, clazz);
out.add(object);
}
}3) Object encoder (output encoder --- encoding protocol object --- placed in the penultimate processor chain)
public class ObjectEncode extends MessageToByteEncoder<Object> {
private Class<?> clazz;
public ObjectEncode(Class<?> clazz) {
super();
this.clazz = clazz;
}
@Override
protected void encode(ChannelHandlerContext ctx, Object msg, ByteBuf out) throws Exception {
if (clazz.isInstance(msg)) {
byte[] data = SerializingUtil.serialize(msg);
out.writeInt(data.length);
out.writeBytes(data);
}
}
}5. Low-level file reading and writing tools
public class FileUtil {
// 默认一次只能读取10k数据
private static final int DEF_BUFF_SIZE = 1024*10;
private static int buff_size = DEF_BUFF_SIZE;
/**
* 客户端根据文件路径和position读取文件，返回文件分片数据FileBurstData对象。
*/
public static FileBurstData readFile(String fileUrl, Integer readPosition) throws IOException {
File file = new File(fileUrl);
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
randomAccessFile.seek(readPosition);
byte[] bytes = new byte[buff_size];
int readSize = randomAccessFile.read(bytes);
if (readSize <= 0) {
randomAccessFile.close();
return new FileBurstData(Constants.FileStatus.COMPLETE);
}
FileBurstData fileBurstData = new FileBurstData();
fileBurstData.setFileUrl(fileUrl);
fileBurstData.setFileName(file.getName());
fileBurstData.setBeginPos(readPosition);
fileBurstData.setEndPos(readPosition + readSize);
// 不足buff尺寸需要拷贝去掉空字节
if (readSize < buff_size) {
byte[] copy = new byte[readSize];
System.arraycopy(bytes, 0, copy, 0, readSize);
fileBurstData.setBytes(copy);
fileBurstData.setStatus(Constants.FileStatus.END);
} else {
fileBurstData.setBytes(bytes);
fileBurstData.setStatus(Constants.FileStatus.CENTER);
}
randomAccessFile.close();
return fileBurstData;
}
/**
* 服务端根据url和客户端的文件分片数据对象，进行写文件，并且返回文件分片指令（由客户端使用）
*/
public static FileBurstInstruct writeFile(String baseUrl, FileBurstData fileBurstData) throws IOException {
if (Constants.FileStatus.COMPLETE == fileBurstData.getStatus()) {
return new FileBurstInstruct(Constants.FileStatus.COMPLETE); // Constants.FileStatus ｛0开始、1中间、2结尾、3完成｝
}
// 服务端写文件
File file = new File(baseUrl + "/" + fileBurstData.getFileName());
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw");
randomAccessFile.seek(fileBurstData.getBeginPos());
randomAccessFile.write(fileBurstData.getBytes());
randomAccessFile.close();
if (Constants.FileStatus.END == fileBurstData.getStatus()) {
return new FileBurstInstruct(Constants.FileStatus.COMPLETE);
}
// 构建文件分片指令
FileBurstInstruct fileBurstInstruct = new FileBurstInstruct();
fileBurstInstruct.setStatus(Constants.FileStatus.CENTER); // 字段：读取状态
fileBurstInstruct.setClientFileUrl(fileBurstData.getFileUrl());
fileBurstInstruct.setReadPosition(fileBurstData.getEndPos() + 1); // 字段(核心)：下次读取位置
return fileBurstInstruct;
}In addition, a construction tool class for supplementary file protocol objects
/**
* 消息构建对象
*/
public class MsgUtil {
/**
* 构建对象；文件传输info(客户端)
*/
public static FileTransferProtocol buildRequestTransferFile(String fileUrl, String fileName, Long fileSize) {
FileDescInfo fileDescInfo = new FileDescInfo();
fileDescInfo.setFileUrl(fileUrl);
fileDescInfo.setFileName(fileName);
fileDescInfo.setFileSize(fileSize);
FileTransferProtocol fileTransferProtocol = new FileTransferProtocol();
fileTransferProtocol.setTransferType(0);//0请求传输文件、1文件传输指令、2文件传输数据
fileTransferProtocol.setTransferObj(fileDescInfo);
return fileTransferProtocol;
}
/**
* 构建对象；文件传输指令(服务端)
* @param status 文件读取状态
* @param clientFileUrl 客户端文件地址
* @param readPosition 读取位置
*/
public static FileTransferProtocol buildTransferInstruct(Integer status, String clientFileUrl, Integer readPosition) {
FileBurstInstruct fileBurstInstruct = new FileBurstInstruct();
fileBurstInstruct.setStatus(status);
fileBurstInstruct.setClientFileUrl(clientFileUrl);
fileBurstInstruct.setReadPosition(readPosition);
FileTransferProtocol fileTransferProtocol = new FileTransferProtocol();
fileTransferProtocol.setTransferType(Constants.TransferType.INSTRUCT); // 0 (info) 1 (INSTRUCT) 2 (data)
fileTransferProtocol.setTransferObj(fileBurstInstruct);
return fileTransferProtocol;
}
/**
* 构建对象；文件传输指令(服务端)
*/
public static FileTransferProtocol buildTransferInstruct(FileBurstInstruct fileBurstInstruct) {
FileTransferProtocol fileTransferProtocol = new FileTransferProtocol();
fileTransferProtocol.setTransferType(Constants.TransferType.INSTRUCT); // 0 (info) 1 (INSTRUCT) 2 (data)
fileTransferProtocol.setTransferObj(fileBurstInstruct);
return fileTransferProtocol;
}
/**
* 构建对象；文件传输数据(客户端)
*/
public static FileTransferProtocol buildTransferData(FileBurstData fileBurstData) {
FileTransferProtocol fileTransferProtocol = new FileTransferProtocol();
fileTransferProtocol.setTransferType(Constants.TransferType.DATA); // 0 (info) 1 (INSTRUCT) 2 (data)
fileTransferProtocol.setTransferObj(fileBurstData);
return fileTransferProtocol;
}
}Second, the client and server implementation code
The code package and class structure that have been completed above are as follows
1. Client-side writing
public class FileTransferClient {
private EventLoopGroup workerGroup = new NioEventLoopGroup();
private Channel channel;
public ChannelFuture connect(String inetHost, int inetPort) {
ChannelFuture channelFuture = null;
try {
Bootstrap b = new Bootstrap();
b.group(workerGroup);
b.channel(NioSocketChannel.class);
b.option(ChannelOption.AUTO_READ, true);
b.handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new ObjectDecode(FileTransferProtocol.class));
ch.pipeline().addLast(new ObjectEncode(FileTransferProtocol.class));
ch.pipeline().addLast(new FileTransferHandler());
}
});
channelFuture = b.connect(inetHost, inetPort).syncUninterruptibly();
this.channel = channelFuture.channel();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (null != channelFuture && channelFuture.isSuccess()) {
System.out.println("client start done");
} else {
System.out.println("client start error");
}
}
return channelFuture;
}
public void destroy() {
if (null == channel)
return;
channel.close();
workerGroup.shutdownGracefully();
}
}We return the client’s channel and initiate an initial request to the server through the channel. The custom processor is only responsible for receiving data. After the decoder is processed, we know that the received data is the protocol object FileTransferProtocol, and this object has three types, What our client receives is the instruction object of the server, and what the server caches is also the instruction object. That is, after the client reads the data after the initial request, the data will be read and sent after the instruction object returned by the server is received. Data and so on cycle operations.
Demonstration of data sent by the client:
public class ClientMain {
public static void main(String[] args) {
FileTransferClient client = new FileTransferClient();
ChannelFuture connect = client.connect("127.0.0.1", 7000);
File file = new File("C:\\test\\src\\测试传输文件.rar");
// 构建传输协议对象 （是包装info对象）
FileTransferProtocol fileTransferProtocol = MsgUtil.buildRequestTransferFile(file.getAbsolutePath(),
file.getName(), file.length());
connect.channel().writeAndFlush(fileTransferProtocol);
}
}
Client custom processor:
/**
* 客户端自定义处理器
*/
public class FileTransferHandler extends ChannelInboundHandlerAdapter{
/**
* 主要逻辑：
* 判断指令是否完成，完成退出，没完成，读数据，构建传输对象，并且写传输对象到服务端。
**/
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
// 客户端接收FileTransferProtocol对象
if (!(msg instanceof FileTransferProtocol)) return;
FileTransferProtocol fileTransferProtocol = (FileTransferProtocol) msg;
switch (fileTransferProtocol.getTransferType()) {
// 客户端只会处理类型为1，即传输对象是FileBurstInstruct(指令)
case 1:
FileBurstInstruct fileBurstInstruct = (FileBurstInstruct) fileTransferProtocol.getTransferObj();
// 服务端返回的instruct的状态已经完成,客户端退出操作
if (Constants.FileStatus.COMPLETE == fileBurstInstruct.getStatus()) {
ctx.flush();
ctx.close();
System.exit(-1);
return;
}
// 客户端读文件数据返回fileBurstData
FileBurstData fileBurstData = FileUtil.readFile(fileBurstInstruct.getClientFileUrl(), fileBurstInstruct.getReadPosition());
System.out.println("客户端读取一次文件，结尾是：" + fileBurstData.getEndPos());
// 构建协议对象传输
ctx.writeAndFlush(MsgUtil.buildTransferData(fileBurstData));
break;
default:
break;
}
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
ctx.close();
System.out.println("异常信息：\r\n" + cause.getMessage());
}
}2. Server-side writing
/**
* 服务端
*/
public class FileTransferServer {
/**
* Main函数
*/
public static void main(String[] args) {
String path = "";
FileTransferServer server = new FileTransferServer(path);
server.bind(7000);
}
private EventLoopGroup parentGroup = new NioEventLoopGroup(1);
private EventLoopGroup childGroup = new NioEventLoopGroup();
private Channel channel;
// 上传文件的服务端存储目标路径
private String dest_path;
public FileTransferServer(String dest_path) {
super();
this.dest_path = dest_path;
}
public ChannelFuture bind(int port) {
ChannelFuture channelFuture = null;
try {
ServerBootstrap b = new ServerBootstrap();
b.group(parentGroup, childGroup).channel(NioServerSocketChannel.class) // 非阻塞模式
.option(ChannelOption.SO_BACKLOG, 128).childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new ObjectDecode(FileTransferProtocol.class));
ch.pipeline().addLast(new ObjectEncode(FileTransferProtocol.class));
ch.pipeline().addLast(new FileTransferServerHandler(dest_path));
}
});
channelFuture = b.bind(port).sync();
this.channel = channelFuture.channel();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (null != channelFuture && channelFuture.isSuccess()) {
System.out.println("server start done");
} else {
System.out.println("server start error");
}
}
return channelFuture;
}
public void close() {
if (channel != null) {
channel.close();
}
parentGroup.shutdownGracefully();
childGroup.shutdownGracefully();
}
public Channel getChannel() {
return channel;
}
public void setChannel(Channel channel) {
this.channel = channel;
}
}We can know from the client's request sending method and the read method that the server processor needs to receive two types of transmission objects, one is the info object, the other is the data object, the info object is used for processing when the request is opened, and the data object is in The data object has been received during the transmission process, and the data data is written to the server.
In the above process, whether we receive the info object or the data object, what we send to the client is an instruct instruction object.
If the info object is received, it is necessary to determine whether there is an instruct cache? If there is, it is a resumable upload, then directly get the cached object and send it to the client, and ask the client to read the file and transfer the data. If not, then the instruct instruction object must be initially constructed, that is, the specified position position is 0 starting.
If the data object is received, the data of the data object must be written to the server storage address, and the insturct instruction is constructed to store in the cache (specify the position as the end of the last read file + 1), and then send the insturct object to the client , The client decides whether to continue reading the file according to whether it is completed.
Server-side custom processor implementation:
public class FileTransferServerHandler extends ChannelInboundHandlerAdapter{
private String dest_path;
public FileTransferServerHandler(String dest_path) {
this.dest_path = dest_path;
}
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
System.out.println("客户端链接" + ctx.channel().localAddress().toString());
}
@Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
System.out.println("客户端断开链接" + ctx.channel().localAddress().toString());
}
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
if (!(msg instanceof FileTransferProtocol))return;
FileTransferProtocol fileTransferProtocol = (FileTransferProtocol) msg;
switch (fileTransferProtocol.getTransferType()) {
case 0:
FileDescInfo info = (FileDescInfo) fileTransferProtocol.getTransferObj();
// 有缓存说明是断点续传，instruct记录的是下次需要读的postion
FileBurstInstruct old = CacheUtil.get(info.getFileName());
if(old!=null) {
if (old.getStatus() == Constants.FileStatus.COMPLETE) {
CacheUtil.remove(info.getFileName());
}
ctx.writeAndFlush(MsgUtil.buildTransferInstruct(old));
return ;
}
// 没缓存就是初始传输，主要是指定instruct的position为0
FileTransferProtocol sendFileTransferProtocol = MsgUtil.buildTransferInstruct(Constants.FileStatus.BEGIN, info.getFileUrl(), 0);
ctx.writeAndFlush(sendFileTransferProtocol);
break;
case 2:
FileBurstData fileBurstData = (FileBurstData) fileTransferProtocol.getTransferObj();
FileBurstInstruct fileBurstInstruct = FileUtil.writeFile(dest_path, fileBurstData);
// 保存断点续传信息
CacheUtil.put(fileBurstData.getFileName(), fileBurstInstruct);
ctx.writeAndFlush(MsgUtil.buildTransferInstruct(fileBurstInstruct));
// 传输完成删除断点信息
if (fileBurstInstruct.getStatus() == Constants.FileStatus.COMPLETE) {
CacheUtil.remove(fileBurstData.getFileName());
}
break;
default:
break;
}
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
System.out.println("异常信息：\r\n" + cause.getMessage());
}
}
end !!!