First, the functional requirements and non-functional requirements
Requirements convenient operation, select multiple files and folders to upload;
support of the full PC operating system platform, Windows, Linux, Mac
Support batch files and folders download, HTTP. Continue to transmit after refresh the page. After closing the browser retention schedule information.
Support batch upload and download folders, server-side retention folder hierarchy, server-side file the same sandwich structure and local level.
Support batch upload large files (20G) and download, and the need to ensure that during uploading of computer users do not appear stuck and so experience;
support the upload folder, the number of files in a folder of more than 10,000, and contains a hierarchy.
Support for HTTP, after closing the browser or refresh the browser is still able to retain progress.
Support folder structure management, support for new folder, the folder directory navigation support
Interactive and friendly, timely feedback progress of the upload;
Security service side, not because uploading files cause JVM memory overflow function affect other functions use;
Maximum use of the upstream bandwidth network, improve the upload speed;
Second, the design analysis
For large files, either the client or server side, if one-time read send, receive, are not desirable, it can easily lead to memory problems. So for large file upload, upload using cut segment
From an efficiency point of view upload, upload concurrent use of multiple threads to achieve maximum efficiency.
Third, the solution:
The front page of the file upload may choose to use some of the more useful upload component, such as Baidu open source components WebUploader, up6 Ze excellent software, some of these components to meet the basic daily needs of the file upload functionality, such as asynchronous upload files, folders , drag and drop uploads, paste upload, upload progress monitoring, thumbnail file, even large files HTTP, transfer large files in seconds.
In a web project upload folder it has now become a mainstream requirement. We have similar needs in OA, ERP or enterprise system. Upload folder hierarchy and can be reserved on-line user's guide as well, the user is also more convenient to use. It can provide more advanced applications support.
Folder data table structure
CREATE TABLE IF NOT EXISTS `up6_folders` (
`f_id` char(32) NOT NULL ,
`f_nameLoc` varchar(255) default '',
`f_pid` char(32) default '',
`f_uid` int(11) default '0',
`f_lenLoc` bigint(19) default '0',
`f_sizeLoc` varchar(50) default '0',
`f_pathLoc` varchar(255) default '',
`f_pathSvr` varchar(255) default '',
`f_pathRel` varchar(255) default '',
`f_folders` int(11) default '0',
`f_fileCount` int(11) default '0',
`f_filesComplete` int(11) default '0',
`f_complete` tinyint(1) default '0',
`f_deleted` tinyint(1) default '0',
`f_time` timestamp NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
`f_pidRoot` char(32) default '',
PRIMARY KEY (`f_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
File data table structure
CREATE TABLE IF NOT EXISTS `up6_files` (
`f_id` char(32) NOT NULL,
`F_pid` char (32) default '', / * parent folder ID * /
`F_pidRoot` char (32) default '', / * root level folder ID * /
`F_fdTask` tinyint (1) default '0', / * whether a folder information * /
`F_fdChild` tinyint (1) default '0', / * whether the folder files * /
`f_uid` int(11) default '0',
`F_nameLoc` varchar (255) default '', / * file name in the local (original file name) * /
`F_nameSvr` varchar (255) default '', / * file server name * /
`F_pathLoc` varchar (512) default '', / * path of the local file * /
`F_pathSvr` varchar (512) default '', / * file in a remote server location * /
`f_pathRel` varchar(512) default '',
`F_md5` varchar (40) default '', / * file MD5 * /
`F_lenLoc` bigint (19) default '0', / * file size * /
`F_sizeLoc` varchar (10) default '0', / * file size (formatted) * /
`F_pos` bigint (19) default '0', / * resume position * /
`F_lenSvr` bigint (19) default '0', / * size uploaded * /
`F_perSvr` varchar (7) default '0%', / * * Percentage uploaded /
`F_complete` tinyint (1) default '0', / * has been uploaded * /
`f_time` timestamp NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
`f_deleted` tinyint(1) default '0',
`f_scan` tinyint(1) default '0',
PRIMARY KEY (`f_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
The core of the project is to upload the file block. Front and rear end to high level of cooperation, the two sides agreed need better data in order to complete large file segment, we focus on in the project to solve the following problems.
* How fragmentation;
* How to synthesize a file;
* Interrupted which slice from the beginning.
How points, using powerful js library to ease our work, have been able to block the wheels on large files on the market, although the nature of the programmer has forced me to re-create the wheel. However, because of the relationship as well as working time, I can only give up the. Finally, I chose Baidu's WebUploader to achieve the desired front-end.
How close, before closing, we must first solve a problem, how do we distinguish the file block belongs. At the beginning, I was using a front-end to generate a unique uuid do sign documents, put on each slice request. But later, when do I give up the second pass, using Md5 to maintain block and file relationships.
Merge file server in question, and record blocking, and in this respect the industry has actually given a good solution. Referring Thunder, you will find that each download time, there will be two files, a file body, another file is a temporary file, temporary file stores the state of each block corresponding byte position.
These are the need to make close contact with front and rear ends, the front end of a fixed size need to file fragmentation, and fragmentation to bring the request number and size. After successful transmission request reaches the front end of the background, according to the server only needs to request data slice number and slice each block size (segment size is fixed and the same) is calculated start position, and the read segment data file, writing into the file.
In order to facilitate the development, I will end service business logic follows divided into initialization, block processing, and the like uploaded file.
The server-side business logic module follows
Functional Analysis:
Folder generating module
After the folder is uploaded by the server scan code is as follows
public class fd_scan
{
DbHelper db;
Connection con;
PreparedStatement cmd_add_f = null;
PreparedStatement cmd_add_fd = null;
public FileInf root = null; // root
public fd_scan()
{
this.db = new DbHelper();
this.con = this.db.GetCon();
}
public void makeCmdF()
{
StringBuilder sb = new StringBuilder();
sb.append("insert into up6_files (");
sb.append(" f_id");//1
sb.append(",f_pid");//2
sb.append(",f_pidRoot");//3
sb.append(",f_fdTask");//4
sb.append(",f_fdChild");//5
sb.append(",f_uid");//6
sb.append(",f_nameLoc");//7
sb.append(",f_nameSvr");//8
sb.append(",f_pathLoc");//9
sb.append(",f_pathSvr");//10
sb.append(",f_pathRel");//11
sb.append(",f_md5");//12
sb.append(",f_lenLoc");//13
sb.append(",f_sizeLoc");//14
sb.append(",f_lenSvr");//15
sb.append(",f_perSvr");//16
sb.append(",f_complete");//17
sb.append(") values(");
sb.append(" ?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(")");
try {
this.cmd_add_f = this.con.prepareStatement(sb.toString());
this.cmd_add_f.setString(1, "");//id
this.cmd_add_f.setString(2, "");//pid
this.cmd_add_f.setString(3, "");//pidRoot
this.cmd_add_f.setBoolean(4, true);//fdTask
this.cmd_add_f.setBoolean(5, false);//f_fdChild
this.cmd_add_f.setInt(6, 0);//f_uid
this.cmd_add_f.setString(7, "");//f_nameLoc
this.cmd_add_f.setString(8, "");//f_nameSvr
this.cmd_add_f.setString(9, "");//f_pathLoc
this.cmd_add_f.setString(10, "");//f_pathSvr
this.cmd_add_f.setString(11, "");//f_pathRel
this.cmd_add_f.setString(12, "");//f_md5
this.cmd_add_f.setLong(13, 0);//f_lenLoc
this.cmd_add_f.setString(14, "");//f_sizeLoc
this.cmd_add_f.setLong(15, 0);//f_lenSvr
this.cmd_add_f.setString(16, "");//f_perSvr
this.cmd_add_f.setBoolean(17, true);//f_complete
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace ();
}
}
public void makeCmdFD()
{
StringBuilder sb = new StringBuilder();
sb.append("insert into up6_folders (");
sb.append(" f_id");//1
sb.append(",f_pid");//2
sb.append(",f_pidRoot");//3
sb.append(",f_nameLoc");//4
sb.append(",f_uid");//5
sb.append(",f_pathLoc");//6
sb.append(",f_pathSvr");//7
sb.append(",f_pathRel");//8
sb.append(",f_complete");//9
sb.append(") values(");//
sb.append(" ?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(",?");
sb.append(")");
try {
this.cmd_add_fd = this.con.prepareStatement(sb.toString());
this.cmd_add_fd.setString(1, "");//id
this.cmd_add_fd.setString(2, "");//pid
this.cmd_add_fd.setString(3, "");//pidRoot
this.cmd_add_fd.setString(4, "");//name
this.cmd_add_fd.setInt(5, 0);//f_uid
this.cmd_add_fd.setString(6, "");//pathLoc
this.cmd_add_fd.setString(7, "");//pathSvr
this.cmd_add_fd.setString(8, "");//pathRel
this.cmd_add_fd.setBoolean(9, true);//complete
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace ();
}
}
protected void GetAllFiles(FileInf inf,String root)
{
File dir = new File(inf.pathSvr);
File [] allFile = dir.listFiles();
for(int i = 0; i < allFile.length; i++)
{
if(allFile[i].isDirectory())
{
FileInf fd = new FileInf();
String uuid = UUID.randomUUID().toString();
uuid = uuid.replace("-", "");
fd.id = uuid;
fd.pid = inf.id;
fd.pidRoot = this.root.id;
fd.nameSvr = allFile[i].getName();
fd.nameLoc = fd.nameSvr;
fd.pathSvr = allFile[i].getPath();
fd.pathSvr = fd.pathSvr.replace("\\", "/");
fd.pathRel = fd.pathSvr.substring(root.length() + 1);
fd.perSvr = "100%";
fd.complete = true;
this.save_folder(fd);
this.GetAllFiles(fd, root);
}
else
{
FileInf fl = new FileInf();
String uuid = UUID.randomUUID().toString();
uuid = uuid.replace("-", "");
fl.id = uuid;
fl.pid = inf.id;
fl.pidRoot = this.root.id;
fl.nameSvr = allFile[i].getName();
fl.nameLoc = fl.nameSvr;
fl.pathSvr = allFile[i].getPath();
fl.pathSvr = fl.pathSvr.replace("\\", "/");
fl.pathRel = fl.pathSvr.substring(root.length() + 1);
fl.lenSvr = allFile[i].length();
fl.lenLoc = fl.lenSvr;
fl.perSvr = "100%";
fl.complete = true;
this.save_file(fl);
}
}
}
protected void save_file(FileInf f)
{
try {
this.cmd_add_f.setString(1, f.id);//id
this.cmd_add_f.setString(2, f.pid);//pid
this.cmd_add_f.setString(3, f.pidRoot);//pidRoot
this.cmd_add_f.setBoolean(4, f.fdTask);//fdTask
this.cmd_add_f.setBoolean(5, true);//f_fdChild
this.cmd_add_f.setInt(6, f.uid);//f_uid
this.cmd_add_f.setString(7, f.nameLoc);//f_nameLoc
this.cmd_add_f.setString(8, f.nameSvr);//f_nameSvr
this.cmd_add_f.setString(9, f.pathLoc);//f_pathLoc
this.cmd_add_f.setString(10, f.pathSvr);//f_pathSvr
this.cmd_add_f.setString(11, f.pathRel);//f_pathRel
this.cmd_add_f.setString(12, f.md5);//f_md5
this.cmd_add_f.setLong(13, f.lenLoc);//f_lenLoc
this.cmd_add_f.setString(14, f.sizeLoc);//f_sizeLoc
this.cmd_add_f.setLong(15, f.lenSvr);//f_lenSvr
this.cmd_add_f.setString(16, f.perSvr);//f_perSvr
this.cmd_add_f.setBoolean(17, f.complete);//f_complete
this.cmd_add_f.executeUpdate();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace ();
}//
}
protected void save_folder(FileInf f)
{
try {
this.cmd_add_fd.setString(1, f.id);//id
this.cmd_add_fd.setString(2, f.pid);//pid
this.cmd_add_fd.setString(3, f.pidRoot);//pidRoot
this.cmd_add_fd.setString(4, f.nameSvr);//name
this.cmd_add_fd.setInt(5, f.uid);//f_uid
this.cmd_add_fd.setString(6, f.pathLoc);//pathLoc
this.cmd_add_fd.setString(7, f.pathSvr);//pathSvr
this.cmd_add_fd.setString(8, f.pathRel);//pathRel
this.cmd_add_fd.setBoolean(9, f.complete);//complete
this.cmd_add_fd.executeUpdate();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace ();
}
}
public void scan(FileInf inf, String root) throws IOException, SQLException
{
this.makeCmdF();
this.makeCmdFD();
this.GetAllFiles(inf, root);
this.cmd_add_f.close();
this.cmd_add_fd.close();
this.con.close();
}
}
Upload block, processing logic block should be the most simple logic, UP6 file has a block, for each block and the identification data, the identification includes the index file block sizes, offsets, file the MD5, file block MD5 (open needed) and other information, the server can receive the information after the process is very convenient. For example the block data stored in the distributed storage system
Block upload can be said that the foundation of our entire project, like HTTP, suspend these are the need to use the block.
This block is relatively straightforward. It is the use of a front end webuploader, and other basic functions block has already been sealed, easy to use.
With webUpload available to our file API, the front end was extremely simple.
Front HTML template
this.GetHtmlFiles = function()
{
var acx = "";
acx += '<div class="file-item" id="tmpFile" name="fileItem">\
<div class="img-box"><img name="file" src="js/file.png"/></div>\
<div class="area-l">\
<div class="file-head">\
<div name="fileName" class="name">HttpUploader程序开发.pdf</div>\
<div name="percent" class="percent">(35%)</div>\
<div name="fileSize" class="size" child="1">1000.23MB</div>\
</div>\
<div class="process-border"><div name="process" class="process"></div></div>\
<div name="msg" class="msg top-space">15.3MB 20KB/S 10:02:00</div>\
</div>\
<div class="area-r">\
<span class="btn-box" name="cancel" title="取消"><img name="stop" src="js/stop.png"/><div>取消</div></span>\
<span class="btn-box hide" name="post" title="继续"><img name="post" src="js/post.png"/><div>继续</div></span>\
<span class="btn-box hide" name="stop" title="停止"><img name="stop" src="js/stop.png"/><div>停止</div></span>\
<span class="btn-box hide" name="del" title="删除"><img name="del" src="js/del.png"/><div>删除</div></span>\
</div>';
acx += '</div>';
acx += '<div class="file-item" name="folderItem">\
<div class="img-box"><img name="folder" src="js/folder.png"/></div>\
<div class="area-l">\
<div class="file-head">\
<div name="fileName" class="name">HttpUploader程序开发.pdf</div>\
<div name="percent" class="percent">(35%)</div>\
<div name="fileSize" class="size" child="1">1000.23MB</div>\
</div>\
<div class="process-border top-space"><div name="process" class="process"></div></div>\
<div name="msg" class="msg top-space">15.3MB 20KB/S 10:02:00</div>\
</div>\
<div class="area-r">\
<span class="btn-box" name="cancel" title="取消"><img name="stop" src="js/stop.png"/><div>取消</div></span>\
<span class="btn-box hide" name="post" title="继续"><img name="post" src="js/post.png"/><div>继续</div></span>\
<span class="btn-box hide" name="stop" title="停止"><img name="stop" src="js/stop.png"/><div>停止</div></span>\
<span class="btn-box hide" name="del" title="删除"><img name="del" src="js/del.png"/><div>删除</div></span>\
</div>';
acx += '</div>';
acx += '<div class="files-panel" name="post_panel">\
<div name="post_head" class="toolbar">\
<Span class = "btn" name = "btnAddFiles"> select multiple files </ span> \
<Span class = "btn" name = "btnAddFolder"> Select Folder </ span> \
<Span class = "btn" name = "btnPasteFile"> paste files and directories </ span> \
<span class="btn" name="btnSetup">安装控件</span>\
</div>\
<div class="content" name="post_content">\
<div name="post_body" class="file-post-view"></div>\
</div>\
<div class="footer" name="post_footer">\
<Span class = "btn-footer" name = "btnClear"> Clear completed file </ span> \
</div>\
</div>';
return acx;
};
It certainly points together. The large file fragmentation, but fragmentation of the original file is no function, so we need to synthesize original fragmentation of files. We just need to fragmentation written to the file by the original location to go. Because the principle of a front that we have talked about, we know the block size and block number, I can see the block start position in the file. So here it is wise to use RandomAccessFile, RandomAccessFile able to move files back and forth inside. But in the vast majority of andomAccessFile function, it has been the NIO JDK1.4 the "memory-mapped files (memory-mapped files)" replaced. I wrote were synthesized using RandomAccessFile and MappedByteBuffer file in the project. Methods and corresponds uploadFileRandomAccessFile uploadFileByMappedByteBuffer. The following two method code.
Sec transfer function
Server-side logic
Sec transfer function, I believe we all reflect over, when the network disk upload, upload the files found in a second pass. In fact, the principle of a little studied students should know, in fact, is to examine the MD5 file, upload it to record the MD5 file system, first obtain the contents of the file MD5 MD5 value or part of the value before uploading a file, and then on the matching system data.
Breakpoint-http achieve transfer principle seconds after the client select the file, click Upload when triggered obtain the file MD5 value, obtain the MD5 calling the system interface (/ index / checkFileMd5), querying the MD5 already exists (I project in redis used to store data, a document MD5 value to make key, value is stored in the address file.) returning to check the interface state, and then to the next step. I believe we will be able to look at the code to understand.
Ah, the front end of the MD5 value is used webuploader comes with features, this is a good tool.
Controls will trigger the end of the calculation file MD5 md5_complete events, and traditional values md5, developers only need to deal with this event,
http
up6 automatically resuming has been processed, and then have no need to develop a separate process.
F_post.jsp receives these parameters and processed, developers need only focus on business logic, other aspects need not be concerned.
HTTP is that an interruption in the process of file uploads, human factors (pause) or force majeure (broken network or networks difference) leads to upload files to half failed. Then when environmental restoration, and re-upload the file, which is not a new start upload.
Front has also been mentioned, the functions of HTTP is based on the block uploaded to achieve, to a large file into many small blocks, each server can be uploaded to the success of the block are ground down, the client upload call interface to quickly verify a file start, conditions are selected to skip a block.
The principle is that before each file upload, file MD5 value on access to, call interface before uploading files (/ index / checkFileMd5, yes also pass the second test interfaces) If the file status acquisition is not completed, All block numbers returns no upload, and front-end block which is not uploaded conditions sieve is calculated, and then upload.
When receiving the file after the file blocks may be written directly to the server
This is the folder after the upload results
This is the folder structure to store uploaded after the end of the service
Reference article: http://blog.ncmem.com/wordpress/2019/08/12/java-http%E5%A4%A7%E6%96%87%E4%BB%B6%E6%96%AD%E7 % 82% B9% E7% BB % AD% E4% BC% A0% E4% B8% 8A% E4% BC% A0 /
Welcome to the discussion group "374 992 201" together