Large file transfer solution: fragmentation upload / download speed limit

Foreword

Will encounter many projects in uploading and downloading videos, update package, applications and other documents, such documents have in common is huge, I met about 4G files simultaneously downloading more than 100 machines in the project, if at this time with post upload and download can not think about, but Baidu said the search words are adjusted to limit post of php.ini, but this is a ridiculous solution, thus we need to use another solution - fragmentation upload and download speed limits

In this band we use php to achieve it, apply a variety of languages and frameworks, this is used in the php laravel, language and ideas to achieve the same

If used in the project part uploading, personal recommendations as to find the corresponding package (AetherUpload-Laravel), with 7 direct conditional cattle cloud, Ali cloud companies and other large fragments upload service

Part uploading

principle

The uploaded file according to a certain division rule, the data is divided into blocks of the same size;
Initializing a slice uploads, this return part uploading unique identifier;
Sending each data block fragments according to a certain policy (serial or parallel);
After the transmission is completed, the server is determined according to the data upload is complete, if incomplete, the data block synthesized original file.

achieve

h5

h5 achieve some, h5 partially achieved the split files, uploading, you tell the total number of pieces and the current file server is the first of several films, various temporary files sent via http request

<!doctype html> <html> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0"> <meta http-equiv="X-UA-Compatible" content="ie=edge"> <title>Document</title> <style> #progress{ width: 300px; height: 20px; background-color:#f7f7f7; box-shadow:inset 0 1px 2px rgba(0,0,0,0.1); border-radius:4px; background-image:linear-gradient(to bottom,#f5f5f5,#f9f9f9); } #finish{ background-color: #149bdf; background-image:linear-gradient(45deg,rgba(255,255,255,0.15) 25%,transparent 25%,transparent 50%,rgba(255,255,255,0.15) 50%,rgba(255,255,255,0.15) 75%,transparent 75%,transparent); background-size:40px 40px; display: inline-block; height: 20px; } form{ margin-top: 50px; } </style> </head> <body> <p id="progress"> <span id="finish" style="width: 0%;" progress="0"></span> </p> <form action=""> <input type="file" name="file" id="file"> <input type="button" value="停止" id="stop"> </form> <script> var fileForm = document.getElementById("file"); var stopBtn = document.getElementById('stop'); var upload = new Upload(); fileForm.onchange = function(){ upload.addFileAndSend(this); } stopBtn.onclick = function(){ this.value = "停止中"; upload.stop(); this.value = "已停止"; } function Upload(){ var xhr = new XMLHttpRequest(); var form_data = new FormData(); const LENGTH = 1024 * 1024 *2; var start = 0; var end = start + LENGTH; var blob; var blob_num = 1; var is_stop = 0 //对外方法，传入文件对象 this.addFileAndSend = function(that){ var file = that.files[0]; blob = cutFile(file); sendFile(blob,file); blob_num += 1; } //停止文件上传 this.stop = function(){ xhr.abort(); is_stop = 1; } //切割文件 function cutFile(file){ var file_blob = file.slice(start,end); start = end; end = start + LENGTH; return file_blob; }; //发送文件 function sendFile(blob,file){ var form_data = new FormData(); var total_blob_num = Math.ceil(file.size / LENGTH); form_data.append('file',blob); form_data.append('blob_num',blob_num); form_data.append('total_blob_num',total_blob_num); form_data.append('file_name',file.name); xhr.open('POST','http://vnn-admin.cc/Api/sliceUpload',false); xhr.onreadystatechange = function () { if (xhr.readyState==4 && xhr.status==200) { console.log(xhr.responseText); } var progress; var progressObj = document.getElementById('finish'); if(total_blob_num == 1){ progress = '100%'; }else{ progress = Math.min(100,(blob_num/total_blob_num)* 100 ) +'%'; // console.log(progress); // console.log('分割'); } progressObj.style.width = progress; var t = setTimeout(function(){ if(start < file.size && is_stop === 0){ blob = cutFile(file); sendFile(blob,file); blob_num += 1; }else{ setTimeout(t); } },1000); } xhr.send(form_data); } } </script> </body> </html>

Server

The server receives the uploaded file sheet, and determine whether the last piece, if it is to merge files, delete files upload block

/**
     * @Desc: 切片上传
     *
     * @param Request $request
     * @return mixed
     */
    public function sliceUpload(Request $request) { $file = $request->file('file'); $blob_num = $request->get('blob_num'); $total_blob_num = $request->get('total_blob_num'); $file_name = $request->get('file_name'); $realPath = $file->getRealPath(); //临时文件的绝对路径 // 存储地址 $path = 'slice/'.date('Ymd') ; $filename = $path .'/'. $file_name . '_' . $blob_num; //上传 $upload = Storage::disk('admin')->put($filename, file_get_contents($realPath)); //判断是否是最后一块，如果是则进行文件合成并且删除文件块 if($blob_num == $total_blob_num){ for($i=1; $i<= $total_blob_num; $i++){ $blob = Storage::disk('admin')->get($path.'/'. $file_name.'_'.$i); // Storage::disk('admin')->append($path.'/'.$file_name, $blob); //不能用这个方法，函数会往已经存在的文件里添加0X0A，也就是\n换行符 file_put_contents(public_path('uploads').'/'.$path.'/'.$file_name,$blob,FILE_APPEND); } //合并完删除文件块 for($i=1; $i<= $total_blob_num; $i++){ Storage::disk('admin')->delete($path.'/'. $file_name.'_'.$i); } } if ($upload){ return $this->json(200, '上传成功'); }else{ return $this->json(0, '上传失败'); } }

Download speed limit

principle

By limiting the output bytes per second
Close buffer cache

achieve

public function sliceDownload() { $path = 'slice/'.date('Ymd') ; $filename = $path .'/'. '周杰伦 - 黑色幽默 [mqms2].mp3' ; //获取文件资源 $file = Storage::disk('admin')->readStream($filename); //获取文件大小 $fileSize = Storage::disk('admin')->size($filename); header("Content-type:application/octet-stream");//设定header头为下载 header("Accept-Ranges:bytes"); header("Accept-Length:".$fileSize);//响应大小 header("Content-Disposition: attachment; filename=周杰伦 - 黑色幽默 [mqms2].mp3");//文件名 //不设置的话要等缓冲区满之后才会响应 ob_end_clean();//缓冲区结束 ob_implicit_flush();//强制每当有输出的时候,即刻把输出发送到浏览器\ header('X-Accel-Buffering: no'); // 不缓冲数据 $limit=1024*1024; $count=0; //限制每秒的速率 while($fileSize-$count>0){//循环读取文件数据 $data=fread($file,$limit); $count+=$limit; echo $data;//输出文件 sleep(1); } }

Large file transfer solution: fragmentation upload / download speed / HTTP

When you need more speed to adjust the value of $ limit

to sum up

So far things on the principle of fragmentation upload and download speed and simple implementation Demo has been finished, should be aware of how to achieve a large part uploading it, we want to help, because the large file uploads and downloads are often encountered in implementation

Original Address: https: //learnku.com/articles/31108