Delphi's code and ideas for handling high-speed file upload and download
Upload and download are a pair of concepts with different directions. The following corresponding client and server codes: turn around, it means download; turn around again, it means upload.
1. Ideas
1. Upload or download large files in sections (that is, "breakpoint" upload or download)
2. Upload or download these segments separately
3. Combine and restore the "segmented" file streams after uploading or downloading
4. Regarding acceleration (you often see "high-speed" upload or download on the Internet): If multiple threads upload or download these segments separately, the speed can be increased.
You go to use Gao Yong's code, it is completely no problem; here is just to provide you with a speed-up solution, Gao Yong's code, you can upload or download at high speed by slightly modifying it according to this idea.
Two, the code
2.1. Client:
///<summary>RestFull上传文件函数(文件流方法)：
///上传单个文件(TFileStream实例化抽象流:文件分段传参给服务器):
///</summary>
function CopyFileTFileStream(const Path: string;
const BufSize :Integer= (1*1024*1024); //:BufSize大小默认值1MB:为1个分段
const ServerPath:string='api\upload\';
const ServerFileName:string='A1.pdf' ):string;
var Buffer: TBytes; ReadCount,FileSize,FileBufferCount,iCircle: Integer;
ATStream,AWriteStream: TFileStream; ACopyStream:TStream;
AResultStr:string; ifResult: Boolean;
begin
if (Path.Trim='')
or (ServerPath.Trim='')
or (ServerFileName.Trim='') then
begin
AResultStr:='文件不能为空！';
Result := AResultStr ;
exit;
end;
if FileExists(Path) =true then
begin
AResultStr:='文件存在！';
//AWriteStream:读本地大文件的流:
AWriteStream := TFileStream.Create( Path,fmOpenRead );
iCircle:=ceil( AWriteStream.Size /BufSize );
//:分几段读取,就写几个临时文件.tmp:
//大文件：已知大于最小设定值,则分段读取,直到读完为止:
if AWriteStream.Size > BufSize then
//:不能用FileSize显式的判断
begin //:需要分数据包处理，如果是服务端是Datasnap默认32k的缓冲区(缓存，太JB小啦):
//文件流无需SetSize,而是设置其Buffer:TBytes字节数组的长度://SetLength(Buffer, BufSize);
SetLength( Buffer,(5*1024*1024) ); //:如果设置的较大：超内存需要控制，单独写例程
//$F000;//$F000:61440:60kb//:System.Classes.fmCreate
//:印象中好像最大缓冲区就这个大小，你自己核实吧，5兆我够了
FileBufferCount:=0; FileSize:=BufSize;
AResultStr:='';
for FileBufferCount := 1 to iCircle do
begin //文件流无需SetSize:
//ATStream:将大文件流AWriteStream拆分成:
//三位数的小临时文件.tmp的文件流：
ACopyStream:= TMemoryStream.Create;
ATStream:=TFileStream.Create( Path
+FillBeforeString(IntToStr(FileBufferCount),10,'0')+'.tmp'
,fmCreate or fmOpenReadWrite);//:FillBeforeString:自写函数,字符串末尾向左补齐字符
//:文件流,没有就产生、有就打开 //10位：足够1T的文件拆分
try
if FileSize < AWriteStream.Size then FileSize:= BufSize;
if FileSize >= AWriteStream.Size then
FileSize:=AWriteStream.Size - (iCircle-1) * BufSize;
ReadCount := AWriteStream.Read( Buffer[0], FileSize );
AResultStr:=AResultStr+','+IntToStr(FileSize);
if ReadCount > 0 then
ATStream.WriteBuffer(Buffer[0], ReadCount);
ACopyStream.CopyFrom(ATStream,0);
//:ACopyStream:由CopyFrom的调用者自己来释放
try
ifResult:=ServerMethods
.upLoadFileFromTStreamBufferRead(ACopyStream,( ServerPath +ServerFileName
+FillBeforeString(IntToStr(FileBufferCount),3,'0')+'.tmp') );
//:服务器方法成功执行需要时间:
//while ifResult<>true do sleep(0);//:需要的场景才这样做
except
AWriteStream.disposeOf; ATStream.disposeOf; ACopyStream.disposeOf;
Result := '流加载错误'; exit;
end;
FileSize:=FileSize + BufSize;
finally
//传完该临时分段文件流释放它的句柄，并删除它对应的临时文件:
ATStream.disposeOf;
System.IOUtils.TFile.Delete(Path
+FillBeforeString(IntToStr(FileBufferCount),3,'0')+'.tmp');
end;
end;
//确保:全部临时分段文件流处理完成之后,发出合并文件请求，
//将分段文件流还原为原始文件:
if FileBufferCount =iCircle then
begin
try
ifResult:=ServerMethods
.mergeFilesUseTFileStream(ServerPath + ServerFileName);
//while ifResult <>true do sleep(0);//:服务器方法成功执行需要时间,需要的场景才做
except
AWriteStream.disposeOf; ATStream.disposeOf; ACopyStream.disposeOf;
Result := '文件流合并错误'; exit;
end;
AResultStr:= '文件上传成功';
AWriteStream.disposeOf;
end;
end else
begin
//小文件：已知小于最小设定值,则直接复制数据:
ACopyStream:= TMemoryStream.Create;
ACopyStream.CopyFrom(AWriteStream,0);
try
ifResult:=ServerMethods
.upLoadFileFromTStreamBufferRead( ACopyStream,(ServerPath+ServerFileName) );
//:服务器方法成功执行需要时间
//while ifResult<>true do sleep(0);//:需要的场景才这样做
AResultStr:=IntToStr(AWriteStream.Size);
except
AWriteStream.disposeOf; ACopyStream.disposeOf;
Result := '服务器出错了'; exit;
end;
if ifResult=true then
begin
AResultStr:= '文件上传成功';
AWriteStream.disposeOf;
end;
end;
end else AResultStr:='文件不存在！';
System.IOUtils.TDirectory.SetCurrentDirectory
(System.IOUtils.TPath.GetLibraryPath);
Result := AResultStr ;
end;
2.2. Server:
/// <summary>将（大文件拆分后的）单个小临时文件上传到服务器的函数；
///参数为文件数据流和服务器端应用库路径下的子路径 </summary>
function TServerMethods1.upLoadFileFromTStreamBufferRead(
const ATStream:TStream; const toFilePath:string): Boolean;
const BufSize = $F000; //:默认$F000:61440:60kb的小文件 //System.Classes.fmCreate
var Buffer: TBytes; ReadCount,times: Integer; FS: TFileStream;
begin
if FileExists(toFilePath)=true then
begin
TThread.Synchronize(nil,
procedure
begin
TFile.Delete(toFilePath);
MainServerForm.Memo_Errors.Lines.Add('删除文件:'+toFilePath.Trim);
end
);
end;//:存在就先删除
//删除后然后产生新文件:
FS := TFileStream.Create(toFilePath, System.Classes.fmCreate);
//:常量默认fmCreate = $FF00; //: TFileStream create mode
TThread.Synchronize(nil,
procedure
begin//测试:
//MainServerForm.Memo_Errors.Lines.Add('新产生文件:'+toFilePath.Trim);
end
);
try
//if ATStream.Size = -1 then//:大小未知则一直读取到没有数据为止:
begin
SetLength( Buffer, BufSize );
repeat
ReadCount := ATStream.Read( Buffer[0], BufSize );
if ReadCount > 0 then
FS.WriteBuffer( Buffer[0], ReadCount );
if ReadCount < BufSize then
break;
until ReadCount < BufSize;
end //else FS.CopyFrom(ATStream, 0)
; //:大小已知则直接复制数据
finally
FS.Free;
Result := True;
end;
end;
/// <summary>多文件流合并(用文件流方法实例化抽象流):</summary>
function TServerMethods1.mergeFilesUseTFileStream(const toFilePath:string): Boolean;
var filesCount :Integer; SearchRec :TSearchRec; currPath :string;
fileName :string; filesName :TStringList;
FileTStream,tmpTStream :TStream;
begin
TDirectory.SetCurrentDirectory(System.IOUtils.TPath.GetLibraryPath);
fileName:=System.IOUtils.TPath.GetLibraryPath + toFilePath;
if TDirectory.Exists(TDirectory.GetParent(fileName)) then
begin
currPath:=TDirectory.GetParent(fileName);
//强制路径产生：//ForceDirectories(currPath)
//DirectoryExists:路径是否存在//FileExists:文件是否存在:uses System.SysUtils;
TDirectory.SetCurrentDirectory(currPath);
//:进入该路径（设置当前路径并进入）
if FileExists(fileName)=true then TFile.Delete(fileName);
end;
filesCount := FindFirst(currPath + '\' + '*.tmp', faAnyFile, SearchRec);//:搜索成功返回0
if filesCount<>0 then
begin
System.SysUtils.FindClose(SearchRec);//?不识别:需要带上前缀:System.SysUtils.
TDirectory.SetCurrentDirectory(System.IOUtils.TPath.GetLibraryPath);
Result :=false; exit;
end;
FileTStream:=TFileStream.Create(fileName,fmCreate);
filesName:=TStringList.Create;
while filesCount = 0 do
begin
filesName.Add(SearchRec.Name);
tmpTStream:=TFileStream.Create(SearchRec.Name,fmOpenRead);
try
//Vcl.Forms.Application.ProcessMessages;
//:不卡界面：VCL将线程池TServerMethods1的消息事件交给应用的主线程
//:不卡界面：FMX用线程同步方法TThread.Synchronize
FileTStream.CopyFrom(tmpTStream,0);
filesCount := FindNext(SearchRec);
finally
tmpTStream.disposeOf;
end;
end;
try
for filesCount := 0 to filesName.Count-1 do
begin
if FileExists(filesName[filesCount].Trim)=true then
TFile.Delete(filesName[filesCount].Trim);
end;
finally
System.SysUtils.FindClose(SearchRec);//?不识别:需要带上前缀:System.SysUtils.
filesName.disposeOf; FileTStream.disposeOf;
TDirectory.SetCurrentDirectory(System.IOUtils.TPath.GetLibraryPath);
end;
Result :=true;
end;
Parallel upload, CPU 8 channels are fully open, almost full utilization and full load upload.
Related articles of this blog: Source download: https://download.csdn.net/download/pulledup/12578989
"TTreeView Complete Enumeration and Recursive Algorithm" https://blog.csdn.net/pulledup/article/details/103687816
If you like it, just click like and favorite below, so that you can watch the next sharing: