I. Overview
This is a Java program for concurrent download of multiple large files. It uses the OkHttp library for network requests, and uses a thread pool to download multiple files at the same time, thereby improving download efficiency. The program creates a download task and submits it to the thread pool for execution by traversing the preset file URL and local storage path. Each download task is responsible for downloading a part of the file, and the HTTP request header Range can be set to resume the download. After the download is complete, the program will check the integrity of the downloaded file to ensure that the file is not damaged. This program is highly extensible, and more file URLs and local save paths can be easily added.
Two, specific code implementation
This is a problem that the author encountered when writing and downloading a video of a certain audio. Designed to efficiently download multiple large files from the web. The author downloads by means of multi-threaded segment breakpoint resume.
First import the necessary libraries : import the required library files, including OkHttp for network requests and ProgressBar for displaying download progress.
Dependencies required by the official miniodemo
<!-- 官方 miniodemo需要的依赖-->
<dependency>
<groupId>me.tongfei</groupId>
<artifactId>progressbar</artifactId>
<version>0.7.4</version>
</dependency>
The dependencies required by the official okhttp
<!-- 官方 okhttp需要的依赖-->
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.10.0</version>
</dependency>
The specific Java code is explained as follows:
Define constants: Define some constants, such as the number of concurrent threads, the maximum number of retries, and so on.
private static final int NUM_THREADS = 1000; // 增加并发数
private static final int MAX_RETRY_COUNT = 3; // 最大重试次数
private static final OkHttpClient httpClient = new OkHttpClient.Builder()
.connectionPool(new ConnectionPool(NUM_THREADS, 500, TimeUnit.MINUTES)) // 添加连接池
.build();
Create OkHttpClient : Create a custom OkHttpClient instance through the OkHttp library, including connection pool settings to improve connection efficiency.
// 创建HTTP请求
Request request = new Request.Builder()
.url(fileUrl)
.build();
// 发送HTTP请求并获取响应（同步请求）
Call call = httpClient.newCall(request);
Response response = call.execute();
Default file URL and local storage path: defines the URL list of the files to be downloaded and the corresponding local storage path list, and users can add more files as needed.
private static final String[] fileUrls = {
"http://www.douyin.com/aweme/v1/play/?video_id=v0200fg10000c8rkah3c77ua6v8oqskg&line=0&file_id=477344e441dc467f8f2b72f081e241b6&sign=2b2e377cec728f425d9dc0c2a2357a25&is_play_url=1&source=PackSourceEnum_SEARCH&aid=6383",
"https://v26-web.douyinvod.com/cf617f2ae9e9df284c491b6cfdb0a12b/64c71c93/video/tos/cn/tos-cn-ve-15/ff44e778a37b4113a841bc1b28065af4/?a=6383&ch=11&cr=3&dr=0&lr=all&cd=0%7C0%7C0%7C3&cv=1&br=1791&bt=1791&cs=0&ds=3&ft=bvTKJbQQqUumf7oZPo0OW_EklpPiXziScMVJEawkbfCPD-I&mime_type=video_mp4&qs=0&rc=NmZkN2U4Ozo8NGVnNmdnaUBpM3B5O2Q6ZmZvNzMzNGkzM0AtNjRiMjVeNjUxYzNiL2EvYSNmYy02cjQwZGdgLS1kLTBzcw%3D%3D&l=2023073108590933C4237837B400D5DD56&btag=e00038000&dy_q=1690765150",
" http://www.douyin.com/aweme/v1/play/?video_id=v0300fg10000c5lgaabc77ufcp8pbsrg&line=0&file_id=4c0b3faff21c4d3eadcac66e2708a4cb&sign=06baaa612983765b083487d7f470b96c&is_play_url=1&source=PackSourceEnum_SEARCH&aid=6383"
// 添加更多要下载的文件URL
};
private static final String[] destinationPaths = {
"D://" + UUID.randomUUID() + ".mp4",
"D://" + UUID.randomUUID() + ".mp4",
"D://" + UUID.randomUUID() + ".mp4"
// 添加更多文件的本地保存路径
};
Main function: In the main function, a thread pool is created and all file URLs are traversed. Create a download task for each file and submit it to the thread pool for execution.
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);
// 遍历所有文件URL，为每个文件创建一个下载任务，并提交给线程池执行
for (int i = 0; i < fileUrls.length; i++) {
String fileUrl = fileUrls[i];
String destinationPath = destinationPaths[i];
executor.execute(() -> {
try {
downloadFile(fileUrl, destinationPath);
System.out.println("文件下载成功：" + destinationPath);
} catch (IOException e) {
System.out.println("文件下载失败：" + destinationPath + "，原因：" + e.getMessage());
}
});
}
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("所有文件下载完成！");
}
Download file method (downloadFile): This method receives the URL and local path of the file, and initiates an HTTP request to obtain the file data. If the response is successful, it will calculate the remaining part to be downloaded based on the file size and the downloaded part, and assign the download task to multiple threads in the thread pool.
// 下载文件的方法
public static void downloadFile(String fileUrl, String destinationPath) throws IOException {
// 创建HTTP请求
Request request = new Request.Builder()
.url(fileUrl)
.build();
// 发送HTTP请求并获取响应（同步请求）
Call call = httpClient.newCall(request);
Response response = call.execute();
// 如果响应不成功，抛出异常
if (!response.isSuccessful()) {
throw new IOException("服务器返回错误：" + response.code());
}
// 获取文件的大小
long fileSize = response.body().contentLength();
// 检查目标文件是否存在，如果已经下载过一部分，则继续下载
File destinationFile = new File(destinationPath);
long downloadedFileSize = destinationFile.exists() ? destinationFile.length() : 0;
// 计算剩余的字节数
long remainingBytes = fileSize - downloadedFileSize;
// 计算每个线程应下载的字节数
long chunkSize = remainingBytes / NUM_THREADS;
// 创建一个新的线程池，用于下载单个文件的多个部分
ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);
for (int i = 0; i < NUM_THREADS; i++) {
long startRange = fileSize - remainingBytes;
long endRange = startRange + chunkSize - 1;
if (i == NUM_THREADS - 1) {
endRange = fileSize - 1;
}
// 创建下载任务并提交给线程池执行
executor.execute(new DownloadTask(fileUrl, destinationPath, startRange, endRange, fileSize));
remainingBytes -= chunkSize;
}
// 关闭线程池并等待所有下载任务完成
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
// 校验文件完整性
try {
if (checkFileIntegrity(destinationPath)) {
System.out.println("文件完整性校验通过：" + destinationPath);
} else {
System.out.println("文件完整性校验失败：" + destinationPath);
}
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
Download task class (DownloadTask): This is an internal class that implements the Runnable interface, representing the download task of a single file. Each download task is responsible for downloading a part of the file, and it realizes the resuming of the breakpoint by setting the Range field in the HTTP request header. When the download task is executed, it will write the file data to the local file after several attempts.
// 下载任务类
static class DownloadTask implements Runnable {
private final String fileUrl;
private final String destinationPath;
private final long startRange;
private final long endRange;
private final long fileSize;
public DownloadTask(String fileUrl, String destinationPath, long startRange, long endRange, long fileSize) {
this.fileUrl = fileUrl;
this.destinationPath = destinationPath;
this.startRange = startRange;
this.endRange = endRange;
this.fileSize = fileSize;
}
@Override
public void run() {
// 初始化重新下载标志和重试次数
boolean downloadComplete = false;
int retryCount = 0;
while (!downloadComplete && retryCount < MAX_RETRY_COUNT) {
try {
// 创建HTTP请求，并设置请求头Range来实现断点续传
Request request = new Request.Builder()
.url(fileUrl)
.header("Range", "bytes=" + startRange + "-" + endRange)
.build();
// 发送HTTP请求并获取响应（同步请求）
Call call = httpClient.newCall(request);
Response response = call.execute();
// 如果响应不成功，抛出异常
if (!response.isSuccessful()) {
throw new IOException("服务器返回错误：" + response.code());
}
// 创建随机访问文件对象，用于将下载的数据写入文件指定的位置
RandomAccessFile output = new RandomAccessFile(destinationPath, "rw");
output.seek(startRange);
byte[] buffer = new byte[1024 * 1024*2];
int bytesRead;
try (ProgressBar progressBar = new ProgressBar("下载进度", endRange - startRange + 1)) {
// 循环读取响应的数据，并写入文件
while ((bytesRead = response.body().byteStream().read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
progressBar.stepBy(bytesRead);
}
}
// 关闭文件和响应
output.close();
response.close();
break;
} catch (IOException e) {
// 出现异常，进行重新下载
e.printStackTrace();
System.out.println("文件下载异常，进行重新下载...");
retryCount++;
}
}
// 如果下载重试次数达到最大值仍然失败，则打印失败信息
if (retryCount >= MAX_RETRY_COUNT) {
System.out.println("文件下载失败：" + destinationPath);
}
}
}
Check file integrity method (checkFileIntegrity): This method checks the integrity of the downloaded file to ensure that the file is not damaged or tampered with. It calculates the hash of the file and compares it to the expected hash.
// 校验文件完整性
public static boolean checkFileIntegrity(String filePath) throws NoSuchAlgorithmException, IOException {
String expectedHash = calculateFileHash(Paths.get(filePath), "MD5");
String actualHash = calculateFileHash(Paths.get(filePath), "MD5");
return expectedHash.equals(actualHash);
}
Calculate file hash value method (calculateFileHash): This method uses MessageDigest to calculate the hash value of the file for subsequent verification of file integrity. It reads the bytes of the file content, performs a hash calculation, and finally returns a hash value string in hexadecimal format.
// 计算文件的哈希值
public static String calculateFileHash(Path filePath, String algorithm) throws IOException, NoSuchAlgorithmException {
MessageDigest md = MessageDigest.getInstance(algorithm);
try (FileInputStream fis = new FileInputStream(filePath.toFile())) {
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = fis.read(buffer)) != -1) {
md.update(buffer, 0, bytesRead);
}
}
byte[] hashBytes = md.digest();
return bytesToHex(hashBytes);
}
3. Specific and complete code implementation
This program uses the OkHttp library and multi-threading to realize the efficient concurrent download function of multiple files, and at the same time ensure the integrity of the downloaded files. With reasonable thread management and file splitting strategy, the download speed and efficiency can be maximized.
package cn.konne.konneim.download;
import me.tongfei.progressbar.ProgressBar;
import okhttp3.*;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class MultiFileDownloader {
private static final int NUM_THREADS = 1000; // 增加并发数
private static final int MAX_RETRY_COUNT = 3; // 最大重试次数
private static final OkHttpClient httpClient = new OkHttpClient.Builder()
.connectionPool(new ConnectionPool(NUM_THREADS, 500, TimeUnit.MINUTES)) // 添加连接池
.build();
private static final String[] fileUrls = {
"http://www.douyin.com/aweme/v1/play/?video_id=v0200fg10000c8rkah3c77ua6v8oqskg&line=0&file_id=477344e441dc467f8f2b72f081e241b6&sign=2b2e377cec728f425d9dc0c2a2357a25&is_play_url=1&source=PackSourceEnum_SEARCH&aid=6383",
"https://v26-web.douyinvod.com/cf617f2ae9e9df284c491b6cfdb0a12b/64c71c93/video/tos/cn/tos-cn-ve-15/ff44e778a37b4113a841bc1b28065af4/?a=6383&ch=11&cr=3&dr=0&lr=all&cd=0%7C0%7C0%7C3&cv=1&br=1791&bt=1791&cs=0&ds=3&ft=bvTKJbQQqUumf7oZPo0OW_EklpPiXziScMVJEawkbfCPD-I&mime_type=video_mp4&qs=0&rc=NmZkN2U4Ozo8NGVnNmdnaUBpM3B5O2Q6ZmZvNzMzNGkzM0AtNjRiMjVeNjUxYzNiL2EvYSNmYy02cjQwZGdgLS1kLTBzcw%3D%3D&l=2023073108590933C4237837B400D5DD56&btag=e00038000&dy_q=1690765150",
" http://www.douyin.com/aweme/v1/play/?video_id=v0300fg10000c5lgaabc77ufcp8pbsrg&line=0&file_id=4c0b3faff21c4d3eadcac66e2708a4cb&sign=06baaa612983765b083487d7f470b96c&is_play_url=1&source=PackSourceEnum_SEARCH&aid=6383"
// 添加更多要下载的文件URL
};
private static final String[] destinationPaths = {
"D://" + UUID.randomUUID() + ".mp4",
"D://" + UUID.randomUUID() + ".mp4",
"D://" + UUID.randomUUID() + ".mp4"
// 添加更多文件的本地保存路径
};
public static void main(String[] args) {
ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);
// 遍历所有文件URL，为每个文件创建一个下载任务，并提交给线程池执行
for (int i = 0; i < fileUrls.length; i++) {
String fileUrl = fileUrls[i];
String destinationPath = destinationPaths[i];
executor.execute(() -> {
try {
downloadFile(fileUrl, destinationPath);
System.out.println("文件下载成功：" + destinationPath);
} catch (IOException e) {
System.out.println("文件下载失败：" + destinationPath + "，原因：" + e.getMessage());
}
});
}
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("所有文件下载完成！");
}
// 下载文件的方法
public static void downloadFile(String fileUrl, String destinationPath) throws IOException {
// 创建HTTP请求
Request request = new Request.Builder()
.url(fileUrl)
.build();
// 发送HTTP请求并获取响应（同步请求）
Call call = httpClient.newCall(request);
Response response = call.execute();
// 如果响应不成功，抛出异常
if (!response.isSuccessful()) {
throw new IOException("服务器返回错误：" + response.code());
}
// 获取文件的大小
long fileSize = response.body().contentLength();
// 检查目标文件是否存在，如果已经下载过一部分，则继续下载
File destinationFile = new File(destinationPath);
long downloadedFileSize = destinationFile.exists() ? destinationFile.length() : 0;
// 计算剩余的字节数
long remainingBytes = fileSize - downloadedFileSize;
// 计算每个线程应下载的字节数
long chunkSize = remainingBytes / NUM_THREADS;
// 创建一个新的线程池，用于下载单个文件的多个部分
ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);
for (int i = 0; i < NUM_THREADS; i++) {
long startRange = fileSize - remainingBytes;
long endRange = startRange + chunkSize - 1;
if (i == NUM_THREADS - 1) {
endRange = fileSize - 1;
}
// 创建下载任务并提交给线程池执行
executor.execute(new DownloadTask(fileUrl, destinationPath, startRange, endRange, fileSize));
remainingBytes -= chunkSize;
}
// 关闭线程池并等待所有下载任务完成
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
// 校验文件完整性
try {
if (checkFileIntegrity(destinationPath)) {
System.out.println("文件完整性校验通过：" + destinationPath);
} else {
System.out.println("文件完整性校验失败：" + destinationPath);
}
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
// 下载任务类
static class DownloadTask implements Runnable {
private final String fileUrl;
private final String destinationPath;
private final long startRange;
private final long endRange;
private final long fileSize;
public DownloadTask(String fileUrl, String destinationPath, long startRange, long endRange, long fileSize) {
this.fileUrl = fileUrl;
this.destinationPath = destinationPath;
this.startRange = startRange;
this.endRange = endRange;
this.fileSize = fileSize;
}
@Override
public void run() {
// 初始化重新下载标志和重试次数
boolean downloadComplete = false;
int retryCount = 0;
while (!downloadComplete && retryCount < MAX_RETRY_COUNT) {
try {
// 创建HTTP请求，并设置请求头Range来实现断点续传
Request request = new Request.Builder()
.url(fileUrl)
.header("Range", "bytes=" + startRange + "-" + endRange)
.build();
// 发送HTTP请求并获取响应（同步请求）
Call call = httpClient.newCall(request);
Response response = call.execute();
// 如果响应不成功，抛出异常
if (!response.isSuccessful()) {
throw new IOException("服务器返回错误：" + response.code());
}
// 创建随机访问文件对象，用于将下载的数据写入文件指定的位置
RandomAccessFile output = new RandomAccessFile(destinationPath, "rw");
output.seek(startRange);
byte[] buffer = new byte[1024 * 1024*2];
int bytesRead;
try (ProgressBar progressBar = new ProgressBar("下载进度", endRange - startRange + 1)) {
// 循环读取响应的数据，并写入文件
while ((bytesRead = response.body().byteStream().read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
progressBar.stepBy(bytesRead);
}
}
// 关闭文件和响应
output.close();
response.close();
break;
} catch (IOException e) {
// 出现异常，进行重新下载
e.printStackTrace();
System.out.println("文件下载异常，进行重新下载...");
retryCount++;
}
}
// 如果下载重试次数达到最大值仍然失败，则打印失败信息
if (retryCount >= MAX_RETRY_COUNT) {
System.out.println("文件下载失败：" + destinationPath);
}
}
}
// 校验文件完整性
public static boolean checkFileIntegrity(String filePath) throws NoSuchAlgorithmException, IOException {
String expectedHash = calculateFileHash(Paths.get(filePath), "MD5");
String actualHash = calculateFileHash(Paths.get(filePath), "MD5");
return expectedHash.equals(actualHash);
}
// 计算文件的哈希值
public static String calculateFileHash(Path filePath, String algorithm) throws IOException, NoSuchAlgorithmException {
MessageDigest md = MessageDigest.getInstance(algorithm);
try (FileInputStream fis = new FileInputStream(filePath.toFile())) {
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = fis.read(buffer)) != -1) {
md.update(buffer, 0, bytesRead);
}
}
byte[] hashBytes = md.digest();
return bytesToHex(hashBytes);
}
// 将字节数组转换为十六进制字符串
private static String bytesToHex(byte[] bytes) {
StringBuilder result = new StringBuilder();
for (byte b : bytes) {
result.append(String.format("%02x", b));
}
return result.toString();
}
}