springboot FTP server upload && download sample demo

Recently, the project needs to use the ftp server to interact with third-party resources, so I wrote a small demo to record it~

basic knowledge

FTP server

FTP (File Transfer Protocol) is a file transfer protocol, which is a protocol based on TCP and adopts the client/server mode. Through the FTP protocol, users can perform operations such as uploading or downloading files on the FTP server. Although there are many sites downloaded through the HTTP protocol, because the FTP protocol can well control the number of users and the allocation of broadband, and upload and download files quickly and conveniently, FTP has become the preferred server for file upload and download in the network. At the same time, it is also an application program, through which users can connect their computers to all servers running the FTP protocol around the world, and access a large number of programs and information on the servers. The function of the FTP service is to realize the remote transmission of complete files. (From Baidu Encyclopedia)

Create files in multi-level directories

I always thought that the multi-level directory creation of the ftp server is the same as the directory creation in java, but it is not the same~ You can refer to the document: About the path switching problem of the ftp upload changeWorkingDirectory() method Upload
files to the directory steps:hello.json/home/upload/

1. Use it first FTPClient.changeWorkingDirectory("home")to determine whether the home directory exists? If not present then 2
2. Use to FTPClient.makeDirectory("home")create a directory, and use to FTPClient.changeWorkingDirectory("home")point the ftp session to the home directory
3. Use FTPClient.changeWorkingDirectory("upload")to determine whether the upload directory exists? If not present then 4
4. Use to FTPClient.makeDirectory("upload")create a directory, and use to FTPClient.changeWorkingDirectory("upload")point the ftp session to the upload directory
5. FTPClient.storeFile(fileName, is);Upload file stream using is

FTP transfer mode

Active mode and passive mode of FTP

Contrast	active mode	passive mode
client	Randomly open a port (above 1024)	-
Server	-	Randomly open a port locally (above 1024)

Real knowledge comes from practice

Import dependencies and add configuration

1. Add dependencies in pom.xml

<dependency>
	<groupId>commons-net</groupId>
	<artifactId>commons-net</artifactId>
	<version>3.6</version>
</dependency>

2. Add ftp server configuration in application.yml (multiple environments need to be configured separately in application-dev/test/prod.yml)

ftp:
  hostname: 172.16.1.1
  port: 21
  username: lizzy
  password: lizzy
  root-path: /upload/

Define configuration beans

package com.lizzy.common.config;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

import lombok.Data;

@ConfigurationProperties(prefix = "ftp")
@Data
@Component
public class FtpConfig {
    
    
	
	private String hostname;
	
	private Integer port;
	
	private String username;
	
	private String password;
	
	private String rootPath;
	
}

Define the FTP tool class

Get FTPClient, set timeout

private FTPClient getFTPClient() {
    
    
		
	FTPClient client = new FTPClient();
	// 设置默认超时时间30s
	client.setDefaultTimeout(30000);
	// 设置链接超时时间
	client.setConnectTimeout(30000);
	// 设置数据传输时间
	client.setDataTimeout(30000);
	return client;
}

upload

/**
 * 上传文件至FTP服务器 
 * @param config ftp服务器配置
 * @param path 相对目录（不包含根目录）
 * @param fileName 文件名 
 * @param is 文件流 
 */
public static void upload(FtpConfig config, String path, String fileName, InputStream is) {
    
    
	
	FTPClient client = getFTPClient();
	try {
    
    
		// 连接服务器
		client.connect(config.getHostname(), config.getPort());
		// 登录
		client.login(config.getUsername(), config.getPassword());
		int reply = client.getReplyCode();
		if (!FTPReply.isPositiveCompletion(reply)) {
    
    
			
			// 连接失败 
			client.disconnect();
			log.info("FTP服务器配置链接失败！请检查配置：{}", config.toString());
			return ;
		}
		
		// 设置被动模式，开通一个端口来传输数据
		client.enterLocalPassiveMode();
		// 切换到上传目录
		final String filePath = config.getRootPath() + path;
		if (!client.changeWorkingDirectory(filePath)) {
    
    
			// 目录不存在则创建
			String[] dirs = filePath.split("/");
			for (String dir : dirs) {
    
    
				
				if (StringUtils.isEmpty(dir)) {
    
    
					continue ;
				}
				if (!client.changeWorkingDirectory(dir)) {
    
    
					client.makeDirectory(dir);
					client.changeWorkingDirectory(dir);
				}
			}
		}
		
		// 设置被动模式，开通一个端口来传输数据
		client.enterLocalPassiveMode();
		//设置上传文件的类型为二进制类型
		client.setFileType(FTP.BINARY_FILE_TYPE);
		client.storeFile(fileName, is);
		log.debug("文件{}成功上传至FTP服务器！", filePath + "/" + fileName);
		client.logout();
		
	} catch (IOException e) {
    
    
		log.info("FTP服务器配置链接失败！错误：{}", e.getMessage());
		e.printStackTrace();
	} finally {
    
    
		try {
    
    
			is.close();
			if (client.isConnected()) {
    
    
				client.disconnect();
			}
		} catch (IOException e) {
    
    
			e.printStackTrace();
		}
	}
	
}

download

/**
 * 从FTP服务器下载文件流 
 * @param config FTP服务器配置
 * @param path 相对目录（不包含根目录）
 * @param fileName 文件名 
 * @return
 */
public static InputStream download(FtpConfig config, String path, String fileName) {
    
    
	
	FTPClient client = getFTPClient();
	try {
    
    
		// 连接服务器
		client.connect(config.getHostname(), config.getPort());
		// 登录
		client.login(config.getUsername(), config.getPassword());
		int reply = client.getReplyCode();
		if (!FTPReply.isPositiveCompletion(reply)) {
    
    
			
			// 连接失败 
			client.disconnect();
			log.info("FTP服务器配置链接失败！请检查配置：{}", config.toString());
			return null;
		}
		
		// 设置被动模式，开通一个端口来传输数据
		client.enterLocalPassiveMode();
		// 切换到下载目录
		final String filePath = config.getRootPath() + path;
		client.changeWorkingDirectory(filePath);
		log.debug("FTP session指向下载目录：{}", filePath);
		//设置上传文件的类型为二进制类型
		client.setFileType(FTP.BINARY_FILE_TYPE);
//			client.setControlEncoding("utf8");
		
		// 中文名会下载失败，文件名需转码 
		InputStream is = client.retrieveFileStream(new String(fileName.getBytes("gbk"), "ISO-8859-1"));
		if (null == is) {
    
    
			log.debug("下载文件：{}失败！读取长度为0！", fileName);
			return null;
		}
		InputStream retIs = copyStream(is);
		is.close();
		client.completePendingCommand();
		log.debug("从FTP服务器下载文件（{}）成功！", fileName);
		return retIs;
		
	} catch (IOException e) {
    
    
		log.info("FTP服务器配置链接失败！错误：{}", e.getMessage());
		e.printStackTrace();
	} finally {
    
    
		
		try {
    
    
			if (client.isConnected()) {
    
    
				log.info("从FTP服务器下载文件{}流程结束，关闭链接！", fileName);
				client.disconnect();
			}
		} catch (IOException e) {
    
    
			e.printStackTrace();
		}
	}
	return null;
}

The copy stream method is as follows:

private InputStream copyStream(InputStream is) throws IOException {
    
    
	
	ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
	byte[] buffer = new byte[1024];
	int len;
	while ((len = is.read(buffer)) > -1 ) {
    
    
	    baos.write(buffer, 0, len);
	}
	baos.flush();
	
	return new ByteArrayInputStream(baos.toByteArray());
}

problems encountered

file overwrite

Configuration of whether to overwrite files with the same name on the FTP server

retrieveFileStream stuck

The code before the error is as follows: I really looked at it for a long time and didn’t know why it failed, so I started to print the size of each file, and finally found a little clue, the size of the printed file of two different files is the same! Can you see it, I am speechless, use inputstream.available()the method carefully to judge the flow=_=||

// 中文名会下载失败，文件名需转码 
InputStream is = client.retrieveFileStream(new String(fileName.getBytes("gbk"), "ISO-8859-1"));
if (null == is || is.available() < 1) {
    
    
	log.debug("下载文件：{}失败！读取长度为0！", fileName);
	return null;
}
log.debug("结束读取文件：{}，大小为：{}", fileName, is.available());
InputStream retIs = copyStream(is);
is.close();
client.completePendingCommand();

Use inputstream.available() with caution when judging stream operations

Precautions (mined pit)

Connect first and then enterLocalActiveMode

Reference document: Ftpclient calls retrieveFileStream to return null, download failure problem in docker

In the connect method, the mode is set to active mode ACTIVE_LOCAL_DATA_CONNECTION_MODE.
The enterLocalActiveMode method is to set the link mode to passive mode

If you call enterLocalActiveMode first and then connect, the connection mode is still active mode!

To get the return stream, you must call completePendingCommad

Reference document: Precautions for using the completePendingCommand method in FTPClient

The problem encountered in the process of writing FTP is that the frame of the process is fixed, and you can't write more or less, otherwise it will be a bunch of blocks!

// 上传方法，之后若调用completePendingCommand会卡死 
public boolean storeFile(String remote, InputStream local)
// 上传方法，之后必须调用completePendingCommand 
public OutputStream storeFileStream(String remote)

// 下载方法，之后若调用completePendingCommand会卡死 
public boolean retrieveFile(String remote, OutputStream local)
// 下载方法，之后必须调用completePendingCommand 
public InputStream retrieveFileStream(String remote)

Destination inputStream.close() re-completePendingCommad

Reference document: Stepping on the pit when using the completePendingCommand method in FTPClient

completePendingCommand() will always wait for the FTP Server to return 226 Transfer complete, but the FTP Server will only return when the InputStream executes the close method. So first execute the close method, and then execute the completePendingCommand

postscript

The demo was written in a hurry, and I will update it if I encounter problems later~