Detailed explanation of FastDFS Docker installation and normal installation {with Java code}

FastDFS

50 pictures detailing VMware Workstation 16 + CentOS 7 download and installation

[Packages required for normal installation of FastDFS] libfastcommon, fastdfs, fastdfs-nginx-module
Baidu cloud disk download

【HHS 工具】Xshell&Xftp、FinalShell、SecureCRT&SecureFX

Baidu cloud disk download

Introduction

FastDFS is a lightweight distributed file management system open sourced by Taobao's Yu Qing in 2008. FastDFS is implemented in C language and supports UNIX systems such as Linux and MacOS. FastDFS is similar to Google FS. It is an application-level file system. It is not a universal file system and can only be accessed through a proprietary API. Currently, it provides C and Java SDK, as well as PHP extension SDK.

FastDFS is specially tailored for Internet applications to solve the problem of large-capacity file storage and pursue high performance and high scalability. It can be regarded as a file-based key/value storage system. The key is the file ID and the value is the file content, so it is called It is more suitable to serve as a distributed file storage service.

Advantage

Large capacity storage and high performance access

Architecture

As a distributed file management system, FastDFS mainly includes four functions:

• File storage
• File synchronization
• File Upload
• Download Document

System architecture diagram of FastDFS official website:

The FastDFS architecture consists of two parts: Tracker and Storage. Tracker is used to track files, which is equivalent to an index of files, while Storage is used to save files.

The uploaded file is eventually saved on Storage, and the metadata information of the file is saved on Tracker. Load balancing of Storage can be achieved through Tracker.

Storage is generally built into a cluster. A Storage Cluster can be composed of multiple groups. Different groups do not communicate with each other. A group is equivalent to a small cluster. The group is composed of multiple Storage Servers. The Storage Servers in the group will File synchronization over the connection ensures high availability.

Install

Docker installation

Install

# 1、拉取镜像
docker pull delron/fastdfs
# 2、使用 docker 镜像构建 tracker 容器（跟踪服务器，起到调度的作用）
docker run -dti --network=host --name tracker -v /var/fdfs/tracker:/var/fdfs -v /etc/localtime:/etc/localtime delron/fastdfs tracker
# 3、使用 docker 镜像构建 storage 容器（存储服务器，提供容量和备份服务）
docker run -dti --network=host --name storage -e TRACKER_SERVER=192.168.226.128:22122 -e GROUP_NAME=group1 -v /var/fdfs/storage:/var/fdfs -v /etc/localtime:/etc/localtime delron/fastdfs storage

# 若不修改端口，跳过【4-6】
# 4、进入 storage 容器，到 storage 的配置文件中配置 http 访问的端口（默认：8888），配置文件在 /etc/fdfs 目录下的 storage.conf
	# 进入容器
	docker exec -it storage /bin/bash
	# 进入目录
	cd /etc/fdfs
	# 编辑文件
	vi storage.conf
# 5、修改 storage 中的 nginx （不需要安装）
	cd /usr/local/nginx/conf
	vi nginx.conf
# 6、修改完配置重启容器，没有修改就不需要重启
docker restart storage

# 开放端口
# nginx
firewall-cmd --zone=public --permanent --add-port=8888/tcp
# tracker
firewall-cmd --zone=public --permanent --add-port=22122/tcp
# storage
firewall-cmd --zone=public --permanent --add-port=23000/tcp
# 重启防火墙
systemctl restart firewalld
# 容器开机自启
docker update --restart=always tracker
docker update --restart=always storage

java client call

structure

rely

<!-- Swagger3 -->
<dependency>
    <groupId>io.springfox</groupId>
    <artifactId>springfox-boot-starter</artifactId>
    <version>3.0.0</version>
</dependency>
<!-- FastDFS -->
<dependency>
    <groupId>com.github.tobato</groupId>
    <artifactId>fastdfs-client</artifactId>
    <version>1.27.2</version>
</dependency>

Configuration

application.properties

# FastDFS 配置
# 读取时间
fdfs.so-timeout=1500
# 连接超时时间
fdfs.connect-timeout=600
# 缩略图
fdfs.thumb-image.width=150
fdfs.thumb-image.height=150
# Tracker 服务配置地址列表，确保 tracker storage nginx 已经启动
fdfs.tracker-list[0]=192.168.226.128:22122
fdfs.web-server-url=http://192.168.226.128:8888/

FdfsConfig

@Configuration
@Import(FdfsClientConfig.class) // 提供拥有 FastDFS 的 Java 客户端
@EnableMBeanExport(registration = RegistrationPolicy.IGNORE_EXISTING) // 解决 JMX 重复注册 Bean 的问题
public class FdfsConfig {
    
    

}

Swagger2Config

@Configuration
public class Swagger2Config {
    
    

    @Bean
    Docket docket() {
    
    
        return new Docket(DocumentationType.OAS_30)
                // 配置网站的基本信息
                .apiInfo(new ApiInfoBuilder()
                        // 网站标题
                        .title("FastDFS接口文档")
                        // 标题后面的版本号
                        .version("v1.0")
                        .description("FastDFS接口文档")
                        // 联系人信息
                        .contact(new Contact("yueyazhui", "https://www.yueyazhui.top", "[email protected]"))
                        .build())
                .select()
                // 指定接口的位置
                .apis(RequestHandlerSelectors.basePackage("top.yueyazhui.learnfastdfs02.controller"))
                .build();
    }
}

FdfsClientUtil

@Component
public class FdfsClientUtil {
    
    

    @Autowired
    private FastFileStorageClient fastFileStorageClient;
    @Value("${fdfs.web-server-url}")
    private String fdfsWebServerUrl;

    /**
     * 文件上传
     *
     * @param file
     * @return 返回文件路径（卷名和文件名）
     * @throws IOException
     */
    public String upload(MultipartFile file) throws IOException {
    
    
        if (file.isEmpty()) {
    
    
            return "文件不存在";
        }
        byte[] bytes = file.getBytes();
        long fileSize = file.getSize();
        String originalFilename = file.getOriginalFilename();
        String extension = originalFilename.substring(originalFilename.lastIndexOf(".") + 1);
        InputStream is = new ByteArrayInputStream(bytes);
        Set<MetaData> metaDataSet = new HashSet<MetaData>();
        metaDataSet.add(new MetaData("date", DateFormatUtils.format(new Date(), "yyyy-MM-dd HH:mm:ss")));
        metaDataSet.add(new MetaData("author", "yueyazhui"));
        StorePath storePath = fastFileStorageClient.uploadFile(is, fileSize, extension, metaDataSet);
        return fdfsWebServerUrl + storePath.getFullPath();
    }

    /**
     * 下载文件
     *
     * @param filePath 文件路径
     * @return 文件字节
     * @throws IOException
     */
    public byte[] download(String filePath) {
    
    
        byte[] bytes = null;
        if (StringUtils.isNotBlank(filePath)) {
    
    
            String group = filePath.substring(0, filePath.indexOf("/"));
            String path = filePath.substring(filePath.indexOf("/") + 1);
            DownloadByteArray byteArray = new DownloadByteArray();
            bytes = fastFileStorageClient.downloadFile(group, path, byteArray);
        }
        return bytes;
    }

    /**
     * 删除文件
     *
     * @param filePath 文件路径
     */
    public void delete(String filePath) {
    
    
        if (StringUtils.isNotBlank(filePath)) {
    
    
            fastFileStorageClient.deleteFile(filePath);
        }
    }
}

FileController

@Controller
@Api(tags = "文件")
public class FileController {
    
    

    @Autowired
    private FdfsClientUtil fdfsClientUtil;

    @GetMapping
    @ApiIgnore
    public String index() {
    
    
        return "index";
    }

    /**
     * 上传文件
     *
     * @param file 文件
     * @return 文件路径
     */
    @PostMapping(value = "upload", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
    @ResponseBody
    public String upload(@RequestPart("file") MultipartFile file) throws IOException {
    
    
        String filePath = fdfsClientUtil.upload(file);
        return filePath;
    }

    /**
     * 下载文件
     *
     * @param filePath 文件路径
     * @return
     */
    @GetMapping(value = "download")
    @ResponseBody
    @ApiOperation("下载")
    @ApiImplicitParams({
    
    
            @ApiImplicitParam(name = "filePath", value = "文件路径", required = true),
            @ApiImplicitParam(name = "fileName", value = "文件名", required = true)
    })
    public ResponseEntity<byte[]> download(@RequestParam("filePath") String filePath, @RequestParam("fileName") String fileName) throws UnsupportedEncodingException {
    
    
        byte[] bytes = fdfsClientUtil.download(filePath);
        HttpHeaders headers = new HttpHeaders();
        headers.setContentDispositionFormData("attachment", new String(fileName.getBytes("UTF-8"), "ISO-8859-1"));
        headers.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        return new ResponseEntity<>(bytes, headers, HttpStatus.CREATED);
    }

    /**
     * 删除文件
     *
     * @param filePath 文件路径
     * @return 删除结果
     */
    @DeleteMapping(value = "delete")
    @ResponseBody
    @ApiOperation("删除")
    @ApiImplicitParam(name = "filePath", value = "文件路径", required = true)
    public void delete(@RequestParam("filePath") String filePath) {
    
    
        fdfsClientUtil.delete(filePath);
    }
}

Normal installation

FastDFS is generally used to upload images. After the image is uploaded successfully, Nginx is generally used to access the image. Therefore, FastDFS installation will be introduced from three aspects:

• Tracker installation
• Storage installation
• nginx installation

Tracker installation

First, you need to prepare an environment, two libraries and an installation package.

an environment

FastDFS is developed in C language and requires gcc environment. The installation command is as follows:

yum -y install gcc-c++

Two libraries

FastDFS depends on the libevent library. The installation command is as follows:

yum -y install libevent

The basic library libfastcommon required for FastDFS to run is officially provided by FastDFS.

libfastcommon download address

Note: You can also download it from Baidu Cloud Disk at the beginning of the article.

Copy the downloaded libfastcommon to the /usr/local/ directory, and then execute the following commands in sequence:

cd /usr/local
tar -zxvf libfastcommon-1.0.43.tar.gz 
cd libfastcommon-1.0.43
./make.sh
./make.sh install

an installation package

The Tracker and Storage installation packages are the same and can be downloaded once.

The installation file can be downloaded from the FastDFS GitHub repository at the download address

Note: You can also download it from Baidu Cloud Disk at the beginning of the article.

Copy the downloaded file to the /usr/local directory, and then execute the following commands to install:

cd /usr/local
tar -zxvf fastdfs-6.06.tar.gz
cd fastdfs-6.06
./make.sh
./make.sh install

After the installation is successful, execute the following command to copy the configuration file in the conf directory of the installation directory to the /etc/fdfs directory:

cd conf
cp ./* /etc/fdfs/

Configuration

Enter the /etc/fdfs directory to configure:

Open the tracker.conf file:

cd /etc/fdfs
vi tracker.conf

Modify the following configuration:

The default port is 22122, which can be modified according to actual needs. Then configure the metadata saving directory (note that the directory must exist).

start up

Start Tracker:

/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

Tracker installed successfully.

Storage installation

For simplicity, just build a Storage instance.

Storage installation also requires libevent and libfastcommon. For the installation of these two libraries, refer to the above.

Storage installation is also the same as Tracker, and the execution commands are also the same. If Tracker and Storage are installed on the same server. (Equivalent to having Storage already installed when installing Tracker)

The only thing you need to do is to enter the /etc/fdfs directory and configure Storage:

cd /etc/fdfs
vi storage.conf

Configure three places, namely base_path, store_path0 and tracker_server; the tracker_server template has two addresses. If there is only one Tracker, configure one and comment out the other.

After the configuration is complete, execute the following command to start Storage:

/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start

After the Tracker and Storage are started, the file is uploaded; but if the uploaded file is an image, we also need to provide an image access function, and the current best solution is Nginx.

nginx installation

Nginx is an important partner of FastDFS.

The installation of Nginx is divided into two steps:

• Install nginx
• Install fastdfs-nginx-module under Storage

Install nginx

cd /root/install-package
wget http://nginx.org/download/nginx-1.22.0.tar.gz
tar -zxvf nginx-1.22.0.tar.gz
cd nginx-1.22.0
yum -y install pcre-devel
yum -y install openssl openssl-devel
./configure
make
make install
# 启动
cd /usr/local/nginx/sbin
./nginx
# 修改 Nginx 配置文件，重新加载 Nginx
./nginx -s reload

After Nginx starts successfully, visit the Nginx address in the browser. If you see this page, it means that Nginx has been installed successfully.

Install fastdfs-nginx-module under Storage

fastdfs-nginx-module download address

Note: You can also download it from Baidu Cloud Disk at the beginning of the article.

Copy the downloaded file to the /usr/local directory. Then enter the /usr/local directory and execute the following commands:

cd /usr/local
tar -zxvf fastdfs-nginx-module-1.22.tar.gz

Then /usr/local/fastdfs-nginx-module-1.22/src/mod_fastdfs.confcopy the file to /etc/fdfs/the directory and modify the contents of the file:

cp /usr/local/fastdfs-nginx-module-1.22/src/mod_fastdfs.conf /etc/fdfs/
vi /etc/fdfs/mod_fastdfs.conf

Next, go back to the decompression directory of the nginx installation file downloaded in the first step and execute the following command to reconfigure the compilation and installation:

cd /root/install-package/nginx-1.22.0
./configure --add-module=/usr/local/fastdfs-nginx-module-1.22/src
make
make install

After the installation is complete, modify the nginx configuration file as follows:

vi /usr/local/nginx/conf/nginx.conf

Configure nginx request forwarding here.

After the configuration is completed, start nginx;

cd /usr/local/nginx/sbin
ps aux | grep nginx
kill -9 25665
kill -9 25666
./nginx

Seeing the following log indicates that nginx started successfully:

ngx_http_fastdfs_set pid=25899

The role of fastdfs-nginx-module

Storage is composed of many groups, and each group is a small cluster. In each group, the data will be synchronized, but if the data has not been synchronized yet, a request will be sent at this time. What should I do? At this time, fastdfs-nginx-module will directly obtain the file from the source Storage.

The installation was successful.

Java client call

rely:

<dependency>
    <groupId>net.oschina.zcx7878</groupId>
    <artifactId>fastdfs-client-java</artifactId>
    <version>1.27.0.0</version>
</dependency>

Add the FastDFS configuration file fastdfs-client.properties in the project's resources directory with the following content:

## fastdfs-client.properties

fastdfs.connect_timeout_in_seconds = 5
fastdfs.network_timeout_in_seconds = 30

fastdfs.charset = UTF-8

fastdfs.http_anti_steal_token = false
fastdfs.http_secret_key = yueyazhui
fastdfs.http_tracker_http_port = 80

fastdfs.tracker_servers = 47.95.1.150:22122

## Whether to open the connection pool, if not, create a new connection every time
fastdfs.connection_pool.enabled = true

## max_count_per_entry: max connection count per host:port , 0 is not limit
fastdfs.connection_pool.max_count_per_entry = 500

## connections whose the idle time exceeds this time will be closed, unit: second, default value is 3600
fastdfs.connection_pool.max_idle_time = 3600

## Maximum waiting time when the maximum number of connections is reached, unit: millisecond, default value is 1000
fastdfs.connection_pool.max_wait_time_in_ms = 1000

The address of fastdfs.tracker_servers Tracker can be configured according to the actual situation.

fastdfs.http_secret_key secret key.

code show as below:

StorageClient1 client1 = null;

@BeforeEach
void before() throws MyException, IOException {
    
    
    // 加载配置文件
    ClientGlobal.initByProperties("fastdfs-client.properties");
    // 构建TrackerClient
    TrackerClient trackerClient = new TrackerClient();
    // 获得TrackerServer
    TrackerServer trackerServer = trackerClient.getConnection();
    // 初始化StorageServer
    StorageServer storageServer = null;
    // 获得一个StorageClient1
    client1 = new StorageClient1(trackerServer, storageServer);
}

@Test
void upload() throws MyException, IOException {
    
    
    // 元数据信息（额外信息）
    NameValuePair pairs[] = null;
    // 上传
    String fileId = client1.upload_file1("C:\\Users\\15235\\Pictures\\Saved Pictures\\images\\avatar_admin.gif", "gif", pairs);
    System.out.println("fileId = " + fileId);
}

@Test
void download() throws MyException, IOException {
    
    
    // 下载
    byte[] bytes = client1.download_file1("group1/M00/00/00/L18BlmNQ7Q-ALxnTAAItxHZKwEI729.gif");
    FileOutputStream fos = new FileOutputStream(new File("C:\\Users\\15235\\Pictures\\Saved Pictures\\images\\avatar_admin-copy.gif"));
    fos.write(bytes);
    fos.close();
}

secure access

When accessing, just add a token from the time of uploading.

First enable token verification on the server:

vi /etc/fdfs/http.conf

After the configuration is complete, remember to restart the server:

cd /usr/local/nginx/sbin
./nginx -s stop
./nginx

Get the token, the code is as follows:

@Test
void token() throws MyException, UnsupportedEncodingException, NoSuchAlgorithmException {
    
    
    // 获取时间戳
    int ts = (int) Instant.now().getEpochSecond();
    // 获取Token
    String token = ProtoCommon.getToken("M00/00/00/L18BlmNQ7Q-ALxnTAAItxHZKwEI729.gif", ts, "yueyazhui");
    StringBuffer sb = new StringBuffer();
    sb
        .append("http://47.95.1.150/")
        .append("group1/M00/00/00/L18BlmNQ7Q-ALxnTAAItxHZKwEI729.gif")
        .append("?token=")
        .append(token)
        .append("&ts=")
        .append(ts);
    System.out.println("sb.toString() = " + sb.toString());
}

The token is mainly obtained based on the ProtoCommon.getToken method. Note that the first parameter of this method is the file ID you want to access. **Note that this address does not contain group; **The second parameter is the timestamp. The three parameters are keys, which must be consistent with the configuration of the server.

Concatenate the generated string and append it to the access path, such as: http://47.95.1.150/group1/M00/00/00/L18BlmNQ7Q-ALxnTAAItxHZKwEI729.gif?token=108711d7651b1b78905896181d7693cb&ts=1666256470. "If there is no token in the access path at this time, the access will fail."

Finish!

Excerpt: Jiangnan Yidianyu FastDFS