Distributed file storage FastDFS service construction and code implementation

Distributed file storage FastDFS

FastDFS is an open source lightweightDistributed file system, It manages files, functions include: file storage, file synchronization, file access (file upload, file download), etc., which solves the problem of mass storage and load balancing. Especially suitable for online services that use files as the carrier, such as photo album websites, video websites, and so on.

FastDFS is tailor-made for the Internet, taking full consideration of mechanisms such as redundant backup, load balancing, and linear expansion, and paying attention to indicators such as high availability and high performance. It is easy to build a high-performance file server cluster with FastDFS to provide file upload and download And other services.

The FastDFS architecture includes Tracker server and Storage server . The client requests the Tracker server to upload and download files, and the Storage server completes the file upload and download through the Tracker server scheduling .

The tracker server is used for load balancing and scheduling. The Tracker server can find the Storage server to provide file upload services according to some strategies when uploading files. The tracker can be called a tracking server or a dispatch server. The function of the Storage server is file storage. The files uploaded by the client are finally stored on the Storage server. The Storage server does not implement its own file system but uses the file system of the operating system to manage files. Storage can be called a storage server.

Vernacular

FastDFS two components

1. Tracker is responsible for load balancing operations of file management, control center (registration center)

2.Storage work (file upload, download, etc.)

3. FastDFS noMaster-slaveConcept. onlySynchronize，Multiple groups together are clusters

work process

Upload process

The upload return format is as follows

group1/M00/02/44/1.png

instruction manual

1.客户端上传文件后存储服务器将文件 ID 返回给客户端，此文件 ID 用于以后访问该文件的索引信息。文件索引信息包括：组名，虚拟磁盘路径，数据两级目录，文件名。
2.组名：文件上传后所在的storage组名称，在文件上传成功后有storage 服务器返回，需要客户端自行保存。
3.虚拟磁盘路径：storage 配置的虚拟路径，与磁盘选项store_path*对应。如果配置了
store_path0 则是 M00，如果配置了 store_path1 则是 M01，以此类推。
4.数据两级目录：storage 服务器在每个虚拟磁盘路径下创建的两级目录，用于存储数据文件。
5.文件名：与文件上传时不同。是由存储服务器根据特定信息生成，文件名包含：源存储服务器 IP 地址、文件创建时间戳、文件大小、随机数和文件拓展名等信息

Server operation

1. Install fastdfs, I am easy to operate based on docker

1.1 Get the mirror

docker pull morunchang/fastdfs

1.2 Run Tracker

docker run -d --name tracker --net=host morunchang/fastdfs sh tracker.sh

1.3 Run storage

docker run -d --name storage --net=host -e TRACKER_IP=192.168.1.1:22122 -e GROUP_NAME=group1 morunchang/fastdfs sh storage.sh

The network mode used is --net=host, 192.168.1.1 is the host's IP

group1 is the group name, that is, the storage group

If you want to add a new storage server, run the command again, paying attention to the new group name

1.4 Set the container to start automatically

docker update --restart=always tracker
docker update --restart=always storage

2. Nginx configuration

Nginx mainly provides support for FastDFS file access, you need to add Nginx_FastDFS_Module

When accessing the picture, the user accesses it through nginx, and ngnix will find the storage group corresponding to this module

2.1 Modify the nginx configuration file
vim /etv/nginx/conf/nginx.conf
and add the following content to the server

 location ~ /M00 {
                root /data/fast_data/data;
                ngx_fastdfs_module;
 }

Code manipulation

The language is not limited, php/java are the same, so for the operation, use java to develop

1. There is no need to explain how to build the project. It uses spring boot to quickly build one
. After the build is completed
, add the fastdfs dependency in pom.xml

 <dependency>
           <groupId>net.oschina.zcx7878</groupId>
           <artifactId>fastdfs-client-java</artifactId>
           <version>1.27.0.0</version>
       </dependency>

2. Add the fdfs_client.conf configuration file in the resource directory

# 链接超时时间
connect_timeout=60
#网络请求超时时间
network_timeout=60
charset=UTF-8
# tracker http端口
http.tracker_http_port=8080
# tracker TCP通信端口  安装服务时的端口号
tracker_server=10.211.55.4:22122

Project code

1. Controller controller class
2. Tool upload
3. File information package class

controller

package com.demo.fast.controller;

import com.demo.fast.file.FastDFSFile;
import com.demo.fast.utils.FastDFSUtil;
import org.springframework.util.StringUtils;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;

/**
 * @Author cxk
 * @date 2020/10/5 01:05
 */
@RestController
@RequestMapping("/upload")
@CrossOrigin
public class FileController {

    @PostMapping
    public String upload(@RequestParam(value = "file")MultipartFile multipartFile) throws Exception{
        //封装文件信息
        FastDFSFile fastDFSFile = new FastDFSFile(
                multipartFile.getOriginalFilename(), //文件名字 1.png
                multipartFile.getBytes(),  //文件字节数组
                StringUtils.getFilenameExtension(multipartFile.getOriginalFilename()) //获取文件扩展名
        );
        //调用封装好的工具类将文件传入进去就可以
        String[] upload = FastDFSUtil.upload(fastDFSFile);
        //拼接访问地址 nginx 访问地址
        // url = http://10.211.55.4:8080/upload[0]/upload[1]
        String url = "http://10.211.55.4:8080/" + upload[0] + "/" + upload[1];
        //在这你就可以吧url存入到你的数据库中去啦
        return url;
    }
}

Tools FastDFSUtil

package com.demo.fast.utils;

import com.demo.fast.file.FastDFSFile;
import org.csource.common.MyException;
import org.csource.common.NameValuePair;
import org.csource.fastdfs.ClientGlobal;
import org.csource.fastdfs.StorageClient;
import org.csource.fastdfs.TrackerClient;
import org.csource.fastdfs.TrackerServer;
import org.springframework.core.io.ClassPathResource;

import java.io.IOException;

/**
 * @Author cxk
 * @date 2020/10/5 01:06
 * 工具类
 */
public class FastDFSUtil {

    /**
     * 初始化tracker
     * //获取配置文件
     */
    static {
        try {
            String filePath = new ClassPathResource("fdfs_client.conf").getPath();
            ClientGlobal.init(filePath);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


    public static String[]  upload(FastDFSFile fastDFSFile) throws Exception{
        // 创建Tracker 访问客户端对象的创建TrackerClient
        TrackerClient trackerClient = new TrackerClient();
        // 通过TrackerClien 访问TrackerServer 服务 获取连接信息
        TrackerServer trackerServer = trackerClient.getConnection();
        //通过TrackerServer连接信息客户已获取Storage信息，创建StorageClient对象c存储StorageClient 信息
        StorageClient storageClient = new StorageClient(trackerServer, null);
        //通过StorageClient 访问Storage 实现文件上传并且获取文件存储后的信息
        /**
         * 参数1  上传文件的字节数字
         * 参数2 文件上传的扩展名
         * 参数3  附加参数
         */
//        NameValuePair[] meta_list = new NameValuePair[1];
//        meta_list[0] = new NameValuePair("author",fastDFSFile.getAuthor());
//        storageClient.upload_file(fastDFSFile.getContent(),fastDFSFile.getExt(),meta_list);
        //附加参数 获取作者
        /**
         * 返回的参数说明
         *  file[0]  返回文件存储在storage的组的名字  group1
         *  file[1]  返回文件存储在storage上的文件民资  M00/02/44/1.png
         */

        String[] file = storageClient.upload_file(fastDFSFile.getContent(), fastDFSFile.getExt(), null);
        return file;
    }
}

File information package FastDFSFile

package com.demo.fast.file;

import java.io.Serializable;

/**
 * @Author cxk
 * @date 2020/10/5 01:07
 *  封装文件上传等信息
 */
public class FastDFSFile implements Serializable {
    //文件名字
    private String name;
    //文件内容
    private byte[] content;
    //文件扩展名
    private String ext;
    //文件MD5摘要值
    private String md5;
    //文件创建作者
    private String author;

    public FastDFSFile(String name, byte[] content, String ext) {
        this.name = name;
        this.content = content;
        this.ext = ext;
    }

    public FastDFSFile(String name, byte[] content, String ext, String md5, String author) {
        this.name = name;
        this.content = content;
        this.ext = ext;
        this.md5 = md5;
        this.author = author;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public byte[] getContent() {
        return content;
    }

    public void setContent(byte[] content) {
        this.content = content;
    }

    public String getExt() {
        return ext;
    }

    public void setExt(String ext) {
        this.ext = ext;
    }

    public String getMd5() {
        return md5;
    }

    public void setMd5(String md5) {
        this.md5 = md5;
    }

    public String getAuthor() {
        return author;
    }

    public void setAuthor(String author) {
        this.author = author;
    }
}

Startup project

postman测试访问
127.0.0.1:10086/upload

As shown in the figure below, return the connection address and it will be OK when stored in the database.

Visit

http://10.211.55.4:8080/group1/M00/00/00/CtM3BF96Ea6AdHhSAAI_WH8DBGE491.png

I wrote a complete demo and those in need can download it and see the
demo address