Distributed file storage system (fastdfs)

1, a distributed file storage system

Upload pictures, due to the more than one server, so you can not find a picture in which server the above, then you need a file server for management picture.

FastDFS is a file server.

2、FastDFS

1 supports uploading -> give us a return path

2 Support Downloads -> Download using the path

Small files distributed file storage system :

Document G is less than a small file, the file can be split without

Need to split large files.

FastDFS is a distributed file storage system consists of two parts.

1 tracker (tracker)

2 storage (memory)

If you want to expand the capacity of FastDFS, very convenient, the group added directly on the line

3, FastDFS to build

3.1, created in the server container FastDFS

3.1.1 Prepare a tracker

docker run -d --name tracker --net=host morunchang/fastdfs sh tracker.sh

3.1.2 Prepare a storage

1 public addresses: When developing, using the public network

2 private address: on-line service, the use of private network

docker run -d --name storage --net=host -e TRACKER_IP=服务器公网:22122 -e GROUP_NAME=g1 morunchang/fastdfs sh storage.sh

You can use docker logs 容器名称commands to view the vessel is operating successfully.

In the same group storage files are backed up.

Located in different groups can use load balancing.

3.2, add project dependencies

<!--spring boot项目导入的依赖-->
<dependency>
    <groupId>com.github.tobato</groupId>
    <artifactId>fastdfs-client</artifactId>
    <version>1.26.7</version>
</dependency>
<!--测试类依赖-->
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-test</artifactId>
	<scope>test</scope>
</dependency>

3.3, yml profile

fdfs:
  tracker-list:            #TrackerList参数,支持多个
  - 192.168.1.105:22122
  - 192.168.1.106:22122 

3.4 Test operation fastdfs

package com.zxm.test;

import com.github.tobato.fastdfs.domain.fdfs.StorageNode;
import com.github.tobato.fastdfs.domain.fdfs.StorageNodeInfo;
import com.github.tobato.fastdfs.domain.proto.storage.DownloadFileWriter;
import com.github.tobato.fastdfs.service.FastFileStorageClient;
import com.github.tobato.fastdfs.service.TrackerClient;
import com.zxm.utils.FastdfsUtil;
import lombok.SneakyThrows;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import java.io.File;
import java.io.FileInputStream;
import java.util.HashSet;

@SpringBootTest
@RunWith(SpringRunner.class)
public class FileUploadTest {

    @Autowired
    private FastFileStorageClient fastFileStorageClient;

    @Autowired
    private TrackerClient trackerClient;

    /**
     * 文件的上传
     */
    @SneakyThrows
    @Test
    public void updateLoadTest(){
        // 也可以上传视频。只需要填入视频路径，然后将fileExtName换为视频的后缀就行。
        File file = new File("C:/Users/86131/Pictures/Saved Pictures/1.jpg");
        String path = fastFileStorageClient.uploadFile(
                new FileInputStream(file),
                file.length(),
                "jpg",
                new HashSet<>(0)).getFullPath();
        System.out.println(path);
    }

    /**
     * 文件的下载
     */
    @Test
    public void testDownload(){
        String path = "C:/Users/86131/Desktop/临时文件夹/d2.jpg";
        fastFileStorageClient.downloadFile("g1","M00/00/00/rBHR8l5h-T6AQTubAApSuL6VXKQ831.jpg", new DownloadFileWriter(path));
    }
    /**
     * 文件的删除
     * 删除之后，还可以访问，但是不能下载了。
     */
    @Test
    public void testDelete(){
        String fullPath = "g1/M00/00/00/rBHR8l5nSzeAS0pmAAVS7KokxMQ062.jpg";
        String groupName = FastdfsUtil.parseGroup(fullPath);
        System.out.println(groupName);
        String path = fullPath.replaceFirst(groupName+"/", "");
        System.out.println(path);
        fastFileStorageClient.deleteFile(groupName,path);
        System.out.println("删除完成");
//        或者直接使用下述方法就可删除。
//        fastFileStorageClient.deleteFile("g1","M00/00/00/rBHR8l5nSzeAS0pmAAVS7KokxMQ062.jpg");
    }

    @Test
    public void trackerTest(){
        /**
         * 文件下载时候用到。
         * fetchStorage对象中包含组名，storage的服务器ip，端口
         */
        StorageNodeInfo fetchStorage = trackerClient.getFetchStorage("g1", "M00/00/00/rBHR8l5nSzeAS0pmAAVS7KokxMQ062.jpg");
        /**
         * 文件上传时候用到
         * 得到当前注册到tracker上面的storege
         * 不同组可以使用负载均衡。
         */
        StorageNode storeStorage = trackerClient.getStoreStorage();
    }

}

3.5, the controller application fastDFS

package com.zxm.controller;

import com.github.tobato.fastdfs.service.FastFileStorageClient;
import lombok.SneakyThrows;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

/**
 * 负责文件上传的控制器
 */
@RestController
@RequestMapping("/admin/file")
public class FileController {
    @Autowired
    private FastFileStorageClient fastFileStorageClient;

    // http://47.94.225.69:8080  fastDFS自定义地址配置。
    // 此处使用@Value注解从yml配置文件中注入。
    @Value("${resources.url}")
    private String serverAddress; 

    /**
     * 上传一个图片
     * @param file
     * @return
     */
    @SneakyThrows
    @PostMapping("/upload/element")
    public ResponseEntity<String> uploadFile(@RequestParam("file") MultipartFile file){
        // 开始上传文件
        String fullPath = fastFileStorageClient.uploadFile(
                file.getInputStream(),
                file.getSize(),
                "jpg",
                null).getFullPath();
        return ResponseEntity.ok(serverAddress+"/"+fullPath);
    }
}

# fastdfs的自定义地址配置
resources:
  url: http://47.94.225.69:8080

4, fastdfs Principle Analysis

4.1, fastDFS file upload process

tracker also has the effect of load balancing, selecting one operation from a plurality of storage.

4.2, fastDFS file download process

5, FastDFS combination and Nginx

5.1, to do load balancing tracker

In order to carry higher amounts of concurrency, tracker cluster can be made. Need to use nginx load balancing.

5.2, the role of nginx fastdfs-nginx-module plug-in

FastDFS by Tracker server, file server storage on a Storage, Storage storage server but between the same group need to enter the file copy, the problem of synchronization delay

Assume Tracker server to upload files to the Storage Server 11, after a successful file upload ID has been returned to the client. At this point FastDFS storage cluster mechanism will synchronize the file to the same set of storage Storage Server 12, if the file has not been completed copy of the client if the file ID in the file take 12 Storage Server, will be unable to access the file mistake. (Storage Server 11 and Storage Server 12 is the same storage group)

But between the Storage Server storage groups need access to the same file copy, the problem of synchronization delay

Assume Tracker server to upload files to the Storage Server 11, after a successful file upload ID has been returned to the client. At this point FastDFS storage cluster mechanism will synchronize the file to the same set of storage Storage Server 12, if the file has not been completed copy of the client if the file ID in the file take 12 Storage Server, will be unable to access the file mistake. (Storage Server 11 and Storage Server 12 is the same storage group)

The fastdfs-nginx-module connections can be redirected to a file when the file upload the source server (Storage Server 11) take the file , avoid file copy client due to the delay caused by unreachable errors.