[E-commerce website 003] FastDFS distributed file system (project practice: store static resources such as pictures, audio and video, e-commerce website has too many static resources, use distributed file system instead of traditional file system)

I. Introduction

FastDFS distributed file system (project practice: store static resources such as pictures, audio and video, e-commerce websites have too many static resources, use a distributed file system to replace the traditional file system)

Second, what is a distributed file system

2.1 Distributed File System

Definition: The physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network.

Traditional file system VS distributed file system The
files managed by the traditional file system are stored on this machine.
The files managed by the distributed file system are stored in many machines. These machines are connected through a network and must be managed uniformly. Regardless of uploading or accessing files, you need to access
the'distributed architecture = distributed lock + distributed transaction + distributed login (single sign-on) + distributed file system through the management center

Distributed file systems include: Taobao's FastDFS, Google's GFS, HDFS (Hadoop), TFS (Taobao), etc.

2.2 FastDFS is a distributed file system developed by Taobao C language

FastDFS
definition: a lightweight, high-performance open source distributed file system developed by Mr. Yu Qing from Taobao, developed in pure C language.
Functions: file storage, file synchronization, file access (uploading, downloading), access load balancing, online expansion, five functions are suitable for applications or systems with large-capacity storage requirements.
ps: Applications and systems with large-capacity storage requirements are also consistent requirements for all distributed storage.

Third, the architecture of FastDFS

3.1 FastDFS architecture diagram

FastDFS architecture diagram

Explanation of the above figure:
FastDFS has two main roles: Tracker Server and Storage Server.
First, Tracker Server: Tracker Server , which is mainly responsible for scheduling communication between storage nodes and clients , playing a load balancing role in access, and recording the running status of storage nodes, and is the hub that connects clients and storage nodes.
Second, Storage Server: storage server , save files and file meta data (meta data), each storage server will start a separate thread to actively report its status information to each tracker server in the Tracker cluster, including disk usage, File synchronization status and file upload and download times statistics and other information
Group: file group, cluster of multiple Storage Servers. After uploading a file to a machine in the same group, FastDFS will instantly synchronize the file to all other machines in the same group to play a backup role. Servers in different groups store different data, are independent of each other, and do not communicate.
Tracker Cluster: A cluster of tracking servers, composed of a group of Tracker Servers.
Storage Cluster: Storage cluster, composed of multiple groups.

3.2 FastDFS upload process

For the explanation of the above figure:

1. Client finds available Storage server through Tracker server.
2. The Tracker server returns the IP address and port number of an available Storage server to the Client.
3. The Client directly establishes a connection with one of the Storage servers through the IP address and port returned by the Tracker server and uploads files.
4. After the upload is complete, the Storage server returns a file ID to the Client, and the file upload ends.

3.3 FastDFS download process

For the explanation of the above figure:

1. The Client uses the Tracker server to find the Storage server where the file to be downloaded is located.
2. The Tracker server returns the IP address and port number of a certain Storage server containing the specified file to the Client.
3. The Client directly establishes a connection with one of the Storage servers through the IP address and port returned by the Tracker server and specifies the file to be downloaded.
4. The file was downloaded successfully.

3.4. Installation and use

Refer to the pre-class information: "centos install FastDFS.md"

3.5 Practice: java client

Mr. Yu Qing provided a Java client, but as a C programmer, the Java code written can be imagined. And it has been out of maintenance for a long time.

Here is an open source FastDFS client that supports the latest SpringBoot2.0.

Configuration and use are extremely simple, support connection pool, support automatic generation of thumbnails, mad and cool, there is nothing.

Address: tobato/FastDFS_client

3.5.1. Introduce maven dependency

In the parent project, we have managed dependencies, the version is:

<fastDFS.client.version>1.26.2</fastDFS.client.version>

Therefore, here we can directly introduce the coordinates:

<dependency>
    <groupId>com.github.tobato</groupId>
    <artifactId>fastdfs-client</artifactId>
</dependency>

3.5.2. Introduction of configuration classes

Pure java configuration:

@Configuration
@Import(FdfsClientConfig.class)
// 解决jmx重复注册bean的问题
@EnableMBeanExport(registration = RegistrationPolicy.IGNORE_EXISTING)
public class FastClientImporter {
    
    
}

3.5.3. Write FastDFS configuration file information

fdfs:
  so-timeout: 1501
  connect-timeout: 601
  thumb-image: # 缩略图
    width: 60
    height: 60
  tracker-list: # tracker地址
    - 192.168.56.101:22122

3.5.4. Test class

@RunWith(SpringRunner.class)
@SpringBootTest(classes = LyUploadService.class)
public class FdfsTest {
    
    

    @Autowired
    private FastFileStorageClient storageClient;

    @Autowired
    private ThumbImageConfig thumbImageConfig;

    @Test
    public void testUpload() throws FileNotFoundException {
    
    
        File file = new File("D:\\test\\baby.png");
        // 上传并且生成缩略图
        StorePath storePath = this.storageClient.uploadFile(
                new FileInputStream(file), file.length(), "png", null);
        // 带分组的路径
        System.out.println(storePath.getFullPath());
        // 不带分组的路径
        System.out.println(storePath.getPath());
    }

    @Test
    public void testUploadAndCreateThumb() throws FileNotFoundException {
    
    
        File file = new File("D:\\test\\baby.png");
        // 上传并且生成缩略图
        StorePath storePath = this.storageClient.uploadImageAndCrtThumbImage(
                new FileInputStream(file), file.length(), "png", null);
        // 带分组的路径
        System.out.println(storePath.getFullPath());
        // 不带分组的路径
        System.out.println(storePath.getPath());
        // 获取缩略图路径
        String path = thumbImageConfig.getThumbImagePath(storePath.getPath());
        System.out.println(path);
    }
}

result:

group1/M00/00/00/wKg4ZVro5eCAZEMVABfYcN8vzII630.png
M00/00/00/wKg4ZVro5eCAZEMVABfYcN8vzII630.png
M00/00/00/wKg4ZVro5eCAZEMVABfYcN8vzII630_60x60.png

Visit the first path:

Visit the last path (thumbnail path), pay attention to adding the group name:

FastDFS and HDFS (Hadoop DFS) learn together, both

3.5.5. Transform upload logic

@Service
public class UploadService {
    
    

    private static final Logger logger = LoggerFactory.getLogger(UploadController.class);

    // 支持的文件类型
    private static final List<String> suffixes = Arrays.asList("image/png", "image/jpeg");

    @Autowired
    FastFileStorageClient storageClient;

    public String upload(MultipartFile file) {
    
    
        try {
    
    
            // 1、图片信息校验
            // 1)校验文件类型
            String type = file.getContentType();
            if (!suffixes.contains(type)) {
    
    
                logger.info("上传失败，文件类型不匹配：{}", type);
                return null;
            }
            // 2)校验图片内容
            BufferedImage image = ImageIO.read(file.getInputStream());
            if (image == null) {
    
    
                logger.info("上传失败，文件内容不符合要求");
                return null;
            }

            // 2、将图片上传到FastDFS
            // 2.1、获取文件后缀名
            String extension = StringUtils.substringAfterLast(file.getOriginalFilename(), ".");
            // 2.2、上传
            StorePath storePath = this.storageClient.uploadFile(
                    file.getInputStream(), file.getSize(), extension, null);
            // 2.3、返回完整路径
            return "http://image.leyou.com/" + storePath.getFullPath();
        } catch (Exception e) {
    
    
            return null;
        }
    }
}

Just remove the original logic of saving the file and upload it to FastDFS.

3.5.6. Testing

Tested by RestClient:

3.6. Page test upload

Found that the upload was successful:

However, when we visit the page:

This is because our picture is uploaded to the virtual machine, the ip is: 192.168.56.101

Therefore, we need to map image.leyou.com to 192.168.56.101

Modify our hosts:

Upload again:

Four, interview golden finger

4.1 FastDFS and HDFS (Hadoop DFS) learn together

First, FastDFS and HDFS (Hadoop DFS) learn together.
FastDFS and HDFS (Hadoop DFS) learn together. Both are distributed file storage systems. They are both applications and systems for large-capacity storage requirements. They are easy to use. Interview If you want to blow, you must introduce the architecture of FastDFS and HDFS.

Second, why use the distributed file system FastDFS?
FastDFS distributed file system (project practice: store static resources such as pictures, audio and video, e-commerce websites have too many static resources, use distributed file system instead of traditional file system)

4.2 Distributed File System + FastDFS

4.2.1 Distributed File System

Definition: The physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network.

Traditional file system VS distributed file system The
files managed by the traditional file system are stored on this machine.
The files managed by the distributed file system are stored in many machines. These machines are connected through a network and must be managed uniformly. Regardless of uploading or accessing files, you need to access
the'distributed architecture = distributed lock + distributed transaction + distributed login (single sign-on) + distributed file system through the management center

Distributed file systems include: Taobao's FastDFS, Google's GFS, HDFS (Hadoop), TFS (Taobao), etc.

4.2.2 FastDFS is a distributed file system developed by Taobao C language

FastDFS
definition: a lightweight, high-performance open source distributed file system developed by Mr. Yu Qing from Taobao, developed in pure C language.
Functions: file storage, file synchronization, file access (uploading, downloading), access load balancing, online expansion, five functions are suitable for applications or systems with large-capacity storage requirements.
ps: Applications and systems with large-capacity storage requirements are also consistent requirements for all distributed storage.

4.3 The architecture of FastDFS

4.3.1 FastDFS architecture diagram

FastDFS architecture diagram

Explanation of the above figure:
FastDFS has two main roles: Tracker Server and Storage Server.
First, Tracker Server: Tracker Server , which is mainly responsible for scheduling communication between storage nodes and clients , playing a load balancing role in access, and recording the running status of storage nodes, and is the hub that connects clients and storage nodes.
Second, Storage Server: storage server , save files and file meta data (meta data), each storage server will start a separate thread to actively report its status information to each tracker server in the Tracker cluster, including disk usage, File synchronization status and file upload and download times statistics and other information
Group: file group, cluster of multiple Storage Servers. After uploading a file to a machine in the same group, FastDFS will instantly synchronize the file to all other machines in the same group to play a backup role. Servers in different groups store different data, are independent of each other, and do not communicate.
Tracker Cluster: A cluster of tracking servers, composed of a group of Tracker Servers.
Storage Cluster: Storage cluster, composed of multiple groups.

4.3.2 FastDFS upload process

For the explanation of the above figure:

1. Client finds available Storage server through Tracker server.
2. The Tracker server returns the IP address and port number of an available Storage server to the Client.
3. The Client directly establishes a connection with one of the Storage servers through the IP address and port returned by the Tracker server and uploads files.
4. After the upload is complete, the Storage server returns a file ID to the Client, and the file upload ends.

4.3.3 FastDFS download process

For the explanation of the above figure:

1. The Client uses the Tracker server to find the Storage server where the file to be downloaded is located.
2. The Tracker server returns the IP address and port number of a certain Storage server containing the specified file to the Client.
3. The Client directly establishes a connection with one of the Storage servers through the IP address and port returned by the Tracker server and specifies the file to be downloaded.
4. The file was downloaded successfully.

V. Summary

The FastDFS distributed file system is complete.
Code every day, make progress every day! ! !