FastDFS Distributed File System

The article comes from Liu Xin's class notes

Introduction to FastDFS Distributed File System

    FastDFS is one of the well-known system-level open source software in China. Its users include Alipay, JD.com, Xunlei, 58.com, Ganji, etc. It is a software developed by individuals. The author is Yu Celebrate.

    This article is just to learn some design ideas of FastDFS and learn how it implements an efficient, concise, and lightweight system.

    We have entered the Internet era. While the Internet has brought convenience to our lives, it has also brought us many challenges.

    For the storage of massive files, one machine is not enough, so use multiple machines to store them.

    If only one copy of a file is stored, then if the machine storing the file is broken, the file will be lost. The solution is to back up the file. I believe most people have the habit of backing up important files. The same is true for FastDFS. In order to prevent a single point of failure, redundant backups are definitely required.

    FastDFS divides the application server into several groups, and each group can have multiple machines (usually 3), and each machine is called a storage server. The data in the same group is mutually backed up, that is to say, when a user transfers a file to any server, it will be backed up on the other two servers in the same group, so the storage space of a group is stored in the group. The machine with the least space is the same (same principle as the barrel). In order not to waste storage space, the three machines in the same group should be the same.

    How about enough storage for each storage server? From the perspective of expansion, it can be divided into multiple directories. Each directory starts with M and is divided by 00, 01, 02.... Generally, there is no need to divide so many directories, only one directory is enough.

    There are two levels of directories under each root directory. As shown in the figure, there are two levels of directories under /data/fastdfs0, each level has 256 directories, so there are 65535 directories in total. When storing a file, it is determined which directory it is placed in by hashing twice.

    Then the question arises, with so many groups, which group of servers should be selected for storage? In other words, which group do you visit when you visit?

    The solution provided by FastDFS is to introduce a tracker server, which is used to record the storage server information in each group. The storage information is actively reported to the tracker by each storage. With this information, the tracker can do scheduling. After work, see who has more storage space, and put the file there.

Features of FastDFS

• Groups are independent of each other
• Storage servers in the same group need to back up each other
  
  • After the file is stored in a storage, it needs to be backed up to another server
• There is no interaction between trackers
  
  • Each storgae server needs to actively report information to all trackers
  • The tracker and the tracker do not know each other's existence

How to upload files

    In order to facilitate the understanding of the downloaded file, it is assumed that the uploaded file is: ==Group1/M00/00/0C/wKjGgVgbV2-ABdo-AAAAHw.jpg==

    As shown in the sequence diagram below, you can see how the client uploads files to the server. First, the client sends an upload link request to the tracker, which is then scheduled by the tracker to query the available storage, and send the ip and port corresponding to the storgae to the client; after getting the storage server information, the client can directly upload the file to the storage; The storage will generate a new file name and write it to the disk. After completion, the new file information will be returned to the client, and the client will finally save the file information locally. It should be noted that the storage will periodically report information to the tracker.

How to choose a server

• There is more than one tracker, which one does the client choose to upload files?
  
  • The trackers are equal, you can choose any one
• How does tracker choose groups?
  
  • round robin
  • load balance (select the largest remaining space group to upload)
  • specify group (specify group upload)
• How to choose storage?
  
  • round robin, used by all servers for polling (default)
  • Sort by ip address and select the first storage (the one with the smallest ip address)
  • Sort by priority (the upload priority is set by stoage, and the parameter is upload_priority)
• How to choose storage path
  
  • round robin, polling (default)
  • load balance, select the storage path that uses the largest remaining space

How to choose a storage directory

• Selected storage directory?
  
  • storage will generate a file_id, encoded in Base64, the fields include: storage ip, file creation time, file size, file CRC32 checksum and random number
  • There are two 256 * 256 subdirectories under each storage directory, storage will hash twice according to the file file_id, and then store the file in the subdirectory with file_id as the file name

It should be noted that the file_id is saved by cilent. If it is not saved, you will not know where the uploaded file is going.

File synchronization between storage servers

• The storages in the same group are equal, and operations such as file upload and deletion can be performed on any storage
• File synchronization is only performed between stroages in the same group, and the push method is used, that is, the source server is synchronized to the target server
• Only the source data needs to be synchronized, and the backup data does not need to be synchronized again, otherwise a loop will be formed
• When a new storage is added, the existing storage will synchronize all existing data (including source data and backup data) to the new server

The last earliest synchronization of the Storage was synchronized

    This title is a bit awkward. Now there are three servers A, B, and C. Each server needs to record the last time that the other two servers synchronize to itself. For example, in the following figure, servers A and B synchronize all files with A at 9:31, and C synchronizes all files with A at 9:33, then the earliest synchronization time of server A is 9:31. Same goes for the other two servers.

    The meaning of the earliest synchronization time at the end is to determine whether a file exists on a storage. For example, the earliest synchronization time of server A here is 9:31, then if the creation time of a file is 9:30, it is certain that the file must exist on server A.

    Stroage will periodically tell the tracker the synchronization time of each machine. When the client needs to download a file, the tracker needs to determine whether a storage has the file. It only needs to parse the creation time of the file, and then compare it with the value. If it is greater than the creation time, it means that the file exists in the storage and can be downloaded from the storage.

    However, this algorithm has flaws, such as the following situation: the creation time of the W file is 9:33, and the server C has synchronized all the files before 9:33 to B, so the B server actually already has the W file, but according to the The earliest synchronization time at the end will consider that there is no W file in B. So this algorithm is simple, but sacrifices some files.

How to download files

    First, the client sends a download connection request. The requested thing is essentially ==Group1/M00/00/0C/wKjGgVgbV2-ABdo-AAAAHw.jpg==; the tracker will query the available storage server (according to the following four principles) Select) and send it to the client; now the client has the file information saved locally, as well as the address and port of the server, then directly access the corresponding server to download the file.

How to choose a storage server available for download

There are a total of four principles below, from top to small, the conditions are becoming more and more relaxed

• The source storage to which the file was uploaded (the file is uploaded directly to this server)
• File creation timestamp < file timestamp to which storage was synced, which means the current file has been synced
• File creation timestamp = file timestamp to which storage is synced, and (current time - file creation timestamp) > the maximum time it takes for a file to be synced (5 minutes)
• (current time - file creation time) > file synchronization delay threshold, for example, if we set the threshold to 1 day, it means that file synchronization can definitely be completed within one day

Use of FastDFS

    The user accesses the web server through a browser or mobile phone, and the web server forwards the request to the application server. After the application server receives the request, it interacts with the FastDFS file system through the fastDFS API. But this design will cause pressure on the application server, because both upload and download go through the application server.

    In order to avoid excessive pressure on the application server, the client can use Http to access directly without going through the application server.

FastDFS other content

Prevent hotlinking

    In order to prevent the files uploaded hard by others from being stolen by others, it can be solved by setting a token to the URL. The anti-hotlink configuration of FastDFS is as follows:

#是否做tokrn检查，缺省值为false

http.anti\_steal.check\_token=true

#生成token的有效时长/秒

http.anti\_steal.token\_ttl=900

#生成token的密钥，尽量设置长一些

http.anti\_steal.secret\_key=@#$%\*+\*&!~

FastDFS生成token策略为：token = md5（文件名，密钥，时间戳）

Consolidated storage

• Disadvantages of Massive Small Files
  
  • Metadata management is inefficient. In the disk file system, directory entries, inodes, and data are stored in different locations on the media
  • Decentralized data storage
  • A large number of random accesses to the disk reduce the efficiency (small files may be in this track and that in that track, which will cause a large number of random accesses, and a large number of small files are very unfriendly to I/O)
• Combined storage feature provided by FastDFS
  
  • Default large file 64M
  • Each file space is called a slot (256bytes <= slot <= 16MB)

    That is to say, for small files, FastDFS will store multiple small files into one large file. By default, a large file with a size of 64M is built, and then divided into multiple slots. The smallest slot is 256bytes, so if If a file is smaller than 256bytes, then it will also occupy the size of 256bytes. Just like the traditional Chinese medicine cabinets we saw in the hospital, each drawer is divided into multiple small grids, and the grids of different sizes are selected according to the size of the medicinal material package.

    file id without merging

    File ID when merging

    The mechanism of storage merging will not be discussed in depth here, because it brings a series of new problems, such as the need to record not only the names of large files, but also the names of small files during synchronization, which suddenly becomes more troublesome; the original The free space management can be calculated directly through the operating system, but it is not possible now, because a 64M block is created, and there is free space in this block, so it is very troublesome to calculate.

Summarize

• FastDFS is the poor man's solution
• FastDFS achieves simplicity and efficiency to the extreme, saves resources very much, and is completely affordable for small and medium-sized websites