Linux system deployment FastDFS distributed file system

I. Overview

Why do we need it?

As we all know, in the microservice architecture, requests coming in from the gateway will Ribbonbe load-balanced, which may cause each request to be processed by a different server, because, in order to improve the throughput of the system, some services are clustered In this case, when a user needs to store a file, if the file is stored in the server that currently processes the request, then next time you want to obtain the file, you may not be able to get it, because you This request may be handled by another server. In order to solve the problem of storing Chinese files in a distributed system, FastDFS came into being

What is FastDFS?

This is an open source distributed file system that is responsible for storing files. Its main functions include: file storage, file synchronization, file access, etc., which solves the problem of mass storage and load balancing. Especially suitable for online services that use files as the carrier, such as photo album websites, video websites, and so on.

FastDFS is tailor-made for the Internet, taking full consideration of redundant backup, load balancing, linear expansion, etc., and focusing on high availability and high performance. Using FastDFS, you can easily build a high-performance file server cluster to upload and download files Wait for service

The structure of FastDFS :

FastDFS server has two roles**: tracker (tracker) and storage node (storage). In a Storage cluster, each Volume is also called a group**

How does FastDFS store files?

Stored procedure

TrackerMainly responsible for scheduling requests, load balancing play a role similar to the micro registry service (with a heartbeat mechanism, etc.), it has a storage information for each point, receipt of client to store files When the request is made, a load balancing algorithm is used to select one Storageto store the file.

Why are all clusters?

As mentioned before, terms such as high availability and load balancing are all guaranteed by the clustering of trackers. When a tracker goes down, other trackers can continue to process storage requests, which ensures high availability. When deciding which storage the file should be stored in, load balancing algorithms such as random or polling are used to ensure that the data stored in each storage is more uniform

I mentioned earlier that redundant backup and linear expansion are guaranteed by groups. First of all, think about why the concept of group appears? **Because, when we store files, we don't mean that everything will be fine after storing the files in a certain storage. **What if a machine fails? **The data in there may be lost. This is a very serious situation. In order to solve this situation, FastDFS can use multiple Storages to store the same files. The purpose of this is to backup data. That is, redundant backup, which solves the problem of file loss when a certain Storage fails. These Storages that store the same files belong to the same group. There is another situation, **how to expand when the number of stored files gradually increases? **According to the structure of FastDFS, we can expand the capacity by adding groups

Two, installation

Docker installation is recommended here, because it is simple and quick, and the steps are simple, suitable for the first time to contact FastDfs and are not very familiar with Linux commands, let alone go directly to the topic.

The general process :安装Docker——拉取FastDFS镜像——使用镜像创建容器并修改配置文件——重启后即可使用

Install Docker

installation

# 1、yum 包更新到最新
yum update
# 2、安装需要的软件包， yum-util 提供yum-config-manager功能，另外两个是devicemapper驱动依赖的
yum install -y yum-utils device-mapper-persistent-data lvm2
# 3、 设置yum源，（如果提示说没有yum-config-manager这个命令：yum -y install yum-utils）
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# 4、 安装docker，出现输入的界面都按
y yum install -y docker-ce
# 5、 查看docker版本，验证是否验证成功
docker -v
# 6、出现docker的版本信息说明成功了

Configure mirror acceleration

Log in to Alibaba Cloud and select Mirror Accelerator in the left menu to get your own exclusive mirror and address

Alibaba Cloud image acquisition address: https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors,

Add your own image acceleration address at the end of the /etc/docker/daemon.json file (if there is no such file, create one)
```
{
      
      
	"registry-mirrors": ["https://你的ID.mirror.aliyuncs.com"]
}
```

Commonly used commands:

systemctl start docker    #启动
systemctl stop docker     #停止
systemctl restart docker  #重启

docker ps		#查看运行的容器
docker images	#查看下载的镜像

Pull FastDFS image

It takes a little time to download the image in this step, just wait patiently

docker pull morunchang/fastdfs

Run Tracker and Storage

Run tracker
- --Name is the container name
- --Net=host means set to host network mode, the container uses the host's ip and does not need to do port mapping
- The sh file is executed after sh. If the storage container is running, it is storage.sh
```
docker run -d --name tracker --net=host morunchang/fastdfs sh tracker.sh
```
Run storage
- In order to facilitate the success of the subsequent test, you also need to create a folder in advance to associate the files stored in the storage. We can change the files in the storage container by changing this file. You can understand these two folders as the same
```
#创建文件夹
mkdir -p /apps/storage/data
```
- -v is followed by the mapping relationship between the virtual machine file and the file in the container
- Change TRACKER_IP to the ip address of your Linux system (can be viewed through ifconfig), if it is a cloud server, change it to the public network IP
- -e is followed by the parameters of the container, GROUP_NAME is the group name, you can set it according to your needs
```
docker run -d --name storage -v /apps/storage/data:/etc/fdfs_data/ --net=host -e TRACKER_IP=192.168.220.100:22122 -e GROUP_NAME=group1 morunchang/fastdfs sh storage.sh
```

Modify the configuration of Nginx (in the storage container)

Enter the storage container
```
docker exec -it storage /bin/bash
```
Open the nginx configuration file nginx.conf
```
vi /etc/nginx/conf/nginx.conf
```
Add the following content (existing, no need to change), the purpose of this step is to ip:port/组名/M00/*map the request to the ngx_fastdfs_modulemodule and store it in the /data/fast_data/datadirectory of the current container
```
location ~ /M00 {
      
      
	root /data/fast_data/data;
    ngx_fastdfs_module;
    add_header Cache-Control no-store;
}
```
After setting, you can exit the container
```
exit
```
Container storage container
```
docker restart storage
```
In addition to the above, if you have other requirements, such as modifying other information of the configuration file, you can modify it according to the following steps
```
docker exec -it storage /bin/bash
cd /etc/fdfs
vi tracker.conf
vi storage.conf
```

Three, test

first step:

Test whether Nginx starts normally, if you did not modify the configuration file of nginx in the above steps, the default is to open port 8080

Open the browser and enter:, Linux的ip:8080if you see Welcom to Nginx, it means that there is no problem with Nginx startup

Unsuccessful solution :

If the access is denied by seconds, it means the reason is the firewall. If an error is displayed after waiting for a period of time, it may be due to other reasons. At this time, you need to enter the storage container to view the error log of nginx , which is in /etc/nginx/the log directory under the container .

Step 2:

If there is no problem in the first step, you can pass in an image to the /apps/storage/data/data/00/00 directory under Linux,

For example: named rBCY81_Rh2eAcZAgAA7o7y_7EUQ049.png(because the file name is too short to be accessible),

Or create a file by yourself

vi /apps/storage/data/data/00/00/rBCY81_Rh2eAcZAgAA7o7y_7EUQ049.txtWhatever you write

Then try to access this file with a browser, enter the URL in the following format

ip:8080/组名(默认group1)/M00/00/00/文件(例如：rBCY81_Rh2eAcZAgAA7o7y_7EUQ049.txt)

Unsuccessful solution :

If you can’t access it, you can enter the container and lock the cause of the error by viewing the configuration file after viewing the container

step:

Enter the container

docker exec -it storage /bin/bash

View storage log file information

cat /data/fast_data/logs/storaged.log

When using SpringBoot or SpringCloud's FdfsClient to upload pictures, if you do not get tracker or storage, open the firewall's 22122, 23000 ports (specific Baidu, according to the operating system). If the error reported is a timeout, you can view the following application.yml Set the timeout period of the configuration file to be longer before testing

Four, summary

For Xiaobai who is new to this, if you encounter a problem, the first thing to do is not to blindly search for a solution on Baidu, but to check the log file as soon as possible, and to know which log file to check, if you say I don’t know which directory the log file is in. It’s okay. Don’t worry. The one on the Internet is to check the log file and then lock the problem according to the error indicated by the log file, and then look for a targeted solution. Many of the problems that you usually encounter are often relatively broad, and you may search for various solutions. Because a superficial problem is often caused by many medium and deep problems, can you find it on the Internet? Solving the problems you encounter is all about luck.

The important thing is said three times:

You must look at the log file if there is a problem! ! ! You must look at the log file if there is a problem! ! ! If there is a problem, you must look at the log file

Quickly build FastDFS distributed file system