Distributed image server FastDFS
related file download address https://download.csdn.net/download/qq_26877377/10401553
1. What is FastDFS
FastDFS is an open source distributed file system written in C language. FastDFS is tailored for the Internet, fully considering mechanisms such as redundant backup , load balancing, and linear expansion , and pays attention to indicators It is easy to build a high-performance file service with FastDFS The server cluster provides services such as file upload and download .
The FastDFS architecture includes Tracker server and Storage server . The client requests the Tracker server to upload and download files, and the Tracker server schedules and finally completes the file upload and download from the Storage server .
The role of the Tracker server is load balancing and scheduling. The Tracker server can find the Storage server to provide file upload services according to some policies when uploading files. A tracker can be referred to as a tracking server or a scheduling server.
The role of the storage server is to store files. The files uploaded by the client are finally stored on the storage server. The storage server does not implement its own file system but uses the file system of the operating system to manage files. Storage can be referred to as a storage server.
The server has two roles:
Tracker : manages the cluster, tracker can also implement the cluster. Each tracker node has equal status. Collect the status of the Storage cluster.
Storage : The actual saved files Storage is divided into multiple groups, and the files saved between each group are different. Each group can have multiple members, the content stored in the group members is the same, the status of the group members is the same, and there is no concept of master and slave.
2. File upload process
After the client uploads the file, the storage server returns the file ID to the client, and this file ID is used to access the index information of the file in the future. The file index information includes: group name, virtual disk path, data two-level directory, and file name.
Group name: The name of the storage group where the file is uploaded. After the file is uploaded successfully, the storage server returns, and the client needs to save it by itself.
Virtual Disk Path: The virtual path of the storage configuration, corresponding to the disk option store_path* . M00 if store_path0 is configured, M01 if store_path1 is configured , and so on.
Data two-level directory: A two-level directory created by the storage server under each virtual disk path to store data files.
Filename: Different from when the file was uploaded. It is generated by the storage server according to specific information, and the file name includes: source storage server IP address, file creation timestamp, file size, random number and file extension and other information.
3. File download process
4. Simple FastDFS Architecture
5 FastDFS installation
5.1 Install fastdfs dependencies
1. Unzip libfastcommon-master.zip
2. Go to the directory of libfastcommon-master
3. Execute ./make.sh
4. Execute sudo ./make.sh install
5.2 Install fastdfs
1. Unzip fastdfs-master.zip
2. Go to the fastdfs-master directory
3. Execute ./make.sh
4. Execute sudo ./make.shinstall
5.3 Configure the tracking server tracker
1. sudo cp/etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
2. Create the directory fastdfs/tracker in the /home/python/ directory
mkdir –p/home/python/fastdfs/tracker
3. Edit the /etc/fdfs/tracker.conf configuration file sudo vim /etc/fdfs/tracker.conf
Modify base_path=/home/python/fastdfs/tracker
5.4 Configuring storage server storage
1. sudo cp/etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
2. Create a directory storage in the /home/python/fastdfs/ directory
mkdir –p /home/python/fastdfs/storage
3. Edit the /etc/fdfs/storage.conf configuration file sudo vim /etc/fdfs/storage.conf
Modifications:
base_path=/home/python/fastdfs/storage
store_path0=/home/python/fastdfs/storage
tracker_server=ip address of your own ubuntu virtual machine: 22122
5.5 Start tracker and storage
sudo servicefdfs_trackerd start
sudo servicefdfs_storaged start
Description: If there is no service, you can start by specifying the configuration file by command
sudo fdfs_trackerd / etc / fdfs / tracker.conf
sudo fdfs_storaged / etc / fdfs / storage.conf
5.6 Test whether the installation is successful
1. sudo cp/etc/fdfs/client.conf.sample /etc/fdfs/client.conf
2. Edit the /etc/fdfs/client.conf configuration file sudo vim /etc/fdfs/client.conf
Modifications:
base_path=/home/python/fastdfs/tracker
tracker_server=ip address of your own ubuntu virtual machine: 22122
3. Upload file test:
fdfs_upload_file/etc/fdfs/client.conf Image file to upload
If a file id like group1/M00/00/00/rBIK6VcaP0aARXXvAAHrUgHEviQ394.jpg is returned, the file upload is successful
5.7 Install nginx and fastdfs-nginx-module
1. Unzip nginx-1.8.1.tar.gz
2. Unzip fastdfs-nginx-module-master.zip
3. Enter the nginx-1.8.1 directory
4. Execute
sudo ./configure--prefix=/usr/local/nginx/ --add-module=fastdfs-nginx-module-master The absolute path of the decompressed directory /src
sudo make
5. sudo cpfastdfs-nginx-module-master mod_fastdfs.conf /etc/fdfs/mod_fastdfs.conf under src in the decompressed directory
6. sudo vim / etc / fdfs / mod_fastdfs.conf
Modifications:
connect_timeout=10
tracker_server=ip address of your own ubuntu virtual machine: 22122
url_have_group_name=true
store_path0=/home/python/fastdfs/storage
7. sudo cp fastdfs-master/conf/http.conf /etc/fdfs/http.conf
8. sudo cp fastdfs-master/conf/mime.types/etc/fdfs/mime.types
9.sudo vim/usr/local/nginx/conf/nginx.conf
Add configuration information in the http section as follows:
server {
listen 8888;
server_name localhost;
location ~/group[0-9]/ {
ngx_fastdfs_module;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
10. start nginx
sudo/usr/local/nginx/sbin/nginx
6. Upload test using python client
1. workon django_py3
2. Go to the directory where fdfs_client-py-master.zip is located
3. pip install fdfs_client-py-master.zip
4. Install dependencies
pipinstall configparser
pipinstall mutagen
pipinstall requests
5.
>>> fromfdfs_client.client import Fdfs_client
>>> client =Fdfs_client('/etc/fdfs/client.conf')
>>> ret =client.upload_by_filename('test')
>>> ret
{'Groupname':'group1','Status':'Upload successed.', 'Remotefile_id':'group1/M00/00/00/
wKjzh0_xaR63RExnAAAaDqbNk5E1398.py','Uploadedsize':'6.0KB','Local file name':'test'
, 'Storage IP':'192.168.243.133'}
Documentation https://github.com/jeffeilly/fdfs_client-py