Article Directory
[go-fastdfs] go-fastdfs distributed file system structures
billion, the origin of
a scratch: FastDFS Introduction
Second, the protagonist played: go-fastdfs (go-language version of FastDFS)
1. Introduction
2. Installation
3. Configuration
4 start
5.
Third, the data landscaping: go-fastdfs-web monitoring tools installed
0. introduction
1. installation
2. configuration
3. Some screenshots
Fourth, written in the last: summary
V. Links: other related articles
[go-fastdfs ] go-fastdfs distributed file system structures
billion, the origin of
some time ago to write a crawling reptile prepare a lot of pictures, there is no cloud server, there is no image server, and then create a virtual machine, opened a picture of HTTP receive services, with the very happy, crawling stand-alone show had a good time, so, after a long time, one day clean up cyberspace, accidentally, the virtual machine did not, ha ha. Let me messy for a while -
lessons learned, then install Distributed File System Image backup data set can customize it. . .
[Selection]
Option 1: B station BFS open source distributed file system access is very suitable for small files, but it is more complex to deploy.
Option 2: C original FastDFS, Alternative
Option 3: Other Distributed File System (GFS, HDFS, Lustre, Ceph , GridFS, mogileFS, TFS , etc.);
Option Four: stumbled FastDFS go to achieve, it is him ...
First, start from scratch: FastDFS Profile
FastDFS is an open source, high-performance distributed file system (DFS). Its main features include: file storage, file synchronization and file access, as well as high-capacity and load balancing. Mainly to solve the problem of mass data storage, especially for small and medium file (recommended range: 4KB <file_size <500MB) as the carrier online services.
Well looked quite right, yes ...
A more detailed description see here;
Project address: https: //github.com/happyfish100/fastdfs
Second, the protagonist played: go-fastdfs (go-language version of FastDFS)
Project address: https: //github.com/sjqzhang/go-fastdfs
compiled version of Download: https: //github.com/sjqzhang/go-fastdfs/ releases
1. Introduction
Chinese Wiki Address: https: //github.com/sjqzhang/go-fastdfs/blob/master/README-en.md
go-fastdfs http protocol is based on a distributed file system, which is based on Road to Jane's design philosophy, everything from simple design, operation and maintenance and makes it easier expansion, it has a high performance, reliable, no center, maintenance-free advantages.
Support curl command uploads
a browser that supports uploading
support HTTP download
support for multi-machine automatic synchronization
support breakpoint download
support for small files automatically merge (inode reduce occupancy
support the configuration automatically generate
support for self-monitoring alarms
support clustered file View
using common HTTP protocol
without special client (support wget, curl and other tools)
based fastdfs
high performance (as used leveldb library kv)
high reliability (extremely simple design, the use of proven components)
without center design (all nodes can simultaneously read and write)
the advantages
No dependence (single file)
automatic synchronization
failure automatic repair
by talented directory easy to maintain
different scenes support
files automatically de-duplication
support directory custom
support to retain the original filename
support automatically generate a unique file name
support browser to upload
supports viewing cluster file information
to support cluster monitoring e-mail alerts
to support token download token = md5 (file_md5 + timestamp)
operation and maintenance simple, there is only one character (unlike fastdfs have three roles Tracker Server, Storage Server, Client) , the configuration is automatically generated
for each node peer (simplified operation and maintenance)
all nodes can read and write simultaneously
deploy FIG.
2. Install
-
First, download the compiled version of the official website
-
Modify permissions
root# chmod +x fileserver- 1
-
Generate a configuration file
// 运行一下，然后杀死程序即可生成各种需要的配置文件，还是很人性化的 root# ./fileserver
3. Configure the
directory structure after // running
root # tree -L 2
// directory structure after running
the root -L Tree # 2
.
├── the conf
│ └── cfg.json
├── Data
│ ├── 20,190,517
│ ├── 20,190,518
│ ├── 20,190,519
│ ├── 20.19052 million
├── 20,190,521 │
│ ├── 20,190,522
│ ├── 20,190,523
│ ├── fileserver.db
│ ├── log.db
│ └── stat.json
├── Files
│ ├── _big
│ └── default
fileserver ├──
├── log
│ ├── access.log
│ ├── fileserver.log
│ └── tusd.log
└── static
└── uppy.html
16 directories, 7 files
Cfg.json need to modify files in the conf directory;
Here we add two machines, two machines need to add the IP address and port number to peers inside the group, so as to achieve mutual backup;
The rest need to modify according to their own, with a default configuration here;
Of course, the second machine the same process also needs to be done;
CAT # conf root / cfg.json
{
"Bind end number": "Port",
"addr": ": 8080",
"PeerID": "The only cluster, use the one-character 0-9, automatic generation of default ",
" peer_id ":" 2 ",
" this host address ":" the machine http address, default automatically generated (note port must be consistent with the port addr in), will segment the network, automatically generated not ask for the network modify, the same below ",
" Host ":" http://172.16.13.103:8080 ",
" cluster ":" cluster list, pay attention to high availability, IP must not be the same, the same does not automatically backed up, and It can not 127.0.0.1, and must be the network IP, automatically generated default ",
" peers ": [" http://172.16.13.103:8080","http://172.16.13.104:8080 "],
" group no ":" is used to distinguish between different clusters (upload or download) and support_group_upload used in conjunction with the download path ",
" Group ":" group1 ",
" a merger of small files ":" default is not consolidation,Merger situation can be resolved inode is not enough (less than 1M for the current file) to merge ",
" enable_merge_small_file ": false,
" retry failed files synchronized time ":" in seconds ",
" refresh_interval ": 1800,
"Automatically rename": "Default is not automatically renamed with the original file name",
"rename_file": false,
"whether to support web upload, easy debugging": "tacit support web upload",
"enable_web_upload": to true,
"whether support non-date path ":" support non-default path to date, that supports custom path, specify the path upload files ",
" enable_custom_path ": to true,
" download domain ":" domain name for the external network to download files, not contains HTTP: // ",
" download_domain ":" ",
" scene list ":" when set, the user must refer to a scene item in the list, not the default limit (Note: If you want to open the scene recognition function, format as follows: 'scene name: googleauth_secret' as default: N7IET373HB2C5M6D ",
" scenes ": [],
" default scene ":" default default ",
" default_scene ":" default ",
" display of directory ":" default display, easy debugging,Turn off in the on-line ",
" show_dir ": to true,
" Configuration message ":" ",
" mail ": {
" User ":" [email protected] ",
" password ":" ABC ",
"Host": "smtp.163.com:25"
},
"warning received mail list": "Receive the number of groups",
"alram_receivers": [],
"warning receiver URL": "Methods post, parameters: subjet, message ",
" alarm_url ":" ",
" whether it is necessary to download with a token ":" true and false ",
" download_use_token ": false,
" download token expiration time ":" in seconds ",
" download_token_expire ": 600,
" whether to automatically fix ":" appears in more than 100 million documents a performance problem, deselect this option, manually by day synchronization, please see the FAQ ",
" auto_repair ": to true,
" file de-duplication algorithms md5 possible conflict, the default md5 ":" sha1 | MD5 ",
" file_sum_arithmetic ":" MD5 ",
" support by group (cluster) management, the main purpose is Nginx supports multi-cluster ":"Not supported by default, when the path is not supported http://10.1.5.4:8080/action, when the support is http://10.1.5.4:8080/group (group parameter configurations) / action, action for the action name such as status, delete, sync, etc. ",
" support_group_manage ": false,
"Management ip list": "ip whitelist for managing the collection,",
"admin_ips": [ "127.0.0.1"],
"whether migration is enabled": "not enabled by default",
"enable_migrate": false,
"files whether to go heavy ":" default go heavy ",
" enable_distinct_file ": to true,
" whether to open cross-site visit ":" enabled by default ",
" enable_cross_origin ": to true,
" whether to open Google certification, and secure uploading and downloading ": "default is not open",
"enable_google_auth": false,
"certified url": "take effect when the url is not empty, pay attention: http parameters used in ordinary uploaded auth_token as an authentication parameter in HTTP through the HTTP header Upload- Metadata in auth_token as an authentication parameter, the certification process with reference to the authentication architecture diagram ",
" auth_url ":" ",
" Download whether certification ":" By default no authentication (Note that this option is to take effect in the case of auth_url not empty), "
"enable_download_auth": false,
"whether the default download": "the default download",
"default_download": to true,
"whether the machine read-only": "The default read-write",
"READ_ONLY": false,
"Whether to open HTTP": "enabled by default",
"enable_tus": to true
}
4. Start
// here to run directly, of course, you can also write your own management script
root & # ./fileserver
// check the operation status
root # PS -ef | grep fileserver | grep -v grep
Ubuntu 38193 37927 0 May23 PTS / 0 00:01:00 ./fileserver
5. Use
At this point you can normally use, open the browser, web upload function under test;
The default port number is: 8080, my address is: 172.16.13.103: 8080
More than that, of course, support command, web, upload the code, as well as HTTP functionality;
On the server deployment, visit: 172.16.13.103: 8080 / stat
you can see the structure of the JSON file status at the moment, as follows:
{
"data": [
{
"date": "20190517",
"fileCount": 1,
"totalSize": 32083
},
{
"date": "20190521",
"fileCount": 1,
"totalSize": 2123748
},
{
"date": "20190523",
"fileCount": 0,
"totalSize": 0
},
{
"date": "all",
"fileCount": 2,
"totalSize": 2155831
}
],
"message": "",
"Status ":" the ok " else nginx deployment, stress testing and other functions, the venue and the official documentation of it.
}
Third, the data landscaping: go-fastdfs-web monitoring tool installation
projects address: https: //github.com/perfree/go-fastdfs-web
each packaged version Download: https: //github.com/perfree/go-fastdfs -web / releases
0. Introduction
go-fastdfs is good, but to monitor the machine and the file is not friendly, I really want to have a tall console, you can view real-time status of the file system ...
accidentally found a go-fastdfs-web this project, I might be able to see directly through the WEB file system state, why be so good ...
1. Install
The official download page to download the required version (with less than jre or jre)
Directory Structure:
root# tree -L 2
.
├── config
│ ├── application-dev.properties
│ ├── application-prod.properties
│ ├── application.properties
│ └── log4j.properties
├── db
│ └── go-fastdfs.db
├── go-fastdfs-web.jar
├── goFastDfsWeb.sh
├── jre
│ ├── bin
│ ├── COPYRIGHT
│ ├── lib
│ ├── LICENSE
│ ├── plugin
│ ├── README
│ ├── THIRDPARTYLICENSEREADME-JAVAFX.txt
│ ├── THIRDPARTYLICENSEREADME.txt
│ └── Welcome.html
├── logs
│ ├── go-fastdfs-web.log
│ └── go-fastdfs-web.log.2019-05-21.log
└── README.md
7 directories, 16 files
run
The author provides management script, it is quite user-friendly ...
root# ./goFastDfsWeb.sh start
1
2
2. Configure
Since the default port number is 8088, so you can access the web,
address: 172.16.13.103: 8088
Login screen need to register an account;
the need to deploy the two servers go-fastdfs added to the file system directory monitoring, automatic pop-up interface added;
3. Some screenshots
--------------- -
Disclaimer: this article is CSDN blogger "catch the train" original article, follow the CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/tmt123421/article/details/90522244
run
The author provides management script, it is quite user-friendly ...
root# ./goFastDfsWeb.sh start
2. Configure
Since the default port number is 8088, so you can access the web,
address: 172.16.13.103: 8088
Login screen need to register an account;
the need to deploy the two servers go-fastdfs added to the file system directory monitoring, automatic pop-up interface added;
3. Some screenshots
Disclaimer: This article is CSDN blogger "catch the train" original article, follow the CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/tmt123421/article/details/90522244
