table of Contents
A detailed .MFS Distributed File System
MFS principle Distributed File System
The composition of MFS file system
MFS file system of the working process
Deploy two .MFS distributed file system
MetaLogger Server Configuration
Configuring Chunk Server1 and Chunk Server2 (same step)
Master Server start monitoring procedures, testing services
MFS Maintenance and Disaster Recovery
A detailed .MFS Distributed File System
Distributed Introduction
- Because many of the storage server can not withstand the demands of business, the distributed file system came into being, the use of many relationships (ie, multi-web servers for multiple file servers), which can cause a dramatic performance upgrade
- Distributed File System (Distributed File System) refers to the physical file system storage resource-management is not necessarily directly connected to the local node, but connected through a computer network and node. It is simply put some scattered (distributed on each computer in the LAN) shared folder, a collection to a folder (virtual shared folder). For the user, when you want to access the shared folders, just open the virtual shared folder, you can see all the links to shared files in virtual folders folder, so users do not feel these shares are distributed among the on the computer
- The benefits of distributed file system is centralized access, simplify operation, data disaster, to provide file access performance to give
MFS principle Distributed File System
- MFS (MooseFS), is a fault-tolerant network distributed file system, he stored the data dispersed across multiple physical servers, and presented to the user is a unified resource. MFS, like other types of UNIX file systems, contains a hierarchical structure, file attributes, you can create a special file (block devices, character devices, pipes, sockets), symbolic links and hard links
The composition of MFS file system
- Master server (server metadata): in the whole system is responsible for managing the file system, maintain metadata
- MetaLogger server (metadata journaling server): Master backup server change log file, the file type is changelog_ml * mfs... When Master server data is lost or corrupted, you can obtain recovery from the log file server
- Chunk server (the data storage server): the server, the real storage for storing data files, the file will save block, and copy the data between the server, the more data servers, can be used "capacity" of the greater reliability the higher, the better the performance
- Client (Client): mount MFS file system, using the MFS resources
MFS file system of the working process
MFS reading processing data
- The client makes a request to the metadata server
- Metadata server the required data storage location (Chunk Server IP address and Chunk Number) tell the client
- It is known as the client requests data chunk server
- Chunk server sends data to the client
MFS data writing process
- The client sends a write request to the metadata server
- After the metadata server interaction in Chunk Server (this interaction is carried out only when there needed to block Chunks), but only the metadata server in the new block Chunks some servers created by the metadata created successfully inform Servers server operation is successful
- Metadata server tells the client, which can write data in chunks which chunk server
- Client data is written to the specified chunk server
- The Chunk Server with other Chunk Server data synchronization, after a successful synchronization Chunk Server tells the client data is written to success
- The client tells the server writes completed
Deploy two .MFS distributed file system
lab environment
| Roles | IP addresses |
| Master Server | 192.168.43.101/24 |
| Meta Logs Server | 192.168.43.102/24 |
| Chunk Server1 | 192.168.43.103/24 |
| Chunk Server2 | 192.168.43.104/24 |
| cilent | 192.168.43.105/24 |
Master Server Configuration
- The basic configuration of the environment
##关闭安全性功能（生产环境中是永久性关闭）
systemctl stop firewalld
setenforce 0
##配置主机名
hostnamectl set-hostname master
##配置地址映射
vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.43.101 master
192.168.43.102 metalogger
192.168.43.103 chunk1
192.168.43.104 chunk2
192.168.43.105 client
-
Installation dependent packages
yum install -y gcc gcc-c++ zlib-devel- Create a user
[root@master ~]# useradd -s /sbin/nologin -M mfs-
Edit compile and install source package mfs
[root@master ~]# cd /opt
[root@master opt]# tar xzvf moosefs-3.0.100-1.tar.gz -C /opt/
[root@master opt]# cd /opt/moosefs-3.0.100/
[root@mfsmaster moosefs-3.0.100]#
./configure \
--prefix=/usr/local/mfs \
--with-default-user=mfs \
--with-default-group=mfs \
--disable-mfschunkserver \
--disable-mfsmount
[root@mfsmaster moosefs-3.0.100]# make && make install-
Copy the MFS profile
[root@master mfs]# ls
mfsexports.cfg.sample mfsmetalogger.cfg.sample
mfsmaster.cfg.sample mfstopology.cfg.sample
[root@master mfs]# cp mfsmaster.cfg.sample mfsmaster.cfg
[root@master mfs]# cp mfsexports.cfg.sample mfsexports.cfg
[root@master mfs]# cp mfsmetalogger.cfg.sample mfsmetalogger.cfg
[root@master mfs]# cp mfstopology.cfg.sample mfstopology.cfg
[root@master mfs]# ls
mfsexports.cfg mfsmaster.cfg.sample mfstopology.cfg
mfsexports.cfg.sample mfsmetalogger.cfg mfstopology.cfg.sample
mfsmaster.cfg mfsmetalogger.cfg.sample
[root@master mfs]# cd /usr/local/mfs/var/mfs
[root@master mfs]# ls
metadata.mfs.empty
[root@master mfs]# cp metadata.mfs.empty metadata.mfs
[root@master mfs]# chown mfs:mfs /usr/local/mfs/var/mfs
-
Start mfsmaster
[root@master ~]# /usr/local/mfs/sbin/mfsmaster start
open files limit has been set to: 16384
working directory: /usr/local/mfs/var/mfs
lockfile created and locked
initializing mfsmaster modules ...
exports file has been loaded
topology file has been loaded
loading metadata ...
metadata file has been loaded
no charts data file - initializing empty charts
master <-> metaloggers module: listen on *:9419
master <-> chunkservers module: listen on *:9420
main master server module: listen on *:9421
mfsmaster daemon initialized properly
[root@master ~]# netstat -natp | grep mfs
tcp 0 0 0.0.0.0:9419 0.0.0.0:* LISTEN 38654/mfsmaster
tcp 0 0 0.0.0.0:9420 0.0.0.0:* LISTEN 38654/mfsmaster
tcp 0 0 0.0.0.0:9421 0.0.0.0:* LISTEN 38654/mfsmaster
MetaLogger Server Configuration
- The basic configuration of the environment
##关闭安全性功能（生产环境中是永久性关闭）
systemctl stop firewalld
setenforce 0
##配置主机名
hostnamectl set-hostname metalogger
##配置地址映射
vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.43.101 master
192.168.43.102 metalogger
192.168.43.103 chunk1
192.168.43.104 chunk2
192.168.43.105 client- Installation dependent packages
yum install gcc gcc-c++ zlib-devel -y- Create a user
[root@metalogger ~]# useradd -s /sbin/nologin -M mfs- Edit Edit source packages installation
[root@metalogger ~]# cd /opt
[root@metalogger opt]# tar xzvf moosefs-3.0.100-1.tar.gz -C /opt/
[root@metalogger opt]# cd /opt/moosefs-3.0.100/
[root@metalogger moosefs-3.0.100]#
./configure \
--prefix=/usr/local/mfs \
--with-default-user=mfs \
--with-default-group=mfs \
--disable-mfschunkserver \
--disable-mfsmount
[root@metalogger moosefs-3.0.100]# make && make install- Metalogger copy and modify the main configuration file
[root@metalogger moosefs-3.0.100]# cd /usr/local/mfs/etc/mfs/
[root@metalogger mfs]# ls
mfsexports.cfg.sample mfsmetalogger.cfg.sample
mfsmaster.cfg.sample mfstopology.cfg.sample
[root@metalogger mfs]# cp mfsmetalogger.cfg.sample mfsmetalogger.cfg
[root@metalogger mfs]# vi /usr/local/mfs/etc/mfs/mfsmetalogger.cfg
MASTER_HOST = 192.168.43.101 //指定master地址- Open service
[root@metalogger mfs]# /usr/local/mfs/sbin/mfsmetalogger start
open files limit has been set to: 4096
working directory: /usr/local/mfs/var/mfs
lockfile created and locked
initializing mfsmetalogger modules ...
mfsmetalogger daemon initialized properly
[root@metalogger mfs]# netstat -natp | grep mfs
tcp 0 0 192.168.43.102:42268 192.168.43.101:9419 ESTABLISHED 60076/mfsmetalogger Configuring Chunk Server1 and Chunk Server2 (same step)
- The basic configuration of the environment
##关闭安全性功能（生产环境中是永久性关闭）
systemctl stop firewalld
setenforce 0
##配置主机名
hostnamectl set-hostname chunk1
##配置地址映射
vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.43.101 master
192.168.43.102 metalogger
192.168.43.103 chunk1
192.168.43.104 chunk2
192.168.43.105 client- Installation dependent packages
yum -y install gcc gcc-c++ zlib-devel- Create a user
[root@chunk1 ~]# useradd -s /sbin/nologin -M mfs- Edit Edit source packages installation
[root@chunk1 ~]# cd /opt
[root@chunk1 opt]# tar xzvf moosefs-3.0.100-1.tar.gz -C /opt/
[root@chunk1 opt]# cd /opt/moosefs-3.0.100/
[root@chunk1 moosefs-3.0.100]#
./configure \
--prefix=/usr/local/mfs \
--with-default-user=mfs \
--with-default-group=mfs \
--disable-mfsmaster \
--disable-mfsmount
[root@chunk1 moosefs-3.0.100]# make && make install- Metalogger copy and modify the main configuration file
[root@chunk1 moosefs-3.0.100]# cd /usr/local/mfs/etc/mfs/
[root@chunk1 mfs]# ll
总用量 32
-rw-r--r-- 1 root root 3491 4月 12 08:13 mfschunkserver.cfg.sample
-rw-r--r-- 1 root root 1648 4月 12 08:13 mfshdd.cfg.sample
-rw-r--r-- 1 root root 2175 4月 12 08:13 mfsmetalogger.cfg.sample
[root@chunk1 mfs]# cp mfschunkserver.cfg.sample mfschunkserver.cfg
[root@chunk1 mfs]# cp mfshdd.cfg.sample mfshdd.cfg
●修改 mfschunk 主配置文件， 指定 master。
[root@chunk1 mfs]# vi mfschunkserver.cfg
MASTER_HOST = 192.168.43.101
[root@chunk1 mfs]# vi mfshdd.cfg
/data //指定数据存放目录
[root@chunk1 mfs]# mkdir /data
[root@chunk1 mfs]# chown -R mfs:mfs /data- Open service
[root@chunk1 ~]# /usr/local/mfs/sbin/mfschunkserver start
open files limit has been set to: 16384
working directory: /usr/local/mfs/var/mfs
lockfile created and locked
setting glibc malloc arena max to 4
setting glibc malloc arena test to 4
initializing mfschunkserver modules ...
hdd space manager: path to scan: /data/
hdd space manager: start background hdd scanning (searching for available chunks)
main server module: listen on *:9422
no charts data file - initializing empty charts
mfschunkserver daemon initialized properly
[root@chunk1 ~]# netstat -natp | grep mfs
tcp 0 0 0.0.0.0:9422 0.0.0.0:* LISTEN 24112/mfschunkserve
tcp 0 0 192.168.43.103:43390 192.168.43.101:9420 ESTABLISHED 24112/mfschunkserve Configuration Client
- The basic configuration of the environment
##关闭安全性功能（生产环境中是永久性关闭）
systemctl stop firewalld
setenforce 0
##配置主机名
hostnamectl set-hostname client
##配置地址映射
vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.43.101 master
192.168.43.102 metalogger
192.168.43.103 chunk1
192.168.43.104 chunk2
192.168.43.105 client- Installation dependent packages
yum -y install gcc gcc-c++ zlib-devel- Edit compile and install the client
[root@client ~]# cd /opt
[root@client opt]# tar xzvf fuse-2.9.2.tar.gz -C /opt
[root@client opt]# cd fuse-2.9.2
[root@client fuse-2.9.2]# ./configure
[root@client fuse-2.9.2]# make && make install
##设置环境变量
[root@client fuse-2.9.2]# vi /etc/profile
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
[root@client fuse-2.9.2]# source /etc/profile- MFS client configuration
[root@client fuse-2.9.2]# cd /opt
[root@client opt]# useradd -s /sbin/nologin -M mfs
[root@client opt]# tar xzvf moosefs-3.0.100-1.tar.gz -C /opt/
[root@client opt]# cd /opt/moosefs-3.0.100/
[root@client moosefs-3.0.100]#
./configure \
--prefix=/usr/local/mfs \
--with-default-user=mfs \
--with-default-group=mfs \
--disable-mfsmaster \
--disable-mfschunkserver \
--enable-mfsmount
[root@client moosefs-3.0.100]# make && make instal- MFS file system mounts
[root@client ~]# mkdir /opt/mfs
[root@client ~]# modprobe fuse //加载fuse模块到内核中去
[root@client ~]# /usr/local/mfs/bin/mfsmount /opt/mfs -H 192.168.43.101 //挂载MFS系统
mfsmaster accepted connection with parameters: read-write,restricted_ip,admin ; root mapped to root:root
[root@client ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 xfs 10G 3.9G 6.2G 39% /
devtmpfs devtmpfs 474M 0 474M 0% /dev
tmpfs tmpfs 489M 0 489M 0% /dev/shm
tmpfs tmpfs 489M 7.1M 482M 2% /run
tmpfs tmpfs 489M 0 489M 0% /sys/fs/cgroup
/dev/sda5 xfs 4.0G 37M 4.0G 1% /home
/dev/sda1 xfs 4.0G 158M 3.9G 4% /boot
tmpfs tmpfs 98M 12K 98M 1% /run/user/42
tmpfs tmpfs 98M 0 98M 0% /run/user/0
192.168.43.101:9421 fuse.mfs 40G 8.8G 32G 22% /opt/mfsNote: If you want to uninstall the MFS system, the direct use umount / opt / mfs
- MFS client operating on
[root@client ~]# vim /etc/profile
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
export PATH=/usr/local/mfs/bin:$PATH ###此处添加
[root@client ~]# source /etc/profile
##命令 mfsgetgoal 用来设置文件被复制的份数， 生产环境 Chunk Server 节点数量应至少
大于 2， 文件副本数小于等于 Chunk Server 服务器的数量
[root@client ~]# mfsgetgoal -r /opt/mfs
/opt/mfs:
directories with goal 2 : 1
[root@client ~]# mfssetgoal -r 2 /opt/mfs/
/opt/mfs/:
inodes with goal changed: 0
inodes with goal not changed: 1
inodes with permission denied: 0
[root@client ~]# cd /opt/mfs/
[root@client mfs]# ls
[root@client mfs]# echo "123" > 1.txt
[root@client mfs]# ls
1.txt
[root@client mfs]# mfsgetgoal 1.txt
1.txt: 2
Master Server start monitoring procedures, testing services
- Open monitoring function, this feature can be turned on after the other to build a good service
[root@master ~]# /usr/local/mfs/sbin/mfscgiserv
lockfile created and locked
starting simple cgi server (host: any , port: 9425 , rootpath: /usr/local/mfs/share/mfscgi)-
Log http://192.168.43.101:9425/mfs.cgi?masterhost=mfsmaste , test monitoring program
MFS Maintenance and Disaster Recovery
- MFS cluster startup sequence is as follows
(1) Start mfsmaster process.
(2) start all mfschunkserver process.
(3) start mfsmetalogger process (if configured mfsmetalogger).
(4) In all client mount MFS file system.
- MFS cluster stopped in the following order
(1) In all client uninstall MFS file system.
(2) Stop chunkserver process with mfschunkserver stop command.
(3) Stop metalogger process with mfsmetalogger stop command.
(4) Stop master process with mfsmaster stop command
- MFS Disaster Recovery
MFS entire system, directly off only Master may fail to start, you can use commands on the Master
/ usr / local / MFS / sbin / mfsmaster -a repairIf this is a new installation of the Master, after recovering data, to change metalogger and chunkserver configuration
MASTER_HOST the IP, the client also needs to be remounted
