MooseFS 3.0 Deployment Notes (3) [Original]

This article introduces MooseFS+DRBD+KEEPALIVED to achieve high availability. The following are deployment notes.
1. DRBD (to realize synchronous writing of MFSMASTER META_DATA)

1. Configure hosts
192.168.3.127 L-AP
-3-127 192.168.3.131 L-AP-3-131

2. Download the dependency package flex and install
ftp://rpmfind. net/linux/centos/6.8/os/x86_64/Packages/flex-2.5.35-9.el6.x86_64.rpm
rpm -ivh flex-2.5.35-9.el6.x86_64.rpm
3. Download the drbd installation package, And install
1) Install the kernel package
http://oss.linbit.com/drbd/8.4/drbd-8.4.6.tar.gz
tar -xzvf drbd-8.4.6.tar.gz
cd drbd-8.4.6
cd drbd #Because
the kernel is less than 2.6.33, it is necessary to install the kernel package. The version after 2.6.33 already contains no need to install
make KDIR=/usr/src/kernels/2.6.32-220.el6.x86_64/
cp drbd.ko /lib /modules/`uname -r`/kernel/lib/
depmod
modprobe drbd
modinfo drbd

2) Install the toolkit
Download http://oss.linbit.com/drbd/drbd-utils-8.9.6.tar.gz
tar -xzvf drbd-utils-8.9.6.tar.gz
cd drbd-utils-8.9. 6
# --with-pacemaker, enable pacemaker cluster support, will create ocf format resource
files./configure --prefix=/usr/local/drbd --sysconfdir=/etc/ --localstatedir=/var/ --with- pacemaker
make && make install
(long wait)

3) Modify configuration
ln -s /usr/local/drbd/sbin/drbdadm /usr/sbin/drbdadm
ln -s /usr/local/drbd/sbin/drbdmeta /usr/sbin /drbdmeta
ln -s /usr/local/drbd/sbin/drbdsetup /usr/sbin/drbdsetup

4) DRBD main configuration file (/etc/drbd.conf)
## The global_common.conf file contains the global and common DRBD global configuration sections ; 
        ## *.res files contain configuration information for DRBD resources.
vi /etc/drbd.d/global_common.conf
global {
        usage-count no; # Whether to join DRBD official statistics
        # minor-count dialog-refresh disable-ip-verification
        # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;
}

common {
        handlers {
                # Define handler routines, there are tons of routines in /usr/lib/drbd/
                # These are EXAMPLE handlers only.
                # They may have severe implications,
                # like hard resetting the node under certain circumstances.
                # Be careful when chosing your poison .

                # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/ sysrq-trigger ; reboot -f";
                # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
        }

        startup {
# Define startup timeout, etc.
                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
        }

        options {
                # cpu-mask on-no-data-accessible
        }

        disk {
# Disk-related public settings, Such as I/O, data status
                # size on-io-error fencing disk-barrier disk-flushes
                # disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c- fill-target c-max-rate
                # c-min-rate disk-timeout
                on-io-error detach; # Configure the I/O error handling policy to detach
        }

        net {
# Set the authentication method and password information used in DRBD synchronization.
                # protocol timeout max-epoch-size max-buffers unplug-watermark
                # connect-int ping-int sndbuf-size rcvbuf-size ko-count
                # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
                # after-sb-1pri after-sb-2pri always-asbp rr-conflict
                # ping-timeout data-integrity-alg tcp-cork on-congestion
                # congestion-fill congestion-extents csums-alg verify-alg
                # use-rle
                cram-hmac-alg sha1;
                shared-secret "drbd";
        }
        syncer {
               verify-alg sha1;
# 定义网络传输速率，一般传输速度的30%
               rate 50M;
        }
}

## Create a resource configuration file (/etc/drbd.d/r0.res)
resource r0{
# The common part can be placed at the top, and each node will automatically inherit
# Use the third synchronization protocol of drbd, indicating that it has received After the writing of the remote host is confirmed, it is considered that the writing is completed
. Protocol C;
device /dev/drbd0; # DRBD logical device path
# drbd metadata storage method, inside the DRBD disk.
meta-disk internal;
# The description of each host starts with "on", followed by the host name. The configuration of this host is in the following {}.
on L-AP-3-127{
  # Here is the host of the node name 'uname -n'
  address 192.168.3.127:7788; # Set DRBD listening port
  disk /var/lib/mfs; # Node physical device
}
on L-AP-3-131{
    address 192.168.3.131:7788;
    disk / var/lib/mfs;
}
}
5) Start
/etc/rc.d/init.d/drbd start
In the step of drbdadm create-md r0
, open(/dev/mapper/vg_drbd-lv_drbd) failed: Device or resource busy. The reason is that the file system format is inconsistent and it is mounted.
Solution:
1. umount /drbd
2. dd if=/dev/zero of=/dev/mapper/vg_drbd-lv_drbd bs=1M count=100
3. Re-execute drbdadm create-md r0
4. Delete auto mount vi /etc/fstab delete the line /drbd Deploy

the exact same one on the 3.131 machine.

service drbd start Start the service

service drbd status Check the status
If Secondary/Unknown appears, close the firewall.

drbdadm primary --force r0 (to set the primary node, only execute 3.127)

service drbd status View the status (you can see the change of the primary node)

cat /proc/drbd You can see the synchronization of the primary and secondary nodes
mkfs -t ext4 /dev/drbd0 (the following Only executed on master node 3.127)
mkdir /mnt/drbd0
mount /dev/drbd0 /mnt/drbd0
chown -R mfs:mfs /mnt/drbd0/

Create a mount directory from the node
mkdir /mnt/drbd0
chown -R mfs:mfs /mnt/drbd0/


6) Modify mfsmaster configuration
vi /etc/mfs/mfsmaster.cfg
Modify DATA_PATH=/mnt/drbd0/mfs
mkdir /mnt/drbd0 /mfs
chmod -R 777 /mnt/drbd0/mfs
cp /var/lib/mfs/* /mnt/drbd0/mfs/


7) Verify synchronization (because the standby node cannot be read, it can only be viewed after the master-slave switchover .)
The original master node umount /mnt/drbd0
is set as the slave node drbdadm secondary r0
The original slave node is set as the master node drbdadm primary r0
The original slave node mount /dev/drbd0 /mnt/drbd0
Check whether the relevant directory data is consistent. (Pay attention to the execution order)

8) After verifying the synchronization, because the umount cannot be umount at first, after rebooting directly, it is found that the two machines have a split brain, and no one knows the other. At this time, fix it as follows:
Execute on the standby machine 3.131:
drbdadm disconnect r0
drbdadm secondary r0
drbdadm --discard-my-data connect r0
Execute on the host 3.127:
drbdadm connect r0
is executed on the standby machine 3.131:
service drbd stop
service drbd start
Check the status:
service drbd status
The red part can not be executed
9) After restarting, restore the master
service drbd start
drbdadm primary --force r0
mount /dev/drbd0 /mnt/drbd0
10) After restarting, restore slave
service drbd start
service drbd status

Error analysis:
command mount /dev/drbd0 /mnt/drbd0
prompts mount: you must specify the filesystem type
Scheme: drbd needs to be set up, and then set as primary to mount