basic environment
-
A total of four virtual machines are installed with CentOS 7 X64 system.
-
Each virtual machine has two network cards, which are connected to two networks, NAT network 192.168.195.0, and only host network 192.168.162.0.
-
The Target host installs the iSCSI TGT service to provide shared storage services.
-
Test1 and Test2 hosts use the shared storage provided by the Target host to provide NFS services of A/P structure to the outside world, and use PaceMaker for cluster management.
-
The Develop host mounts the NFS service provided by the Test host and tests its functions.
Network Configuration
IP address
Use a static IP address and configure it as follows.
Target
eth0 : 192.168.195.131
eth1 : 192.168.162.131
Test1
eth0 : 192.168.195.132
eth1 : 192.168.162.132
Test2
eth0 : 192.168.195.133
eth1 : 192.168.162.133
Develop
eth0 : 192.168.195.136
eth1 : 192.168.162.136
The floating IP used by NFS is: 192.168.195.200.
DNS
Add the following sections under the domain name resolution files of the four machines, and ensure that they can ping each other.
$ vi /etc/hosts
192.168.195.131 target
192.168.195.132 test1
192.168.195.133 test2
192.168.195.136 develop
Configure iSCSI
install tgt service
Do the following on the Target host:
Install the package
$ yum install -y scsi-target-utils iscsi-initiator-utils
Configure the tgt service, use the /dev/sdd1 partition as the first LUN, and the password is nfsha.
$ cp -v /etc/tgt/targets.conf /etc/tgt/targets.conf.bak
$ vi /etc/tgt/targets.conf
<target iqn.2016-12.org.lr:lvm>
backing-store /dev/sdd1
initiator-address 192.168.0.0/16
incominguser nfsha nfsha
</target>
The /dev/sdd1 partition is ignored in LVM.
$ vi /etc/lvm/lvm.conf
global_filter = [ "a|/dev/sdc|", "r|.*/|" ]
restart the service
$ systemctl enable tgtd.service
$ systemctl restart tgtd.service
$ systemctl status tgtd.service
$ tgtadm -L iscsi -m target -o show
Configure initiator client
Perform the following operations on the three hosts Test1, Test2, and Develop:
Install the package.
$ yum install -y iscsi-initiator-utils
Set the client name and replace XXX with test1, test2, and develop respectively.
$ vi /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2016-12.com.lr:XXX
Set the authentication protocol and password.
$ vi /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = nfsha
node.session.auth.password = nfsha
Restart the service.
$ systemctl enable iscsid.service
$ systemctl restart iscsid.service
$ systemctl status iscsid.service
Discover and add storage.
$ iscsiadm -m discovery -t st -p 192.168.195.131
$ iscsiadm -m node -T iqn.2016-12.org.lr:lvm -p 192.168.195.131:3260 -l
$ iscsiadm -m discovery -t st -p 192.168.162.131
$ iscsiadm -m node -T iqn.2016-12.org.lr:lvm -p 192.168.162.131:3260 -l
Check out the newly added storage.
$ iscsiadm -m node –l
192.168.195.131:3260,1 iqn.2016-12.org.lr:lvm
192.168.162.131:3260,1 iqn.2016-12.org.lr:lvm
$ ll /dev/sd*
brw-rw---- 1 root disk 8, 0 11月 24 16:25 /dev/sda
brw-rw---- 1 root disk 8, 1 11月 24 16:25 /dev/sda1
brw-rw---- 1 root disk 8, 2 11月 24 16:25 /dev/sda2
brw-rw---- 1 root disk 8, 3 11月 24 16:25 /dev/sda3
brw-rw---- 1 root disk 8, 16 11月 24 16:25 /dev/sdb
brw-rw---- 1 root disk 8, 17 11月 24 16:25 /dev/sdb1
brw-rw---- 1 root disk 8, 32 11月 24 16:25 /dev/sdc
brw-rw---- 1 root disk 8, 48 11月 24 16:25 /dev/sdd
brw-rw---- 1 root disk 8, 64 11月 24 16:25 /dev/sde
brw-rw---- 1 root disk 8, 80 11月 24 16:25 /dev/sdf
The newly generated devices are sde and sdf.
add multipath
Perform the following operations on the three hosts Test1, Test2, and Develop:
Install the package and load the driver.
$ yum install -y device-mapper*
$ modprobe dm-multipath
View the WWID of the iSCSI device.
$ /lib/udev/scsi_id -g -u /dev/sde
360000000000000000e00000000010001
Configure the multipath name.
$ vi /etc/multipath.conf
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^sd[a-d][0-9]*"
}
defaults {
user_friendly_names yes
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
multipaths {
multipath {
wwid 360000000000000000e00000000010001
alias lvm
}
}
start the service
$ mkdir /run/multipathd/ -pv
$ touch /run/multipathd/multipathd.pid
$ chmod 777 /run/multipathd/multipathd.pid
$ systemctl enable multipathd.service
$ systemctl restart multipathd.service
$ systemctl status multipathd.service
$ reboot
flush multipath cache
$ multipath -F
$ multipath -v2
$ multipath -ll
lvm (360000000000000000e00000000010001) dm-2 IET ,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 4:0:0:1 sde 8:64 active ready running
`- 3:0:0:1 sdf 8:80 active ready running
Configure LVM
Allow LVM to scan for iscsi devices
$ vi /etc/lvm/lvm.conf
filter = [ "a|/dev/mapper/lvm|", "r|.*/|" ]
Create LVM partition
$ pvcreate /dev/mapper/lvm
$ vgcreate storage /dev/mapper/lvm
$ lvcreate -L 10G -n test storage
Configure LVM, set locking_type to 1, set use_lvmetad to 0, and disable the lvmetad service.
$ lvmconf --enable-halvm --services --startstopservices
Set volume_list to exclude the storage volume group we want to use to avoid automatic system activation.
$ vgs --noheadings -o vg_name
storage
$ vi /etc/lvm/lvm.conf
volume_list = [ ]
Rebuild the initramfs to avoid the automatic activation of the volume group when the kernel starts.
$ dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)
Restart the system for the configuration to take effect.
$ reboot
Configure cluster services
Initialize settings
Perform the following configurations on Test1 and Test2:
disable firewall
$ systemctl disable firewalld
$ systemctl stop firewalld
Or
$ firewall-cmd --permanent --add-service=high-availability
$ firewall-cmd --add-service=high-availability
$ firewall-cmd --reload
Disable SeLinux
$ setenforce 0
$ sed -i.bak "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
$ reboot
Install the package.
$ yum install pcs pacemaker fence-agents-all
Modify the cluster account password
$ passwd hacluster
$ cat /etc/passwd | grep hacluster
hacluster:x:189:189:cluster user:/home/hacluster:/sbin/nologin
restart the service
$ systemctl start pcsd.service
$ systemctl enable pcsd.service
$ systemctl status pcsd.service
Initialize the cluster
Perform the following configuration on any Test node:
Authenticate cluster account
$ pcs cluster auth test1 test2
Username: hacluster
Password:
test1: Authorized
test2: Authorized
Create a cluster
$ pcs cluster setup --start --name my_cluster test1 test2
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
test1: Succeeded
test2: Succeeded
Starting cluster on nodes: test1, test2...
test2: Starting Cluster...
test1: Starting Cluster...
Synchronizing pcsd certificates on nodes test1, test2...
test1: Success
test2: Success
Restaring pcsd on the nodes in order to reload the certificates...
test1: Success
test2: Success
Start the cluster manually
$ pcs cluster start
Set the cluster to start up
$ pcs cluster enable --all
View cluster status
$ pcs cluster status
test1: Success
test2: Success
$ pcs cluster enable --all
test1: Cluster Enabled
test2: Cluster Enabled
$ pcs cluster status
Cluster Status:
Last updated: Tue Dec 13 14:58:32 2016 Last change: Tue Dec 13 14:57:07 2016 by hacluster via crmd on test2
Stack: corosync
Current DC: test2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 0 resources configured
Online: [ test1 test2 ]
PCSD Status:
test1: Online
test2: Online
Configure fence devices
Install the plugin package
$ yum install -y fence-*
View available fence plugins
$ pcs stonith list
View the instructions for using the fence plugin
$ pcs stonith describe fence_kdump
fence_kdump - Fence agent for use with kdump
The fence_kdump agent is intended to be used with with kdump service.
Stonith options:
nodename: Name or IP address of node to be fenced
ipport: Port number
family: Network family
action: Fencing action
timeout: Timeout in seconds
verbose: Print verbose output
version: Print version
usage: Print usage
priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names.
pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
pcmk_host_check: How to determine which machines are controlled by the device.
pcmk_delay_max: Enable random delay for stonith actions and specify the maximum of random delay
Create fence_kdump.
$ pcs stonith create my_fence fence_kdump test1
Check fence status.
$ pcs stonith show my_fence
Resource: my_fence (class=stonith type=fence_kdump)
Operations: monitor interval=60s (my_fence-monitor-interval-60s)
Configure NFS service
Create file system resources
Create an NFS partition and format it with the ext4 filesystem.
$ lvcreate -L 50G -n nfs storage
$ mkfs.ext4 /dev/storage/nfs
Mount the NFS partition.
$ mkdir /mnt/nfs
$ mount /dev/storage/nfs /mnt/nfs
Create NFS directory and test files.
$ mkdir -pv /mnt/nfs/exports
$ mkdir -pv /mnt/nfs/exports/export1
$ mkdir -pv /mnt/nfs/exports/export2
$ touch /mnt/nfs/exports/export1/clientdatafile1
$ touch /mnt/nfs/exports/export2/clientdatafile2
Unmount the NFS partition and deactivate the LVM volume group.
$ umount /dev/storage/nfs
$ vgchange -an storage
Create an LVM resource so that the volume group can be activated.
$ pcs resource create my_lvm LVM volgrpname=storage exclusive=true --group nfsgroup
Create file system resources.
$ pcs resource create nfsshare Filesystem \
device=/dev/storage/nfs directory=/mnt/nfs fstype=ext4 --group nfsgroup
Create export resource
Create an NFS service resource.
$ pcs resource create nfs-daemon nfsserver \
nfs_shared_infodir=/mnt/nfs/nfsinfo nfs_no_notify=true --group nfsgroup
Create exportfs resource.
$ pcs resource create nfs-root exportfs clientspec=192.168.0.0/255.255.0.0 \
options=rw,sync,no_root_squash directory=/mnt/nfs/exports fsid=0 --group nfsgroup
$ pcs resource create nfs-export1 exportfs clientspec=192.168.0.0/255.255.0.0 \
options=rw,sync,no_root_squash directory=/mnt/nfs/exports/export1 fsid=1 --group nfsgroup
$ pcs resource create nfs-export2 exportfs clientspec=192.168.0.0/255.255.0.0 \
options=rw,sync,no_root_squash directory=/mnt/nfs/exports/export2 fsid=2 --group nfsgroup
Create a floating IP resource.
$ pcs resource create nfs_ip IPaddr2 ip=192.168.195.200 cidr_netmask=24 --group nfsgroup
Create nfsnotify resource.
$ pcs resource create nfs-notify nfsnotify source_host=192.168.195.200 --group nfsgroup
View current status.
$ pcs status
Cluster name: my_cluster
Last updated: Tue Dec 13 15:09:27 2016 Last change: Tue Dec 13 15:09:15 2016 by root via cibadmin on test1
Stack: corosync
Current DC: test2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 9 resources configured
Online: [ test1 test2 ]
Full list of resources:
my_fence (stonith:fence_kdump): Started test1
Resource Group: nfsgroup
my_lvm (ocf::heartbeat:LVM): Started test2
nfsshare (ocf::heartbeat:Filesystem): Started test2
nfs-daemon (ocf::heartbeat:nfsserver): Started test2
nfs-root (ocf::heartbeat:exportfs): Started test2
nfs-export1 (ocf::heartbeat:exportfs): Started test2
nfs-export2 (ocf::heartbeat:exportfs): Started test2
nfs_ip (ocf::heartbeat:IPaddr2): Started test2
nfs-notify (ocf::heartbeat:nfsnotify): Started test2
PCSD Status:
test1: Online
test2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
Validation and testing
Perform the following tests on the Develop node.
View the currently exported file system
$ showmount -e 192.168.195.200
Export list for 192.168.195.200:
/mnt/nfs/exports/export1 192.168.0.0/255.255.0.0
/mnt/nfs/exports 192.168.0.0/255.255.0.0
/mnt/nfs/exports/export2 192.168.0.0/255.255.0.0
Verify NFSv4
$ mkdir nfsshare
$ mount -o "vers=4" 192.168.195.200:export1 nfsshare
$ mount | grep nfsshare
192.168.195.200://export1 on /root/nfsshare type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.195.136,local_lock=none,addr=192.168.195.200)
$ ls nfsshare
clientdatafile1
$ umount nfsshare
Verify NFSv3
$ mkdir nfsshare
$ mount -o "vers=3" 192.168.195.200:/mnt/nfs/exports/export2 nfsshare
$ mount | grep nfsshare
192.168.195.200:/mnt/nfs/exports/export2 on /root/nfsshare type nfs (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.195.200,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.195.200)
$ ls nfsshare
clientdatafile2
$ umount nfsshare
Failover test
Mount export1 using NFSv4.
$ mkdir nfsshare
$ mount -o "vers=4" 192.168.195.200:export1 nfsshare
$ ls nfsshare
Check the running status of the cluster. Currently, all services are running on the Test2 node.
$ Cluster name: my_cluster
Last updated: Tue Dec 13 15:20:53 2016 Last change: Tue Dec 13 15:20:50 2016 by root via crm_attribute on test1
Stack: corosync
Current DC: test2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 9 resources configured
Online: [ test1 test2 ]
Full list of resources:
my_fence (stonith:fence_kdump): Started test1
Resource Group: nfsgroup
my_lvm (ocf::heartbeat:LVM): Started test2
nfsshare (ocf::heartbeat:Filesystem): Started test2
nfs-daemon (ocf::heartbeat:nfsserver): Started test2
nfs-root (ocf::heartbeat:exportfs): Stopped
nfs-export1 (ocf::heartbeat:exportfs): Stopped
nfs-export2 (ocf::heartbeat:exportfs): Stopped
nfs_ip (ocf::heartbeat:IPaddr2): Stopped
nfs-notify (ocf::heartbeat:nfsnotify): Stopped
PCSD Status:
test1: Online
test2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
Run the test loop code.
$ while :; do date +"%s %c -- $i" | tee -a ~/nfsshare/nfstest.log; sleep 1; (( ++i )); done
Suspend the Test2 host.
$ pcs cluster standby test2
Check the running status of the cluster. Currently, all services are running on the Test1 node.
$ pcs status
Cluster name: my_cluster
Last updated: Tue Dec 13 15:26:16 2016 Last change: Tue Dec 13 15:25:47 2016 by root via crm_attribute on test1
Stack: corosync
Current DC: test2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
2 nodes and 9 resources configured
Node test2: standby
Online: [ test1 ]
Full list of resources:
my_fence (stonith:fence_kdump): Started test1
Resource Group: nfsgroup
my_lvm (ocf::heartbeat:LVM): Started test1
nfsshare (ocf::heartbeat:Filesystem): Started test1
nfs-daemon (ocf::heartbeat:nfsserver): Started test1
nfs-root (ocf::heartbeat:exportfs): Started test1
nfs-export1 (ocf::heartbeat:exportfs): Started test1
nfs-export2 (ocf::heartbeat:exportfs): Started test1
nfs_ip (ocf::heartbeat:IPaddr2): Started test1
nfs-notify (ocf::heartbeat:nfsnotify): Started test1
PCSD Status:
test1: Online
test2: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
on the Test2 host.
$ dmesg -c
[ 3145.065649] nfsd: last server has exited, flushing export cache
On the Test1 host.
$ dmesg -c
[ 4423.429239] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[ 4424.169604] NFSD: starting 90-second grace period (net ffffffff81a26d80)
On the Develop host.
$ cat ~/nfsshare/nfstest.log
...
1481613943 2016年12月13日 星期二 15时25分43秒 -- 71
1481613944 2016年12月13日 星期二 15时25分44秒 -- 72
1481613945 2016年12月13日 星期二 15时25分45秒 -- 73
1481613946 2016年12月13日 星期二 15时25分46秒 -- 74
1481613947 2016年12月13日 星期二 15时25分47秒 -- 75
1481614054 2016年12月13日 星期二 15时27分34秒 -- 76
1481614055 2016年12月13日 星期二 15时27分35秒 -- 77
1481614056 2016年12月13日 星期二 15时27分36秒 -- 78
1481614057 2016年12月13日 星期二 15时27分37秒 -- 79
1481614058 2016年12月13日 星期二 15时27分38秒 -- 80
1481614059 2016年12月13日 星期二 15时27分39秒 -- 81
...
$ dmesg -c
...
[39697.057079] nfs: server 192.168.195.136 not responding, still trying
[39701.648036] nfs: server 192.168.195.136 not responding, still trying
[39738.902904] nfs: server 192.168.195.136 not responding, timed out
[39918.946359] nfs: server 192.168.195.136 not responding, timed out
[39923.889380] nfs: server 192.168.195.136 not responding, timed out
...
$ cat /proc/fs/nfsfs/servers
NV SERVER PORT USE HOSTNAME
v4 c0a8c3c8 801 1 192.168.195.200
$ cat /proc/fs/nfsfs/volumes
NV SERVER PORT DEV FSID FSC
v4 c0a8c3c8 801 0:42 0:0 no
It can be seen that before and after switching the host, the writing process on the Develop host is blocked for about 107 seconds (multiple repeated tests, all between 95 seconds and 110 seconds), and then reading and writing can be performed normally.
Perform active/standby switchover during reading and writing to check whether the file can be read and written correctly.
$ cp /Data/Software/OS/CentOS-7-x86_64-Everything-1511.iso .
$ md5sum /Data/Software/OS/CentOS-7-x86_64-Everything-1511.iso CentOS-7-x86_64-Everything-1511.iso
dba29c59117400b111633be2bf2aaf0e /Data/Software/OS/CentOS-7-x86_64-Everything-1511.iso
dba29c59117400b111633be2bf2aaf0e CentOS-7-x86_64-Everything-1511.iso
Reference documentation
Red Hat Enterprise Linux 7 High Availability Add-On Administration