First, the host information
| hostname | ip |
| nfs1 | 192.168.63.148 |
| nfs2 | 192.168.63.150 |
Second, the department drbd
1, drbd Introduction
1 DRBD基本功能 Distributed Replicated Block Device(DRBD)是一种基于软件的，无共享，复制的存储解决方案，在服务器之间的对块设备（硬盘，分区，逻辑卷等）进行镜像。 DRBD是将不同机器上的两块大小相同的硬盘或是两个分区让它们的每一位都对齐，从而当用户空间有数据要存储到主节点磁盘时，工作在内核空间的DRBD会监控数据，一旦发现数据是要存储到定义了DRBD的分区上后，就会把数据复制一份通过网络传送到备用节点上来。备用节点上运行一个服务时刻可以接收对方发来的数据，然后接到内核中，内核中的DRBD接收到数据后通过内核保存到磁盘上。双方式通过DRBD协议按位存储数据。 在高可用(HA)中使用DRBD功能，可以代替使用一个共享盘阵。本地(主节点)与远程主机(备节点)的数据可以保证实时同步。当本地系统出现故障时，远程主机上还会保留有一份相同的数据，可以继续使用。需要说明一点的是DRBD只支持两个节点不支持多节点。 2、DRBD部署模式 ①、主从模式 DRBD主从架构中，主节点可以读写数据而从节点不能读写，连挂载都不允许，否则会造成文件系统崩溃，但是主从的节点可以相互切换，如可以把主节点分区卸载后把主节点转为从，然后把分区挂在到从节点上，再把从转为主。 ②、双主模式 由于DRBD是在两台独立的机器上实现数据块同步，所以单凭在一个节点上写数据时施加锁机制而另外一个节点要写数据时看不到对方施加的锁，因此会照成数据损坏。但是如果把DRBD用在高可用集群中就可以实现双主模型，在高可用中把DRBD定义成主从资源基于分布式文件锁DLM加集群文件系统gfs或ocfs，这样一来当一端在写入数据时因为是DLM+GFS，所以就会通知给另外一端的DRBD从而避免数据损坏（双主模型并不是并行读写）。 3、DRBD数据同步模式 由于DRBD将数据发送给对端服务器之后还要确定对方的回应以确定数据是否安全存储，然后DRBD程序才退出，最终应用程序完成数据存储。这样一来DRBD发送数据并接收到回应的时间就是应用程序执行的过程时间。所以又涉及到了数据安全跟性能之间平衡了，DRBD提供了异步、半同步、同步等工作方式。 ①、异步复制 异步复制协议。一旦本地磁盘写入已经完成，数据包已在发送队列中，则写被认为是完成的。在一个节点发生故障时，可能发生数据丢失，因为被写入到远程节点上的数据可能仍在发送队列。尽管，在故障转移节点上的数据是一致的，但没有及时更新。这通常是用于地理上分开的节点。 ②、半同步复制 内存同步（半同步）复制协议。一旦本地磁盘写入已完成且复制数据包达到了对等节点则认为写在主节点上被认为是完成的。数据丢失可能发生在参加的两个节点同时故障的情况下，因为在传输中的数据可能不会被提交到磁盘。 ③、同步复制 同步复制协议。只有在本地和远程节点的磁盘已经确认了写操作完成，写才被认为完成。没有任何数据丢失，所以这是一个群集节点的流行模式，但I / O吞吐量依赖于网络带宽。 一般使用协议C，但选择C协议将影响流量，从而影响网络时延。为了数据可靠性，我们在生产环境使用时须慎重选项使用哪一种协议。
2, prepare two virtual machines nfs1 and nfs2 (does not describe here)
3, two virtual machines were added to a 100G hard disk (not describe here, vmware operation)
4, respectively, set the hostname
#nfs1
hostnamectl set-hostname nfs1
#nfs2 hostnamectl set-hostname nfs2
5, both hosts turn off the firewall and selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config systemctl stop firewalld systemctl disable firewalld setenforce 0
6, see the new disk information
[root @ NFS1 ~] # fdisk the -l | grep sdb Disk / dev / sdb: 107.4 GB, 107 374 182 400 bytes, 209715200 sector [root @ NFS2 ~] # fdisk the -l | grep sdb Disk / dev / sdb: 107.4 GB, 107 374 182 400 bytes, 209715200 sector
7, respectively, modify the hosts file
[root@nfs2 ~]# cat << EOF >> /etc/hosts 192.168.63.148nfs1 192.168.63.150nfs2 EOF
8, installation drbd
# Mounted reliance may be required yum the install -Y glib2 the libxml2-devel-devel bzip2 -devel devel-Flex the OpenIPMI Bison-devel-devel devel-NET-SNMP IPMItool the ipmiutil ipmiutil- devel AsciiDoc # yum source mounted RPM -Uvh HTTPS: // www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm # installation drbd yum -y install drbd90-utils kmod- drbd90 # drbd load the kernel module modprobe drbd # View drbd module lsmod | grep drbd # join startup echo " modprobe DRBD " >> / etc / rc.local chmod +x /etc/rc.d/rc.local
9, the configuration file
/ etc / drbd.conf # master configuration file /etc/drbd.d/ global_common.conf # global configuration file /etc/drbd.conf Description
10, change the configuration file
Music Videos /etc/drbd.d/global_common.conf /etc/drbd.d/ global_common.conf.bak CAT > << /etc/drbd.d/global_common.conf the EOF # The Result of IS over the DRBD A Decade of Development by . cially available from Linbit # an In Case you need Professional Services for DRBD or have have # the Feature Requests from Visit HTTP: // www.linbit.com , Ltd. Free Join { Usage - COUNT NO; # whether to participate in DRBD usage statistics, the default is yes. Drbd installed capacity of official statistics, to NO # Decide session What kind of the udev symlinks you want for " Implicit " Volumes # (the without Explicit Those Volume <VNR> {} Block, tIMPLIED VNR = 0): # /dev/drbd/by-resource/<resource>/<vnr> (explicit volumes) # /dev/drbd/by-resource/<resource> (default for implict) udev-always-use-vnr; # treat implicit the same as explicit volumes # minor-count dialog-refresh disable-ip-verification # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600; } common { protocol C; #使用DRBD的同步协议，添加这一行 handlers { # These are EXAMPLE handlers only. # They may have severe implications, # like hard resetting the node under certain circumstances. # Be careful when choosing your poison. pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;stop -f"; # fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; # split-brain "/usr/lib/drbd/notify-split-brain.sh root"; # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root"; # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k"; # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh; # quorum-lost "/usr/lib/drbd/notify-quorum-lost.sh root"; } startup { # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb } options { # cpu-mask on-no-data-accessible # RECOMMENDED for three or more storage nodes with DRBD 9: # quorum majority; # on-no-quorum suspend-io | io-error; } disk { on-io-error detach; #配置I/O错误处理策略为分离 #size on-io-error fencing disk-barrier disk-flushes # disk-drain md-flushes resync-rate resync-after al-extents # c-plan-ahead c-delay-target c-fill-target c-max-rate # c-min-rate disk-timeout } net { # protocol timeout max-epoch-size max-buffers # connect-int ping-int sndbuf-size rcvbuf-size ko-count # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri # after-sb-1pri after-sb-2pri always-asbp rr-conflict # pingData Integrity-ALG--timeout TCP-ON- Cork Congestion # Congestion -fill Congestion extents csums-ALG-Verify- ALG # use - RLE } the syncer { Rate 1024M; # set the rate at which the network node synchronization standby } } the EOF Notes : oN -io- possible error policy for one of the following separation detach: this is the default and recommended option if the underlying disk I happened on the node / O error, it will run the equipment in diskless diskless mode pass_on: DRBD will be the I / O error reporting to the top, on the primary node, it will report to the mounted file system, but it is often overlooked on this node (upper So on this node can not report) -Local- in -error: call the local disk I / O command processing program defined; it requires a resource handler corresponding local-io-error process called at the wrong command; which gives administrators the freedom enough power to order or command script calls local-io-error processing the I / O error The definition of a resource
11, create a resource file
CAT > << /etc/drbd.d/data.res the EOF Resource Resource Name Data # { protocol C; # protocol using Meta - Disk Internal; Device / dev / drbd1; #DRBD device name the syncer { Verify - ALG SHA1; # encryption algorithm } NET { the allow -two- primaries is; } ON NFS1 { Disk / dev / SDB; address 192.168 . 63.148 : 7789 ; # DRBD listen address and port disposed } ON NFS2 { Disk / dev / SDB; address 192.168.63.150:7789; } } EOF
12, enabled drbd
drbdadm create-md data drbdadm up data 报错：Command 'drbdmeta 1 v08 /dev/vdb internal apply-al' terminated with exit code 255 解决：dd if=/dev/zero of=/dev/sdb bs=1M count=100
13, the configuration nfs1
drbdadm primary data --force
14, create a file system
File system can only be mounted on the primary (Primary) node, it can be formatted for the operating system formatted file DRBD device set the master node after Primary: the main current node; in front of the current node Secondary: standby node for the time to view the current node role [root @ nfs1 ~ ] # drbdadm role the Data Primary [root @ NFS2 ~ ] # drbdadm role the Data secondary can be seen on the Primary nfs1
Format / dev / drbd1
[NFS1 the root @ ~] # mkfs.ext4 / dev / drbd1 the mke2fs 1.42 . . 9 ( 28 -Dec- , 2013 ) file system label = the OS type: the Linux block size = 4096 (log = 2 ) block size = 4096 (= log 2 ) Stride = 0 blocks, Stripe width = 0 blocks 6.5536 million of inodes, 26,213,591 blocks 1,310,679 blocks ( 5.00 %) Reserved for The Super User first data block = 0 Maximum filesystem blocks=2174746624 800 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: completed is being written inode table: complete Creating TECHNOLOGY ( 32768 Blocks): complete Writing superblocks and filesystem accounting information: complete
Mount / dev / drbd1 and view status
mkdir /data mount /dev/drbd1 /data #查看drbd状态 [root@nfs1 ~]# drbdadm status data role:Primary disk:UpToDate nfs2 role:Secondary replication:SyncSource peer-disk:Inconsistent done:55.27
15. Switch the test if the synchronization was successful
# Nfs1 file created on the downgrade and to switch to the standby node Touch / Data / testfile. { . 1 .. . 4 } LL / Data umount / Data drbdadm Secondary Data # upgrade operation on NFS2, switch to the master node drbdadm primary Data drbdadm Role Data [NFS2 the root @ ~ ] # drbdadm Status Data Role: Primary Disk: an Inconsistent NFS1 Role: Secondary Replication: Use the peer SyncTarget -disk: to UpToDate DONE : 91.24
[root@nfs2 ~]# mkdir /data
[root@nfs2 ~]# mount /dev/drbd1 /data
[root@nfs2 ~]# ll /data/
总用量 16
drwx------ 2 root root 16384 1月 2 17:34 lost+found
-rw-r--r-- 1 root root 0 1月 2 17:41 testfile.1
-rw-r--r-- 1 root root 0 1月 2 17:41 testfile.2
-rw-r--r-- 1 root root 0 1月 2 17:41 testfile.3
-rw-r--r-- 1 root root 0 1月 2 17:41 testfile.4
To be continued. . .