nfs主主模式

1.解决了nfs单点故障；

2.使用nfs+keepalived做了主备；

　　1.当nfs服务器宕机时~主备切换，依据监控keepalived主上的nfs挂了，就停掉主keepalived；依靠keepalived的nfs_check.sh监控（脚本：nfs_check.sh）

　　nfs_check.sh：

　　　　1.监控keepalived是否为主，为主：检查挂载是可用

　　　　2.检查此机器上的nfs是否正常，不正常，关闭keepalived

　　2.通过keepalived的notify_master、notify_backup、notify_fault、notify_stop的4个状态，分别对应的脚本来决定数据的同步方向

3.通过rsync+inotify实现主备间的共享目录进行同步。

VIP：192.168.109.136

master-nfs：192.168.109.137

backup-nfs：192.168.109.138

client：192.168.109.139

nfs与所有挂载机做免密安全认证，为后面判断挂载读写正常检测提供(略）

ssh-keygen

ssh-copy-id IP

一、master-nfs、backup-nfs 安装nfs服务；client安装客户端（略）：参考：https://www.cnblogs.com/NGames/p/12088954.html

二、master-nfs、backup-nfs在双nfs主机上安装keepalived

yum -y install keepalived

修改配置文件：2台一样（BACKUP），主主

vim keepalived.conf

! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
[email protected]
[email protected]
router_id nfs
}
notification_email_from [email protected]
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}

vrrp_script chk_nfs {
script "/etc/keepalived/nfs_check.sh >> /etc/keepalived/logs/check.log"
interval 3
fall 3
rise 3
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nfs
}
virtual_ipaddress {
192.168.109.136
}
notify_master /etc/keepalived/nfs_master.sh
notify_backup /etc/keepalived/nfs_backup.sh
notify_fault /etc/keepalived/nfs_fault.sh
notify_stop /etc/keepalived/nfs_stop.sh
}

mkdir -pv /etc/keepalived/logs

---------------------------------------------------------------

vim nfs_backup.sh

vip=192.168.109.136

NUM=`ip a | grep $vip | wc -l`

LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"

echo "[slave]" >> $LOGFILE

date >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

if [ $NUM -eq 1 ]

then

echo "$vip from nfs_backup.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1

nohup bash /scripts/inotifi.sh &

else

echo "无VIP from nfs_backup.sh 停止自动同步脚本" >> $LOGFILE 2>&1

kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`

kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`

vim nfs_check.sh

VIP=192.168.109.136

ip=`/sbin/ip a|grep "ens33"|grep "$VIP"`

share_dir=/apprun/nfsdata

nfs_ceshi_file=/apprun/nfsdata/a.txt

if [ "$ip" != "" ];then

for i in `cat /etc/keepalived/web_IP`;

do echo ${i};

TXT=`/usr/bin/ssh ${i} "cat $nfs_ceshi_file"`

if [ "$TXT" != "NFS server OK..." ];then

/usr/bin/ssh ${i} "umount -l $share_dir && mount -a"

done

port=`netstat -ntlp | grep 2049`

if [ "$port" == "" ];then

systemctl stop keepalived

echo "NFS server OK..." > /apprun/nfsdata/a.txt

vim nfs_fault.sh

vip=192.168.109.136

NUM=`ip a | grep $vip | wc -l`

LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"

echo "[slave]" >> $LOGFILE

date >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

if [ $NUM -eq 1 ]

then

echo "$vip from nfs_fault.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1

nohup bash /scripts/inotifi.sh &

else

echo "无VIP from nfs_fault.sh 停止自动同步脚本" >> $LOGFILE 2>&1

kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`

kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`

vim nfs_master.sh

vip=192.168.109.136

NUM=`ip a | grep $vip | wc -l`

LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"

echo "[master]" >> $LOGFILE

date >> $LOGFILE

echo "Being master...." >> $LOGFILE 2>&1

sleep 1

if [ $NUM -eq 1 ]

then

echo "$vip from nfs_master.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1

nohup bash /scripts/inotifi.sh &

else

echo "无VIP from nfs_master.sh 停止自动同步脚本" >> $LOGFILE 2>&1

kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`

kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`

vim nfs_stop.sh

vip=192.168.109.136

NUM=`ip a | grep $vip | wc -l`

LOGFILE="/etc/keepalived/logs/keepalived-nfs-state.log"

echo "[slave]" >> $LOGFILE

date >> $LOGFILE

echo "Being slave...." >> $LOGFILE 2>&1

if [ $NUM -eq 1 ]

then

echo "$vip from nfs_stop.sh 开始启用自动同步脚本" >> $LOGFILE 2>&1

nohup bash /scripts/inotifi.sh &

else

echo "无VIP from nfs_stop.sh 停止自动同步脚本" >> $LOGFILE 2>&1

kill -9 `ps aux | grep inotifi.sh | grep -v grep | awk '{print $2}'`

kill -9 `ps aux | grep inotifywait | grep -v grep | awk '{print $2}'`

web_IP 记录需要挂载的机器ip

-------------------------------------------------------------------------------------------

配置文件同步：

rsync -avz --progress -e ssh /etc/keepalived/* 192.168.109.138:/etc/keepalived/

1.此时启动keepalived会因没有同步数据脚本/scripts/inotifi.sh ，可以先配置下面的数据同步

2.要先nfs，不然检测会关闭keepalived

systemctl restart keepalived.service

systemctl enable keepalived.service

=====================================================================================================

三、创建共享目录

mkdir -pv /apprun/nfsdata

四、共享目录资源同步

1.情况一：master_nfs为（192.168.109.137）；backup_nfs为（192.168.109.138）

安装：backup_nfs为（192.168.109.138）

\\安装rsync

yum -y install rsync

vim /etc/rsyncd.conf

pidfile = /var/run/rsyncd.pid

lock file = /var/run/rsyncd.lock

log file = /var/log/rsyncd.log

[backup]

path = /apprun/nfsdata

comment = client(192.168.109.138)

uid = root

gid = root

port = 873

ignore errors

use chroot = no

read only = no

list = no

max connections = 200

timeout = 600

auth users = wangbocheng

secrets file = /etc/rsync.pass

\\创建用户认证文件

echo "wangbocheng:wang123" > /etc/rsync.pass

chmod 600 /etc/rsync.pass

\\启动

systemctl restart rsyncd

systemctl enable rsyncd

---------------------------------------------------

master_nfs为（192.168.109.137）

yum -y install rsync inotify-tools

依据系统版本选择源：

wget http://mirrors.aliyun.com/repo/epel-7.repo

yum -y install inotify-tools

echo "wang123" > /etc/rsync.pass

chmod 600 /etc/rsync.pass

手动同步命令：（手动数据同步）

rsync -avzp --port=873 --progress --delete /apprun/nfsdata/ [email protected]::backup --password-file=/etc/rsync.pass

自动同步

\\编写脚本：vim /scripts/inotifi.sh

host=192.168.109.138
src=/apprun/nfsdata/
des=backup
password=/etc/rsync.pass
user=wangbocheng
inotifywait=/usr/bin/inotifywait
$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src | while read files ;do
rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1

done

//启动脚本

nohup bash /scripts/inotifi.sh &

\\查看是否同步

在master-nfs共享目录下写入文件，查看日志，查看 backup-nfs目录同步情况

######设置脚本的开机自启动（依各人环境需求是否配置，此处无需）

######touch /etc/rc.d/rc.local

######chmod +x /etc/rc.d/rc.local

######echo "nohup /bin/bash /scripts/inotify.sh" >> /etc/rc.d/rc.local

######重启后查看

2.情况二：master_nfs为（192.168.109.138）；backup_nfs为（192.168.109.137）

安装：backup_nfs为（192.168.109.137）

\\安装rsync

yum -y install rsync

vim /etc/rsyncd.conf

pidfile = /var/run/rsyncd.pid

lock file = /var/run/rsyncd.lock

log file = /var/log/rsyncd.log

[backup]

path = /apprun/nfsdata

comment = client(M138)

uid = root

gid = root

port = 873

ignore errors

use chroot = no

read only = no

list = no

max connections = 200

timeout = 600

auth users = wangbocheng2

secrets file = /etc/rsync2.pass

\\创建用户认证文件

echo "wangbocheng2:wang123" > /etc/rsync2.pass

chmod 600 /etc/rsync2.pass

\\启动

systemctl restart rsyncd

systemctl enable rsyncd

---------------------------------------------------

master_nfs为（192.168.109.138）

yum -y install rsync inotify-tools

依据系统版本选择源：

wget http://mirrors.aliyun.com/repo/epel-7.repo

yum -y install inotify-tools

echo "wang123" > /etc/rsync2.pass

chmod 600 /etc/rsync2.pass

手动同步命令：（手动数据同步）

rsync -avzp --port=873 --progress --delete /apprun/nfsdata/ [email protected]::backup --password-file=/etc/rsync2.pass

自动同步

\\编写脚本：vim /scripts/inotifi.sh

host=192.168.109.137
src=/apprun/nfsdata/
des=backup
password=/etc/rsync2.pass
user=wangbocheng2
inotifywait=/usr/bin/inotifywait
$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src | while read files ;do
rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1

done

//启动脚本（1.2情况只能是一边启动同步脚本，不然数据混乱）

nohup bash /scripts/inotifi.sh &

\\查看是否同步

在master-nfs共享目录下写入文件，查看日志，查看 backup-nfs目录同步情况

===============================================================================

nfs服务安装（略）参考：https://www.cnblogs.com/NGames/p/12088954.html

日志：

keepalived 状态切换日志相关记录：tail -f /etc/keepalived/logs/keepalived-nfs-state.log

健康检查相关日志记录：tail -f /etc/keepalived/logs/check.log

数据源同步相关日志记录：tail -f /tmp/rsync.log

==============================================

已验证，

情况1：nfs意外关闭；状态切换正常，数据同步正常

情况2：keepalived意外关闭；状态切换正常，数据同步正常

猜你喜欢