rsync + inotify one million files in real-time synchronization

They are mounted on two nodes rsync

yum -y install rsync

The source server 107 to install inotify real-time monitoring file changes

wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
tar zxf inotify-tools-3.14.tar.gz
cd inotify-tools-3.14
./configure --prefix=/usr/local/inotify
make -j2
make install
ln -s /usr/local/inotify/bin/inotifywait /usr/local/bin/

The source server password authentication file

 cat /etc/rsync.pwd
123456

Modify the file permissions to 600

chmod 600 /etc/rsync.pwd

The target server password authentication file, webrsync users need to create and set a password in advance

cat /etc/rsync.pwd
webrsync:123456

Modify the file permissions to 600

chmod 600 /etc/rsync.pwd

106 target server configuration file

 cat /etc/rsyncd.conf
# /etc/rsyncd: configuration file for rsync daemon mode

# See rsyncd.conf man page for more options.

# configuration example:

# uid = nobody
# gid = nobody
# use chroot = yes
# max connections = 4
# pid file = /var/run/rsyncd.pid
# exclude = lost+found/
# transfer logging = yes
# timeout = 900
# ignore nonreadable = yes
# dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2

# [ftp]
#        path = /home/ftp
#        comment = ftp export area


uid = 0
gid = 0
use chroot = no
max connections = 0            #0表示没有限制
File = /var/run/rsyncd.pid PID
lock file = /var/run/rsyncd.lock
File /var/log/rsyncd.log = log 
log the format T =% m%%% A% B F 
# = the exclude found Lost + / 
# = Yes Transfer the logging 
timeout time = 600 # 600 is preferably disposed 
Reverse Lookup NO = 
# Yes = nonreadable the ignore 
# = * dont the compress .gz .tgz * * * .Z * .Z .zip .rpm * * * .deb the .bz2 
 
[ayd01] # modules to be synchronized 
path = / Data / 
Read = to false only 
the ignore io errors # ignore some errors 
hosts the allow = 10.11.1.0/24 
auth = webrsync the Users 
Secrets File = /etc/rsync.pwd

Creating rsync.sh source server 107 Scripts

src=/data/
des=test01
rsync_passwd_file=/etc/rsync.pwd
ip1=10.11.1.106
user=webrsync
cd ${src}
/usr/local/bin/inotifywait -mrq --format  '%Xe %w%f' -e modify,create,delete,attrib,close_write,move ./ | while read file
do
        INO_EVENT=$(echo $file | awk '{print $1}')
        INO_FILE=$(echo $file | awk '{print $2}')
        echo "-------------------------------$(date)------------------------------------"
        echo $file
        if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]]         # 判断事件类型
        then
                echo 'CREATE or MODIFY or CLOSE_WRITE or MOVED_TO'
                rsync -avzcR --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
        fi
        if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]]
        then
                echo 'DELETE or MOVED_FROM'
                rsync -avzR --delete --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
        fi
        if [[ $INO_EVENT =~ 'ATTRIB' ]]
        then
                echo 'ATTRIB'
                if [ ! -d "$INO_FILE" ]
                then
                        rsync -avzcR --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
                fi
        fi
done

　Note that the synchronization of directory permissions to 755 or owner as webrsync

Reference: https://blog.csdn.net/yanzhenjingfan/article/details/88667263

rsync + inotify one million files in real-time synchronization

Guess you like