Linux (CentOS 8) real-time automatic file synchronization backup (rsync+inotify)

1. Introduction to rsync+inotify

1. Introduction to rsync

rsync (remote synchronize) is a remote data synchronization tool under Liunx/Unix, which can quickly synchronize files and directories between multiple hosts through LAN/WAN.

There are generally two ways to synchronize files between Linux, namely rsync and scp. scp is equivalent to copying and pasting, creating a new file if it does not exist, and overwriting it if it exists, while rsync compares whether the files on both sides are the same, and updates if they are not the same. Therefore, rsync and scp are very different in the presence of folders, because scp is for copying and overwriting, and rsync is better in terms of execution performance. And rsync can also save information such as folders and file permissions.

However, rsync also has certain shortcomings. When synchronizing data, all files need to be scanned and compared. If the number of files is quite large, scanning files is very time-consuming and performance-consuming. Secondly, rsync cannot monitor and synchronize data in real time, which may lead to data inconsistency in some time periods. The solution to this problem is real-time synchronization, so you need to use the combination of rsync+inotify.

2. Introduction to inotify

inotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism. The Linux kernel has added support for inotify since version 2.6.13. Through inotify, you can monitor various events such as adding, deleting, modifying, and moving in the file system. Using this kernel interface, inotify-tools can monitor various changes in files under the file system.

First check whether the system kernel supports inotify. The following three files appear to indicate that the system supports inotify by default, as shown below.

uname -r  #查询系统内核版本
ll /proc/sys/fs/inotify
-rw-r--r-- 1 root root 0 Mar 11 09:34 max_queued_events
-rw-r--r-- 1 root root 0 Mar 11 09:34 max_user_instances
-rw-r--r-- 1 root root 0 Mar 11 09:34 max_user_watches

2. Overall structure

Here, two Linux server nodes are used for demonstration to realize real-time synchronization of files between the two nodes. 192.168.100.46 is the client node , which is the node that needs to synchronize data. Deploy rsync+inotify, and 192.168.100.47 is the server node , that is, receive synchronization Data nodes only need to deploy rsync.

3. Server-side deployment (rsync)

1. Install rsync

dnf install rsync rsync-daemon -y

2. Configure rsync

vim /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = yes
max connections = 10
strict mode=yes
pid file = /var/run/rsyncd.pid
lock file=/var/run/rsync.lock
log file=/var/log/rsyncd.log
[backup]
        path = /backup129/
        comment = backup file
        ignore errrors
        read only=no
        write only=no
        hosts allow=192.168.100.46
        hosts deny=*
        list=false
        uid=root
        gid=root
        auth users=ai
        secrets file=/etc/rsync.password

2.2. Create a password file, that is, the value of the secrets file configured above /etc/rsync.password, the content format is: user:password, user is the root configured above, and password is the password.

[root@localhost ~]# echo "ai:admin@123" > /etc/rsync.password

2.3. Modify password file permissions

[root@localhost ~]# chmod 600 /etc/rsync.password 

2.4. Start the rsync service

[root@localhost ~]# systemctl restart rsyncd 
[root@localhost ~]# systemctl enable rsyncd 
Created symlink /etc/systemd/system/multi-user.target.wants/rsyncd.service → /usr/lib/systemd/system/rsyncd.service.

2.5. Firewall open port 873

[root@localhost ~]# firewall-cmd --add-port=873/tcp --permanent --zone=public 
success
[root@localhost ~]# firewall-cmd --reload 
success

2.6. Create a sync folder

mkdir /backup129/ #和配置文件中的文件名一致

4. Client deployment (rsync+inotify)

1. Install rsync

dnf install rsync rsync-daemon -y

2. Configure rsync

2.1. Only the authentication password file needs to be configured in the client node. First, create the rsync.password file under the etc folder. Only the password is required, and no user is required. The password needs to be consistent with that of the synchronization node node2.

[root@localhost ~]# vim /etc/rsync.password
[root@localhost ~]# cat /etc/rsync.password 
password

2.2. Modify password file permissions

[root@localhost /]# chmod 600 /etc/rsync.password 

2.3. Start the rsync service

[root@localhost /]# systemctl restart rsyncd 
[root@localhost /]# systemctl enable rsyncd 
Created symlink /etc/systemd/system/multi-user.target.wants/rsyncd.service → /usr/lib/systemd/system/rsyncd.service.

2.4. Firewall open port 873

[root@localhost /]# firewall-cmd --add-port=873/tcp --permanent --zone=public
success
[root@localhost /]# firewall-cmd --reload 
success

3. Manual synchronization test

3.1. Create a test folder

mkdir -p /root/data/backuptest/test

3.2. Use the following command to perform a synchronization test, some of the parameters should correspond to those in the configuration file of the synchronization node, such as the authentication module name backup, user name ai, etc.

 rsync -avH --port 873 --delete /root/data/backuptest/ [email protected]::backup --password-file=/etc/rsync.password 

The file synchronization is successful, and then deploy inotify to realize real-time file synchronization backup

4. Deploy inotify

4.1. Install Ali epel source

yum install -y https://mirrors.aliyun.com/epel/epel-release-latest-8.noarch.rpm

4.2. Install inotify-tools

[root@localhost ~]# dnf install inotify-tools -y 

4.3. Create rsync synchronization shell script

[root@localhost backuptest]# cat inotifyrsync.sh 
#!/bin/bash
hst1=192.168.100.47
src=/root/data/backuptest/
dst1=backup
user1=ai
/usr/bin/inotifywait -mrq --timefmt '%d/%m/%y %H:%M' --format '%T %w%f%e' -e close_write,delete,create,attrib $src \
        | while read files
do
                /usr/bin/rsync -vzrtopg --delete --progress --password-file=/etc/rsync.password $src $user1@$host1::$dst1 > /dev/null 2>&1
                        echo "${files} was rsynced." >> /tmp/rsync.log 2>&1
                done

host is the ip of the server, src is the directory to be monitored by the client in real time, des is the authentication module name, which needs to be consistent with the server, and user is the authentication user in the password file.

4.4. Grant script permissions

chmod 755 /root/data/backuptest/inotifyrsync.sh 

4.5. Run the script file in the background

/root/data/backuptest/inotifyrsync.sh &

4.6. Add the script to the system self-startup file

echo "/root/data/backuptest/inotifyrsync.sh &" >> /etc/rc.local

5. Real-time synchronization backup verification

5.1. Add or delete files or folders in the client node to see if the server will automatically synchronize. After verification, the automatic synchronization backup function is realized.