rsync remote synchronization and rsync source server + inotify monitoring tool to achieve real-time synchronization

One, rsync introduction

Remote Sync, remote synchronization, it is an open source fast incremental backup tool that can mirror and synchronize the entire directory tree between different hosts.
Supports incremental backups, maintains connections and permissions, and uses optimized synchronization algorithms to perform compression before transmission, so it is very suitable for applications such as remote backups and mirror servers.
Supports local replication, or synchronizes
rsync and scp FTP with other SSH, rsync hosts, etc. The advantage of the tool backup mechanism is that rsync synchronous backup first compares the changed data in the copy, which saves resources. If 1T of data has only 1K of data changes, rsync basically only needs to synchronize 1k of data while scp is a Dumb copy, all copies.

1.1, rsync service mode

1. Synchronize by ssh

2. C/S mode, rsync has server-side daemon module and rsync client

1.2, the principle of rsync service

In the remote synchronization task, the client responsible for initiating the rsync synchronization operation is called the initiator, and the server responsible for the corresponding rsync synchronization operation from the client is called the synchronization source.
During the synchronization process, the synchronization source is responsible for providing the original location of the document, and the initiator should have read permission to this location.
As shown below:

rsync is a fast incremental backup tool that supports:
(1) local replication;
(2) synchronization with other SSH;
(3) synchronization with the rsync host.

Later, I will demonstrate how to use the rsync backup tool from these three aspects.

1.3 rsync common operations

1.4, the idea of ​​configuring rsync source

The configuration of the rsync source server is roughly divided into three steps:
(1) Establish an rsync configuration file;
(2) Create a data file for the backup account;
(3) Start the rsync service.

Two, build rsync service

Prepare two virtual machines, one as the synchronization source and one for the client to initiate synchronization.

2.1, create rsync configuration file

Synchronization source server:

[root@server-9~]# yum -y install httpd
//如果虚拟机是最小化安装，还需要rsync包安装。
[root@localhost ~]# yum -y install rsync

[root@server-9 ~]# vi /etc/rsyncd.conf               ####在同步源服务器上配置
uid = nobody
 gid = nobody
 use chroot = yes                            ####禁锢在源目录
 #address = XXXXXXXX                  ####监听地址
 port 873                                             ####监听端口号
 log file = /var/log/rsyncd.log            ####日志文件位置
 pid file = /var/run/rsyncd.pid            ####存放进程ID的文件位置
 hosts allow = 192.168.100.0/24          ####允许访问的客户机地址
[wwwroot]                                           ####共享模块名称
 path = /var/www/html                       ####源目录的实际路径
 comment = Document Root of www.51xit.top
 read only =yes                                   #####是否只读(手动同步为yes ,做自动同步需改为no)
 dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z             ####同步时不在压缩的文件类型
 auth users =backuper                               #####授权账户
 secrets file = /etc/rsyncd_users.db                                         ####存放账户信息的数据文件

For security reasons, the synchronization source should only allow read-only synchronization. In addition, the synchronization can also be anonymous, as long as the "auth users" "secrets file" configuration is commented out.

2.2. Create a data file for the backup account

//创建rsync账号文件
采用“用户名:密码”的记录格式，每行一个用户记录，独立的账号数据，不依赖于系统账号
[root@server-9~]# vi /etc/rsyncd_users.db
backuper:pwd123

//由于账号信息采用明文存放，因此需要调整文件权限，避免账号信息泄露
[root@server-9 ~]# chmod 600 /etc/rsyncd_users.db 

2.3 Restart the rsync service to make it effective

rsync --dameon 

//此处如果想要停止这个服务 ，有两个方式：
方式一：
[root@server-9 ~]# kill $(cat /var/run/rsyncd.pid)   #使用这个停止服务必须删除存放rsync服务进程的文件
[root@lserver-9 ~]# rm -rf /var/run/rsyncd.pid

方式二：直接使用“netstat -anpt | grep rsync”命令查出进程号，使用“kill 进程号”。

Three. rsync synchronization test (three methods)

3.1 Write the test file first

[root@server-9 ~]# cd /var/www/html/
[root@server-9 html]# ls
[root@server-9 html]# echo "this is one" > /var/www/html/index.html
[root@server-9 html]# echo "this is web" > /var/www/html/web.html
[root@server-9 html]# ls
index.html  web.html

3.2 Perform operation synchronization commands on other nodes

3.2.1 The first synchronization method

[root@SERVER 10 ~]# rsync -avz backuper@192.168.100.9::wwwroot /opt
Password:
receiving incremental file list
./
index.html
web.html

sent 65 bytes  received 199 bytes  40.62 bytes/sec
total size is 24  speedup is 0.09

3.2.2 The second synchronization method

[root@SERVER 10 ~]# rsync -avz rsync://backuper@192.168.100.9/wwwroot /opt
Password:
receiving incremental file list

sent 20 bytes  received 90 bytes  12.94 bytes/sec
total size is 24  speedup is 0.22

3.2.3 The third synchronization method, without interaction

[root@SERVER 10 opt]# rm -rf *.html

创建密码文件
[root@SERVER 10 opt]# vim server.pass

abc123
~
~增加权限
[root@SERVER 10 opt]# chmod 600 server.pass
[root@SERVER 10 opt]# rsync -az --delete --password-file=/opt/server.pass backuper@192.168.100.9::wwwroot /opt
查看同步情况，同步成功
[root@SERVER 10 opt]# ls
index.html  web.html

3.2.4, set up regular synchronization

[root@SERVER 10 wwwroot]# crontab -e   ####每天晚上10点半对服务器网站目录更新一次
30 22 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass backuper@192.168.100.41::wwwroot /opt/myweb/

[root@SERVER 10 wwwroot]# systemctl restart crond
[root@SERVER 10wwwroot]# systemctl enable crond

Four. rsync + inotify monitoring tool to achieve automatic synchronization

Introduction to real-time synchronization
1. Periodic synchronization has some shortcomings, such as:

The backup execution time is fixed, the delay is obvious, and the real-time performance is poor.
When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary.
2. The advantages of real-time synchronization

Once the synchronization source changes, immediately start the backup
As long as the synchronization source does not change, the backup will not be performed
3. Introduction to inotify

Starting from 2.6.13, the Linux kernel introduced the inotify mechanism.
It is a file system change notification mechanism, which
can monitor files and directories.
When monitoring a directory, it can simultaneously monitor the directory and each subdirectory and file in the directory.
It can assist rsync, monitor data changes, and trigger rsync to synchronize data.

4.1 Configure on the server

vim /etc/rsync.conf

uid = nobody
gid = nobody
port 873
log file = /var/log/rsyncd.log
# use chroot = yes
# max connections = 4
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.100.0/24
# exclude = lost+found/
# transfer logging = yes
# timeout = 900
# ignore nonreadable = yes
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2

[wwwroot]
path = /var/www/html
comment = www
read only = no                   (自动同步改NO   ，手动同步改为yes )
auth users = backuper
secrets file = /etc/rsyncd_users.db

4.2 Create password file text

vim /etc/rsync_users.db

backuper:abc123

chmod 600 /etc/rsyncd_users.db  //设置权限

4.3 Restart service

 kill 16360
netstat -ntap | grep rsync
rsync --daemon
 netstat -ntap | grep rsync

Install httpd, generate, var/www/html folder, and grant permissions

yum  install httpd -y 

 cd /var/www
chmod 777 html


ll
总用量 0
drwxr-xr-x 2 root root 6 4月   2 21:14 cgi-bin
drwxrwxrwx 2 root root 6 4月   2 21:14 html

4.4 Setting inotify-tools on the node side

4.4.1 Install monitoring software

tar -zxvf inotify-tools-3.14.tar.gz
yum install  make  gcc gcc-c++ httpd -y
cd 进入目录
./configure
make && make install

4.4.2 Setting up the kernel file

When the number of directories and files to be monitored is large or changes are frequent, it is recommended to increase the values ​​of these three parameters.
You can directly modify the configuration file of /etc/sysctl.conf to set the management queue, the number of instances, and the number of monitoring.

vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

 sysctl -p  // 立即生效

4.4.3 Create password text

vim /etc/server.pass
写入 abc123
[root@SERVER 10 /]# chmod 600 /etc/server.pass

4.4.4 Add permissions to the directory

[root@SERVER 10 /]# cd /var/www
[root@SERVER 10 www]# chmod 777 html

4.4.5 Monitoring test

[root@SERVER 10 ~]# cd /var/www/html/
[root@SERVER 10 html]# touch abc
[root@SERVER 10 html]# rm -rf abc


[root@SERVER 10 www]# inotifywait -mrq -e create,delete,move,modify,attrib,move,delete /var/www/html/
/var/www/html/ CREATE abc
/var/www/html/ ATTRIB abc
/var/www/html/ DELETE abc

4.4.6 Create script files to realize automatic monitoring synchronization

vim /opt/intoify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ [email protected]::wwwroot/"

$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
        $RSYNC_CMD
#       echo "${FILE} was rsynced" >>/opt/inotify_rsync.log

    fi
done

Increase execution authority and execute

chmod +x intoify.sh
./inotify.sh

4.4.7 Create a file on the directory side and view and test it on another server side

[root@SERVER 10 html]# echo "this is test" > test.html
[root@SERVER 10 html]# echo "this is test" > test2.html
[root@SERVER 10 html]# echo "this is test3" > test3.html
[root@SERVER 10 html]#

[root@SERVER 10 opt]# ./inotify.sh
rsync: failed to set times on "/." (in wwwroot): Operation not permitted (1)
rsync: chgrp "/.test.html.CMUx1o" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.test2.html.QsPEBC" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: failed to set times on "/." (in wwwroot): Operation not permitted (1)
rsync: chgrp "/.test.html.WlvSvG" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.test2.html.uKf5IU" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: failed to set times on "/." (in wwwroot): Operation not permitted (1)
rsync: chgrp "/.test.html.T8HIwo" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.test2.html.QPx7rz" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.test3.html.ZmVwnK" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: failed to set times on "/." (in wwwroot): Operation not permitted (1)
rsync: chgrp "/test3.html" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.test.html.VG7wIt" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.test2.html.xuuoiF" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

4.4.8 Check the server and find that the synchronization is successful

[root@server-9 html]# ls
test2.html  test3.html  test.html
[root@server-9 html]# cat test3.html
this is test3