rsync remote synchronization+inotify

1. Basic introduction to Rsync service

1. About rsync

• An incremental backup tool
• Remote Sync, remote synchronization
• Support local replication, or synchronize with other SSH, rsync hosts
• Official website: http://rsync.samba.org

Two, configure rsync backup source

2.1 Configure rsync source server

rsync synchronization source

• The remote server for the backup operation, also known as the backup source

2.2 Configure rsync source

The basic idea

• Establish rsyncd.conf configuration file and independent account file
• Enable rsync --daemon mode

Application example

• User backup, allowing downlink synchronization
• The operating directory is /var/www/html (install HTTP)

Configuration file rsyncd.conf

• Need to be established manually, the syntax is similar to Samba configuration
• Authentication configuration auth users, secrets file, if not added, it is anonymous

rsync account file

• Adopt the record format of "Username: Password", one user record per line
• Independent account data, not dependent on system account

Enable rsync service

• Provide services alone through --daemin

Three, the basic usage of rsync command

3.1 Use rsync backup tool

rsync format:

• rsync [options] original location target location

Common options:

• -a: archive mode, recursive and preserve object attributes
• -v: Display detailed information about the synchronization process
• -z: compress when transferring files
• -H keep hard link files
• -A: retain ACL attribute information
• -Delete: delete files in the target location but not in the original location
• --Checksum: Determine whether to skip the file according to the checksum of the object

Two representation methods of configuration source Format:
Format 1: Username@Host Address::Shared module name

[root@localhost etc]# rsync backuper@192.168.1.10::wwwroot /opt/

Format 2: rsync://username@host address/shared module name

[root@localhost etc]#  rsync -avz rsync://backuper@192.168.1.10::wwwroot /opt/

Four, rsync backup operation example

源地址：192.168.1.10

同步目录：/var/www/html

客户端地址：192.168.1.11

4.1 rsync source station configuration

1. Determine whether rsync is installed

[root@server1 ~]# rpm -qa rsync
rsync-3.0.9-18.el7.x86_64

2. Modify the configuration file

uid = nobody
gid = nobody
use chroot = yes             #禁锢家目录
address = 192.168.1.10       #提供同步服务的地址     
port 873
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.1.0/24     #允许同步的网段      

[wwwroot]
path = /var/www/html     #同步的目录
comment = www.lcx.com   #描述信息
read only = yes      #只读模式开启
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2  #这些结尾的文件同步时不再压缩
auth users = backuper      #指定来访用户
secrets file = /etc/rsyncd_users.db      #用户密码存放在secrets file中

3. Create a password file for the backuper user

[root@server1 ~]# vi /etc/rsyncd_users.db
backuper:123456

4. The password file on the server needs 600 permissions

[root@server1 ~]# chmod 600 /etc/rsyncd_users.db 

5. Start the service to view the status

[root@server1 ~]# rsync --daemon   #启动rsync守护进程
[root@server1 ~]# netstat -anpt | grep rsync
tcp        0      0 192.168.1.10:873        0.0.0.0:*               LISTEN      8108/rsync          
[root@server1 ~]# yum -y install httpd   #安装apache
[root@server1 ~]# cd /var/www/html/ #此时会创建这个目录

[root@server1 html]# vi index.html
<h1>This is web</h1>

[root@server1 html]# cd ../
[root@server1 www]# chmod 777 html/ #给/var/www/html所有权限

4.2 Client Verification

Common options:

• -r: Recursive mode, including all files in the directory and subdirectories
• -l: For symbolic link files are still copied as symbolic link files
• -v: Display detailed information about the synchronization process
• -a: Archive mode, retain file permissions, attributes and other information, which is equivalent to the combined option "-rlptgoD"
• -z: compress when transferring files
• -p: keep the permission flag of the file
• -t: keep the time stamp of the file
• -g: Keep the group mark of the file (only for super users)
• -o: Keep the owner flag of the file (only for super users)
• -H: Keep hard link files
• -A: retain ACL attribute information
• -D: Keep device files and other special files

method one:

[root@server1 ~]# yum -y install httpd
[root@server1 ~]# cd /var/www/
[root@server1 www]# chmod 777 html/
[root@server1 www]# rsync -avz backuper@192.168.1.10::wwwroot /var/www/html/
Password: 
receiving incremental file list

sent 61 bytes  received 113 bytes  20.47 bytes/sec
total size is 22  speedup is 0.13

Method Two,

[root@server1 www]# rsync -avz rsync://backuper@192.168.1.10/wwwroot /var/www/html/
Password: 
receiving incremental file list

sent 61 bytes  received 113 bytes  49.71 bytes/sec
total size is 22  speedup is 0.13

Sync files in secret-free mode:
first create a password file /etc/server.pass locally on the client

[root@server1 ~]# vi /etc/server.pass
123456
[root@server1 ~]# rsync -avz --delete --password-file=/etc/server.pass backuper@192.168.1.10::wwwroot /var/www/html

Five, rsync real-time synchronization

5.1 Why real-time synchronization

The lack of regular synchronization

• The time to perform the backup is fixed, the delay is obvious, and the real-time performance is poor
• When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary

Advantages of real-time synchronization

• Once the synchronization source changes, immediately start the backup
• As long as there is no change in the synchronization source, no backup is performed

5.2 About inotify

• Linux kernel inotify mechanism
• Available from version 2.6.13
• Can monitor changes in the file system and make notifications and responses
• Auxiliary software: inotify-tools

5.3 rsync+inotify real-time synchronization

Client adjust inotify kernel parameters

[root@server2 ~]# vim /etc/sysctl.conf 
fs.inotify.max_queued_events = 16384  ###监控事件队列大小
fs.inotify.max_user_instances = 1024  ###最多监控实例数
fs.inotify.max_user_watches = 1048576  ###每个实例最多监控文件数

[root@server2 ~]# sysctl -p #刷新生效
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

Source site modify configuration file

[root@server1 ~]# vi /etc/rsyncd.conf 
添加写的权限：
read only = no

Install inotify-tools on the client

[root@server2 ~]# yum -y install gcc gcc-c++ make #安装编译环境
[root@server2 ~]# tar zxvf inotify-tools-3.14.tar.gz 
[root@server2 ~]# cd inotify-tools-3.14/
[root@server2 inotify-tools-3.14]# ./configure 
[root@server2 inotify-tools-3.14]# make && make install
[root@server2 inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /var/www/html #开启监控

#
-m持续监控、r递归、-q简要型输出，-e操作（更新），modify（修改），create（更新），move（移动）,delete（删除），/var/www/html（监控地址）
此时监控台不可操作，可以通过远程登录多开页面进行操作
#

The second page of the client

[root@server2 html]# cd /var/www/html/
[root@server2 html]# touch abc
[root@server2 html]# rm -rf abc

Client

[root@server2 inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /var/www/html/
/var/www/html/ CREATE abc #显示刚才操作
/var/www/html/ DELETE abc #显示刚才操作

Write a script on the client to combine inotify monitoring and rsync remote synchronization

[root@server2 ~]# vi inotify.sh
#!/bin/bash
INOTIFY="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html"
RSYNC="rsync -avz --delete --password-file=/etc/server.pass /var/www/html [email protected]::wwwroot/" #本地路径在前为源与本地同步
$INOTIFY | while read DIRECTORY EVENT FILE   #逐条读取监控记录
do
        if [ $(pgrep rsync | wc -l) -le 0 ];then
            $RSYNC
        fi
done

The directory permissions on both sides of the synchronization are 777
synchronization sources

[root@server1 ~]# chmod 777 /var/www/html/ 添加权限
[root@server1 ~]# ls -lh /var/www
总用量 352K
drwxr-xr-x. 2 root root    6 10月  2 00:52 cgi-bin
drwxrwxrwx. 3 root root   47 12月 30 23:27 html

Client

[root@server1 ~]# ls -lh /var/www/
drwxr-xr-x. 2 root root  6 8月   4 2017 cgi-bin
drwxrwxrwx. 2 root root 57 12月 30 23:35 html

Run the script, create a file in the client /var/www/html directory, and check whether the source /var/www/html directory is synchronized to the
client

[root@server2 ~]# cd /var/www/html/
[root@server2 html]# echo '<h1>This is web</h1>' > index.html

Client monitoring

[root@server2 ~]# ./inotify.sh 
sending incremental file list
html/
html/index.html

sent 120 bytes  received 31 bytes  302.00 bytes/sec
total size is 22  speedup is 0.15
sending incremental file list

sent 58 bytes  received 9 bytes  134.00 bytes/sec
total size is 22  speedup is 0.33

Source station

[root@server1 ~]# cd /var/www/html/
[root@server1 html]# ls
abc  html  index.html
[root@server1 html]# cd html/
[root@server1 html]# ls
index.html
[root@server1 html]# cat index.html 
<h1>This is web</h1>