rsync remote synchronization (downlink synchronization) + inotify real-time synchronization deployment

1. Introduction to rsync

A fast incremental backup tool

rsync (Remote Sync, remote synchronization)

• An open source quick backup tool
• Remote Sync, remote synchronization
• Support local replication, or synchronize with other SSH, rsync hosts

Two, rsync synchronization source

In the remote synchronization task, the client responsible for initiating the rsync synchronization operation is called the initiator, and the server responsible for responding to the rsync synchronization operation from the client is called the synchronization source (backup source). During the synchronization process, the synchronization source is responsible for providing the original location of the file, and the initiator should have read permission to this location.
Example:
Server A synchronizes the data of Server B, and Server B is the backup source.
Conversely, Server B synchronizes the data of Server A, then Server A is the backup source.

Three, configure the rsync source

1. Basic ideas

• Establish rsyncd.conf configuration file and independent account file
• Enable rsync --daemon mode

2. Configuration file rsyncd.conf

• Need to be established manually, the syntax is similar to Samba configuration
• Authentication configuration auth users, secrets file, if not added, it will be anonymous

3. rsync account file

• Adopt the record format of "username:password", one user record per line
• Independent account data, not dependent on the account system

4. Start the rsync service

• Independently provide services through --daemon

• The service can be shut down by executing kill $(cat /var/run/rsyncd.pid)

Four, rsync command

基本格式：rsync [选项] 原始位置 目标位置

• -r: Recursive mode, including all files in the directory and subdirectories.
• -l: The symbolic link file is still copied as a symbolic link file.
• -v: Display detailed (verbose) information about the synchronization process.
• -z: Compress when transferring files (compress).
• -a: Archive mode, retain file permissions, attributes and other information, which is equivalent to the combined option "-rlptgoD".
• -p: Keep the permission mark of the file.
• -t: Keep the time stamp of the file.
• -g: Keep the group mark of the file (only for super users).
• -o: Keep the owner mark of the file (only for super users).
• -H: Keep hard-linked files.
• -A: Keep ACL attribute information.
• -D: Keep device files and other special files.
• --Delete: Delete files in the target location but not in the original location.
• --Checksum: Decide whether to skip files based on checksum (not file size, modification time).

The function of -delete is simply to delete the difference file and keep the consistency

Five, two expression methods of configuration source

Format 1:

用户名@主机地址::共享模块名
例如：
backuper@192.168.163.10::wwwroot /opt

Format 2:

rsync://用户名@主机地址/共享模块名
例如：
rsync://backuper@192.168.163.10/wwwroot /opt

Six, rsync real-time synchronization

(1) Insufficiency of regular synchronization

1. The time to perform the backup is fixed, the delay is obvious, and the real-time performance is poor.
2. When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary

(2) Advantages of real-time synchronization

1. Once the synchronization source changes, start the backup immediately
2. As long as the synchronization source does not change, the backup will not be performed

Seven, inotify introduction

Inotify is a feature of the Linux kernel that canMonitor changes in the file system and respond to notifications, Auxiliary software: inotify-tools

1. Adjust the inotify kernel parameters (optimization)

• /etc/sysctl.conf (kernel parameter configuration file)

max_queue_events    	#监控事件队列大小
max_user_instances  	#最多监控实例数
max_user_watches    	#每个实例最多监控文件数  
配置的监控数量应该大于监控目标的总文件数

2. Use the inotify-tools auxiliary tool

• inotifywait: for continuous monitoring, real-time output results
• inotifywatch: used for short-term monitoring, output the results after the task is completed

例：
inotifywait -mrq -e modify,create,attrib,move,delete 文件或目录

#---------参数解释------------
-m	持续进行监控
-r	递归监控所有子对象
-q	简化输出信息
-e	指定要监控哪些事件类型
modify	修改
create	创建
attrib  属性更改
move	移动
deletc	删除

3. Write synchronization scripts

Writing ideas:
(1) First set two variables: monitor and perform backup
(2) Use while and read to continuously obtain monitoring results
(3) Perform different operations based on the results

vim /opt/inotify_rsynx.sh
#!/bin/bash
#定义两个变量：监控文件，执行备份
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete 需要监控的目录或文件"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/密码文件 刚才监控的目录或文件 用户名@主机地址::共享模块名"
 
 #while read获取监控结果
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do 
    #如果rsync没有运行，执行rsync进行备份操作
    if [ $(pgrep rsync | wc -l) -eq 0 ] ; then
        $RSYNC_CMD
	fi
done

Eight, configure rsync downlink synchronization

Downlink synchronization: backup master server data to slave server

inotify-tools-3.14.tar.gz

Host	operating system	IP	Required installation package, software
Master	CentOS7	192.168.182.11	rsync
Slave	CentOS7	192.168.182.22	rsync / inotify-tools-3.14.tar.gz

Environment configuration：

Master（192.168.182.11 ）

1. Turn off the firewall and install the software

systemctl stop firewalld.service 
setenforce 0
#检查是否安装，一般系统已默认安装rsync
rpm -q rsync
yum -y install rsync

2. Create the /etc/rsyncd.conf configuration file

• If the anonymous method is used, just remove the "auth users" and "secrets file" configuration items in the following configuration.

vim /etc/rsyncd.conf
uid = root
gid = root
use chroot = yes
address = 192.168.163.10
port 873
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.163.0/24
[wwwroot]
path = /var/www/html
comment = Document Root of www.test.com
read only = yes
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z
auth users = backuper lisi
secrets file = /etc/rsyncd_users.db
#---------配置解释----------------------------------------------
uid = root					     
gid = root					    
use chroot = yes					#禁锢在源目录
address = 192.168.163.10			#监听地址，监听本机地址
port 873						    #监听端口 tcp/udp 873，可通过cat /etc/services | grep rsync查看
log file = /var/log/rsyncd.log		#日志文件位置
pid file = /var/run/rsyncd.pid		#存放进程 ID 的文件位置
hosts allow = 192.168.163.0/24		#允许同步的客户机网段
[wwwroot]					        #共享模块名称
path = /var/www/html				#源目录的实际路径（同步的目录）
comment = Document Root of www.test.com
read only = yes					    #是否为只读
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z  #同步时不再压缩的文件类型
auth users = kk			#授权账户，多个账号以空格分隔
secrets file = /etc/rsyncd_users.db			      #存放账户信息的数据文件

3. Create a data file for the backup account

• No need to create a system user with the same name

vim /etc/rsyncd_users.db
lisi:123456

chmod 600 /etc/rsyncd_users.db

4. Ensure that all users have read access to the source directory /var/www/html (the file directory that needs to be backed up)

yum -y install httpd

chmod +r /var/www/html
ls -ld /var/www/html

5. Start rsync

#启动 rsync 服务程序
rsync --daemon					#启动 rsync 服务，以独立监听服务的方式（守护进程）运行 

netstat -anpt | grep rsync

#关闭 rsync 服务的方法
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid

verification

Master（192.168.1682.11）

cd /var/www/html/
vim 1.html

Slave（192.168.182.22）

rsync -az --delete --password-file=/etc/server.pass kk@192.168.182.11::wwwroot /opt/haha
ls haha

It is impossible to execute manually in an enterprise, and periodic tasks are generally used

#设置周期性任务
crontab -e
0 2 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass lisi@192.168.163.10::wwwroot /opt/abc

systemctl restart crond
systemctl enable crond

Nine, rsync + inotify real-time synchronization

1、Master（192.168.182.11）

vim /etc/rsyncd.conf
read only = no

kill `cat /var/run/rsyncd.pid`
netstat -natp | grep rsync

rsync --daemon
netstat -natp | grep rsync
 
chmod 777 /var/www/html

2、Slave（192.168.182.22）

(1) Adjust the inotify kernel parameters

• max_queue_events (monitor event queue, the default value is 16384)
• max_user_instances (maximum number of monitoring instances, the default value is 128)
• max_user_watches (the maximum number of monitored files per instance, the default value is 8192)
• When the number of directories and files to be monitored is large or the changes are frequent, it is recommended to increase the values ​​of these three parameters.

cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances 
cat /proc/sys/fs/inotify/max_user_watches 

vim /etc/sysctl.conf 

fs.inotify.max_queued_events = 32768
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

sysctl -p

yum -y install gcc gcc-c++ 

#放入安装包
tar zxvf inotify-tools-3.14.tar.gz -C /opt
cd /opt	

cd /opt/inotify-tools-3.14/

./configure
make && make install

vim /opt/inotify_rsync.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /opt/abc/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /opt/abc/ [email protected]::wwwroot"

$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
        $RSYNC_CMD
	fi
done

cd /opt/
chmod +x inotify_rsync.sh
. /opt/inotify_rsync.sh &

#加入开机自动执行
chmod +x /etc/rc.d/rc.local
echo '/opt/inotify_rsync.sh' >> /etc/rc.d/rc.local

cd /opt/
chmod +x inotify.sh 
. /inotify.sh 

cd /opt/haha
touch ccc.html
rm -rf aaa.html

Master (192.168.182.11) verification

cd /var/www/html
ls

Slave（192.168.182.22）=

Master（192.168.182.11）