Rsync downlink synchronization, rsync+inotify real-time synchronization
Article Directory
- Rsync downlink synchronization, rsync+inotify real-time synchronization
- 1. Introduction to rsync
- Two, rsync synchronization source
- Three, configure the rsync source
- Four, inotify introduction
- Five, configure rsync downlink synchronization
-
- 1、Master（192.168.2.4）
-
- (1) Turn off the firewall and install the corresponding software
- (2) Establish /etc/rsyncd.conf configuration file
- (3) Create a data file for the backup account
- (4) Ensure that all users have read access to the source directory /var/www/html (the file directory that needs to be backed up)
- (5) Start the rsync service program
- 2、Slave（192.168.2.5）
- 3. Verification
- Six, rsync+inotify real-time synchronization
1. Introduction to rsync
rsync (Remote Sync, remote synchronization)
1) An open source fast backup tool
2) Support local replication
3) You can also mirror and synchronize the entire directory tree between different hosts (such as other SSH, rsync hosts), support incremental backups, and maintain clamping and permissions.
4) The optimized synchronization algorithm is used to perform compression before transmission, so it is very suitable for applications such as remote backup and mirroring servers.
Official website: https://rsync.samba.org/
Two, rsync synchronization source
- In the remote synchronization task, the client responsible for initiating the rsync synchronization operation is called the initiator, and the server responsible for responding to the rsync synchronization operation from the client is called the synchronization source (backup source). During the synchronization process, the synchronization source is responsible for providing the original location of the file, and the initiator should have the read permission to this location.
Example:
Server A synchronizes the data of Server B, and Server B is the backup source.
Conversely, Server B synchronizes the data of Server A, then Server A is the backup source.
Three, configure the rsync source
1. Basic ideas
-
Establish rsyncd.conf configuration file and independent rsync account file
- Configuration file rsyncd.conf
- Need to be manually configured, the syntax is similar to Samba configuration
- Authentication configuration auth users, secrets file, if not added, it will be anonymous
- rsync account file
- Record in the format of "Username: Password", one user record per line
- Independent account data, independent of system account
- Configuration file rsyncd.conf
-
Enable rsync service
- Provide services alone through --daemon: rsync --daemon
- The service can be shut down by executing kill $(cat /var/run/rsyncd.pid)
2, rsync command
#命令的用法
rsync [选项] 原始位置 目标位置
#----------常用选项--------------------------
-r：递归模式，包含目录及子目录中的所有文件。
-l：对于符号链接文件仍然复制为符号链接文件。
-v：显示同步过程的详细（verbose）信息。
-z：在传输文件时进行压缩（compress）。
-a：归档模式，保留文件的权限、属性等信息，等同于组合选项“-rlptgoD”。
-p：保留文件的权限标记。
-t：保留文件的时间标记。
-g：保留文件的属组标记（仅超级用户使用）。
-o：保留文件的属主标记（仅超级用户使用）。
-H：保留硬连接文件。
-A：保留 ACL 属性信息。
-D：保留设备文件及其他特殊文件。
--delete：删除目标位置有而原始位置没有的文件,即删除差异文件，保留一致性。
--checksum：根据校验和（而不是文件大小、修改时间）来决定是否跳过文件。
--password-file=file：从file中得到密码，用于免交互处理，file文件的权限要是600
3. Two expressions of configuration source
Download the specified resources to the local /root directory for backup.
Format 1:
用户名@主机地址::共享模块名
例如：
[email protected]::wwwroot /opt
Format 2:
rsync://用户名@主机地址/共享模块名
例如：
rsync://[email protected]/wwwroot /opt
4. Free interactive format
echo "密码" > /etc/密码文件
chmod 600 /etc/密码文件
#设置周期性任务
crontab -e
30 22 * * * /usr/bin/rsync -az --delete --password-file=/etc/密码文件 [email protected]::wwwroot /opt
systemctl restart crond
systemctl enable crond
Four, inotify introduction
Inotify is a feature of the Linux kernel that can monitor changes in the file system and respond to notifications. Auxiliary software: inotify-tools
1. Adjust the inotify kernel parameters (optimization)
- /etc/sysctl.conf (kernel parameter configuration file), need to be configured
- max_queue_events: monitor the size of the event queue
- max_user_instances: the most monitored instances
- max_user_watches: Maximum number of monitored files per instance The
configured number of monitoring should be greater than the total number of files of the monitoring target
例如：
vim /etc/sysctl.conf
max_queue_events = 16384
max_user_instances = 1024
max_user_watches = 1048576
2. Use the inotify-tools auxiliary tool
1) inotifywait: used for continuous monitoring, real-time output results
- Various events such as modify, create, move, delete, and attrib can be monitored, and the results will be output immediately when there is a change.
2) inotifywatch: used for short-term monitoring, output the results after the task is completed
- It can be used to collect file system changes and output summary changes after the operation is over.
例：
inotifywait -mrq -e modify,create,attrib,move,delete 文件或目录
#---------参数解释------------
-m 持续进行监控
-r 递归监控所有子对象
-q 简化输出信息
-e 指定要监控哪些事件类型
modify 修改
create 创建
attrib 属性更改
move 移动
deletc 删除
3. Write synchronization scripts
Writing ideas:
(1) First set two variables: monitor and perform backup
(2) Use while and read to continuously obtain monitoring results
(3) Perform different operations based on the results
vim /opt/inotify_rsynx.sh
#!/bin/bash
#定义两个变量：监控文件，执行备份
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete 需要监控的目录或文件"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/密码文件 刚才监控的目录或文件 用户名@主机地址::共享模块名"
#while read获取监控结果
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
#如果rsync没有运行，执行rsync进行备份操作
if [ $(pgrep rsync | wc -l) -eq 0 ] ; then
$RSYNC_CMD
fi
done
Five, configure rsync downlink synchronization
Downlink synchronization: Back up the master server data to the slave server
inotify-tools-3.14.tar.gz
related software package
Extraction code: k8ib
environment configuration
| Host | operating system | IP address | Installation package |
|---|---|---|---|
| Master | CentOS7 | 192.168.2.4 | rsync |
| Slave | CentOS7 | 192.168.2.5 | rsync / inotify-tools-3.14.tar.gz |
1、Master（192.168.2.4）
(1) Turn off the firewall and install the corresponding software
systemctl stop firewalld.service
setenforce 0
#检查是否安装，一般系统已默认安装rsync
rpm -q rsync
yum -y install rsync
(2) Establish /etc/rsyncd.conf configuration file
If you use the anonymous method, just remove the "auth users" and "secrets file" configuration items in the following configuration.
vim /etc/rsyncd.conf
uid = root
gid = root
use chroot = yes
address = 192.168.2.4
port 873
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.2.0/24
[wwwroot]
path = /var/www/html
comment = Document Root of www.test.com
read only = yes
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z
auth users = backuper lisi
secrets file = /etc/rsyncd_users.db
#---------配置解释----------------------------------------------
uid = root
gid = root
use chroot = yes #禁锢在源目录
address = 192.168.2.4 #监听地址，监听本机地址
port 873 #监听端口 tcp/udp 873，可通过cat /etc/services | grep rsync查看
log file = /var/log/rsyncd.log #日志文件位置
pid file = /var/run/rsyncd.pid #存放进程 ID 的文件位置
hosts allow = 192.168.2.0/24 #允许同步的客户机网段
[wwwroot] #共享模块名称
path = /var/www/html #源目录的实际路径（同步的目录）
comment = Document Root of www.test.com
read only = yes #是否为只读
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #同步时不再压缩的文件类型
auth users = backuper lisi #授权账户，多个账号以空格分隔
secrets file = /etc/rsyncd_users.db #存放账户信息的数据文件
(3) Create a data file for the backup account
No need to create a system user with the same name
vim /etc/rsyncd_users.db
lisi:123abc
chmod 600 /etc/rsyncd_users.db
(4) Ensure that all users have read access to the source directory /var/www/html (the file directory that needs to be backed up)
yum -y install httpd
chmod +r /var/www/html
ls -ld /var/www/html
(5) Start the rsync service program
Start the rsync service and run as an independent monitoring service (daemon)
rsync --daemon
netstat -natp | grep rsync
Turn off the rsync service
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid
2、Slave（192.168.2.5）
systemctl stop firewalld.service
setenforce 0
yum -y install rsync
cd /opt
mkdir abc
chmod 777 abc
vim /etc/server.pass
123abc
chmod 600 /etc/server.pass
3. Verification
Master（192.168.2.4）
cd /var/www/html/
vim 1.html
ls
Slave（192.168.2.5）
rsync -az --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/abc
ls abc
Note: It is impossible to execute manually in an enterprise, and periodic tasks are generally used
#设置周期性任务
crontab -e
0 2 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/abc
systemctl restart crond
systemctl enable crond
Six, rsync+inotify real-time synchronization
1、Master（192.168.2.4）
Modify the rsync source server configuration file
vim /etc/rsyncd.conf
uid = root #注意这里属主和属组都要改为root，防止后面报错
gid = root
#关闭只读，上行同步需要可以写
read only = no
#重启服务
kill `cat /var/run/rsyncd.pid`
netstat -natp | grep rsync
rsync --daemon
netstat -natp | grep rsync
chmod 777 /var/www/html
Note: Here uid and gid must be changed to root, otherwise an error will be reported later
2、Slave（192.168.2.5）
(1) Adjust the inotify kernel parameters
max_queue_events (monitoring event queue, the default value is 16384)
max_user_instances (the maximum number of monitored instances, the default value is 128)
max_user_watches (the maximum number of monitored files per instance, the default value is 8192)
When the number of directories or files to be monitored is large or changes When it is more frequent, it is recommended to increase the value of these three parameters.
cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches
vim /etc/sysctl.conf
fs.inotify.max_queued_events = 17000
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
sysctl -p
(2) Install inotify-tools
yum -y install gcc gcc-c++ make
#放入安装包
tar zxvf inotify-tools-3.14.tar.gz -C /opt
cd /opt/inotify-tools-3.14/
./configure && make && make install
(3) Write trigger synchronization script
vim /opt/inotify_rsync.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /opt/abc/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /opt/abc/ [email protected]::wwwroot"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
if [ $(pgrep rsync | wc -l) -le 0 ] ; then
$RSYNC_CMD
fi
done
cd /opt/
chmod +x inotify_rsync.sh
. /opt/inotify_rsync.sh &
#加入开机自动执行
chmod +x /etc/rc.d/rc.local
echo '/opt/inotify_rsync.sh' >> /etc/rc.d/rc.local
3. Verification
Slave（192.168.2.5）
cd /opt/abc
touch test.html
ls
rm -rf test.html
ls
Master（192.168.2.4）
cd /var/www/html
ls
