Distributed application - rsync remote synchronization
1. The background and principle of rsync
rsync (Remote Sync, remote synchronization) is an open source software developed by Andrew Tridgell in 1996. It is an open source fast backup tool that can mirror and synchronize the entire directory tree between different hosts, supports incremental backup, and maintains links and permissions, and uses an optimized synchronization algorithm to perform compression before transmission, so it is very suitable for remote backup, applications such as mirror servers.
In a remote synchronization task, the client responsible for initiating the rsync synchronization operation is called the initiator, and the server responsible for responding to the rsync synchronization operation from the client is called the synchronization source. During the synchronization process, the synchronization source is responsible for providing the original location of the file, and the initiator should have read permission for the location.
1. The function of rsync
| Function | illustrate |
|---|---|
| incremental backup | Sync only transfers the changed file blocks, not the entire file, which greatly saves transmission time and bandwidth. |
| file synchronization | rsync can ensure that the source and target folders are completely consistent, whether it is local synchronization or remote synchronization, it can efficiently and accurately maintain file consistency. |
| Flexible filtering mechanism | rsync supports a variety of filtering rules, which can exclude or include specified files based on file name, size, time and other conditions. |
| Support compressed transmission | rsync supports compression algorithms such as gzip, which can compress files during transmission and reduce the amount of transmitted data. |
| safety | rsync can use the SSH protocol for transmission, and ensures data security by encrypting data transmission. |
| http | When a network interruption or abnormality occurs during rsync transmission, the transmission can be resumed, reducing unnecessary retransmission time. |
2. Application scenarios of rsync
file backup: rsync can complete file backup quickly and efficiently, and save storage space and transmission time through the incremental backup function.
data synchronization: rsync can ensure that files between different servers or devices are kept in sync, so that data consistency between multiple nodes can be guaranteed.
network transmission: rsync is very suitable for transferring large files or a large number of small files across the network through the characteristics of compression and incremental transmission.
3. Basic commands using rsync
| scenes to be used | grammatical format |
|---|---|
| Copy local files to target machine | rsync [options] origin location destination location |
| Copy the local directory to the target machine | rsync -r [options] origin location destination location |
| Copy files from remote server | rsync [options] [username@] HOST: origin location destination location |
| Copy directory from remote server | rsync -r [options] [username@] HOST: origin location destination location |
4. The difference between scp and rsync
| scp | rsync | |
|---|---|---|
| transmission efficiency | scp is a file transfer tool based on the SSH protocol, which uses encryption to transfer data. Although it also supports compressed transmission (using option -C), it can only transfer the entire file and cannot perform incremental transfer, so it takes a long time to transfer a large number of files or large files over the network. | rsync is an algorithm-based tool for incremental backup and file synchronization. It only transfers changed file blocks, thus greatly improving transfer efficiency and speed. At the same time, rsync also supports compressed transmission, and can resume transmission from breakpoints. When the transmission interruption resumes, it can continue, reducing the retransmission time. |
| Synchronization ability | scp only provides file transfer function and cannot perform file synchronization. When there is a difference between the source and the target, scp will replace the entire file with the source file, and changes in the file cannot be automatically recognized and processed. | rsync can ensure that the source and target folders are completely consistent, whether it is local synchronization or remote synchronization, it can efficiently and accurately maintain file consistency. Through incremental backup and file comparison, rsync can identify and transfer only changed file blocks to realize incremental synchronization of files. |
| How to use | The usage of scp is similar to the cp command, which is relatively simple and suitable for copying small-scale files. For example, to copy a local file to a remote server: scp local_file username@remote_host:remote_folder/. | The usage of rsync is more flexible, and various filtering and control can be performed according to the needs. For example, to synchronize the contents of two folders: rsync -avz source_folder/destination_folder/. |
To sum up, scp is suitable for simple file copy and transfer, while rsync is more suitable for scenarios that require efficient, incremental synchronization and backup. The choice of which tool to use depends on your specific needs and intended functionality.
2. Configure rsync source server
1. Turn off the firewall
systemctl stop firewalld
setenforce 0
rpm -q rsync #一般系统已默认安装rsync
2. Create /etc/rsyncd.conf configuration file
vim /etc/rsyncd.conf #添加以下配置项
uid = root
gid = root
use chroot = yes #禁锢在源目录
address = 192.168.30.60 #监听地址
port = 873 #监听端口 tcp/udp 873，可通过cat /etc/services | grep rsync查看
log file = /var/log/rsyncd.log #日志文件位置
pid file = /var/run/rsyncd.pid #存放进程 ID 的文件位置
hosts allow = 192.168.30.0/24 #允许访问的客户机地址
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #同步时不再压缩的文件类型
[wwwroot] #共享模块名称
path = /var/www/html #源目录的实际路径
comment = Document Root of www.fxk.com
read only = yes #是否为只读
auth users = backuper #授权账户，多个账号以空格分隔
secrets file = /etc/rsyncd_users.db #存放账户信息的数据文件
#如采用匿名的方式，只要将其中的“auth users”和“secrets file”配置项去掉即可。
#为备份账户创建数据文件
vim /etc/rsyncd_users.db
backuper:123456 #无须建立同名系统用户,backuper为用户名，123456为密码。
chmod 600 /etc/rsyncd_users.db
3. Ensure that all users have read access to the source directory /var/www/html
chmod +r /var/www/html/
ls -ld /var/www/html/
4. Start the rsync service program
rsync --daemon #启动 rsync 服务，以独立监听服务的方式（守护进程）运行
netstat -anpt | grep rsync
5. Close the rsync service
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid
3. Initiator
基本格式：rsync [选项] 原始位置 目标位置
1. Common options
| options | illustrate |
|---|---|
| -r | Recursive mode, including all files in the directory and subdirectories. |
| -l | Files with symbolic links are still copied as symbolic link files. |
| -v | Display verbose information about the synchronization process. |
| -z | Compress when transferring files. |
| -a | Archiving mode, retaining file permissions, attributes and other information, which is equivalent to the combination option "-rlptgoD". |
| -p | Permission flags for files are preserved. |
| -t | File timestamps are preserved. |
| -g | Preserve the file's group flags (only for superusers). |
| -o | Preserve the ownership of the file (only for superusers). |
| -H | Keep hardlinked files. |
| -A | Preserves ACL attribute information. |
| -D | Keep device files and other special files. |
| –delete | Delete files that exist in the target location but not in the original location. |
| –checksum | Skip files based on checksum (not file size, modification time). |
2. Download the specified resources to the local /opt directory for backup.
格式一：
rsync -avz [email protected]::wwwroot /opt/ #密码123456
格式二：
rsync -avz rsync://[email protected]/wwwroot /opt/
3. Interaction-free format configuration
echo "123456" > /etc/server.pass
chmod 600 /etc/server.pass
rsync -az --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/
定时同步
crontab -e
30 0 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/
#为了在同步过程中不用输入密码，需要创建一个密码文件，保存 backuper 用户的密码，如 /etc/server.pass。在执行 rsync 同步时使用选项 “--password-file=/etc/server.pass” 指定即可。
systemctl restart crond
systemctl enable crond
Fourth, the initiator configures rsync+inotify
使用inotify通知接口，可以用来监控文件系统的各种变化情况，如文件存取、删除、移动、修改等。利用这一机制，可以非常方便地实现文件异动告警、增量备份，并针对目录或文件的变化及时作出响应。
将inotify机制与rsync工具相结合，可以实现触发式备份（实时同步），即只要原始位置的文档发生变化，则立即启动增量备份操作；否则处于静默等待状态。这样，就避免了按固定周期备份时存在的延迟性、周期过密等问题。
因为 inotify 通知机制由 Linux 内核提供，因此主要做本机监控，在触发式备份中应用时更适合上行同步。
1.修改rsync源服务器配置文件
vim /etc/rsyncd.conf
......
read only = no #关闭只读，上行同步需要可以写
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid
rsync --daemon
netstat -anpt | grep rsync
chmod 777 /var/www/html/
2.调整 inotify 内核参数
在Linux内核中，默认的inotify机制提供了三个调控参数：max_queue_events（监控事件队列，默认值为16384）、max_user_instances（最多监控实例数，默认值为128）、max_user_watches（每个实例最多监控文件数，默认值为8192）。当要监控的目录、文件数量较多或者变化较频繁时，建议加大这三个参数的值。
cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches
vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 102400
sysctl -p
3.安装 inotify-tools
用 inotify 机制还需要安装 inotify-tools，以便提供 inotifywait、inotifywatch 辅助工具程序，用来监控、汇总改动情况。
inotifywait：可监控modify（修改）、create（创建）、move（移动）、delete（删除）、attrib（属性更改）等各种事件，一有变动立即输出结果。
inotifywatch：可用来收集文件系统变动情况，并在运行结束后输出汇总的变化情况。
tar zxvf inotify-tools-3.14.tar.gz -C /opt/
cd /opt/inotify-tools-3.14
./configure
make && make install
#可以先执行“inotifywait”命令，然后另外再开启一个新终端向 /var/www/html 目录下添加文件、移动文件，在原来的终端中跟踪屏幕输出结果。
inotifywait -mrq -e modify,create,move,delete /var/www/html
#选项“-e”：用来指定要监控哪些事件
#选项“-m”：表示持续监控
#选项“-r”：表示递归整个目录
#选项“-q”：简化输出信息
4.在另外一个终端编写触发式同步脚本
注意，脚本名不可包含 rsync 字符串，否则脚本可能不生效
vim /opt/inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ [email protected]::wwwroot/"
#使用while、read持续获取监控结果，根据结果可以作进一步判断是否读取到输出的监控记录
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
if [ $(pgrep rsync | wc -l) -le 0 ] ; then
#如果rsync未在执行，则立即启动
$RSYNC_CMD
fi
done
chmod +x /opt/inotify.sh
chmod 777 /var/www/html/
chmod +x /etc/rc.d/rc.local
echo '/opt/inotify.sh' >> /etc/rc.d/rc.local #加入开机自动执行
The above script is used to detect changes in the /var/www/html directory of the local machine. Once there is an update, the rsync synchronization operation will be triggered, and the backup will be uploaded to the wwwroot shared directory of the server 192.168.30.60.
The verification process of triggered uplink synchronization is as follows:
- Run the /opt/inotify.sh script program locally.
- Switch to the /var/www/html directory of the machine, and perform operations such as adding, deleting, and modifying files.
- View the changes in the wwwroot directory on the remote server.
5. Use rsync to quickly delete a large number of files.
If you want to delete a large number of files under linux, such as 1 million, 10 million, like the nginx cache of /usr/local/nginx/proxy_temp, etc., then rm -rf * may not work well, because it takes a long time to wait. In this case we can use rsync to handle it neatly. rsync actually uses the replacement principle.
First create an empty folder
mkdir /root/ztm
and use rsync to delete the target directory
rsync --delete-before -a -H -v --progress --stats /root/ztm /usr/local/nginx/proxy_temp/
so that the target directory will be emptied soon
. Option description:
| options | illustrate |
|---|---|
| –delete-before | The receiver performs a delete operation in the transmission |
| -a | Archive mode, which means recursively transfer files and keep all file attributes |
| -H | Files that remain hardlinked |
| -v | verbose output mode |
| –progress | Show transfer progress while transferring |
| –stats | Gives the transfer status of certain files |
