Table of contents
1. Overview of rsync
1.1 rsync server
①.rsync is an open source, fast, multi-functional, excellent tool that can realize full and incremental local or remote data synchronization backup. Moreover, the attribute information of the original data may not be changed, and the data backup and migration characteristics may be realized.
②.rsync software is suitable for various operating system platforms such as unix/linux/windows
③.rsync is a fast and very similar file copy tool. It can copy instinctively, remotely, or as a remote daemon, it provides a large number of parameters to control various aspects of its behavior, and allows a very flexible way to implement file transfer replication
④. Known for its delta-transfer algorithm.
⑤.rsync listening port: 873
⑥.rsync operation mode: C/S
1.2 Synchronization method
1.2-1 Full backup
- All original data is sent
- Send the original file together with the new file
(full copy, low efficiency)
1.2-2 Incremental backup
Before transmitting the data, use some algorithms to compare the data you have with the data I have, and transmit different data through the network.
(Incremental replication, high efficiency)
1.2-3 rsync sync source server
In a remote synchronization task, the client responsible for initiating the rsync synchronization operation is called the initiator, and the server responsible for responding to the rsync synchronization operation from the client is called the synchronization source.
In downlink synchronization (download), the synchronization source is responsible for providing the original location of the document, and the initiator should have read access to the location.
In an upstream sync (upload), the sync source is responsible for providing the target location of the document, to which the initiator should have write access.
1.2-4 The difference between scp and rsync
(1) Functional gap
- rsync remote copy can be attached with soft/hard links. (parameter -l keeps soft links, -H keeps hard links)
- scp does not support copying of links.
(2) Efficiency difference
Simply analyze scp and rsync, the former is copying, and the latter is synchronization.
When rsync and scp do not exist in the folder, the execution time is not much different, but the difference is very large when the folder exists. The reason is that scp is a copy: if the destination file does not exist, it will be created, and if it exists, it will be overwritten. And rsync is a synchronization, compare whether the files on both sides are the same, if they are the same, do nothing, if there is a difference, update it directly.
It will be faster to use rsync when it plays the role of synchronization, and both can be used when it plays the role of copying (there is no file at the destination). Choose rsync or scp depending on the situation.
2. Configure the rsync source
2.1 Basic idea
- Create rsyncd.conf configuration file and independent account file
- Enable rsync's --daemon mode
2.2 Configuration file rsyncd.conf
Authentication configuration auth users, secrets file, if not added, it will be anonymous
2.3 Independent account file
- Username Password
- One user record per line
- Independent account data, independent of system account
2.4 Enable rsync service
- Serve alone through --daemon, rsync --daemon
- Execute kill $(cat /var/run/rsyncd.pid) to close the service
2.5 rsync functions and features
2.5-1 rsync function
As a command: realize local-remote file synchronization
As a service: realize local-remote file synchronization
2.5-2 rsync features
- Can mirror save the entire directory tree and file system
- Can retain the original permissions (permission, mode), owner, group, time (modification time, modify time), soft and hard links, file acl, file attributes (attributes) information, etc.
- High transmission efficiency, use synchronization algorithm, only compare changes
- Supports anonymous transmission, which is convenient for website mirroring; it can also be verified to enhance security
2.6 The same type of service as rsync
- sync Synchronization: Refresh the file system cache, force the modified data blocks to be written to disk, and update the super block.
- async Asynchronous: put the data in the buffer first, and then periodically (usually 30s) to synchronize to the disk.
- rsync remote synchronization: remote synchronous
2.7 Comparison between cp and rsync
- cp copies the source file completely to the specified path for full backup
- When rsync does local copy
- First compare the source file and the target file to find the difference
- Perform consistent synchronization according to the difference between the file at the destination location and the source file
- scp: secure copy is also a command for remote copying, but each backup data is a full backup
Three, rsync command
The syntax used by the command
rsync 【选项】原始位置 目标位置
The function of common option
–delete is simply to delete the difference file and keep the consistency
Four, two ways to configure the source
Format one:
用户名@主机地址::共享模块名
Format two:
rsync://用户名@主机地址/共享模块名
5. Overview of inotify
Can monitor changes in the file system and respond to notifications
- Adjust inotify kernel parameters (optimization)
- /etc/sysctl.conf (kernel parameter configuration file)
max_queue_events #监控事件队列大小
max_user_instances #最多监控实例数
max_user_watches #每个实例最多监控文件数
- notifywait: for continuous monitoring, real-time output results
- inotifywatch: used for short-term monitoring, output the result after the task is completed
inotifywait -mrq -e modify,create,move,delete /var/www/html
6. Configure rsync
rsync source server 192.168.154.10
rsync initiator 192.168.154.11
6.1 Configure rsync source server
#关闭防火墙和核心防护
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
rpm -q rsync #一般系统已默认安装rsync
#建立/etc/rsyncd.conf 配置文件
vim /etc/rsyncd.conf #添加以下配置项
uid = root
gid = root
use chroot = yes #禁锢在源目录
address = 192.168.154.10 #监听地址
port = 873 #监听端口 tcp/udp 873，可通过cat /etc/services | grep rsync查看
log file = /var/log/rsyncd.log #日志文件位置
pid file = /var/run/rsyncd.pid #存放进程 ID 的文件位置
hosts allow = 192.168.154.0/24 #允许访问的客户机地址
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #同步时不再压缩的文件类型
[wwwroot] #共享模块名称
path = /var/www/html #源目录的实际路径
comment = Document Root of www.kgc.com
read only = yes #是否为只读
auth users = backuper #授权账户，多个账号以空格分隔
secrets file = /etc/rsyncd_users.db #存放账户信息的数据文件
如采用匿名的方式，只要将其中的“auth users”和“secrets file”配置项去掉即可。
#为备份账户创建数据文件
vim /opt/rsyncd_users.db
backuper:abc123 #无须建立同名系统用户
chmod 600 /opt/rsyncd_users.db
#保证所有用户对源目录/var/www/html 都有读取权限
mkdir -p /var/www/html
chmod +r /var/www/html/
ls -ld /var/www/html/
#启动 rsync 服务程序
rsync --daemon #启动 rsync 服务，以独立监听服务的方式（守护进程）运行
netstat -anpt | grep rsync
cat /var/run/rsyncd.pid
cd html/
cp /etc/passwd /etc/shadow /etc/fstab ./
6.2 Initiator
Basic format:rsync [选项] 原始位置 目标位置
Commonly used options:
-r: recursive mode, including all files in the directory and subdirectories.
-l: For symbolic link files, still copy as symbolic link files.
-v: Display detailed (verbose) information about the synchronization process.
-z: Compress (compress) when transferring files.
-p: Preserve the permission flags of the file.
-t: Keep the time stamp of the file. (atime access time, mtime modify content time, ctime modify file time)
-g: retain the group mark of the file (only for super users).
-o: Keep the owner mark of the file (superuser only).
-H: Keep hardlink files.
-a: Archive mode, retain file permissions, attributes and other information, which is equivalent to the combination option "-rlptgoD".
-A: Keep ACL attribute information.
-D: Keep device files and other special files.
--delete: Delete files that exist in the target location but not in the original location.
–checksum: decide whether to skip files based on checksum (instead of file size, modification time).
#192.168.154.11
mkdir /data
rsync -avc [email protected]::wwwroot /data/
#rsync源服务器 192.168.154.10
rm -rf fstab
cp /etc/hostname /etc/os-release ./
#192.168.154.11
rsync -avz rsync://[email protected]/wwwroot /data
The fstab of the rsync source server has been deleted, but the originator has not. It needs to be synchronized with the rsync source server
rsync -avz --delete rsync://[email protected]/wwwroot /data
--delete Delete files that exist in the target but not in the original location
6.3 Interaction-free format configuration
rsync origin server
rm -f *
echo a > a
echo b > b
echo c > c
echo d > d
initiator
cd /opt
vim rsync_pass
cat rsync_pass
chmod 600 rsync_pass
#发起端
rsync -az --delete --password-file=/opt/rsync_pass [email protected]::wwwroot /data/
crontab -e -u root
crontab -l
systemctl restart crond
systemctl enable crond
rsync -avz -e 'ssh -p 22' /var/www/html/index.html [email protected]:/opt/
#发起端
rsync -az --password-file=/opt/rsync_pass /etc/yum.repos.d/ [email protected]::wwwroot
The rsync server has received the file from the initiator
. There are two commonly used authentication methods for rsync, one is rsync-daemon, and the other is ssh. In some occasions, using the rsync-daemon method will be relatively inflexible, and the ssh method will become the first choice.
-e 'ssh -p 22': rsync uses ssh to specify the port. If the port is the default port of 22, this parameter does not need to be specified.
rsync -avz -e 'ssh -p 22' /etc/yum.repos.d/ [email protected]::/opt/rh
Seven, the initiator configures rsync+inotify
Using the inotify notification interface can be used to monitor various changes in the file system, such as file access, deletion, movement, modification, etc. Using this mechanism, it is very convenient to realize file change alarms, incremental backups, and respond to changes in directories or files in a timely manner.
Combining the inotify mechanism with the rsync tool can realize triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; otherwise, it will be in a silent waiting state. In this way, problems such as delay and periodic over-intensity that exist when backups are made in a fixed period are avoided.
Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring, and it is more suitable for upstream synchronization when applied in triggered backup.
7.1 Modify the rsync source server configuration file
vim /etc/rsyncd.conf
......
read only = no #关闭只读，上行同步需要可以写
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid
rsync --daemon
netstat -anpt | grep rsync
chmod 777 /var/www/html/
7.2 Adjust inotify kernel parameters
In the Linux kernel, the default inotify mechanism provides three control parameters: max_queue_events (monitoring event queue, the default value is 16384), max_user_instances (the maximum number of monitoring instances, the default value is 128), max_user_watches (the maximum number of monitoring files per instance , the default is 8192). When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the value of these three parameters.
cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches
vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
sysctl -p
7.3 Install inotify-tools
To use the inotify mechanism, you need to install inotify-tools to provide inotifywait and inotifywatch auxiliary tool programs for monitoring and summarizing changes.
inotifywait: It can monitor various events such as modify (modify), create (create), move (move), delete (delete), attrib (attrib change), and output the result immediately when there is a change.
inotifywatch: It can be used to collect file system changes and output the summary changes after the operation ends.
tar zxvf inotify-tools-3.14.tar.gz -C /opt/
cd /opt/inotify-tools-3.14
./configure
make && make install
#You can execute the "inotifywait" command first, and then open a new terminal to add files to the /var/www/html directory, move files, and track the screen output results in the original terminal.
inotifywait -mrq -e modify,create,move,delete /var/www/html
Option "-e": Used to specify which events to monitor
Option "-m": Indicates continuous monitoring
Option "-r": Indicates recursive entire directory
Option "-q": Simplifies the output information
7.4 Write a trigger synchronization script in another terminal
(Note that the script name cannot contain rsync strings, otherwise the script may not take effect)
vim /opt/inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/opt/rsyncd_users.db /var/www/html/ [email protected]::wwwroot/"
#使用while、read持续获取监控结果，根据结果可以作进一步判断是否读取到输出的监控记录
$INOTIFY_CMD | while read DIRECTORY FILE EVENT
do
if [ $(pgrep rsync | wc -l) -le 0 ] ; then
#如果rsync未在执行，则立即启动
$RSYNC_CMD
fi
done
chmod +x /opt/inotify.sh
chmod 777 /var/www/html/
chmod +x /etc/rc.d/rc.local
echo 'nohup /opt/inotify.sh &' >> /etc/rc.d/rc.local #加入开机自动执行
The above script is used to detect changes in the /var/www/html directory of the local machine. Once there is an update, the rsync synchronization operation will be triggered, and the backup will be uploaded to the wwwroot shared directory of the server 192.168.154.10.
The verification process of triggered uplink synchronization is as follows:
(1) Run the /opt/inotify.sh script program on the local machine.
(2) Switch to the /var/www/html directory of the local machine, and perform operations such as adding, deleting, and modifying files.
(3) View the changes in the wwwroot directory in the remote server.
Use rsync to achieve fast deletion of large numbers of files.
If you want to delete a large number of files under linux, such as 1 million, 10 million, like the nginx cache of /usr/local/nginx/proxy_temp, etc., then rm -rf * may not work well, because it takes a long time to wait. In this case we can use rsync to handle it neatly. rsync actually uses the replacement principle.
Create an empty folder first:
mkdir /home/blank
Use rsync to delete the target directory:
rsync --delete-before -a -H -v --progress --stats /home/blank/ /usr/local/nginx/proxy_temp/
so that the target directory will be emptied soon
. Option description:
–delete-before the receiver deletes during transmission
-a archive mode, means to transfer files recursively, and keep all file attributes
-H keep hard-linked files
-v verbose output mode
–progress shows the transmission process during transmission
–stats Gives the transfer status of certain files