1. Analysis of rsync + inotify-tools and rsync + sersync architecture
#### Why use rsync + sersync architecture?
1. Sersync is developed based on inotify, a tool similar to inotify-tools
2. Sersync can record the name of a specific file or a directory that has changed (including addition, deletion, modification) in the monitored directory, and then use rsync to synchronize only the changed file or directory
#### What is the difference between rsync + inotify-tools and rsync + sersync architecture?
1、rsync+inotify-tools
a. Inotify can only record that the monitored directory has changed (added, deleted, changed) and did not record which file or which directory has changed;
b. When rsync is synchronizing, it does not know which file or directory has changed. Every time it synchronizes the entire directory. When the amount of data is large, the synchronization of the entire directory is very time-consuming (rsync requires the entire directory Traverse to find the comparison file), so the efficiency is very low
2、rsync+sersync
a. Sersync can record the name of a specific file or directory that has changed (added, deleted, changed) in the monitored directory;
b. During synchronization, rsync only synchronizes the changed files or directories (each changed data is relatively small compared to the entire synchronized directory data, and rsync is very fast when iterating through the search for comparison files), so it is very efficient.
Synchronization process:
1. Start the sersync service on the synchronization server, and sersync is responsible for monitoring the changes of file system events in the configuration path;
2. Call the rsync command to synchronize the updated file to the target server;
3. Sersync needs to be configured on the main server and rsync server on the synchronization target server (note: it is an rsync service)
Synchronization process and principle:
1. The user writes updated file data to the sersync server in real time;
2. At this time, the sersync service needs to be configured on the synchronization master server;
3. Start the rsync daemon service on another server to pull data from the sersync server synchronously;
Through the service of the rsync daemon, you can find that in fact, sersync is to monitor the local data write or update event; then, after calling the rsync client command, the file corresponding to the write or update event is pushed to the target server through rsync
The reference server for this configuration is: source server A 192.168.100.150 backup synchronization server B 192.168.100.151 evenly set the password-free login users are root
Server A's secret-free authentication and login to server B through the SSH key pair. The secret-free configuration is as follows:
.ssh key pair authentication login:
a. The key pair must be generated on the client and copy the public key to the corresponding directory on the server. Note: The client user must have a private key, and the server user must have a public key;
b. Server A-150 is configured with the following commands:
ssh-keygen -t rsa ## The client creates a key pair and presses Enter all the way. The key pair is stored in the ~ / .ssh directory, id_rsa is the private key, and id_rsa.pub is the public key;
ssh-copy-id [email protected] ## Upload the public key to the target host, and import it into the user home directory of the target host, ~ / .ssh / authorized_keys file, if the upload fails, use scp to copy and import : Scp .ssh / id_rsa.pub user @ target host ip address;
ssh [email protected] ## You can log in directly without password
Note: When multiple users logging in to the server without a password on the same client only need to upload the public key to the home directory of multiple users on the server side;
Execute on the terminals of server A and server B respectively:yum install rsync -y
2. Rsync local and remote synchronization
1. The role of rsync:
Initiator: the host using rsync is the initiator;
Backup source: the host that responds to the initiator is the backup source;
2. Rsync synchronization direction:
Uplink synchronization: upload, data is at the initiator, backup source provides directory, remote users must have write permission to the backup source directory, and the user logged in at the initiator must have read permission to the data;
Downlink synchronization: download, the data is on the backup source, the initiator provides the directory, the user logged in at the initiator must have write permission to the directory, and the remote user must have read permission to the data from the backup source
3. Syntax: rsync option original location target location
Common options: -a archive mode, retain file permissions, time, link, attribution, special files
-v print detailed information
-z compressed transmission
-H Keep hard links
-r recursive mode, recursive subfiles and subfolders
--delete delete files in the target directory but not in the original location
Common options combination: -av -avzH --delete
4. Local synchronous copy file syntax: rsync -av / source file / target directory /
Note: When the source file is / tmp for example, then the entire tmp directory and the files under the directory will be copied;
When the source file is / tmp /, for example, all files in the tmp directory will be copied, excluding the tmp directory itself;
Multiple files can be copied at the same time, separated by spaces;
Examples:
rsync -av / etc / data #Synchronize all files under etc to / data directory (including etc directory itself)
rsync -av / etc / / data #Sync all files under etc to / data directory (not including etc directory itself)
5. Empty local directory syntax: rsync -av --delete / empty directory // target directory /
Note: There are files in the target directory that are deleted, and there are files in the target directory that are not in the empty directory. The empty directory can be created by yourself, then the purpose of deleting is achieved;
6. The syntax of remote synchronous copy file:
Uplink synchronization (upload): rsync -avzH local data user @backup source ip address: backup directory
Example: rsync -avzH / etc [email protected]: / data / ## #root user login without password 92.168.100.151 Server B simultaneously synchronizes all files under the current etc to the / data directory (including the etc directory itself)
Example: If there is a port rsync -e 'ssh -p 2222' -avzH / etc [email protected]: / data / ### root user logs in to the 92.168.100.151 server through port 2222, and simultaneously synchronizes all files under the current etc. Go to / data directory (including etc directory itself)
Downlink synchronization (download): rsync -avzH user @ backup source ip address: backup directory local backup directory
Example: rsync -avzH [email protected]: / data / / etc / ## #The root user logs in to 92.168.100.151 server without password, and all files in the / data directory of the server B to the local / etc directory
3. Rsync + inotify real-time synchronization
Rsync is only a file copy tool, it cannot monitor the addition, deletion and modification of source files. After making changes at the source, you need to execute the rsync command to synchronize the changes to the target.
Rsync needs to scan the entire directory before each synchronization. If there are more files in the source directory, scanning may take more time.
In order to meet the requirements of real-time monitoring, we need to introduce another tool: inotify.
File system event monitoring tool inotify
inotify-tools provides a simple interface for inotify. It is a library written in C language and also contains command line tools.
For a detailed introduction of inotify-tools, please click: https://github.com/rvoicilas/inotify-tools/wiki
Installation of inotify-tools
For centos7 system, execute in order:
|
1
2
|
yum
install
-y epel-release
yum --enablerepo=epel
install
inotify-tools
|
Use the inotifywait command for event monitoring
1. The role of inotify: the kernel module, monitors the changes of files and directories, and provides two modules for monitoring: inotify-wait (continuous monitoring) and inotify-watch (short-term monitoring) through the inotify-tools tool;
2. Optimize kernel parameters:
max_queued_events ## Maximum time queue
max_user_instances ## The largest instance
max_user_watchs ## Maximum number of monitoring files
Note: Real-time synchronization is based on the monitoring file, so it needs to be at the end of the file, and rsync can only be sent at the originating end;
Summary: Real-time synchronization can only handle upstream, not downstream;
3. Inotify + rsync real-time synchronization case: uplink synchronization
a. Install inotify-tools; initiator 192.168.100.150
tar zxvf inotify-tools-*.tar.gz -C /usr/src/
cd /usr/src/inotify-tools-*/
./configure &&make &&make install
ls /usr/local/bin/inotify*
b. Use of inotify: initiator 192.168.100.150
vi /etc/sysctl.conf
fs.inotify.max_queued_events = 16384 ## Monitor the number of event queues
fs.inotify.max_user_instances = 1024 ## Number of monitoring instances
fs.inotify.max_user_watches = 1048576 ## Number of monitored files
:wq
sysctl -p
c. Test the inotify command; the initiator 192.168.100.150
mkdir /root/data
inotifywait -mrq -e modify,create,attrib,move,delete /root/data
Option notes: -m continuous monitoring, r recursive directory, q simplified output, -e specify monitoring events: modify modify, create create, attribute permission modification, move move, delete delete; operate files in other terminals to view changes
Parameter analysis
- -m Keep continuous monitoring state, if you do not write this parameter, inotifywait will exit after listening to an event.
- -r Monitor directories recursively.
- -q Quiet mode, print less content.
- --timefmt Specifies the output format of time.
- --format Specifies the format of event output.
- -e Set the type of monitored events. Here we monitor changes, additions, deletions and metadata.
d. Write the monitoring script 1 as follows (inotifywait-rsync.sh):
|
1
2
3
4
5
6
|
inotifywait -mrq --timefmt
'%d/%m/%y %H:%M'
--
format
'%T %w%f'
-e modify,delete,create,attrib /root/data/rsync/
|
while
read
file
do
rsync
-avPz --progress
/root/data/rsync/ root@
192.168.100.151:/root/data/rsync/
rsync
-avPz --delete /root/data/rsync/
root@
192.168.100.151:/root/data/rsync/
echo
"${file} was synchronized"
done
|
Write script 2: initiator 192.168.100.150
vim rsync_inotify.sh ## Real-time synchronization script
#!/bin/bash
RSYNC="rsync -avzH /root/data/ [email protected]:/tmp/ --delete"
INT_CMD="inotifywait -mrq -e modify,create,move,delete,attrib /root/data/"
$INT_CMD |while read DIRECOTRY EVENT FILE;do
$RSYNC
done
:wq
chmod +x rsync_inotify.sh
./rsync_inotify.sh & ## Start script running in the background
For the monitoring time of each trigger, inotifywait will execute the code between do and done. Here, we call the previously mentioned rsync command for file synchronization.
e. Add crontab to the monitoring script
|
1
2
|
crontab
-e
* * * * * sh /root/data/
inotifywait-rsync
.sh
|
f. Test verification, the initiator creates test files. Initiator 192.168.100.150
cd /root/data
touch 1.txt
g. The backup source verifies that the files are synchronized; the backup terminal 192.168.100.151
ls /tmp/
Run the following command on server A to explain in detail:
|
1
2
3
4
|
# (1)
rsync -avPz --progress /root/data/rsync/ root@192.168.100.151:/root/data/rsync/
# (2)
rsync -avPz --delete --progress /root/data/rsync/ root@192.168.100.151:/root/data/rsync/
|
You will find that demo.txt also appears in the / root / data / rsync / directory on Server B.
Analysis of the above two commands:
(1) Copy the files in the / root / data / rsync directory on Server A to / root / data / rsync on Server B (192.168.100.151).
(2) Compare the files on the target side and the source side. If the file on the target side does not exist on the source side, delete the file on the target side.
4. R sync + sersync real-time synchronization