Detailed interpretation of rsync and inotify remote synchronization

Detailed interpretation of rsync remote synchronization

1. About rsync

A fast incremental backup tool

Remote Sync, remote synchronization

Support local replication, or synchronize with other SSH, rsync hosts

Two, configure the rsync source

1. Basic ideas

Create rsync.conf configuration file, independent account file

Enable rsync --daemon mode

2. Application examples

User backuper, allowing synchronization

The operating directory is /var/www/html

3. Configuration file rsyncd.conf

Need to be established manually, the syntax is similar to Samba configuration

Authentication configuration auth users, secrets file, if not added, it will be anonymous

4. rsync account file

Adopt the record format of "username:password", one user record per line

Independent account data, not dependent on the account system

5. Enable rsync service

Independently provide services through -dawmon

Three, rsync command usage

Basic format: rsync [options] original location target location

Common options

-r	Recursive mode, including all files in the directory and subdirectories
-l	For symbolic link files are still copied as symbolic link files
-v	Show detailed information about the synchronization process
-from	Compress when transferring files
-a	Archive mode, which retains file permissions, attributes and other information, which is equivalent to the combined option "-riptgoD"
-p	Keep file permission mark
-t	Keep the time stamp of the file
-g	Keep the group mark of the file (only for super user)
-O	Keep the owner mark of the file (only for super user)
-H	Keep hard-linked files
-A	Keep ACL attribute information
-D	Keep equipment files and other special files
–delete	Delete files in the target location but not in the original location
–checksum	Decide whether to skip files based on checksum (not file size, modification time)

Five, rsync real-time synchronization

1. The shortcomings of regular synchronization

①The time to perform the backup is fixed, the delay is obvious, and the real-time performance is poor.
②When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary

2. Advantages of real-time synchronization

①Once the synchronization source changes, start the backup immediately
②As long as the synchronization source does not change, the backup will not be performed

Six, about inotify

1. The inotify mechanism of the Linux kernel

①Provided from version 2.6.13 ②It
can monitor the changes of the file system and respond to notifications
③Auxiliary software: inotify-tools

2. rsync + inotify real-time synchronization

notifywait: used for continuous monitoring, real-time output results

inotifywatch: used for short-term monitoring, the results will be released after the task is completed

-m: continuous monitoring

-r: Recursively monitor all child objects

-q: Simplify output information

-e: specify which event types to monitor

Six, rsync experiment steps

1. Environmental layout

Master：rsync、httpd

Slave：rsync、httpd、inotify

2. Master configuration (192.168.200.11)

①Turn off the firewall

systemctl stop firewalld

setenforce 0

②Install httpd and rsync software packages

yum install -y httpd rsync

③Add rsync configuration file

vim /etc/rsyncd.conf #Add the following configuration item
uid = root #It can also be nobody
gid = root #It can also be nobody
use chroot = yes #Contained in the source directory
address = 192.168.200.11 #Monitor address, monitor the local address
port 873 #Monitor port tcp/udp 873, you can view
log file through cat /etc/services | grep rsync = /var/log/rsyncd.log #Log file location
pid file = /var/run/rsyncd.pid #Storage process ID file location
hosts allow = 192.168.200.0/24
#Client network segment allowed for synchronization [wwwroot] #shared module name
path = /var/www/html #The actual path of the source directory (the synchronized directory)
comment = Document Root of www.fyf.com #This has little effect on our configuration of rsync
read only = yes #Whether it is read-only
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #No more during synchronization Compressed file type
auth users = backuper #Authorization account, multiple accounts separated by spaces
secrets file = /etc/rsyncd_users.db #Data file storing account information

④Backup account data files

#If you are anonymous, just remove the "auth users" and "secrets file" configuration items

vim /etc/rsyncd_users.db

backuper:abc123 #No need to create a system user with the same name

chmod 600 /etc/rsyncd_users.db #Add permissions

⑤Install httpd service

yum -y install httpd

systemctl start httpd

systemctl enable httpd

⑥Create shared directories and files

cd / var / www / html

touch aaa bbb

chmod +r /var/www/html/ #Give all readable permissions

⑦Restart the service

rsync --daemon #Start the rsync service, run as an independent monitoring service (daemon)

netstat -anpt | grep rsync #View port number

#Close the rsync method

kill $(cat /var/run/rsyncd.pid)

rm -rf /var/run/rsyncd.pid

3. Configuration of the initiator (slave)

①Turn off the firewall

systemctl stop firewalld

setenforce 0

② Install rsync and httpd

yum install -y httpd rsync

③Realize sharing

#Download the specified resources to the local /opt directory for backup

Format one

rsync -avz [email protected]::wwwroot /opt/ #密码abc123

Format two

rsync -avz rsync://[email protected]/wwwroot /opt/

④Realize no interaction

echo “abc123” > /etc/server.pass

chmod 600 /etc/server.pass

crontab -e 30 22 * * * /usr/bin/rsync -avz --delete --password-file=/etc/server.pass [email protected]::wwwroot /opt/

systemctl restart crond

systemctl enable crond

4. Initiate automatic monitoring inoyify

①Modify the configuration file of the rsync source server (192.168.200.12)

vim /etc/rsyncd.conf

read only = no #Close read-only, upstream synchronization needs to be writable

②Restart the service

kill $(cat /var/run/rsyncd.pid)

rm -rf /var/run/rsyncd.pid

rsync --daemon

③Adjust the inotify kernel parameters

In the Linux kernel, the default inotify mechanism provides three control parameters: max_queue_events (monitoring event queue, the default value is 16384), max_user_instances (the maximum number of monitoring instances, the default value is 128), max_user_watches (the maximum number of files to be monitored per instance , The default value is 8192). When the number of directories and files to be monitored is large or the changes are frequent, it is recommended to increase the values ​​of these three parameters

cat /proc/sys/fs/inotify/max_queued_events #Monitoring event queue
cat /proc/sys/fs/inotify/max_user_instances #Maximum number of monitored instances
cat /proc/sys/fs/inotify/max_user_watches #Maximum number of monitored files per instance

vim /etc/sysctl.conf
#Increase each parameter fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

sysctl -p

④According to the inotify package

Use inotify mechanism also need to install inotify-tools, in order to provide inotifywait, inotifywatch auxiliary tool program

notifywait: can monitor various events such as modify (modify), create (create), move (move), delete (delete), attrib (attribute change), etc., and output the result immediately when there is a change

inotifywatch: can be used to collect file system changes, and output summary changes after the end of the run

#Depending on the environment
mount /dev/cdrom /mnt #Mount the disk
yum install gcc gcc gcc-c++ make -y
#Upload the compressed package inotify-tools-3.14.tar.gz to the /opt directory, and then unzip
tar zxvf inotify- tools-3.14.tar.gz -C /opt/

cd /opt/inotify-tools-3.14
./configure
make && make install

#Execute the "inotifywait" command, and then add files and move files to the /var/www/html directory in another terminal, and track the screen output results. #After
executing the following command, it will enter the monitoring mode and cannot be operated, so you need to open another terminal page for
inotifywait -mrq -e modify,create,move,delete /opt #option
"-e": used to specify which events to monitor
#Option "-m": means continuous monitoring
# option "-r": means recursively the entire directory
# option "-q": simplified output information

⑤Create script

#!/bin/bash
INOTIFY_CMD=“inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/”
RSYNC_CMD=“rsync -apzH --delete --password-file=/etc/server.pass /var/www/html/ [email protected]::wwwroot/”
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
if [ $(pgrep rsync | wc -l) -le 0 ] ; then
$RSYNC_CMD
fi
done

chmod +x inotify.sh #给脚本添加执行权限
sh inotify.sh