rsync remote synchronization (rsync+inotify)

In order to avoid entering a password during the synchronization process, you need to create a password file to save the password of the backuper user, such as /etc/server.pass.

#Use the option "--password-file=/etc/server.pass" to specify when performing rsync synchronization.

systemctl restart crond

systemctl enable crond

4. inotify tool:

1. Install the inotify tool on the sending end:

tar zxvf inotify-tools-3.14.tar.gz -C /opt/

cd /opt/inotify-tools-3.14

./configure

make -j2 && make install

2. inotify kernel parameters

In the Linux kernel, the default inotify mechanism provides three control parameters

1) max_queue_events (monitoring event queue, default value is 16384)

2) max_user_instances (maximum number of monitoring instances, default value is 128)

3) max_user_watches (maximum number of monitored files per instance, default value is 8192)

When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the values of these three parameters.

cat /proc/sys/fs/inotify/max_queued_events

cat /proc/sys/fs/inotify/max_user_instances

cat /proc/sys/fs/inotify/max_user_watches

3. Set up monitoring instances and modify instance files:

because /etc/sysctl.conf

fs.inotify.max_queued_events = 16384

fs.inotify.max_user_instances = 1024

fs.inotify.max_user_watches = 1048576

rsync synchronization speed is very fast, suitable for synchronizing large files, and can be used in conjunction with database synchronization

inotify: Notification interface, which can be used to monitor various changes in the file system. File access, deletion, movement and modification can all be monitored.

The inotify mechanism is used together with rsync configuration. It can both notify changes and achieve synchronization.

Only monitoring, synchronization is rsync

4. inotify command:

inotifywait -mrq -e modify,create,move,delete /opt/test/

-m: continuous monitoring

-r: Recurse the entire directory

-q: information prompt

-e: Specify the events to be monitored. Multiple events are separated by commas.

5. Experiment:

Architecture:

test1 source 20.0.0.21

test2 initiator (client) 20.0.0.22

Turn off firewall security mechanism

systemctl stop firewalld

setenforce 0

Check the installation status:

rpm -q rsync

rsync is the software that comes with Linux

rpm -qc rsync

1. Downstream synchronization:

Downstream synchronization format:

rsync [options] source server location local location

for example

Format one:

rsync -avz [email protected]::message /opt/

Format two:

rsync -avz rsync://[email protected]/message /opt/

test is the authorized account in the configuration file

The IP address is the synchronization source address

message is a shared module defined in the configuration file

Configure source-side rsync:

Change configuration file:

vim /etc/rsyncd.conf #Add the following configuration items

uid = root

gid = root

use chroot = yes #imprisoned in the source directory

address = 20.0.0.21 #Listening address

port = 873

#Listening port tcp/udp 873, can be viewed through cat /etc/services | grep rsync

log file = /var/log/rsyncd.log #Log file location

pid file = /var/run/rsyncd.pid #The file location where the process ID is stored

hosts allow = 20.0.0.0/24 #Client address allowed to be accessed

dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #File types that are no longer compressed during synchronization

[test] #Shared module name

path = /opt/test #The actual path of the source directory

comment = test

write only = yes #Whether it is read-only

auth users = backuper #Authorized accounts, multiple accounts separated by spaces

secrets file = /etc/rsyncd_users.db #Data file to store account information

#If you use anonymous mode, just remove the "auth users" and "secrets file" configuration items.

#Create data files for backup accounts

Add password to authorized account

because /etc/rsyncd_users.db

backuper:123456 #No need to create a system user with the same name

Set that only the owner of the file can read and modify the password file

chmod 600 /etc/rsyncd_users.db

Ensure that all users have read permissions to the source directory /data

mkdir /opt/test

chmod 777 /opt/test

Start the rsync service and run it as an independent listening service (daemon process)

rsync --daemon

Observe whether the startup is successful

ss -voltage | grep rsync

Synchronize data from the server to the client:

rsync -avz [email protected]::test /opt

-avz: retain file permissions vdisplay detailed process zcompress files during synchronization

20.0.0.21::test

Future synchronization will be incremental synchronization, and only new additions will be synchronized.

rsync -avz [email protected]::test /opt

2. Uplink synchronization:

Modify the initiator (client) and update it to the source

Or execute the command on the client:

rsync -azH --delete --password-file=/etc/server.pass /opt/data [email protected]::test/

Perform monitoring and synchronization together

Script:

because inotify.sh

#!/bin/bash

CMD="inotifywait -mrq -e modify,create,move,delete /opt/data"

rsync_cmd="rsync -azH --delete --password-file=/etc/server.pass /opt/data [email protected]::test/"

$CMD | while read DIRECTORY EVENT FILE

if [ $(pgrep rsync | wc -l) -gt 0 ]

then

$reync_cmd

done

You can also set up scheduled tasks

3. Use rsync to quickly delete a large number of files

In the case of a large number of files, such as millions and tens of millions of files, rm -rf * is very slow.

rsync can achieve mass deletion

At this time, using the replacement principle of rsync and combining it with the --delete option, you can quickly delete a large number of files, such as the service cache.

Now the host simulation generates a large number of junk files:

mkdir test1

cd test1/

touch {1..9999}.txt

touch demo.txt

#demo.txt is used for testing

Create another empty file

mkdir test2

cd test2

touch demo.txt

perform synchronous deletion

rsync --delete-before -avH --progress --stats /opt/test2/ /opt/test1

--delete-before: delete during transfer

-a: archive mode

-H: hard link

-v: output information

--progress: Display the transfer process

--stats: gives the transfer status of the file