rsync remote synchronization (sync regularly, real-time synchronization combat!)

This chapter structure

1.rsync Synchronization Profile
Configuration rsync backup source
3.rsync command basic usage
4.rsync exemplary backup operation
5.rsync + inotify combination

About rsync.

A fast incremental backup tool

1.Remote Sync, remote synchronization
2. To support local replication, or with other SSH, rsync host synchronization
3 official website: http://rsync.samba.org

Configuring rsync server source

rsync synchronization source:

It refers to a remote server backup operations, also referred to as a backup source

Configuring rsync source

The basic idea:

1. Establish rsyncd.conf profile, independent of the account files
. Rsync-enabled mode of --daemon

Application examples:

1. households backuper, allowing downlink synchronization
2. Operation of the directory / var / www / html /

Profile rsyncd.conf

1. The need to manually create, syntax similar to the Samba configuration
2. Authentication auth users, secrets file, without anonymity, compared with

rsync account file

1. The use of "username: password" recording format, a user record per line
2. Independent account data, the system does not depend on account

Enabling rsync service

1. alone served by --daemon [performing kill $ (catIvar / run / rsyncd.pid) Close service rsync]

Use rsync backup tool

rsync command usage:

rsync [options] original target position

1. Common options:

-a: archive mode, and retain recursive object properties equivalent to -rlptgoD
-v: Show (verbose) information synchronization procedure
-z: compressed when the file transfer (the compress)
-H: Reserved hard connection file
-A: reserved ACL attribute information
--delete: delete the target location and the original location does not have a file
--checksum: whether to skip the checksum file to determine the object based on

Configure two source identification methods:

Demo：

Preparing the environment: two hosts

CentOS 7-4 as a synchronization source: 192.168.18.148

CentOS 7-5 as a client: 192.168.18.145

7-4 CentOS source of operation:

[root@localhost ~]# hostnamectl set-hostname rsyncd
[root@localhost ~]# su
[root@rsyncd ~]# rpm -q rsync
rsync-3.0.9-18.el7.x86_64
[root@rsyncd ~]# vim /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = yes
pid file = /var/run/rsyncd.pid
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2
#以上内容去掉注释

address = 192.168.18.148        #本机IP地址
port 873        #开端口
log file = /var/rsyncd.log      #指定日志文件 
hosts allow = 192.168.18.0/24   #允许网段访问
#在pid下行添加以上内容

[wwwroot]
path = /var/www/html
comment = www.kgc.com
read only = yes
auth users = backuper
secrets file = /etc/rsyncd_users.db
#在dont下一行插入以上内容：共享模块
#修改完成后按Esc退出插入模式，输入:wq保存退出

#添加密码文件
[root@rsyncd ~]# vim /etc/rsyncd_users.db
backuper:abc123     #需要和共享模块中的auth_users名称一致
#添加完成后按Esc退出插入模式，输入:wq保存退出
[root@rsyncd ~]# chmod 600 /etc/rsyncd_users.db
[root@rsyncd ~]# rsync --daemon
[root@rsyncd ~]# netstat -ntap | grep rsync
tcp        0      0 192.168.18.148:873      0.0.0.0:*          LISTEN      6150/rsync
#此时873端口开启

[root@rsyncd ~]# systemctl stop firewalld.service
[root@rsyncd ~]# setenforce 0
[root@rsyncd ~]# yum install httpd -y
[root@rsyncd html]# cd ..
[root@rsyncd www]# chmod 777 html/
[root@rsyncd www]# ls -l
总用量 0
drwxr-xr-x. 2 root root  6 8月   8 19:42 cgi-bin
drwxrwxrwx. 2 root root 24 12月 16 08:41 html

CentOS 7-5 client operations:

[root@localhost ~]# hostnamectl set-hostname client
[root@localhost ~]# su
[root@client ~]# systemctl stop firewalld.service
[root@client ~]# setenforce 0
[root@client ~]# rpm -q rsync
rsync-3.0.9-18.el7.x86_64
[root@client ~]# yum install httpd -y
[root@client ~]# cd /var/www/html/
#此时文件中是空的没有文件的
[root@client html]# cd ..
[root@client www]# chmod 777 html/
[root@client www]# ls -l
总用量 0
drwxr-xr-x. 2 root root 6 8月   8 19:42 cgi-bin
drwxrwxrwx. 2 root root 6 8月   8 19:42 html

#同步方法一：
[root@client www]#  rsync -avz [email protected]::wwwroot /var/www/html/
Password:       #此时输入密码abc123，敲回车
receiving incremental file list
./
index.html

sent 83 bytes  received 172 bytes  12.44 bytes/sec
total size is 17  speedup is 0.07
[root@client www]# cd html/
[root@client html]# ls      #此时index.html文件被同步
index.html
[root@client html]# cat index.html
this is test web

#同步方法二：
[root@client www]#  rsync -avz rsync://[email protected]::wwwroot /var/www/html/
Password:       #此时输入密码abc123，敲回车
receiving incremental file list
./
index.html

sent 83 bytes  received 172 bytes  12.44 bytes/sec
total size is 17  speedup is 0.07
[root@client www]# cd html/
[root@client html]# ls      #此时index.html文件被同步
index.html
[root@client html]# cat index.html
this is test web

At this point, if we use the planned tasks, you need to take into account the interaction problem-free:

[root@client html]# vim /etc/server.pass
abc123
#写入密码信息后按Esc退出插入模式，输入:wq保存退出
[root@client html]# chmod 600 /etc/server.pass
[root@client html]# rsync -avz --delete --password-file=/etc/server.pass [email protected]::wwwroot /var/www/html/        #用此条命令可以直接进入
receiving incremental file list
./
index.html

sent 83 bytes  received 172 bytes  510.00 bytes/sec
total size is 17  speedup is 0.07
[root@client html]# ls
index.html
[root@client html]# cat index.html
this is test web

#后面就可以在crontab -e中添加计划任务了

rsync real-time synchronization

Regular synchronization inadequate

1. Perform backup time is fixed, delayed significantly, poor real-time
2. When the long-term does not change the synchronization source, intensive periodic task is not necessary

The advantage of real-time synchronization

1. Once the synchronization source changes, starts immediately back
2. As long as no change in the synchronization source, the backup is not performed

About inotify

Linux kernel inotify mechanism

1. From the version 2.6.13 began offering
2. Changes can monitor the file system, and make the notification response
3. supporting software: inotify-tools

rsync + inotify real-time synchronization

Adjust inotify kernel parameters:

max_queue_events: monitoring a queue size
max User instances: monitoring the maximum number of instances
max_ user_watches :: maximum number of monitoring files for each instance

Inotify-tools installation aids:

inotifywait: for continuous monitoring, real output
inotifywatch: for short-term monitoring, then the task is completed the results

-m: Ongoing Monitoring

-r: Recursive monitor all child objects

-q: simplified output

-e: Specify which type of event to monitor

Demo：

Operating end in the client sends:

[root@client html]# vim /etc/sysctl.conf
#需要在发起端开启监控
#在末行下插入以下内容
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
#添加完成后按Esc退出插入模式，输入:wq保存退出

[root@client html]# sysctl -p       #刷新数据
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

#加载inotofy管理工具
[root@client html]# mkdir /aaa
[root@client html]# mount.cifs //192.168.10.190/rpm /aaa
Password for root@//192.168.10.190/rpm:
[root@client html]# cd /aaa
[root@client aaa]# ls
Discuz_X3.4_SC_UTF8.zip           nginx-1.12.2.tar.gz
error.png                         php
extundelete-0.2.4.tar.bz2         redis-5.0.7.tar.gz
haproxy-1.5.19.tar.gz             ruby-2.4.1.tar.gz
httpd-2.4.29.tar.bz2              ruby.png
hzw.jpeg                          squid
inotify-tools-3.14.tar.gz         TC
[root@client aaa]# tar zxvf inotify-tools-3.14.tar.gz -C /opt/
[root@client opt]# cd /opt/inotify-tools-3.14/
[root@client inotify-tools-3.14]# ls
aclocal.m4    config.h.in   COPYING     libinotifytools  man      src
AUTHORS       config.sub    depcomp     ltmain.sh        missing
ChangeLog     configure     INSTALL     Makefile.am      NEWS
config.guess  configure.ac  install-sh  Makefile.in      README
[root@client inotify-tools-3.14]# yum install gcc gc-c++ make -y
[root@client inotify-tools-3.14]# ./configure
[root@client inotify-tools-3.14]# make && make install
[root@client inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /var/www/html/
#进入监控状态，监控本地的html文件中的更新，创建，移动，删除

At this time can not operate, we need to open a remote connection to operate

[root@client ~]# cd /var/www/html/
[root@client html]# ls
index.html
[root@client html]# touch abc       #创建新的abc文件
[root@client html]# rm -rf abc      #删除abc

At this time, the monitor will display the user interface synchronized to this operation:

/var/www/html/ CREATE abc       #同步到创建动作
/var/www/html/ DELETE abc       #同步到删除动作

We can use: monitoring of trigger action, and then synchronize the transfer of rsync

In the first use Ctrl + c to stop monitoring client monitoring, and then create a script, as follows:

[root@client inotify-tools-3.14]# cd /opt/
[root@client opt]# ls
inotify-tools-3.14  rh
[root@client opt]# vim inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ [email protected]::wwwroot/"
$INOTIFY_CMD | while read DIRECTORY EVEVT FILE
do
        if [ $(pgrep rsync | wc -l) -le 0 ] ; then
            $RSYNC_CMD
        fi
done
#添加完成后按Esc退出插入模式，输入:wq保存退出
[root@client opt]# chmod +x inotify.sh
[root@client opt]# ls -l /var/www/
总用量 0
drwxr-xr-x. 2 root root  6 8月   8 19:42 cgi-bin
drwxrwxrwx. 2 root root 24 12月 16 10:00 html

At this time also you need to be aware CentOS 7-4 rsync server configuration file:

[root@rsyncd www]# vim /etc/rsyncd.conf
read only = no      #关闭只读功能
#修改完成后按Esc退出插入模式，输入:wq保存退出

执行脚本
[root@rsyncd ~]# netstat -ntap | grep rsync
tcp        0      0 192.168.18.148:873      0.0.0.0:*          LISTEN      2768/rsync
[root@rsyncd ~]# kill -9 2768       #杀死该进程
[root@rsyncd ~]# rsync --daemon     #启动rsync
[root@rsyncd ~]# failed to create pid file /var/run/rsyncd.pid: File exists
#提示有pid文件存在
[root@rsyncd ~]# cd /var/run/
[root@rsyncd run]# ls
abrt          dhclient-ens33.pid  lock            radvd           syslogd.pid
alsactl.pid   dmeventd-client     log             rpcbind         systemd
atd.pid       dmeventd-server     lsm             rpcbind.sock    tmpfiles.d
auditd.pid    faillock            lvm             rsyncd.pid      tuned
avahi-daemon  firewalld           lvmetad.pid     samba           udev
certmonger    gdm                 mdadm           sepermit        udisks2
chrony        gssproxy.pid        media           setrans         user
chronyd.pid   gssproxy.sock       mount           setroubleshoot  utmp
console       httpd               named           sm-notify.pid   vmware
crond.pid     initramfs           netreport       spice-vdagentd  xtables.lock
cron.reboot   ksmtune.pid         NetworkManager  sshd.pid
cups          libvirt             plymouth        sudo
dbus          libvirtd.pid        ppp             sysconfig
[root@rsyncd run]# cat rsyncd.pid
2768
[root@rsyncd run]# rm -rf rsyncd.pid        #删除此pid文件
[root@rsyncd run]# rsync --daemon       #再次启动
[root@rsyncd run]# netstat -ntap | grep rsync       #此时会生成新的pid号
tcp        0      0 192.168.18.148:873      0.0.0.0:*         LISTEN      5416/rsync
[root@rsyncd run]# cat rsyncd.pid
5416
#此时正常运转rsync

Monitor is turned on at 7-5 client end CentOS:

[root@client opt]# ./inotify.sh
#此时监控开启

Then open another 7-5 start to write the contents of a remote connection:

[root@client html]# echo "this is test" > test.txt

The file is synchronized to the CentOS 7-4 rsync server, we can query:

[root@rsyncd run]# cd /var/www/html/
[root@rsyncd html]# ls
index.html  test.txt
[root@rsyncd html]# cat test.txt
this is test

Above to achieve real-time synchronization!