How to implement Rsync remote synchronization, do you understand?

One: Overview of rsync

1.1: About rsync

• A fast incremental backup tool
  • Remote Sync, remote synchronization
  • Support local replication, or synchronize with other SSH, rsync hosts
  • r (remote) sync is a data mirroring backup software under unix and unix-like platforms. It does not require full backup like ftp. rsync can perform differential backups according to data changes, thereby reducing data traffic and improving work efficiency
  • Rsync can implement incremental backups remotely or locally. rsync can realize file synchronization between local host and remote host (including local push to remote and remote pull to local synchronization), and it can also realize file synchronization under different local paths (between different directories and partitions), but it cannot Realize synchronization between remote path 1 and remote path 2 (scp can be achieved)
  • rsync cooperates with planned tasks to achieve timing or periodic synchronization
  • Rsync configures inotify or sersync to achieve triggered real-time synchronization
  • Official website: http://rsync.samba.org

1.2: The three main configuration files of rsync

• tsyncd.conf (main configuration file)
• rsync.secrets (password file)
• rsyncd.motd (server information file)

1.3: rsync features

• Can mirror and save the entire directory tree and file system

• You can easily maintain the original file permissions, time, soft and hard links, etc., and you can install it without special permissions

• High efficiency, rsync will copy all content during the first synchronization, but only the modified files will be transferred next time. Rsync can perform compression and decompression operations during data transfer, using less bandwidth

• High security, rsync supports anonymous transmission to facilitate website mirroring, and ssh encrypted transmission can be used when transmitting data

Note: In the centos system, rsync comes with the system, no additional installation is required, but if you use the minimal installation, you can use yum install -y rsync to install it.

• Support speed limit, support breakpoint resume

1.4: rsync synchronization process

• It consists of two parts, check mode (decide which files need to be synchronized) and sync mode (when files are synchronized)
  • Check mode
    • The checking mode refers to checking which files need to be synchronized according to specified rules, for example, which files are explicitly excluded from transmission. By default, rsync uses the "quick check" algorithm to quickly check whether the size and mtime (modification time) of the source file and the target file are consistent. If they are inconsistent, they need to be transferred. Of course, you can also change the check mode of quick check by specifying certain options in the rsync command line. For example, the "--size-only" option means that "quick check" will only check files with different file sizes as files to be transferred. rsync supports a lot of options, among which the customization of the check mode is very flexible.
  • Sync mode
    • The synchronization mode refers to what additional work is done before the synchronization process occurs after the file is determined to be synchronized. For example, as mentioned above, whether to delete files that are not on the source host but on the target host, whether to back up the existing target files, and whether to follow additional operations such as link files. rsync also provides a lot of options to make the synchronization mode more flexible.
• Relatively speaking, the option to manually specify the synchronization mode for rsync is more common. The check mode is only specified when there are special needs, because most check mode options may affect the performance of rsync.

Two: rsync source server

2.1: rsync synchronization source

• Refers to the remote server of the backup operation, also known as the backup source

2.2: basic rsync commands

• Start the rsync service: rsync --daemon

• Turn off the rsync service: kill $(cat /var/run/rsyncd.pid)

• Synchronize local file system data: rsync [option] original location target location

例如：
rsync /etc/fstab /opt '同步本机的fstab文件到opt目录'
rsync -rl /boot/grub /opt	'同步本机的grub目录到opt目录'
'如果想要在/opt目录下也创建一个boot目录，那么命令需要为：rsync -R rl /boot/grub /opt '
常用选项：
-a：归档模式，递归并保留对象属性，等同于 -rlptgoD
-r 对子目录以递归模式处理，主要是针对目录来说的，如果单独传一个文件不需要加-r，但是传输的是目录必须加-r选项
-l 保留软链接
-p 保持文件权限
-v：显示同步过程的详细（verbose）信息
-z：在传输文件时进行压缩（compress）
-H：保留硬连接文件
-A：保留ACL属性信息
--delete：删除目标位置有而原始位置没有的文件
--checksum：根据对象的校验和来决定是否跳过文件
--progress 在同步的过程中可以看到同步的过程状态，比如统计要同步的文件数量、同步的文件传输速度等等
路径的格式可以是本地路径，也可以是使用user@host:path或user@host::path的远程路径，如果主机和path路径之间使用单个冒号隔开，表示使用的是远程shell通信方式，而使用双冒号隔开的则表示的是连接rsync daemon

• Two ways of downlink synchronization (use the client to synchronize the content under the wwwroot shared module under the rsync server to the local /opt directory (the real shared path under the shared module needs to have'r' permission for other users))

  • Command format用户名@主机地址：：共享模块名

  E.g:[root@rsyncClient ~]# rsync -avz [email protected]::wwwroot /opt

  • Command format:rsync://用户名@主机地址/共享模块名

  E.g:[root@slave opt]# rsync -avz rsync://[email protected]/wwwroot /root

• rsync synchronizes via ssh

  • The command is similar to the usual scp command
  • For example: upload the local /opt/abc.txt to the /opt directory of the target server:rsync -avz /opt/abc.txt [email protected]:/opt
  • For example: download the target server /opt/qwe.txt file to the local /opt directory:rsync -avz [email protected]:/opt/qwe.txt /opt

2.3: Ideas for configuring rsync source

• The basic idea
  • Establish rsyncd.conf configuration file and independent account file
  • Enable rsync's --daemon mode
• Application example
  • User backup, allowing downlink synchronization
  • The operating directory is /var/www/html
• Configuration file rsyncd.conf
  • Need to be established manually, grammar NVC and Samba configuration
  • Authentication configuration auth users, secrets file, if not added, it is anonymous
• rsync account file
  • Adopt the record format of "Username: Password", one user record per line
  • Independent account data, not dependent on system account
• Enable rsync service
  • Provide services alone through --daemon
• Close service
  • Execute kill $(cat /var/run/rsyncd.pid)
  • Note that you cannot use kill -9 to end the process number, otherwise an error will be reported when you restart it. For details, please see my other blog.

Three: Experiment

3.1: Experimental environment

• VMware software, centos 7.6
• Source server: 20.0.0.51, installation service: rsync, httpd
• Client: 20.0.0.52, installation service: rsync, inotify-tools, httpd

3.2: Experimental purpose

• Realize remote synchronization

3.3: Experimental schematic diagram

3.4: Experimental process

• Turn off the firewall and core protection functions, both source and client have to do

[root@localhost ~]# systemctl stop firewalld	'关闭防火墙'
[root@localhost ~]# systemctl disable firewalld	'关闭防火墙开机自启'
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@localhost ~]# setenforce 0	'关闭核心防护'
[root@localhost ~]# vi /etc/sysconfig/selinux 	'禁止核心防护开启自启'
SELINUX=disabled

3.4.1: Configure the source server

• The rsync tool comes with the system and does not need to be installed, directly modify the main configuration file

vim /etc/rsyncd.conf
'开启以下功能,将#注释，以及添加'
uid = nobody
gid = nobody
use chroot = yes	'禁锢在家目录,用chroot，在传输文件之前，服务器守护程序在将chroot 到文件系统中的目录中，这样做的好处是可能保护系统被安装漏洞侵袭的可能'
address = 20.0.0.51                  '监听地址'
port 873		'监听端口号'
log file = /var/log/rsyncd.log            '日志文件位置'
pid file = /var/run/rsyncd.pid		'存放进程ID的文件位置'
hosts allow = 20.0.0.0/24	'设置白名单，允许哪些地址可以访问'
'添加共享模块'
[wwwroot]		'共享模块名称'
path = /var/www/html	'源目录的实际路径'
comment = www.kevin.com		'描述'
read only = yes			'是否只读'
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2  '同步时不在压缩的文件类型'
auth users = backuper	'授权账户'
secrets file = /etc/rsyncd_user.db		'存放账户信息的数据文件'

• Created in user password file

vim /etc/rsyncd_users.db	'创建存放账户信息的数据文件'
backuper:abc123		'采用“用户名:密码”的记录格式，每行一个用户记录独立的账号数据，不依赖于系统账号'
cd /etc
chmod 600 /etc/rsyncd_users.db  '给数据文件设置权限是为了安全，不让其他用户进行操作'

• Start rsync service

rsync --daemon   '启动服务'
netstat -ntap | grep rsync

• Create a new file in the /var/www/html directory

[root@localhost etc]# yum install httpd -y
[root@localhost etc]# cd /var/www/html/
[root@localhost html]# touch 111.html
[root@localhost html]# touch 222.html

3.4.2: Configure the client

• Two ways of downlink synchronization, the password is abc123

'方法一'
[root@slave opt]# rsync -avzH --delete lisi@192.168.233.131::wwwroot /opt
Password: 
receiving incremental file list
./
111.html
222.html

sent 98 bytes  received 207 bytes  87.14 bytes/sec
total size is 0  speedup is 0.00

'方法二'
[root@slave opt]# rsync -avz rsync://lisi@192.168.233.131/wwwroot /root
Password: 
receiving incremental file list
./
111.html
222.html

sent 98 bytes  received 207 bytes  87.14 bytes/sec
total size is 0  speedup is 0.00

• Secret-free interactive processing of rsync sources

[root@localhost opt]# vim /etc/server.pass		'创建密码文件'
abc123 '写对方密码'
[root@localhost opt]# chmod 600 /etc/server.pass
[root@localhost opt]# rsync -zva --delete --password-file=/etc/server.pass backuper@20.0.0.51::wwwroot /opt '指定刚刚创建的密码文件，发现已经不需要输入密码了'

3.4.3: Configure rsync real-time synchronization

• source

[root@master html]# vi /etc/rsyncd.conf 
read only = no '改为no'
[root@master html]# kill $(cat /var/run/rsyncd.pid)	'关闭rsync服务'
[root@master html]# netstat -ntap |grep rsync
[root@master html]# rsync --daemon	'开启rsync服务'
[root@master html]# netstat -ntap |grep rsync
tcp        0      0 20.0.0.51:873     0.0.0.0:*               LISTEN      88302/rsync         
[root@master html]# chmod 777 /var/www/html

• Client

[root@localhost opt]# yum install httpd -y
[root@localhost opt]# cd /var/www/html/
[root@localhost html]# vim /etc/sysctl.conf
'文件末尾添加'
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@localhost html]# sysctl -p  '立即生效'
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@localhost html]# cd /opt/
[root@localhost opt]# rz -E
rz waiting to receive.
[root@localhost opt]# tar zxvf inotify-tools-3.14.tar.gz '安装inotify-tools辅助工具'
[root@localhost opt]# cd inotify-tools-3.14/
[root@localhost inotify-tools-3.14]# yum install gcc gcc-c++ -y
[root@localhost inotify-tools-3.14]# ./configure
[root@localhost inotify-tools-3.14]# make && make install
[root@localhost inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete  /var/www/html

• Reopen another client terminal, edit the script and start

'测试'
[root@localhost ~]# cd /var/www/html/
[root@localhost html]# touch index.html
[root@localhost html]# echo "this is test" > test.html
'看执行的那台终端'
/var/www/html/ CREATE index.html
/var/www/html/ CREATE test.html
/var/www/html/ MODIFY test.html
'配置启动脚本，并将上一个测试关闭'
[root@localhost opt]# vim /opt/inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html
"RSYNC_CMD="rsync -avz --delete --password-file=/etc/server.pass /var/www/html/ backuper@20.0.0.51::wwwroot/"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE 
 do
   if [ $(pgrep rsync | wc -l) -le 0 ]; then
      $RSYNC_CMD
   fi
done
[root@localhost opt]# chmod +x inotify.sh 
[root@localhost opt]# chmod 777 /var/www/html/
[root@localhost opt]# ./inotify.sh  '开启监控'

• test

• About rsync error

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9] 
'发现会有这个报错，但是文件是正常传过去的，这里的这个报错我没有解决掉，如果有朋友解决了可以私信我或者评论告诉我，多谢！'