table of Contents
Two specific operation command .rsync
Three .rsync configuration file parsing
V. with inotify tools to achieve triggered transmission
A .rsync Overview
- r (remote) sync data mirroring software under unix and unix-platform, it is not required as ftp full backup, differential backup can be carried out according to the rsync variation data, thereby reducing data traffic to improve efficiency
- rsync can be local or remote to achieve incremental backup. rsync file synchronization can be implemented on the local host and the remote host (local or remote to push, pull remote local two synchronization mode) can be realized at different local paths (a different directory, between partitions) synchronization file, but not path to a remote synchronous remote path (SCP can be realized) between 2
- rsync with the planned tasks, you can achieve timing or cycle synchronization
- rsync configuration inotify or sersync, can achieve real-time synchronization trigger
- This is the official website interface rsync
- rsync divided into three profiles: tsyncd.conf (main profile); rsync.secrets (password file); rsyncd.motd (file server information)
- rsync has the following characteristics:
1. Mirror can save an entire directory tree and file system
2. You can easily keep the original file's permissions time, soft and hard links, etc., can be installed without special permission
3. The high efficiency, the first synchronization rsync will copy the entire content, but the next transmission only modified files, rsync compression and decompression operation may be carried out during the transmission of data, to use less bandwidth
4. safe, the rsync support anonymous transport, to facilitate site mirroring, may be used when transmitting data encrypted transport ssh
Note: In centos system rsync system comes with no additional installation, but if you use to minimize installation, you can use yum install -y rsync install it.
5. Support the speed limit, support for HTTP
-
rsync synchronization pattern composed of two parts, inspection mode (decide which files need synchronization) and synchronization pattern (sync files)
(1) Check the profile is in accordance with the specified rule to check what files need to be synchronized, such as which files are not transmitted explicitly excluded. By default, rsync using the "quick check" algorithm to quickly check the size of the source file and the target file, mtime (modification time) whether, and if not then need to transfer. Of course, may be changed by a quick check of the inspection mode rsync specify certain options in the command line, such as "--size-only" option represents a "quick check" only checks the file size of the file as a different file to be transmitted. rsync supports a lot of options, custom check pattern is very elastic.
(2) The synchronization mode refers to the file is determined to be synchronized, what additional work to be done before the synchronization process occurs. For example, whether the above-mentioned first delete the target file on the source host does not but some files on the target host, whether the first backup already exists, whether or not to follow links and other additional file operations. rsync also offers a lot of options for making synchronization patterns become more flexible.
Relatively speaking, the synchronization mode specified for some of the more common rsync manual option, only the specified check mode only when there are special needs, because most check mode options are likely to affect the performance of rsync.
-
Three modes of rsync
(1). Synchronize the local file system. Command-line syntax format segment of "Local".
Local: rsync [OPTION...] SRC... [DEST]
(2) a local host and a remote host using the remote shell communication, command syntax is
Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST(3) the local host is connected rsync daemon on the remote host through a network socket, the command syntax is:
Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
Two specific operation command .rsync
Local copy
- Format: rsync [options] original target position
常用选项有：
-a //归档模式，递归保留对象属性，等同于 -rlptgoD
-v //显示像详细信息
-z //在传输文件时进行压缩
-H //保留硬链接文件
-A //保留ACL属性
-t //暴露修改时间属性
-p //保留文件权限属性
-g //保留文件所属组属性
--progress //显示数据传输的进度信息
--password-file=FILE //指定密码文件，将密码写入文件，实现免交互数据库同步，需要修改权限,使用这个选项需要开启rsync daemon模式
--delete //删除那些仅在目标路径中存在的文件（源路径中不存在），在脚本中的数据同步中经常加上这个参数
--list-only //仅列出服务器模板列表，需要rsync服务器设置list=true
--checksum //根据对象的校验和来决定是否跳过文件- Examples
[root@192 ~]# ls
1.txt initial-setup-ks.cfg 视频 下载
anaconda-ks.cfg 公共 图片 音乐
harbor-offline-installer-v1.2.2.tgz 模板 文档 桌面
[root@192 ~]# rsync -avz harbor-offline-installer-v1.2.2.tgz /opt //保留文件属性、压缩、详细信息等
sending incremental file list
harbor-offline-installer-v1.2.2.tgz
sent 533945011 bytes received 31 bytes 71192672.27 bytes/sec
total size is 533765727 speedup is 1.00
[root@192 ~]# cd /opt
[root@192 opt]# ls
harbor-offline-installer-v1.2.2.tgz rhCopy via remote shell
- Examples of data formats and downloads
rsync [选项] [user@a]HOST:SRC...[DEST] #不加user@表示用root用户进行登陆远程主机下载数据到 本地的DEST路径
##client为
[root@client ~]# echo "123" > 1.txt
[root@client ~]# ls
1.txt initial-setup-ks.cfg 模板 图片 下载 桌面
anaconda-ks.cfg 公共 视频 文档 音乐
[root@client ~]#
##从client上下载数据
[root@server ~]# rsync -avz [email protected]:1.txt /root/
[email protected]'s password:
receiving incremental file list
1.txt
sent 36 bytes received 72 bytes 30.86 bytes/sec
total size is 4 speedup is 0.04
[root@server ~]# ls
1.txt initial-setup-ks.cfg 视频 下载
anaconda-ks.cfg 公共 图片 音乐
harbor-offline-installer-v1.2.2.tgz 模板 文档 桌面
[root@server ~]#
- Data formats, and upload examples
##格式为：
上传数据：rsync[选项] SRC...[user@]HOST:DEST #这里的SRC表示本地数据，DEST表示远端主机目录
##实例如下：
[root@server ~]# rsync -avz --progress harbor-offline-installer-v1.2.2.tgz [email protected]:/root/
[email protected]'s password:
sending incremental file list
harbor-offline-installer-v1.2.2.tgz
533765727 100% 50.62MB/s 0:00:10 (xfer#1, to-check=0/1)
sent 533945011 bytes received 31 bytes 39551484.59 bytes/sec
total size is 533765727 speedup is 1.00
[root@server ~]#
--progress显示进度条Copy via rsync process
- Download data format
rsync [选项] [user@] HOST::SRC...[DEST] #这里双冒号后的SRC表示远端服务器端的模块名
rsync [选项] rsync://[user@]HOST[:port]/src...[DEST] #这里的SRC表示实际的同步目录名，可以指定端口- Upload data format
rsync [选项] SRC...[user]@HOST::DEST #上传本地客户端数据到远端服务端的DEST模块名指定的路径
rsync [选项] SRC...rsync://@HOST[:port]/DEST
Three .rsync configuration file parsing
- Rsync.conf profiles are configured by a plurality of modules and the overall configuration composition,
- Configuration file syntax is:
模块以 [模块名] 开始
参数配置行的格式是 name = value ，其中 value 可以有两种数据类型：
字符串（可以不用引号定界字符串）
布尔值（1/0 或 yes/no 或 true/false）
以 # 或 ; 开始的行为注释
\ 为续行符
- All lines are global configuration parameter [Module1] other than, of course, also possible, in case the value of the parameter is a global parameter module parameters it is part of the definition file in the default values for all modules.
- The main module parameters used to define which directories rsync server to be synchronized. Format module declaration must be [module] form; the name is rsync client to see the name. Similar to the share name Samba server. The data server is synchronized by the real path to specify, according to their own needs, to specify multiple modules
- rsync authentication password file permissions must be 600, otherwise the client can not connect server
- rsync authentication password file in each line specify a user name: password pairs in the format: username: passwd
vim /etc/rsyncd.conf
motd file = /etc/rsyncd.motd #设置服务器信息提示文件，在该文件中编写提示信息
transfer logging = yes #开启rsync数据传输日志功能
log file = /var/log/rsyncd.log #设置日志文件名，可通过log format参数设置日志格式
pid file = /var/run/rsyncd.log #设置rsync进程号保存文件名称
lock file = /var/run/rsync.lock #设置锁文件名称
port = 873 #设置服务器监听的端口号，默认是873
address = 192.168.0.230 #设置本服务器所监听网卡接口的ip地址
uid = nobody #设置进行数据传输时所使用的帐户名或ID号，默认使用nobody
gid = nobody #设置进行数据传输时所使用的组名或GID号，默认使用nobody
#若为yes, rsync会首先进行chroot设置，将根映射在下面的path参数路径下，对客户端而言，系统的根就是path参数指定的路径。但这样做需要root权限，并且在同步符号连接资料时只会同步名称，不会同步内容。
use chroot = no
read only = yes #是否允许客户端上传数据，yes表示不允许
max connections =10 #设置并发连接数，0表示无限制
[common] #自定义模块名，rsync通过模块定义同步的目录，可定义多个
comment = web content #定义注释说明字串
path = /common #同步目录的真是路径通过path指定
ignore errors #忽略一些IO错误
#exclude = test/ #exclude指定common目录下某个目录可以不同步数据
auth users = tom, jerry #设置允许连接服务器的账户，此账户可以是系统中不存在的用户
secrets file = /etc/rysncd.secrets #密码验证文件名，该文件权限要求为只读，建议为600，仅在设置auth users后有效
hosts allow = 192.168.0.0/255.255.255.0 #设置哪些主机可以同步数据，多ip和网段之间使用空格分隔
hosts deny=* #除了hosts allow定义的主机外，拒绝其他所有
list = false #客户端请求显示模块列表时，本模块名称是否显示，默认为trueIV. Turned rsync daemon works
Configuring server
- Apache install a service, providing a site directory
[root@server ~]# yum install httpd -y
[root@server ~]# cd /var/www/html/
[root@server html]# ls
[root@server html]# echo "this is test web" > index.html
[root@server html]# chmod 777 index.html
[root@server html]# - Rsync modify configuration files
vim /etc/rsyncd.conf
1、开启以下功能：
uid = nobody
gid = nobody
use chroot = yes
//注：用chroot，在传输文件之前，服务器守护程序在将chroot 到文件系统中的目录中，这样做的好处是可能保护系统被安装漏洞侵袭的可能。
pid file = /var/run/rsyncd.pid //存放进程ID的文件位置
dont compress = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2 //同步时不再压缩的文件类型
2、添加以下选项：
address = 192.168.43.101 //添加本机IP地址
log file = /var/log/rsyncd.log //日志文件位置
port 893 //端口号
hosts allow = 192.168.43.0/24 //设置白名单，允许哪些地址可以访问
3、添加共享模块名：
[wwwroot] //共享文件名，这个自定义
path = /var/www/html //源目录的实际路径
comment = www.kgc.com //描述
read only = yes //是否为只读
auth users = backuper //授权账户名
secrets file = /etc/rsyncd_users.db //存放账户信息的数据文件- Data file editing account information
[root@server ~]# cat /etc/rsyncd_users.db
backuper:abc123
//权限设置为600，必须设置为600，否则客户端认证会失败
[root@server ~]# chmod 600 /etc/rsyncd_users.db
[root@server ~]# - Start Service
[root@server ~]# rsync --daemon
[root@server ~]# netstat -natp | grep rsync
tcp 0 0 192.168.43.101:873 0.0.0.0:* LISTEN 37983/rsync
[root@server ~]#
client configuration
- Interactive mode, download data
[root@client ~]# rsync -avz [email protected]::wwwroot /opt
Password:
receiving incremental file list
./
index.html
sent 83 bytes received 172 bytes 102.00 bytes/sec
total size is 17 speedup is 0.07
[root@client ~]# cd /opt
[root@client opt]# ls
index.html rh
[root@client opt]# cat index.html
this is test web
[root@client opt]# 报错：rsync: failed to connect to 192.168.43.101 (192.168.43.101): No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(122) [Receiver=3.0.9]Solution: close the server firewall and selinux (systemctl stop firewalld / sentenforce 0)
Or firewall-cmd --permanent --add-port = 873 / tcp # add firewall rules allow access to the data port 873
- Free interactive mode, write a password file in the client, using the --password-file = FILE option to interact free
##在server的/index.html追加文本
[root@server html]# cat index.html
this is test web
[root@server html]# echo "123" >> index.html
[root@server html]# cat index.html
this is test web
123
[root@server html]#
##在client上下载数据，会覆盖原有的文件
[root@client ~]# echo "abc123" > /etc/server.pass
[root@client ~]# chmod 600 /etc/server.pass
[root@client ~]# rsync -avz --password-file=/etc/server.pass [email protected]::wwwroot /opt/
receiving incremental file list
index.html
sent 86 bytes received 177 bytes 526.00 bytes/sec
total size is 21 speedup is 0.08
[root@client ~]# cat /opt/index.html
this is test web
123
[root@client ~]# Shell scripts, regular data backup rsync server on the client side
vi /root/run.sh
#!/bin/bash
export PATH=/bin:/usr/bin:/usr/local/bin
SRC=common #模块名
DEST=/data
server=192.168.0.230
user=tom
passfile=/root/rsync.pass
#if the DEST directory not found, then create one
[ ! -d $DEST ] && mkdir $DEST
[ ! -e $passfile ] && exit 2
rsync -az --delete --password-file=$passfile ${user}@${server}::$SRC $DEST/$(data +%Y%m%d) #加上日期
##在使用crotab定时任务执行这个脚本
crotab -e
0 0 * * * bash /root/run.sh
V. with inotify tools to achieve triggered transmission
rsync to synchronize the pros and cons
- rsync shortcomings
A fixed time to perform a backup, the delay detailed, real-time difference;
When the sync source does not change the long-term, intensive periodic task is not necessary
- rysnc advantage
Once the synchronization source change, to enable the backup immediately;
Does not change as long as the synchronization source, the backup is not performed
inotify Introduction
- Inotify Linux kernel is a mechanism that can monitor file system operations, such as read, write, create and so on.
- Inotify responsive, usage is very simple, and busy polling task efficiently than cron
- Available starting version 2.6.13;
- You can monitor the file system changes, and make notification response;
- Auxiliary software: inotify-tools
Configuring rsync + inotify real-time synchronization
- Modify the kernel parameters in the client
[root@client ~]# vim /etc/sysctl.
sysctl.conf sysctl.d/
[root@client ~]# vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@client ~]# sysctl -p //生效
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@client ~]#
//参数解析
max_queue_event : 监控队伍大小
max_user_instances : 最多监控实例数
max_user_watches : 每个实例最多监控文件数- The client compile and install inotify-tools aid
yum install gcc gcc-c++ make -y //安装编译工具
tar zxvf inotify-tools-3.14.tar.gz -C /opt/ //解压
cd /opt/inotify-tools-3.14/
./configure //编译
make && make install //安装
- Monitoring client files
[root@client opt]# mkdir myweb
[root@client opt]# ls
index.html inotify-tools-3.14 myweb rh
[root@client opt]# inotifywait -mrq -e modify,create,move,delete /opt/myweb
##
inotifywait：用于持续监控，实时输出结果；
inotifywatch：用于短期监控，任务完成后再出结果- In addition to open a terminal client, write a shell script
[root@client opt]# ls
index.html inotify.sh inotify-tools-3.14 myweb rh
[root@client opt]# cat inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,move,delete /opt/myweb"
RSYNC_CMD="rsync -avz --delete --password-file=/etc/server.pass /opt/myweb/ [email protected]::wwwroot/"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE ##读取输出的监控记录
do
if [ $(pgrep rsync | wc -l) -le 0 ]; then ##如果rsync未在执行，则立即启动
$RSYNC_CMD
fi
done
[root@client opt]# chmod +x inotify.sh- On the server, turn off the read-only mode, and restart the service
[root@server html]# vi /etc/rsyncd.conf
......
read only = no
......
[root@server html]# kill $(cat /var/run/rsyncd.pid)
[root@server html]# rsync --daemon
[root@server html]# netstat -anpt |grep rsync
tcp 0 0 192.168.43.101:873 0.0.0.0:* LISTEN 38855/rsync
[root@server html]# chmod 777 /var/www/html/
[root@server html]# - Startup script on the client
[root@client opt]# chmod 777 /opt/myweb/
[root@client opt]# ./inotify.sh Test whether the data synchronization
- Adding data in a client / opt / myweb / directory, see the server / var / www / hmtl no data
- Delete this data in client. View server data will not be deleted
