Realization of file timing synchronization:
Using rsync combined with cron scheduled tasks to achieve:
rsync -av --delete /data/ 10.0.0.12:/back
-a:保留文件属性
-v:显示过程
-delete:如果源文件没有的,目标文件里面有,就把目标文件里面的删除掉
Implementation of real-time file synchronization:
premise:
Synchronization is triggered when a file changes, but triggering synchronization requires a function that depends on file status changes.
inotify
inotify is a monitoring service of the system kernel, which belongs to a unique mechanism of the operating system kernel and is used to monitor file information changes.
Check if the kernel supports inotify;
[root@LAP1 data]# ls -l /proc/sys/fs/inotify
ls: cannot access ' ': No such file or directory
/proc/sys/fs/inotify:
total 0
-rw-r--r-- 1 root root 0 Oct 24 23:39 max_queued_events
-rw-r--r-- 1 root root 0 Oct 24 23:39 max_user_instances
-rw-r--r-- 1 root root 0 Oct 24 23:39 max_user_watches
inotify kernel parameters:
max_queued_events:inotify 事件队列最大长度,如值太小会出现 Event Queue Overflow 错误,默认值:16384, 生产环境建议调大,比如:327679
max_user_instances:每个用户创建inotify实例最大值,默认值:128
max_user_watches:可以监视的文件的总数量(inotifywait 单进程),默认值:8192,建议调大
Note:
The parameters in the proc can be changed through the sysctl tool.
The main tools of the inotify-tools package:
-
inotifywait: Wait for specific file system events (open, close, delete, etc.) to occur on the monitored file or directory, often used for real-time synchronous directory monitoring (this tool is mainly used)
-
inotifywatch: Collect statistics on the usage of the monitored file system, referring to the number of times file system events occur
inotifywait command:
inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]
options:
-m, --monitor 始终保持事件监听
-d, --daemon 以守护进程方式执行,和-m相似,配合-o使用
-r, --recursive 递归监控目录数据信息变化
-q, --quiet 输出少量事件信息
--exclude <pattern> 指定排除文件或目录,使用扩展的正则表达式匹配的模式实现
--excludei <pattern> 和exclude相似,不区分大小写
-o, --outfile <file> 打印事件存到文件中,注意:使用绝对路径
-s, --syslogOutput 发送错误到syslog相当于标准错误输出
--timefmt <fmt> 指定时间输出格式
--format <fmt> 定义输出格式;即实际监控输出内容
-e 指定监听指定的事件,如果省略,表示所有事件都进行监听
For example:
# 10.0.0.11
[root@LAP1 data]# cat file1
[root@LAP1 data]# echo hello > file1
[root@LAP1 data]# ll file1
-rw-r--r-- 1 root root 6 Oct 24 23:50 file1
[root@LAP1 data]# chmod 666
# 10.0.0.11
[root@LAP1 data]# inotifywait -m file1
Setting up watches.
Watches established.
file1 OPEN
file1 CLOSE_NOWRITE,CLOSE
file1 MODIFY
file1 OPEN
file1 MODIFY
file1 CLOSE_WRITE,CLOSE
file1 ATTRIB
The event type specified by the inotifywait -e option
create #文件或目录创建
delete #文件或目录被删除
modify #文件或目录内容被写入
attrib #文件或目录属性改变
close_write #文件或目录关闭,在写入模式打开之后关闭的
close_nowrite #文件或目录关闭,在只读模式打开之后关闭的
close #文件或目录关闭,不管读或是写模式
open #文件或目录被打开
lsdir #浏览目录内容
moved_to #文件或目录被移动到监控的目录中
moved_from #文件或目录从监控的目录中被移动
move #文件或目录不管移动到或是移出监控目录都触发事件
access #文件或目录内容被读取
delete_self #文件或目录被删除,目录本身被删除
unmount #取消挂载
--timefmt time format for inotifywait
%Y #年份信息,包含世纪信息
%y #年份信息,不包括世纪信息
%m #显示月份,范围 01-12
%d #每月的第几天,范围是 01-31
%H #小时信息,使用 24小时制,范围 00-23
%M #分钟,范围 00-59
%S #秒,范例 0-60
--format format definition for inotifywait
%T #输出时间格式中定义的时间格式信息,通过 --timefmt option 语法格式指定时间信息
%w #事件出现时,监控的文件或目录的名称信息,相当于dirname
%f #事件出现时,将显示监控目录下触发事件的文件或目录信息,否则为空,相当于basename
%e #显示发生的事件信息,不同的事件默认用逗号分隔
%Xe #显示发生的事件信息,不同的事件指定用X进行分隔
For example: monitoring changes in the /data/ directory
[root@CentOS8 data]# inotifywait -m --timefmt "%Y-%m-%d %H:%M:%S" --format="%T %w---%f event: %;e" /data
Setting up watches.
Watches established.
2022-10-24 17:12:57 /data/--- event: OPEN;ISDIR
2022-10-24 17:12:57 /data/--- event: ACCESS;ISDIR
2022-10-24 17:12:57 /data/--- event: CLOSE_NOWRITE;CLOSE;ISDIR
2022-10-24 17:13:06 /data/---file3 event: CREATE
2022-10-24 17:13:06 /data/---file3 event: OPEN
2022-10-24 17:13:06 /data/---file3 event: ATTRIB
2022-10-24 17:13:06 /data/---file3 event: CLOSE_WRITE;CLOSE
rsynctools
rsync works in three ways:
-
Local mode: Synchronization is performed on the local file system. The syntax format of the command line is the format of the above "Local" section
-
Based on the traditional ssh protocol, the local host uses the remote shell to communicate with the remote host
-
As an independent service, the local host connects to the rsync daemon on the remote host through a network socket
the difference:
The essence of the first two is through a local or remote shell, while the third way is to let the rsyncd service run on the remote host, make it listen on a port, and wait for the connection from the client.
local mode:
rsync [OPTION...] SRC... [DEST]
For example:
[root@LAP1 data]# rsync file1 file111
[root@LAP1 data]# ls
file1 file11 file111 file2
Based on the traditional ssh protocol usage format:
Pull:
rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push:
rsync [OPTION...] SRC... [USER@]HOST:DEST
For example:
rsync -av --delete /data/ 10.0.0.12:/back
#不写用户名默认使用的就是当前主机使用的用户
As a standalone service:
Pull:
rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST] #协议的形式访问,效果等同于上面
Push:
rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
Common options for rsync:
-v:显示rsync过程中详细信息。可以使用"-vvvv"获取更详细信息。
-a --archive :归档模式,表示递归传输并保持文件属性。
-t --times:保持mtime属性。强烈建议任何时候都加上"-t",否则目标文件mtime会设置为系统时间,导致下次更新,检查出mtime不同从而导致增量传输无效
--delete :以SRC为主,对DEST进行同步。多则删之,少则补之
rsync package:
Official site: rsync
Packages: rsync, rsync-daemon (CentOS 8)
Service file: /usr/lib/systemd/system/rsyncd.service
Configuration file: /etc/rsyncd.conf
Port: 873/tcp
Two ways to implement rsync server
rsync can be used as a server or as a client program.
Method 1: Realize the rsync service through the rsync daemon process
#在备份服务器启动 rsync 进程
[root@bakup_server ~]# rsync --daemon #--daemon选项表示启动为守护进程
Failed to parse config file: /etc/rsyncd.conf #必须要有这个配置文件才能启动成功
[root@bakup_server ~]# touch /etc/rsyncd.conf #需要创建这个配置文件才能正常启动
[root@bakup_server ~]# rsync --daemon #启动rsync守护进程
[root@bakup_server ~]# ss -ntl #守护进程启动后会监听873端口
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 5 0.0.0.0:873 0.0.0.0:*
#设置rsync服务器的共享信息
[root@bakup_server ~]# cat /etc/rsyncd.conf #等号之间可以有空格
[backup] #定义存放数据共享的名字
path = /bakup #真实的路径,存放共享文件的路径 (利用rsync将这个目录共享出去,共享出去的名字叫做bakup)
read only = no #指定可读写,默认只读
[root@bakup_server ~]# rsync --daemon #更改问配置文件以后需要重新开启守护进程才会生效
[root@bakup_server ~]# setfacl -m u:nobody:rwx /bakup/ #指定目录给nobody权限,默认用户以nobody访问此目录
使用客户端连接rsync备份服务器:
#格式 rsync rsync://host 或者 rsync host::
客户端查看服务器的情况:
[root@data_server ~]# rsync rsync://10.0.0.12 #以协议的形式访问
backup #共享出来的名字
[root@data_server ~]# rsync 10.0.0.12:: #以服务的形式访问
backup
#实现客户端将文件拷贝到rsync共享的目录中
注意:传输的时候不管以谁的身份,都会映射为nobody,所以不用写用户名都行
[root@data_server ~]# rsync /root/anaconda-ks.cfg [email protected]::backup
[root@bakup_server bakup]# ll
total 8
-rw------- 1 nobody nobody 1526 Oct 24 17:48 anaconda-ks.cfg
-rw-r--r-- 1 nobody nobody 658 Oct 24 17:49 fstab
Run rsync as an independent service and implement the verification function
rsync-daemon: After installing this package, a service service will be provided, which will listen to its own independent port
[root@bakup_server ~]# yum install rsync-daemon
Installed:
rsync-daemon-3.1.3-9.el8.noarch
[root@bakup_server ~]# systemctl enable rsyncd.service
Created symlink /etc/systemd/system/multi-user.target.wants/rsyncd.service → /usr/lib/systemd/system/rsyncd.service.
illustrate:
rsync does not require verification when transferring files by default
How to enable the verification function of rsync:
[root@backup-centos8 ~]#dnf -y install rsync-daemon
#安装服务的时候自动创建rsync服务器的配置文件
[root@centos8 ~]#vi /etc/rsyncd.conf
uid = root #远程用户映射到本机的用户,默认为nobody
gid = root #默认为nobody
#port = 874 #可指定非标准端口,默认873/tcp
#use chroot = no
max connections = 0 #不限制最大连接数
ignore errors #如果有些错误,就跳过这些错误
exclude = lost+found/ #跳过指定的目录,不去复制
log file = /var/log/rsyncd.log # 日志所在位置
pid file = /var/run/rsyncd.pid # 存放进程的pid文件
lock file = /var/run/rsyncd.lock # 存放锁文件
reverse lookup = no # 拒绝反向解析,不把ip解析为主机名
#hosts allow = 10.0.0.0/24 # 允许连接的主机
[backup] #每个模块名对应一个不同的path目录,如果同名后面模块生效 共享名
path = /data/backup/ #共享的真实路径
comment = backup dir #描述信息
read only = no #默认是yes,即只读
auth users = rsyncuser #默认anonymous可以访问rsync服务器 用于验证的账号,只有这个账号才能去访问
secrets file = /etc/rsync.pas #存放密码的文件 格式: 用户名: 密码
Example: Implement password authentication
[root@CentOS8 ~]# yum install rsync-daemon
[root@CentOS8 ~]# systemctl enable rsyncd --now
[root@CentOS8 ~]# cat /etc/rsyncd.conf
uid = root
gid = root
max connections = 0
ignore errors
exclude = lost+found/
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsyncd.lock
reverse lookup = no
[backup]
path = /data/backup/
comment = backup dir
read only = no
auth users = tom
secrets file = /etc/rsync.pas
[root@CentOS8 ~]# systemctl restart rsyncd.service
[root@CentOS8 ~]# mkdir /data/backup/ -p
[root@CentOS8 ~]# echo "tom:redhat" > /etc/rsync.pas
[root@CentOS8 ~]# chmod 600 /etc/rsync.pas
#必须要修改密码文件权限,不然客户端访问的时候会提示密码错误
客户端访问:
[root@CentOS8 ~]# rsync /etc/fstab [email protected]::backup
Password: #交互输入tom的密码
或者:
[root@CentOS8 ~]# rsync /root/anaconda-ks.cfg rsync://[email protected]/backup
Password:
注意:不指定用户名默认就是当前系统的用户
You can create a file in advance and put the password in the file
#非交互式查看共享目录
[root@CentOS8 ~]# echo "redhat" >/etc/rsync.pas #客户端存放rsync的密码信息
[root@CentOS8 ~]# chmod 600 /etc/rsync.pas #密码文件权限修改(必须,不然会报错)
测试:
[root@CentOS8 ~]# rsync --password-file=/etc/rsync.pas /root/file111 rsync://[email protected]/backup
inotify+rsync+shell script realizes real-time data synchronization
working principle:
- Use the monitoring service (inotify) to monitor changes in information in the synchronization data server directory
- When the data in the directory is found to change, use the rsync service to push it to the backup server
Example: inotify+rsync+shell script to achieve real-time data synchronization
数据服务器:存放数据信息的服务器 10.0.0.11
备份服务器:存放备份信息的服务器 10.0.0.12
Idea: use inotidy to monitor whether the event changes,
[root@CentOS8 ~]# cat inotify_rsync.sh
#!/bin/bash
SRC='/data/www/' #需要同步的目录文件 #注意最后的/
DEST='[email protected]::backup' #同步到备份数据器的指定位置
rpm -q rsync &> /dev/null || yum -y install rsync #如果不存在rsync这个工具就安装它
inotifywait -mrq --exclude=".*\.swp" --timefmt '%Y-%m-%d %H:%M:%S' --format '%T %w %f' -e create,delete,moved_to,close_write,attrib ${SRC} | while read DATE TIME DIR FILE; do
FILEPATH=${DIR}${FILE} #需要同步的文件
rsync -az --delete --password-file=/etc/rsync.pas $SRC $DEST && echo "At ${TIME} on ${DATE}, file $FILEPATH was backuped up via rsync" >> /var/log/changelist.log
done
#通过DATE TIME DIR FILE 这四个变量记录发生的变化 日期 事件 目录 文件