table of Contents

Real-time synchronization sersync combat

Real-time synchronization sersync combat

What is the real-time synchronization

Real-time synchronization is a variation of one as long as the current directory occurrence of an event is triggered, the event will trigger changes in the directory synchronization to the remote server.

sersync and Comparative rsync + inotify

Data synchronization is bound to mention it comes to rsyncgeneral simple server data transfer will use ftp/sftpother ways, but this way is not efficient, does not support the differentiation incremental synchronization does not support real-time transmission. For real-time data synchronization needs of the majority of people will choose rsync+inotify-toolsthe solution, but this program also has some drawbacks (the article will specifically noted), sersyncare the first two people based tool development, not only retains the advantages but also strengthens the real-time monitoring , file filtering, simplify configuration functions to help users improve operational efficiency, saving time and network resources.

project address

sersync project combat

1) Preparing the Environment

Character	External network IP (NAT)	Network IP (LAN)	Installation Tools
web01	eth0: 10.0.0.7	eth1:172.16.1.7	Deploy code (submit jobs)
nfs-server	eth0: 10.0.0.31	eth1:172.16.1.31	rsync+inotify+sersync
backup	eth0: 10.0.0.41	eth1:172.16.1.41	rsync-server

1. Real-time directory synchronization which server, which then installed on the serversersync

2. As long as the installation sersyncmust be installed rsyncandinotify

Installation of rsync server (backup)

1) install rsync service

[root@backup ~]# yum install -y rsync

2) Profile

[root@backup ~]# vim /etc/rsyncd.conf 
uid = www
gid = www
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
#####################################
[zls]
comment = welcome to oldboyedu backup!
path = /backup

[nfs]
comment = welcome to oldboyedu backup!
path = /data

3) Create a user

[root@backup ~]# groupadd www -g 666
[root@backup ~]# useradd www -u 666 -g 666 -s /sbin/nologin -M

4) Create a directory and authorization

[root@backup ~]# mkdir /data  /backup
[root@backup ~]# chown -R www.www /data/ /backup/

5) Create a virtual user's password file and authorized

[root@backup ~]# echo 'rsync_backup:123' > /etc/rsync.passwd 
chmod 600 /etc/rsync.passwd

6) start the rsync service

[root@backup ~]# systemctl start rsyncd

NFS server deployment sersync

1) installation relies sersync rsyncandinotify

[root@nfs ~]# yum install -y rsync inotify-tools

2) Download sersync

[root@nfs ~]# wget https://raw.githubusercontent.com/wsgzao/sersync/master/sersync2.5.4_64bit_binary_stable_final.tar.gz

3) deployment sersync

Source Package: generating compiled and installed codecs

Decompression:

[root@nfs ~]# tar xf sersync2.5.4_64bit_binary_stable_final.tar.gz

4) Mobile and renamed

[root@nfs ~]# mv GNU-Linux-x86 /usr/local/sersync

5) Edit the configuration file

[root@nfs ~]# vim /usr/local/sersync/confxml.xml
    <inotify>
        <delete start="true"/>
        <createFolder start="true"/>
        <createFile start="true"/>
        <closeWrite start="true"/>
        <moveFrom start="true"/>
        <moveTo start="true"/>
        <attrib start="true"/>
        <modify start="true"/>
    </inotify>
-----------------------------------------------------------------------------------------
    <sersync>
        #监控的目录,改成/data
        <localpath watch="/opt/tongbu">
            #推送的IP（backup服务的IP）172.16.1.41 ，name是模块名
            <remote ip="127.0.0.1" name="tongbu1"/>
            <!--<remote ip="192.168.8.39" name="tongbu"/>-->
            <!--<remote ip="192.168.8.40" name="tongbu"/>-->
        </localpath>
        <rsync>
            #执行rsync的参数改成 -az
            <commonParams params="-artuz"/>
            #虚拟用户的用户名和密码文件,开启认证start=true  rsync_backup    /etc/rsync.pass
            <auth start="false" users="root" passwordfile="/etc/rsync.pas"/>
            <userDefinedPort start="false" port="874"/><!-- port=874 -->
            #设置超时时间
            <timeout start="true" time="100"/><!-- timeout=100 -->
            <ssh start="false"/>
        </rsync>
        <failLog path="/tmp/rsync_fail_log.sh" timeToExecute="60"/><!--default every 60mins execute once-->
        <crontab start="false" schedule="600"><!--600mins-->
            <crontabfilter start="false">
                <exclude expression="*.php"></exclude>
                <exclude expression="info/*"></exclude>
            </crontabfilter>
        </crontab>
        <plugin start="false" name="command"/>
    </sersync>


#完整配置文件
[root@nfs ~]# cat /usr/local/sersync/confxml.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<head version="2.5">
    <host hostip="localhost" port="8008"></host>
    <debug start="false"/>
    <fileSystem xfs="false"/>
    <filter start="false">
    <exclude expression="(.*)\.svn"></exclude>
    <exclude expression="(.*)\.gz"></exclude>
    <exclude expression="^info/*"></exclude>
    <exclude expression="^static/*"></exclude>
    </filter>
    <inotify>
    <delete start="true"/>
    <createFolder start="true"/>
    <createFile start="true"/>
    <closeWrite start="true"/>
    <moveFrom start="true"/>
    <moveTo start="true"/>
    <attrib start="true"/>
    <modify start="true"/>
    </inotify>

    <sersync>
    <localpath watch="/data">
        <remote ip="172.16.1.41" name="nfs"/>
        <!--<remote ip="192.168.8.39" name="tongbu"/>-->
        <!--<remote ip="192.168.8.40" name="tongbu"/>-->
    </localpath>
    <rsync>
        <commonParams params="-az"/>
        <auth start="true" users="rsync_backup" passwordfile="/etc/rsync.pass"/>
        <userDefinedPort start="false" port="874"/><!-- port=874 -->
        <timeout start="true" time="100"/><!-- timeout=100 -->
        <ssh start="false"/>
    </rsync>
    <failLog path="/tmp/rsync_fail_log.sh" timeToExecute="60"/><!--default every 60mins execute once-->
    <crontab start="false" schedule="600"><!--600mins-->
        <crontabfilter start="false">
        <exclude expression="*.php"></exclude>
        <exclude expression="info/*"></exclude>
        </crontabfilter>
    </crontab>
    <plugin start="false" name="command"/>
    </sersync>

    <plugin name="command">
    <param prefix="/bin/sh" suffix="" ignoreError="true"/>  <!--prefix /opt/tongbu/mmm.sh suffix-->
    <filter start="false">
        <include expression="(.*)\.php"/>
        <include expression="(.*)\.sh"/>
    </filter>
    </plugin>

    <plugin name="socket">
    <localpath watch="/opt/tongbu">
        <deshost ip="192.168.138.20" port="8009"/>
    </localpath>
    </plugin>
    <plugin name="refreshCDN">
    <localpath watch="/data0/htdocs/cms.xoyo.com/site/">
        <cdninfo domainname="ccms.chinacache.com" port="80" username="xxxx" passwd="xxxx"/>
        <sendurl base="http://pic.xoyo.com/cms"/>
        <regexurl regex="false" match="cms.xoyo.com/site([/a-zA-Z0-9]*).xoyo.com/images"/>
    </localpath>
    </plugin>
</head>

6) Create a virtual user's password file, and authorized

[root@nfs sersync]# echo '123' > /etc/rsync.pass
[root@nfs sersync]# chmod 600 /etc/rsync.pass

7) Check Help

[root@nfs sersync]# /usr/local/sersync/sersync2 -h
set the system param
execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
_______________________________________________________
参数-d:启用守护进程模式
参数-r:在监控前，将监控目录与远程主机用rsync命令推送一遍
c参数-n: 指定开启守护线程的数量，默认为10个
参数-o:指定配置文件，默认使用confxml.xml文件
参数-m:单独启用其他模块，使用 -m refreshCDN 开启刷新CDN模块
参数-m:单独启用其他模块，使用 -m socket 开启socket模块
参数-m:单独启用其他模块，使用 -m http 开启http模块
不加-m参数，则默认执行同步程序
________________________________________________________________

8) Start sersync

[root@nfs data]# /usr/local/sersync/sersync2 -rdo /usr/local/sersync/confxml.xml

------------------------------------------------ rsync service end------------------------------------------------- -

1) Installation

[root@backup ~]# yum install -y rsync

2) change the configuration

[root@backup ~]# vim /etc/rsyncd.conf
uid = rsync
gid = rsync
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
#####################################
[zls]
comment = welcome to oldboyedu backup!
path = /backup

3) Create System User

[root@backup ~]# useradd rsync -s /sbin/nologin  -M

4) Create a virtual user and password files and authorize

[root@backup ~]# echo 'rsync_backup:123' > /etc/rsync.passwd
[root@backup ~]# chmod 600 /etc/rsync.passwd

5) Create a directory and authorization

[root@backup ~]# mkdir /backup
[root@backup ~]# chown -R rsync.rsync /backup

6) to start the service and add the boot from Kai

[root@backup ~]# systemctl start rsyncd
[root@backup ~]# systemctl enable rsyncd

-------------------------------------------------- -------- rsync client (nfs) ----------------------------------- -------------------------------

1) Installation sersync (rsync + inotify)

[root@nfs ~]# yum install -y rsync inotify-tools

2) Installation sersync

download:

[root@nfs ~]# wget https://raw.githubusercontent.com/wsgzao/sersync/master/sersync2.5.4_64bit_binary_stable_final.tar.gz

Decompression:

[root@nfs ~]# tar xf sersync2.5.4_64bit_binary_stable_final.tar.gz

Moved and renamed:

[root@nfs ~]# mv GNU-Linux-x86 /usr/local/sersync

3) modify the configuration file

[root@nfs ~]# vim /usr/local/sersync/confxml.xml
    <inotify>
        <delete start="true"/>
        <createFolder start="true"/>
        <createFile start="true"/>
        <closeWrite start="true"/>
        <moveFrom start="true"/>
        <moveTo start="true"/>
        <attrib start="true"/>
        <modify start="true"/>
    </inotify>

    <sersync>
        <localpath watch="/zls">
            <remote ip="172.16.1.41" name="zls"/>
            <!--<remote ip="192.168.8.39" name="tongbu"/>-->
            <!--<remote ip="192.168.8.40" name="tongbu"/>-->
        </localpath>
        <rsync>
            <commonParams params="-az"/>
            <auth start="true" users="rsync_backup" passwordfile="/etc/rsync.pas"/>
            <userDefinedPort start="false" port="874"/><!-- port=874 -->
            <timeout start="true" time="100"/><!-- timeout=100 -->
            <ssh start="false"/>
        </rsync>

4) Create a directory

[root@nfs ~]# mkdir /zls

5) Create a password file and authorization

[root@nfs ~]# echo '123' > /etc/rsync.pas
[root@nfs ~]# chmod 600 /etc/rsync.pas

6) Start sersync

[root@nfs ~]# /usr/local/sersync/sersync2 -rdo /usr/local/sersync/confxml.xml

sersync is rsync client

Low-level calls: rsync and inotify