rsync backup server set up study notes

rsync data synchronization, backup
timer: rsync + cron scheduled tasks
in real time: rsync + sersync (inotify)

https://www.samba.org/ftp/rsync/rsync.html official documents

• ssh scp command with the remote copy (encrypted copies)
• Local copy cp
• Delete function rm

Some of the additional features of rsync are: rsync additional features

• support for copying links, devices, owners, groups, and permissions to support special copy documents, such as links, equipment, etc.
• exclude and exclude-from options similar to GNU tar can exclude specified file or directory synchronization function, equivalent to the exclusion packaged tar command
• a CVS exclude mode for ignoring the same files that CVS would ignore
  
• can use any transparent remote shell, including ssh or rsh by ssh, rsh encrypted transmission (the rsync without encryption)
• does not require super-user privileges
  
• pipelining of file transfers to minimize latency costs
  
• support for anonymous or authenticated rsync daemons (ideal for mirroring) support for anonymous or authenticated mode transmission process

rsync can be a network service (Port, socket)
rsync command is a command to the client

rsync three operating modes

• The first mode of operation: local

rsync -avz /root/aaa /tmp/  --->cp
rsync -avz --delete /root/aaa/ /tmp/    --->rm

• The second mode of operation: remote shell, generally with ssh, key keyless transferred, combined regular tasks

rsync -avzP -e 'ssh -p 22' /opt/ [email protected]:/tmp/     push
rsync -avzP -e "ssh -p 22" [email protected]:/tmp/ /opt/ 带斜线代表获取的是tmp下的文件    pull
rsync -avzP -e "ssh -p 22" [email protected]:/tmp /opt/  不带斜线代表获取的是tmp这个目录   pull

• The third mode of operation: the network does not need rsync daemon, encryption performance loss vpn (pptp, openvpn, ipsec)

  Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
        rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
  Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
        rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

Key Syntax:
1, -avz equivalent -vzrtopgDl, said in sync file and directory attributes unchanged.
2, -P synchronous display process, may be replaced by --progress
. 3, -e '22 is ssh -p', indicates the channel data transmission via ssh, -p 22 may be omitted
4, root @ 192.168.105.135: / tmp / remote host system user, address, path
5, / opt local path
6, synchronize the contents of a file, there are some of these changes, the first comparison resynchronization

Parameter options	description
-v --verbose	Detailed output mode, and progress during the transmission of information
-z --compress	Compression to improve the transmission efficiency of the transmission, - compress-level = NUM ​​level of compression may be
-a --archive	Archive mode, expressed recursively distribute files, and keep all document properties, equivalent to -rlptgoD
-r --recursive	All subdirectories recursively mode, that directory are equally transmission
-t --times	Keeping steady time information
-o --owner	Keep the file owner information
-p --perms	To keep file permissions
-g --group	Keep the file is a set of information
-P --progress	Process synchronization and display information such as the time schedule of transmission
-D --devices	Keep the device file information
-l --links	Reserved soft links
--delete	No difference synchronization (dangerous)
--exclude	Exclude a single file exclude multiple files {a, b}, {a..d } exclude --exclude-from the plurality of files
--partial	HTTP large file
--bwlimit	--bwlimit = 100 speed limit

Enterprise application scenarios:

1. Data synchronization between the two servers
   
2. All client-server data synchronization to a backup server
   
3. rsync functionality combined with inotify do real-time data synchronization rsync + inotify or sersync

Production scenarios cluster architecture server backup solutions:

cron + rsync the whole network server data backup solutions proposed and responsible for the implementation

1. Propose solutions to back up the whole network data backup important data for the company and the leadership of confusion
   
2. Packaged by local backup, and then combined with rsync inotify application to the whole network back up data to a unified fixed storage server, storage server and then on through the script and checking the results of the backup alarm administrator
   
3. IDC room regularly back up data onto different servers within the company, to prevent engine room fire and earthquake issues lead to data loss

rsync advantages:

1, incremental backup, supporting socket (daemon), centralized backup (sliding support, are client-reference)
2, may also be remote shell mode channel encryption (ssh) transmission, socket (daemon) needs to be encrypted transmission, ipsec vpn service or use of services

rsync Disadvantages:

1. When a large number of small files synchronized than a long time, sometimes, rsync process may terminate
2. Synchronize large files, 10G such large files sometimes go wrong, terminal, while not complete synchronization finished, it will become a hidden file, the transmission can be achieved through HTTP and other parameters. Remote Copy may be disposable scp

---

Rsync backup combat

Step rsync server configuration

1, configure the server, see the current version number and installation package rsync

rsync --version     rsync-3.0.6-12.el6.x86_64

2, vim /etc/rsyncd.conf Add Profile

rsync configuration file official website: https://rsync.samba.org/ftp/rsync/rsyncd.conf.html

#配置文件详解，更多参数查看man rsyncd.conf
#Rsync server
#created by hejian 19:50 2019-3-17
##rsyncd.conf start##
uid = rsync     //客户端具备的权限（不存在，需要创建）
gid = rsync
use chroot = no     //安全相关
max connections = 2000   //客户端连接数
timeout = 600       //超时
pid file = /var/run/rsyncd.pid      #进程ID文件
lock file = /var/run/rsync.lock     #锁文件
log file = /var/log/rsyncd.log      #日志文件
ignore errors           #忽略错误
read only = false   #可读写
list = false        #不可以远程列表
hosts allow = 192.168.105.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup   //虚拟用户
secrets file = /etc/rsync.password   //存放用户账号密码的文件
#####################################
[backup]    //模块调用
#comment = backup by hejian 19:50 2019-3-17
path = /backup   //共享目录
[hejian]
path = /hejian

3, rsync users to create and share the directory / backup, and give permission to the directory

useradd rsync -s /sbin/nologin -M
id rsync
mkdir /backup
chown -R rsync.rsync /backup/

4, create a password file

echo "rsync_backup:123" > /etc/rsync.password
chmod 600 /etc/rsync.password

5, start the rsync service and view

rsync --daemon

lsof -i :873  
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME  
rsync   1647 root    3u  IPv4  12479      0t0  TCP *:rsync (LISTEN)  
rsync   1647 root    5u  IPv6  12480      0t0  TCP *:rsync (LISTEN)  

netstat -tunlp|grep rsync  
tcp        0      0 0.0.0.0:873                 0.0.0.0:*                     LISTEN      1647/rsync            
tcp        0      0 :::873                      :::*                          LISTEN      1647/rsync    

ss -tunlp|grep rsync    
tcp    LISTEN     0      5                     :::873                  :::*        users:(("rsync",1647,5))  
tcp    LISTEN     0      5                      *:873                   *:*        users:(("rsync",1647,3))   

ps -ef|grep rsync  
root       1647      1  0 20:11 ?        00:00:00 rsync --daemon  
root       1660   1505  0 20:12 pts/0    00:00:00 grep --color=auto rsync    

6, adding boot from the start

echo "rsync --daemon" >>/etc/rc.local  
cat /etc/rc.local

---

rsync client configuration step

1, create a password file, and modify permissions

echo "123" > /etc/rsync.password （与服务端密码保持一致，免密登陆）
chmod 600 /etc/rsync.password（与服务端密码权限保持一致）
ll /etc/rsync.password

2, transmitting files rsync (push or pull is operated from the client)

(push)rsync -avz /backup/ [email protected]::backup/ --password-file=/etc/rsync.password
(pull)rsync -avz  rsync://[email protected]:873/backup/ /backup/ --password-file=/etc/rsync.password
echo $? = 0 就代表程序运行是正确的

Push excluded

rsync -avz --exclude=a /root/test/ [email protected]::backup/ --password-file=/etc/rsync.password

Troubleshooting

#Rsync server
#created by hejian 19:50 2019-3-17
##rsyncd.conf start##
uid = rsync
gid = rsync
use chroot = no
max connections = 2000
timeout = 600
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 192.168.0.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
#####################################
[backup]
comment = backup by hejian 19:50 2019-3-17
path = /backup  

Virtual machine snapshots
rsync real test preparation environment, modify the IP, modify the host name, modify the parameters of the color themes, the main backup configuration files

Client error

A error:

[root@lamp01:/tmp]# rsync -avz /tmp/ [email protected]::backup --password-file=rsync.password
rsync: failed to connect to 192.168.105.136: No route to host (113)
rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6]  

Troubleshooting: server firewall turned on

Error two:

[root@lamp01:/tmp]# rsync -avz /tmp/ [email protected]::backup --password-file=rsync.password
rsync: could not open password file "rsync.password": No such file or directory (2)  
Password:  
@ERROR: auth failed on module backup
rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6]  

The first error is the wrong path --password-file = / etc / rsync.password
second error is the server password file and writing backwards client, server written account and password, the client write-only password; there are passwod file server root, backup files the user is rsync; there is a problem misspell a word has led to the validation fails