Foreword:
-
The security of the database needs to be guaranteed, so the local backup strategy is obviously not safe. If there is a sudden damage to the local disk, the database will not be restored.
Therefore, remote backup, that is, copying backup files to a remote server, is a necessary measure.
- Comparison of Several Schemes of Remote Backup Strategy
For the specific implementation of remote backup, we can use many tools, among which the more efficient ones are Regeneron, scp, rsync and pg_dump commands to remotely connect to the database for backup.
So, which of these tools is better to use?
- 1. The configuration of Reborn Dragon is more complicated, and there are more intrusions into the system. Therefore, this tool is not suitable for our scenario.
- 2. The Scp command needs to configure the server to be password-free or use the server’s password in plain text, which will damage the security of the server. Therefore, scp is not suitable for our scenario. ,
- 3. The Pg_dump command also requires the privileged account and password of the postgresql database, which is basically in plain text, and has certain damage to the security of the server. damage.
- 4. Rsync remote backup
As a professional data synchronization software, Rsync supports incremental synchronization. Therefore, this tool is very suitable for remote backup. This tool has the characteristics of simple configuration, guaranteed security (password can be set), and high backup efficiency.
one,
Realization case of remote backup of database
It is planned to use two servers, one server is installed postgresql, the IP address of this server is 192.168.123.60, the role in the remote backup is client, and the push data
The server only backs up the single library test3:
postgres=# \l+
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
-----------+----------+----------+-------------+-------------+-------------------+----------+------------+--------------------------------------------
postgres | pg1 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 10001 kB | pg_default | default administrative connection database
template0 | pg1 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/pg1 +| 7809 kB | pg_default | unmodifiable empty database
| | | | | pg1=CTc/pg1 | | |
template1 | pg1 | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/pg1 +| 7809 kB | pg_default | default template for new databases
| | | | | pg1=CTc/pg1 | | |
test | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 7817 kB | mytbs |
test2 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 7809 kB | pg_default |
test3 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 7817 kB | pg_default |
(6 rows)
postgres=# \dt
List of relations
Schema | Name | Type | Owner
--------+------+-------+----------
public | ssss | table | postgres
(1 row)
The backup single library file is test3.dump file:
[pg1@EULER1 ~]$ pg_dump -Upostgres -d test3 -Fc -f test3.dump
[pg1@EULER1 ~]$ pg_restore -l test3.dump
;
; Archive created at 2023-06-10 10:51:53 CST
; dbname: test3
; TOC Entries: 7
; Compression: -1
; Dump Version: 1.14-0
; Format: CUSTOM
; Integer: 4 bytes
; Offset: 8 bytes
; Dumped from database version: 12.5
; Dumped by pg_dump version: 12.5
;
;
; Selected TOC Entries:
;
202; 1259 49157 TABLE public test3 postgres
3086; 0 49157 TABLE DATA public test3 postgres
2959; 2606 49161 CONSTRAINT public test3 test3_pkey postgres
The other server only has rsync installed, the server’s IP address is 192.168.123.61, and its role in the remote backup is the server, which accepts data, that is to say, the local backup files and physical files on the 60 server are pushed to this server To save, so as to achieve the purpose of remote backup.
two,
Implementation steps of Rsync remote backup
1，
Execute the following on 61 to install the rsync server that receives backup files:
Install rsync, if it is a minimal installation, usually there is no rsync, you need yum installation, yum install rsync -y
2，
Edit /etc/sysconfig/rsyncd, the content is as follows:
OPTIONS="/etc/rsyncd.conf"3，
Edit /etc/rsyncd.conf, the content is as follows:
uid = root
gid = root
port = 11873
use chroot = yes
max connections = 4
hosts allow = *
pid file = /var/run/rsyncd.pid
log file = /var/log/rsyncd/rsyncd.log
lock file =/var/run/rsync.lock
exclude = lost+found/
transfer logging = yes
timeout = 900
ignore nonreadable = yes
dont compress = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2
[pgsql_danku]
path = /data/pgsql_danku
comment = pgsql danku
read only = no
auth users = rsync
secrets file = /etc/rsyncd.passwd
[pgsql_all]
path = /data/pgsql_all
comment = pgsql all
read only = no
auth users = rsync
secrets file = /etc/rsyncd.passwd4. Generation of log files
mkdir -p /var/log/rsyncd/ && touch /var/log/rsyncd/rsyncd.log && chmod 755 /var/log/rsyncd/rsyncd.log5. Generate account and password save files according to the configuration file /etc/rsyncd.conf
vim /etc/rsyncd.passwd #The content of the file is as follows:
rsync:你要设定的密码，客户端60使用的。6. Set the account password file permissions to 600
chmod 600 /etc/rsyncd.passwd7. According to the configuration file /etc/rsyncd.conf, generate a directory for storing synchronization and backup files or directories
[root@centos61 ~]# mkdir -p /data/pgsql_danku
[root@centos61 ~]# mkdir -p /data/pgsql_all
8. Start the service and join the startup to check the service status
systemctl enable rsyncd && systemctl start rsyncd && systemctl status rsyncdOn the server 60, that is, the server where the backup file test3.dump is stored, the push is performed:
vim /etc/rsync.passwd
Password, which is consistent with the password set on the server side, only write the password without writing anything
chmod 600 /etc/rsync.passwd
Write a push script (fill in according to the actual situation):
#!/bin/bash
#!auther zsk
rsync -avz --port=11873 /home/pg1/test3.dump [email protected]::pgsql_danku --password-file=/etc/rsync.passwd
Execute this push script, the general output is as follows:
[root@EULER1 ~]# bash rsync_pgsql.sh
sending incremental file list
./
.bash_history
.bash_logout
.bash_profile
.bashrc
.psql_history
.viminfo
logfile
test3.dump
sent 7,808 bytes received 170 bytes 15,956.00 bytes/sec
total size is 50,944 speedup is 6.39
#### Note: The configuration file of the server can not use Windows, otherwise it will report an error:
[root@EULER1 ~]# bash rsync_pgsql.sh
@ERROR: no path setting.
rsync error: error starting client-server protocol (code 5) at main.c(1648) [sender=3.1.2]
View the log of the rsyncd service:
2023/06/10 11:05:55 [2964] No path specified for module pgsql_danku
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: "dont compress "
2023/06/10 11:06:59 [2967] IGNORING unknown parameter "dont compress "
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " path"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " path"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " comment"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " comment"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " read only"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " read only"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " auth users"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " auth users"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " secrets file"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " secrets file"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " comment"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " comment"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " read only"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " read only"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " auth users"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " auth users"
2023/06/10 11:06:59 [2967] Unknown Parameter encountered: " secrets file"
2023/06/10 11:06:59 [2967] IGNORING unknown parameter " secrets file"
2023/06/10 11:06:59 [2967] name lookup failed for 192.168.123.60: Name or service not known
2023/06/10 11:06:59 [2967] connect from UNKNOWN (192.168.123.60)
2023/06/10 11:06:59 [2967] No path specified for module pgsql_danku
2023/06/10 11:10:22 [2958] sent 0 bytes received 0 bytes total size 0
2023/06/10 11:10:22 [2984] rsyncd version 3.0.9 starting, listening on port 11873
2023/06/10 11:10:26 [2985] Unknown Parameter encountered: " read only"
2023/06/10 11:10:26 [2985] IGNORING unknown parameter " read only"
2023/06/10 11:10:26 [2985] Unknown Parameter encountered: " auth users"
2023/06/10 11:10:26 [2985] IGNORING unknown parameter " auth users"
2023/06/10 11:10:26 [2985] Unknown Parameter encountered: " secrets file"
2023/06/10 11:10:26 [2985] IGNORING unknown parameter " secrets file"
Therefore, on the 61 server, re-type the space in /etc/rsyncd.conf, and then restart the rsyncd service to return to normal.
60 The physical files of the server are under the path /usr/local/pgsql/data
[root@EULER1 ~]# ps aux |grep postgres
root 3241 0.0 0.0 112724 992 pts/0 S+ 11:31 0:00 grep --color=auto postgres
pg1 79417 0.0 1.0 317636 41804 ? Ss 10:53 0:00 /usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
pg1 79418 0.0 0.0 126996 704 ? Ss 10:53 0:00 postgres: logger
pg1 79420 0.0 0.0 318028 3040 ? Ss 10:53 0:00 postgres: checkpointer
pg1 79421 0.0 0.0 317896 2336 ? Ss 10:53 0:00 postgres: background writer
pg1 79422 0.0 0.1 317636 4740 ? Ss 10:53 0:00 postgres: walwriter
pg1 79423 0.0 0.0 318912 1800 ? Ss 10:53 0:00 postgres: autovacuum launcher
pg1 79424 0.0 0.0 129272 1028 ? Ss 10:53 0:00 postgres: stats collector
pg1 79425 0.0 0.0 318740 1384 ? Ss 10:53 0:00 postgres: logical replication launcher
Add the following content to the push script of the 60 server:
#!/bin/bash
#!auther zsk
rsync -avz --port=11873 /home/pg1/ [email protected]::pgsql_danku --password-file=/etc/rsync.passwd
rsync -avz --port=11873 /usr/local/pgsql/data/ [email protected]::pgsql_all --password-file=/etc/rsync.passwd
Push the file to the server again:
。。。。。前面的略略略。。。。。。。。。
pg_stat_tmp/pgss_query_texts.stat
pg_subtrans/
pg_subtrans/0000
pg_tblspc/
pg_tblspc/32771 -> /opt/custome-tablespace
pg_twophase/
pg_wal/
pg_wal/00000003.history
pg_wal/00000004.history
pg_wal/000000040000000000000010
pg_wal/000000040000000000000011
pg_wal/archive_status/
pg_wal/archive_status/00000003.history.done
pg_xact/
pg_xact/0000
sent 6,189,113 bytes received 32,673 bytes 1,777,653.14 bytes/sec
total size is 78,376,166 speedup is 12.60
Go to server 61 to check the status of the remote backup:
[root@centos61 ~]# cd /data/pgsql_all/
[root@centos61 pgsql_all]# ls
backup_label.old pg_commit_ts pg_ident.conf pg_serial pg_tblspc postgresql.auto.conf
base pg_dynshmem pg_logical pg_snapshots pg_twophase postgresql.conf
current_logfiles pg_enterprise_views.explain pg_multixact pg_stat PG_VERSION postmaster.opts
global pg_enterprise_views.stat pg_notify pg_stat_tmp pg_wal postmaster.pid
log pg_hba.conf pg_replslot pg_subtrans pg_xact tablespace_map.old
[root@centos61 pgsql_all]# cd /data/pgsql_danku/
[root@centos61 pgsql_danku]# ll
total 40
-rw-------. 1 postgres postgres 35091 Jun 10 10:53 logfile
-rw-------. 1 postgres postgres 1517 Jun 10 10:51 test3.dump
Then, the /data/pgsql_all/ directory on the 61 server can be directly started as a database, but the lib directory and the bin directory are missing now.
After these two directories are ready, you can start a new postgresql with the backup file as data:
[root@centos61 pgsql_all]# su - postgres -c "pg_ctl -D /data/pgsql_all/ start"
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2023-06-10 12:39:46.401 CST 3090 @ from [vxid: txid:0] [] LOG: starting PostgreSQL 12.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16), 64-bit
2023-06-10 12:39:46.406 CST 3090 @ from [vxid: txid:0] [] LOG: listening on IPv4 address "0.0.0.0", port 5432
2023-06-10 12:39:46.406 CST 3090 @ from [vxid: txid:0] [] LOG: listening on IPv6 address "::", port 5432
2023-06-10 12:39:46.413 CST 3090 @ from [vxid: txid:0] [] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-06-10 12:39:46.920 CST 3090 @ from [vxid: txid:0] [] LOG: could not open directory "pg_tblspc/32771/PG_12_201909212": No such file or directory
2023-06-10 12:39:46.921 CST 3090 @ from [vxid: txid:0] [] LOG: redirecting log output to logging collector process
2023-06-10 12:39:46.921 CST 3090 @ from [vxid: txid:0] [] HINT: Future log output will appear in directory "log".
done
server started