vertica backup and restore

I've been using vertica recently, but I can't find a lot of information on the Internet. I only have to look at the official documentation to do it myself. Today, I have done a backup and recovery of vertica. The following is the process of sorting out and sharing it with you. If you have any questions, please correct me~ You can send it. Email to [email protected] or add communication: 412191494

1. vertica backup

1.1 vertica backup configuration:

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py  --setupconfig
Snapshot name (backup_snapshot): full_backup20160505
Number of restore points (1): 1
Specify objects (no default): 
Object restore mode (coexist, createOrReplace or create) (createOrReplace): 
Vertica user name (dbadmin): dbadmin
Save password to avoid runtime prompt? (n) [y/n]: y
Database user password to save in vbr password config file (no default):
Node v_windy1990_node0001
Backup host name (no default): 192.168.231.110
Backup directory (no default): /home/dbadmin/backup
Change advanced settings? (n) [y/n]: y
Temp directory (/tmp/vbr): 
Number of times to retry (2): 
Seconds between retry attempts (1): 
Encrypt data during transmission? (n) [y/n]: 
Use checksum for data integrity (not file data and size)? (n) [y/n]: 
Port number for rsync daemon (50000): 
User name to access rsync daemon (no default): 
Password of the user who accesses rsync daemon:
Backup transfer bandwidth limit in KBps or 0 for unlimited (0): 
Number of concurrency for backup (1): 
Restore transfer bandwidth limit in KBps or 0 for unlimited (0): 
Number of concurrency for restore (1): 
Password file name (no default): password
Saved vbr password to password.
Config file name (full_backup20160505.ini): 
Saved vbr config to full_backup20160505.ini.

After the backup is successful, check the contents of the backed up files as follows:

[dbadmin@localhost ~]$ more full_backup20160505.ini 
[Misc]
snapshotName = full_backup20160505
restorePointLimit = 1
objectRestoreMode = createOrReplace
tempDir = /tmp/vbr
retryCount = 2
retryDelay = 1
passwordFile = password

[Database]
dbName = Windy1990
dbUser = dbadmin

[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
serviceAccessUser = None
total_bwlimit_backup = 0
concurrency_backup = 1
total_bwlimit_restore = 0
concurrency_restore = 1

[Mapping]
v_windy1990_node0001 = 192.168.231.110:/home/dbadmin/backup

In the backup, I chose a file to save the password:

/home/dbadmin/password

It contains the password of the data path:

[dbadmin@localhost ~]$ more password 
[Passwords]
dbPassword = dbadmin
1.2 vertica 全备份
官方文档：
A full backup is a complete copy of the database catalog, its schemas, tables, and other objects. It is a consistent image of the database at the time the backup occurred. You can use a full backup for disaster recovery to restore a damaged or incomplete database. 

Execute the backup command:

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task backup --config-file full_backup20160505.ini
Error: Manifest not found at rsync://[192.168.231.110]:50000/home/dbadmin/backup/ -- is the backup location initialized? Hint: Execute '--task init' first.
Backup FAILED.

It is found that the error is reported. According to the prompt, the path must be initialized first (personal understanding is to declare the path first)

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task init --config-file full_backup20160505.ini
Initializing backup locations.
Backup locations initialized.

Path declaration succeeded! Check that there is one more file under the backup path.

[dbadmin@localhost backup]$ ls -l
total 4
-rw-r--r--. 1 dbadmin verticadba 22 May  5 13:47 backup_manifest

Then execute the backup command again:

/opt/vertica/bin/vbr.py --task backup --config-file full_backup20160505.ini

You can see that the backup was successful this time:

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task backup --config-file full_backup20160505.ini
Starting backup of database Windy1990.
Participating nodes: v_windy1990_node0001.
Snapshotting database.
Snapshot complete.
Approximate bytes to copy: 60269241 of 60269241 total.
[==================================================] 100%
Copying backup metadata.
Finalizing backup.
Backup complete!

As we can see, the vertica database has been fully backed up. At this point we should be able to guess that the backed up data exists under the path /home/dbamin/backup we just declared:

[dbadmin@localhost backup]$ ll
total 12
-rw-r--r--. 1 dbadmin verticadba 1044 May  5 14:08 backup_manifest
drwxr-xr-x. 8 dbadmin verticadba 4096 May  5 14:07 Objects
drwxr-xr-x. 3 dbadmin verticadba 4096 May  5 14:08 Snapshots

You can see two more folders /Objects / Snapshots We open the Objects folder and we can see that our vertica database file is in this directory. The /home/dbadmin/backup/Snapshots/full_backup20160505_20160505_060600/v_windy1990_node0001 directory stores the following information about the original vertica database we backed up.

Check the data flow for each node:

select node_name,sum(used_bytes) as size_in_bytes from v_monitor.storage_containers group by node_name;

1.3 The official document of vertica's incremental backup introduces full and incremental backups together, called Full and Incremental Backups. Here I list the incremental backups separately. restorePointLimit controls the number of incremental backup sets. I have only one node. I just set restorePointLimit = 1 when configuring the backup file, and execute the full backup command in 1.2 again to achieve incremental backup~

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task backup --config-file full_backup20160505.ini
Starting backup of database Windy1990.
Participating nodes: v_windy1990_node0001.
Snapshotting database.
Snapshot complete.
Approximate bytes to copy: 0 of 60269241 total.
[==================================================] 100%
Copying backup metadata.
Finalizing backup.
Backup complete!

Some people will ask, how do I know that this is an incremental backup instead of a full backup? ? This is a good question. I was thinking about it when I was backing up. After the backup was complete, I opened the backup folder again and saw:

[dbadmin@localhost ~]$ cd backup/
[dbadmin@localhost backup]$ ll
total 12
-rw-r--r--. 1 dbadmin verticadba 1103 May  5 14:36 backup_manifest
drwxr-xr-x. 8 dbadmin verticadba 4096 May  5 14:07 Objects
drwxr-xr-x. 4 dbadmin verticadba 4096 May  5 14:36 Snapshots

When you look at the Objects directory, because I did not operate on the vertica database during this process, the vertica database file is still the original and has not been updated. So it is very simple to know that what we are doing above is incremental. I insert a piece of data into any table in the vertica database, and then I execute the above backup script. I insert a phone number:

dbadmin=> insert into tb_serv_num values('13488888888','qin**');
 OUTPUT 
--------
      1
(1 row)

dbadmin=> commit;
COMMIT
dbadmin=> \q

Then re-execute the backup statement, and check the contents of the backup file after execution:

[dbadmin@localhost ~]$ cd backup/
[dbadmin@localhost backup]$ ll
total 12
-rw-r--r--. 1 dbadmin verticadba 1171 May  5 14:46 backup_manifest
drwxr-xr-x. 9 dbadmin verticadba 4096 May  5 14:46 Objects
drwxr-xr-x. 4 dbadmin verticadba 4096 May  5 14:46 Snapshots

Looking at the file time, does it mean that our reasoning above is correct?~~~ So, don't doubt, our incremental backup is no problem! 1.4 vertica object-level backup official website documentation: The database administrator user can create object-level backups consisting of one or more schemas and tables. Object-level backups are especially useful for multi-tenanted database sites. In fact, object-level backup is similar to full backup. Knowledge differs when generating backup configuration files (note the difference):

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --setupconfigSnapshot name (backup_snapshot): backup_object_tb_serv
Number of restore points (1): 
Specify objects (no default): tb_serv_num
Object restore mode (coexist, createOrReplace or create) (createOrReplace): 
Vertica user name (dbadmin): 
Save password to avoid runtime prompt? (n) [y/n]: y
Database user password to save in vbr password config file (no default):
Node v_windy1990_node0001
Backup host name (no default): 192.168.231.110
Backup directory (no default): /home/dbadmin/backup
Change advanced settings? (n) [y/n]: 
Password file name (no default): password
Saved vbr password to password.
Config file name (backup_object_tb_serv.ini): 
Saved vbr config to backup_object_tb_serv.ini.

Then execute the backup command:

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task backup --config-file backup_object_tb_serv.ini 
Starting backup of database Windy1990.
Objects: ['tb_serv_num']
Participating nodes: v_windy1990_node0001.
Snapshotting database.
Snapshot complete.
Approximate bytes to copy: 0 of 244 total.
[==================================================] 100%
Copying backup metadata.
Finalizing backup.
Backup complete!

Backup completed~~ 1.5 vertica Hard Link Local backup official documentation: A hard link local backup is a full or object-level backup consisting of a complete copy of the database catalog, and a set of hard file links to corresponding data files. Direct Just modify the configuration file of the full backup and add a line:

[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
serviceAccessUser = None
total_bwlimit_backup = 0
concurrency_backup = 1
total_bwlimit_restore = 0
concurrency_restore = 1
hardLinkLocal = True

Then execute the backup command:

/opt/vertica/bin/vbr.py --task backup --config-file HardLinkLocal_backup20160505.ini
[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task backup --config-file HardLinkLocal_backup20160505.ini 
Starting backup of database Windy1990.
Participating nodes: v_windy1990_node0001.
Snapshotting database.
Snapshot complete.
Approximate bytes to copy: 0 of 60269355 total.
[==================================================] 100%
Copying backup metadata.
Finalizing backup.
Backup complete!

1.6 View backup View the contents of backup:

/opt/vertica/bin/vbr.py --task listbackup --config-file HardLinkLocal_backup20160505.ini
[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task listbackup --config-file HardLinkLocal_backup20160505.ini 
backup                                epoch   objects   hosts(nodes)                            file_system_type  
full_backup20160505_20160505_074850   16                v_windy1990_node0001(192.168.231.110)   [Linux]           
full_backup20160505_20160505_064640   16                v_windy1990_node0001(192.168.231.110)   [Linux]   

The backup information can be queried in the vertica database:

dbadmin=> select * from database_backups;
       backup_timestamp        |      node_name       |     snapshot_name     | backup_epoch | node_count | file_system_type |   objects   
-------------------------------+----------------------+-----------------------+--------------+------------+------------------+-------------
 2016-05-05 14:08:03.369797+08 | v_windy1990_node0001 | full_backup20160505   |           15 |          1 | [Linux]          | 
 2016-05-05 14:36:39.496688+08 | v_windy1990_node0001 | full_backup20160505   |           15 |          1 | [Linux]          | 
 2016-05-05 14:42:35.517465+08 | v_windy1990_node0001 | full_backup20160505   |           15 |          1 | [Linux]          | 
 2016-05-05 14:44:43.043536+08 | v_windy1990_node0001 | full_backup20160505   |           15 |          1 | [Linux]          | 
 2016-05-05 14:46:57.958863+08 | v_windy1990_node0001 | full_backup20160505   |           16 |          1 | [Linux]          | 
 2016-05-05 15:03:19.580159+08 | v_windy1990_node0001 | backup_object_tb_serv |           16 |          1 | [Linux]          | tb_serv_num
 2016-05-05 15:50:53.143446+08 | v_windy1990_node0001 | full_backup20160505   |           16 |          1 | [Linux]          | 
(7 rows)

2, vertica recovery

2.1 vertica recovery

In order to see that recovery is useful, again I destroy the data, drop a table, delete a piece of data

dbadmin=> select * from tb_rp_st_user_day ;
 serv_number | user_name 
-------------+-----------
(0 rows)

dbadmin=> drop table tb_rp_st_user_day;
DROP TABLE

dbadmin=> select * from tb_serv_num where name='ss';
  serv_num   | name 
-------------+------
 186371***** | ss
(1 row)

dbadmin=> delete from tb_serv_num where name='ss';
 OUTPUT 
--------
      1
(1 row)

dbadmin=> commit;
COMMIT

Start executing the restore command:

/opt/vertica/bin/vbr.py --task restore --config-file full_backup20160505.ini
[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task restore --config-file full_backup20160505.ini Error: Full database restore requires the nodes be DOWN.
Restore FAILED.

Report an error~ Prompt that it can only be restored when the vertica database is down...

Check out the official documentation:

To restore a full database backup, you must ensure that:
•   The database is down. You cannot restore a full backup when the database is running.
•   All of the backup hosts are up and available.
•   The backup directory exists and contains the backups from which to restore.
•   The cluster to which you are restoring the backup has the same number of hosts as the one used to create the backup. The node names and the IP addresses must also be identical.
•   The database you are restoring must already exist on the cluster to which you are restoring data. The database can be completely empty without any data or schema. As long as the database name matches the name in the backup, and all of the node names match the names of the nodes in the configuration file, you can restore to it.
•   The user performing the restore is the database administrator.

So here, let's try the second recovery method (object-level recovery) to recover data. We just made an object-level backup of the tb_serv_num table at the same time:

To perform an object-level restore:

/opt/vertica/bin/vbr.py --task restore --config-file backup_object_tb_serv.ini
[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task restore --config-file backup_object_tb_serv.ini 
Starting object restore of database Windy1990.
Participating nodes: v_windy1990_node0001.
Objects to restore: tb_serv_num.
Restoring from restore point: backup_object_tb_serv_20160505_070249
Loading snapshot catalog from backup.
Syncing data from backup to cluster nodes.
[==================================================] 100%
Finalizing restore.
Restore complete!

OK~ Seeing that we have recovered successfully above, then board the vertica database to verify whether the data is recovered.

dbadmin=> select * from tb_serv_num ;
  serv_num   | name  
-------------+-------
 138***** | windy
 186***** | ss
 134***** | qin**
(3 rows)

You can see that the data has been restored~, wait, we dropped a table just now, is this table restored?

dbadmin=> \d tb_rp_st_user_day;
Did not find any relation.

As you can see, the table is not restored, why? You must know that our recovery is an object-level recovery. Our object-level backup only backed up the data of the tb_serv_num table. Free recovery can only restore all the data in the table. Then we use the third Hard Link Local method to restore data:

/opt/vertica/bin/vbr.py --task restore --config-file HardLinkLocal_backup20160505.ini
[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task restore --config-file HardLinkLocal_backup20160505.ini 
Error: Full database restore requires the nodes be DOWN.
Restore FAILED.

Also see that the vertica database also needs to be DOWN

It seems to stop the vertica database~ (stop the database, omitted here)
After stopping the vertica database, try full recovery again:

[dbadmin@localhost ~]$ /opt/vertica/bin/vbr.py --task restore --config-file full_backup20160505.ini 
Starting full restore of database Windy1990.
Participating nodes: v_windy1990_node0001.
Restoring from restore point: full_backup20160505_20160505_074850
Computing the size of data to be synced from backup to cluster nodes.
Syncing data from backup to cluster nodes.
[==================================================] 100%
Restoring catalog.
Restore complete!

As you can see, the recovery is not bad.
Restart vertica, and check whether the table tb_rp_st_user_day is restored after logging in to vertica:

dbadmin=> \d tb_rp_st_user_day;
                                            List of Fields by Tables
 Schema |       Table       |   Column    |    Type     | Size | Default | Not Null | Primary Key | Foreign Key 
--------+-------------------+-------------+-------------+------+---------+----------+-------------+-------------
 public | tb_rp_st_user_day | serv_number | varchar(13) |   13 |         | f        | f           | 
 public | tb_rp_st_user_day | user_name   | varchar(50) |   50 |         | f        | f           | 
(2 rows)

As you can see, the recovery was successful.

That's it for vertica backups and simple restores.

Through this vertica recovery, we can see that many problems can be solved in the official documentation. When stopping and starting vertica, I encountered a little problem, but the solution can be seen according to the error prompt.

<script type="text/javascript"> $(function () { $('pre.prettyprint code').each(function () { var lines = $(this).text().split('\n').length; var $numbering = $('<ul/>').addClass('pre-numbering').hide(); $(this).addClass('has-numbering').parent().append($numbering); for (i = 1; i <= lines; i++) { $numbering.append($('<li/>').text(i)); }; $numbering.fadeIn(1700); }); }); </script>