Mongodb backup and recovery to any point in time

description:

According to the technical feedback, it is someone accidently delete a line mongodb table data, user feedback to customer service, and now need to restore user data.

First determine the data to be deleted database schema is what model

1.1, a single point:
Is there a backup with standby situations, start a instance, will import the backup data to the new instance
to find a new instance has been deleted data using mongodump export data, and then imported into the source of deleted data instances ,Data recovery.
See details below in this chapter: 2.2 2.5 Backup and restore
a single node mongodb no oplog concept, if there is no backup, data will be lost, try to use the formal environment copied set architecture.

1.2, replication set:
1.2.1, may take the form of a delay node, a node PRIMARY, SECONDARY two of which make a delay node,
such as node latency delay us eight hours, we found the data to be deleted within 8 hours, and recovery (guarantee oplog not covered by the lower case)
1.2.2, the total amount + delta: the amount of data and the importance of selecting the amount of backup + Vincent oplog incremental backup hours, days or comprehensive quantity incremental backup + oplog backup.
Mongodb backup and recovery to any point in time
1.3, cluster fragmentation
fragment are a cluster composed of a plurality of copy sets, backup copy can be set in accordance with the backup policy

2. Copy set architecture simulation accidentally deleted:

We use the full backup + oplog incremental backup
2.1, we insert 1000 Data: for (var i = 0; i <1000; i ++) {db.user.save ({ "userid": i})}

[root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongo  10.1.1.159:27020
rs02:PRIMARY> use jia_test
switched to db jia_test
rs02:PRIMARY> for (var i=0;i<1000;i++){ db.user.save({"userid":i})  }
rs02:PRIMARY> db.user.count()
1000
rs02:PRIMARY> db.user.find()
{ "_id" : ObjectId("5e69f820ce1bb2285a21a343"), "userid" : 0 }
{ "_id" : ObjectId("5e69f820ce1bb2285a21a344"), "userid" : 1 }
{ "_id" : ObjectId("5e69f820ce1bb2285a21a345"), "userid" : 2 }
{ "_id" : ObjectId("5e69f820ce1bb2285a21a346"), "userid" : 3 }

2.2, we make a full backup:

[root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongodump -h 10.1.1.159:27020 -d jia_test -o /root/
2020-03-12T16:52:37.418+0800    writing jia_test.user to
2020-03-12T16:52:37.421+0800    done dumping jia_test.user (1000 documents)

2.3, the first case:
If we delete operation in a few hours after the backup, as long as the imported data to the new instance will be able to retrieve the data, this is too simple.
The premise is that we have a backup to recover.
2.4, we continue to the second case

We continue to insert data:

[root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongo  10.1.1.159:27020
rs02:PRIMARY> use jia_test
rs02:PRIMARY> db.user.insert({"userid" :1000})
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY> db.user.insert({"userid" :1001})
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY> db.user.insert({"userid" :1002})           
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY> db.user.insert({"userid" :1003})          
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY> db.user.insert({"userid" :1004})
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY> db.user.insert({"userid" :1005})           #1、我们又插入了三条数据         
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY> db.user.remove({"userid" :1003})           #2、我们删除了一条数据       
WriteResult({ "nRemoved" : 1 })
rs02:PRIMARY> db.user.insert({"userid" :1006})            #3、我们又插入一条数据    
WriteResult({ "nInserted" : 1 })
rs02:PRIMARY>

Let's restore the "userid": before this point in time: ({1003 "userid"}) 1003 of this data, we need a moment, that is, to restore deleted before the execution db.user.remove recovery.

2.5, we create a new replica set an example, the full backup import

[root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongorestore -h 10.1.1.77:27010 -d jia_test /root/jia_test/
2020-03-12T17:03:27.054+0800    the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2020-03-12T17:03:27.055+0800    building a list of collections to restore from /root/jia_test dir
2020-03-12T17:03:27.056+0800    reading metadata for jia_test.user from /root/jia_test/user.metadata.json
2020-03-12T17:03:27.074+0800    restoring jia_test.user from /root/jia_test/user.bson
2020-03-12T17:03:27.099+0800    no indexes to restore
2020-03-12T17:03:27.099+0800    finished restoring jia_test.user (1000 documents)
2020-03-12T17:03:27.099+0800    done
/data/mongodb3.6.9/bin/mongo 10.1.1.77:27010
rs02:PRIMARY&gt; db.user.count()
1000
rs02:PRIMARY&gt;

2.6 Export oplog log:

### 首先我们查一下什么时间执行删除操作的、连上执行删除操作的数据实例
[root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongo  10.1.1.159:27020
rs02:PRIMARY> use local
switched to db local
rs02:PRIMARY> db.oplog.rs.find({"op":"d","ns":"jia_test.user"})                                               #op是操作，d是删除，ns命名空间就是某个库的某个表，oplogs日志解释详见oplog介绍
{ "ts" : Timestamp(1584003536, 1), "t" : NumberLong(1), "h" : NumberLong("5476681688775461425"), "v" : 2, "op" : "d", "ns" : "jia_test.user", "ui" : UUID("2527737e-eeea-415b-95de-061b6615a23c"), "wall" : ISODate("2020-03-12T08:58:56.056Z"), "o" : { "_id" : ObjectId("5e69f9b02e42450b5489dd4f") } }
rs02:PRIMARY>

Given: We full backup time: 2020-03-12T16: 52: 37.421 + 0800 done dumping jia_test.user (1000 documents)

Delete time also know that we need the full amount of time the backup was successful, what were the conversion, we get a time stamp,

Try the backup time in ten minutes earlier, we would not have the missing logs, import duplicates automatically overwritten.
Mongodb backup and recovery to any point in time
We got two timestamps:

Backup timestamp: 1584002557
Delete timestamp: "ts": Timestamp (1584003536 , 1)

Exporting data instances oplog log deleted (deletion time is greater than the backup time is less than this period)

 [root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongodump -h  10.1.1.159:27020 -d local -c oplog.rs -q '{ts:{$gt:Timestamp(1584002557, 1),$lt: Timestamp(1584003536, 1)},"ns":{"$regex":"jia_test.user"}}' -o .

2.7, oplog into a new instance:

 [root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongorestore -h 10.1.1.77:27010 –oplogReplay local/oplog.rs.bson

2.8, connect the new recovery examples see:

[root@10-1-1-77 ~]# /data/mongodb3.6.9/bin/mongo 10.1.1.77:27010
rs02:PRIMARY> use jia_test
switched to db jia_test
rs02:PRIMARY> db.user.count()
1006
rs02:PRIMARY> db.user.find({"userid":1003})                                                    #查看1003数据存在             
{ "_id" : ObjectId("5e69f9b02e42450b5489dd4f"), "userid" : 1003 }
rs02:PRIMARY> db.user.find({"userid":1006})                                                     #查看1006数据不存在，因为这条数据是在删除之后插入的，我们恢复的时间点是删除之前的。
rs02:PRIMARY>

2.9, the old instance of View:

[root@10-1-1-159 ~]# /data/mongodb3.6.9/bin/mongo  10.1.1.159:27020
rs02:PRIMARY> use jia_test
switched to db jia_test
rs02:PRIMARY> db.user.count()
1006
rs02:PRIMARY> db.user.find({"userid" :1003})                                                #1003数据不存在
rs02:PRIMARY> db.user.find({"userid" :1006})                                               #1006数据存在，删除只要又写入的                                   
{ "_id" : ObjectId("5e69f9d42e42450b5489dd52"), "userid" : 1006 }  
rs02:PRIMARY>

2.10 We put a new instance of the data by exporting, importing into the online environment can be a formal

Tips: Mongodb official business, at least use a replication set, not a single point, the data must be backed up.

Some time ago a company deletion of data resulting in the loss of hundreds of millions,