Recently, a customer encountered a data file corruption caused by a power outage without a backup of a single-instance mongodb database. Due to the customer’s business needs
And the insensitivity of data, it is required to restore the business as soon as possible, using Mongdb's automatic repair repair command to repair. Fortunately, to help users do
The service has been restored soon, and the sad thing is that the data in the relevant data file is lost under the condition that the customer can accept it. Here, to make a summary of this process, and explain
Why is the data lost after repair.
-
Normal mongodb data query
> show dbs;
admin 0.000GB
config 0.000GB
dns_testdb 0.009GB
local 0.000GB
> use dns_testdb
switched to db dns_testdb
> db.test_collection.find();
{ "_id" : ObjectId("5fedd03d9d2569ee04ab62e1"), "name" : "elephant", "user_id" : 0, "boolean" : false, "added_at" : ISODate("2020-12-31T13:21:01.226Z"), "number" : 5129 }
{ "_id" : ObjectId("5fedd03d9d2569ee04ab62e2"), "name" : "dog", "user_id" : 1, "boolean" : false, "added_at" : ISODate("2020-12-31T13:21:01.237Z"), "number" : 9699 }
{ "_id" : ObjectId("5fedd03d9d2569ee04ab62e3"), "name" : "lion", "user_id" : 2, "boolean" : false, "added_at" : ISODate("2020-12-31T13:21:01.238Z"), "number" : 1783 }
Type "it" for more
> 2. The simulation data file is damaged
[mongo@centos7 dns_testdb]$ du -sh *
28M collection-8--6736947369024546614.wt
9.5M index-9--6736947369024546614.wt
[mongo@centos7 dns_testdb]$
[mongo@centos7 dns_testdb]$
[mongo@centos7 dns_testdb]$ pwd
/opt/mongo/data/single/dns_testdb
[mongo@centos7 dns_testdb]$ dd if=/dev/null of=/opt/mongo/data/single/dns_testdb/collection-8--6736947369024546614.wt bs=1024k count=5
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000132203 s, 0.0 kB/s
[mongo@centos7 dns_testdb]$3. Restart mongodb
> use admin
switched to db admin
> db.shutdownServer();
[mongo@centos7 data]$ mongod --dbpath /opt/mongo/data/single --port 50001 --oplogSize 512 --fork --bind_ip 0.0.0.0 --logpath /opt/mongo/logs/single.log --logappend --journal --directoryperdb --profile=1
about to fork child process, waiting until server is ready for connections.
forked process: 102882
child process started successfully, parent exiting4. Although the mongodb process can be started, the data collection after the data file is damaged will cause mongod to hang
[mongo@centos7 data]$ mongo --port 50001
MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:50001/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("09b6c6aa-059d-4a41-9e0d-e6553966399b") }
MongoDB server version: 4.2.3
Server has startup warnings:
> show dbs;
admin 0.000GB
config 0.000GB
dns_testdb 0.037GB
local 0.000GB
> use dns_testdb;
switched to db dns_testdb
> db.test_collection.find();
2020-12-31T08:43:45.115-0500 I NETWORK [js] DBClientConnection failed to receive message from 127.0.0.1:50001 - HostUnreachable: Connection closed by peer
Error: error doing query: failed: network error while attempting to run command 'find' on host '127.0.0.1:50001'
2020-12-31T08:43:45.118-0500 I NETWORK [js] trying reconnect to 127.0.0.1:50001 failed
2020-12-31T08:43:45.118-0500 I NETWORK [js] reconnect 127.0.0.1:50001 failed failed
> 5. Observe the mongodb log, suggesting that the data file is damaged and it is recommended to use repair to repair
2020-12-31T08:43:45.103-0500 E STORAGE [conn1] WiredTiger error (-31802) [1609422225:103947][102882:0x7f96713b5700], file:dns_testdb/collection-8--6736947369024546614.wt, WT_SESSION.open_cursor: __desc_read, 351: dns_testdb/collection-8--6736947369024546614.wt does not appear to be a WiredTiger file: WT_ERROR: non-specific WiredTiger error Raw: [1609422225:103947][102882:0x7f96713b5700], file:dns_testdb/collection-8--6736947369024546614.wt, WT_SESSION.open_cursor: __desc_read, 351: dns_testdb/collection-8--6736947369024546614.wt does not appear to be a WiredTiger file: WT_ERROR: non-specific WiredTiger error
2020-12-31T08:43:45.104-0500 E STORAGE [conn1] Failed to open a WiredTiger cursor. Reason: UnknownError: -31802: WT_ERROR: non-specific WiredTiger error, uri: table:dns_testdb/collection-8--6736947369024546614, config:
2020-12-31T08:43:45.104-0500 E STORAGE [conn1] This may be due to data corruption. Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
2020-12-31T08:43:45.104-0500 F - [conn1] Fatal Assertion 50882 at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 101
2020-12-31T08:43:45.104-0500 F - [conn1]
***aborting after fassert() failure6. Follow the mongod log to repair the database
[mongo@centos7 data]$ mongod --dbpath /opt/mongo/data/single --port 50001 --oplogSize 512 --fork --bind_ip 0.0.0.0 --logpath /opt/mongo/logs/single.log --logappend --journal --directoryperdb --profile=1 --repair
about to fork child process, waiting until server is ready for connections.
forked process: 102942
child process started successfully, parent exiting
[mongo@centos7 data]$ 7. During the repair process, the mongod log prompts that the damaged data collection and index are rebuilt
2020-12-31T08:44:45.646-0500 I STORAGE [initandlisten] repairDatabase dns_testdb
2020-12-31T08:44:45.646-0500 I STORAGE [initandlisten] Repairing collection dns_testdb.test_collection
2020-12-31T08:44:45.647-0500 E STORAGE [initandlisten] WiredTiger error (-31802) [1609422285:647413][102942:0x7fca99ec8c40], file:dns_testdb/collection-8--6736947369024546614.wt, WT_SESSION.verify: __desc_read, 351: dns_testdb/collection-8--6736947369024546614.wt does not appear to be a WiredTiger file: WT_ERROR: non-specific WiredTiger error Raw: [1609422285:647413][102942:0x7fca99ec8c40], file:dns_testdb/collection-8--6736947369024546614.wt, WT_SESSION.verify: __desc_read, 351: dns_testdb/collection-8--6736947369024546614.wt does not appear to be a WiredTiger file: WT_ERROR: non-specific WiredTiger error
2020-12-31T08:44:45.647-0500 I STORAGE [initandlisten] Verify failed on uri table:dns_testdb/collection-8--6736947369024546614. Running a salvage operation.
2020-12-31T08:44:45.647-0500 E STORAGE [initandlisten] WiredTiger error (-31802) [1609422285:647930][102942:0x7fca99ec8c40], file:dns_testdb/collection-8--6736947369024546614.wt, WT_SESSION.salvage: __desc_read, 351: dns_testdb/collection-8--6736947369024546614.wt does not appear to be a WiredTiger file: WT_ERROR: non-specific WiredTiger error Raw: [1609422285:647930][102942:0x7fca99ec8c40], file:dns_testdb/collection-8--6736947369024546614.wt, WT_SESSION.salvage: __desc_read, 351: dns_testdb/collection-8--6736947369024546614.wt does not appear to be a WiredTiger file: WT_ERROR: non-specific WiredTiger error
2020-12-31T08:44:45.648-0500 W STORAGE [initandlisten] Salvage failed for uri table:dns_testdb/collection-8--6736947369024546614: Salvage failed: -31802: WT_ERROR: non-specific WiredTiger error. The file will be moved out of the way and a new ident will be created.
2020-12-31T08:44:45.648-0500 W STORAGE [initandlisten] Moving data file /opt/mongo/data/single/dns_testdb/collection-8--6736947369024546614.wt to backup as /opt/mongo/data/single/dns_testdb/collection-8--6736947369024546614.wt.corrupt
2020-12-31T08:44:45.648-0500 W STORAGE [initandlisten] Rebuilding ident dns_testdb/collection-8--6736947369024546614
2020-12-31T08:44:45.708-0500 I STORAGE [initandlisten] Successfully re-created table:dns_testdb/collection-8--6736947369024546614.
2020-12-31T08:44:45.718-0500 I INDEX [initandlisten] index build: starting on dns_testdb.test_collection properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "dns_testdb.test_collection" } using method: Foreground
2020-12-31T08:44:45.718-0500 I INDEX [initandlisten] build may temporarily use up to 200 megabytes of RAM
2020-12-31T08:44:45.718-0500 I STORAGE [initandlisten] Index build initialized: 2ddee833-ea97-4964-98c0-7137e71a99c9: dns_testdb.test_collection: indexes: 1
2020-12-31T08:44:45.722-0500 I STORAGE [initandlisten] Index builds manager starting: 2ddee833-ea97-4964-98c0-7137e71a99c9: dns_testdb.test_collection
2020-12-31T08:44:45.724-0500 I INDEX [initandlisten] index build: inserted 0 keys from external sorter into index in 0 seconds
2020-12-31T08:44:45.727-0500 I INDEX [initandlisten] index build: done building index _id_ on ns dns_testdb.test_collection
2020-12-31T08:44:45.727-0500 I STORAGE [initandlisten] Index builds manager completed successfully: 2ddee833-ea97-4964-98c0-7137e71a99c9: dns_testdb.test_collection. Index specs requested: 1. Indexes in catalog before build: 1. Indexes in catalog after build: 18. Restart mongod service after repair
[mongo@centos7 data]$ mongod --dbpath /opt/mongo/data/single --port 50001 --oplogSize 512 --fork --bind_ip 0.0.0.0 --logpath /opt/mongo/logs/single.log --logappend --journal --directoryperdb --profile=1
about to fork child process, waiting until server is ready for connections.
forked process: 102975
child process started successfully, parent exiting
[mongo@centos7 data]$ 9. After the mongod service is started, the service accepts normal data queries, but after the repair, the collection data that is damaged by the data file has been lost
[mongo@centos7 data]$ mongo --port 50001
MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:50001/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("d88894c4-16bf-4013-a993-d29e2493fbdf") }
MongoDB server version: 4.2.3
Server has startup warnings:
> show dbs;
admin 0.000GB
config 0.000GB
dns_testdb 0.000GB
local 0.000GB
> use dns_testdb;
switched to db dns_testdb
> db.test_collection.find();
> 10. Summary
The mongodb database repair command repair, if there is no backup and the data file is damaged, it will cause all the collection data related to the damaged data file to be lost.
However, after the repair, it will not prevent the mongod service from starting normally. Combined with the log of the modification process, it is not difficult to see that repair has performed on damaged data files and related collections.
The index file is rebuilt, and the rebuilt data file and collection file are re-initialized, so the data is lost. So, using mongodb database, the most
Use the replica set of mongodb as a data redundancy security strategy in a reasonable manner. While using the replica set of mongodb, you can also be a delayed synchronization node to prevent
Misoperation.