Author: zhouyu
Links: cnblogs.com/zhouyu629/p/3734494.html
Experienced two days of unremitting efforts, finally resumed production server data misuse once deleted.
The process of the accident and solutions in this record, alert himself and others suggesting Mo made this mistake.
I hope the problems encountered friends can find a trace of inspiration to solve the problem.
01 accident background
Arrange a sister installed on a production server Oracle, sister edge research side installation, the installation does not feel ready to uninstall reinstall.
Find uninstall from the Internet, where the command line to be executed remove the Oracle installation directory, the command is as follows:
rm -rf $ORACLE_BASE/*
If ORACLE_BASE variable is not assigned, then the command becomes:
rm -rf /*
And so on, but my sister used Root account ah. In this way, the entire disk files deleted all, including the application of Tomcat, MySQL database and so on ......
MySQL database is not running? Linux can delete the file being executed? Because it is completely removed, and finally left a Tomcat of Log file, it is estimated that the file is too large, and sometimes not deleted successfully.
Watching sister remorse eyes, it is this thing that I arranged for her to do, did not tell her the stakes clear, without any training, liability can only be one person back, and besides how to make it beautiful bear this responsibility ?
A call to the room, hang the disk to another server, SSH upswing view all files are cleared, this server is running, but a customer's production system, ah, has been running for six months, was restored as soon as possible ah.
Then sent for offline backup of the database, find the backup file only 1KB, which is only a few lines of comment familiar mysqldump (backup script Crontab Is executed in question), the closest backup is in December 2013 of a really leaky house every night rain ah.
Think of it a leader said the case: When a production system hang, find all backups have questions, DVD also has scratches, tape drives is also bad (a senior industry, earlier estimates also do a backup CD-ROM ), I did not expect to really fulfilled my body, how do?
After the department heads aware of the situation, the worst has been done Plan B: AA and product leadership personally led Sunday where customers rushed to the city Monday to leadership communication; BB and CC go over there to find ways to customer administrator convince customers ......
02 straw: ext3grep
To the Internet to find information quickly be mistakenly deleted data recovery, find a really ext3grep can recover files deleted by rm -rf, we also ext3 disk format, and there are many online success stories.
So kindled a glimmer of hope, as soon as possible on the disk umount, to prevent re-written to make deleted files sector. Download ext3grep, install (compile and install the painstaking process for the time being is not the table).
First perform a scan file name command:
ext3grep /dev/vgdata/LogVol00 --dump-names
Print out all the deleted files and paths, rejoicing, without performing a Plan B, the files are in it.
This software can restore files by directory, all the commands can be executed only recovery:
ext3grep /dev/vgdata/LogVol00 --restore-all
The results of the current lack of disk space, no way can only restore files, try a few files, but still partially successful partial failure:
ext3grep /dev/vgdata/LogVol00 --restore-file var/lib/mysql/aqsh/tb\_b\_attench.MYD
You can not help a cold, is it was written to delete files on the disk? The probability of recovery is not, ah, you can restore a few count a few, maybe just important data files MYD file can be restored.
So first of all file names redirected to a file in the file:
ext3grep /dev/vgdata/LogVol00 --dump-names >/usr/allnames.txt
Filter out all MySQL database file name saved as mysqltbname.txt.
Scripting recover files:
while read LINE
do
echo "begin to restore file " $LINE
ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
if \[ $? != 0 \]
then
echo "restore failed, exit"
fi
done < ./mysqltbname.txt
Execution, run about 20 minutes, more than 40 file recovery, but not enough ah, we have nearly 100 tables, each table frm, myd, myi three documents, how to say there are about more than 300 ah!
Will come back files attached to an existing database, but also to the file permissions to 777, restart MySQL, can be considered part of the data back, but the customer is important to sign the attendance data, the phone side reporting of data (these data are said by customer do employee performance) not to come back ah.
Supposed to? Middle and tried another tool extundelete, with ext3grep grammar basically the same principle should be the same, but is said to restore by directory.
Well, give it a try:
extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh
Right on cue, the recovery does not come out! ! ! ! ! ! ! ! Those files have been destroyed. With the leadership of the report, the implementation of Plan B now ...... desperation home from work. (Weekend, go back and take a break, think of a way)
03 had an idea: Binlog
The next morning woke up early in the morning (heart problems ah), the back of the computer, go to the company (this weekend be reimbursed, not criticized, notification, fines, dismissal on the good, but also what the weekend ah).
Still run ext3grep, extundelete, ah recipe that will put to the test server rack systems, can look at the data to find ways to mend it.
Carried out on a test server mysqldump, recover files, recover back cover of the file, add permissions to the file, restart MySQL.
Wait, Wait, is not there Binlog it? Our services are required to open Binlog, maybe you can recover data Binlog in it?
So find Binlog Dump files from the file name out of a total of three:
-
mysql-binlog0001
-
mysql-bin.000009
-
mysql-bin.000010
Restore about 0001:
ext3grep /dev/vgdata/LogVol00 --restore-file var/lib/mysql/mysql-bin.000001
Actually failed ...... look at the other two files, mysql-bin.000010 probably a few hundred MB, should be a little tricky, perform the restore command, actually a success!
SCP as soon as possible to the test server. Execution Binlog restore:
mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p
Enter the password, stuck (a good sign), after a long wait is finally over. Open the application, oh, thank CCTV, MTV, data back!
04 Postscript
Also want to remember the accident, it will no longer make the same mistake. Reflection accident as follows:
-
They did not advance to arrange her time of this server maintenance MM explain the situation worse, and he did not attach importance to chaos management, process confusion. An online production system, any changes must first plan and then move.
-
Automatic backup problems, no one checks. Offline backup personnel 1K per download files from the server, but never seriously. We need a clear responsibility in the workplace.
-
After the accident, it did not discover, causing part of the data written to disk, resulting in unrecoverable problem. Need to write application monitoring program, once the service is abnormal, SMS alerts the responsible person.
-
According to comments reminders, plus a: You can not use the Root user to operate. You should offer different levels of user privileges on the server.
Through this accident
Sharing under the Tools link used herein:
1.https://code.google.com/p/ext3grep/
2.http://extundelete.sourceforge.net/
Ext3grep function with similar principles should also be similar. Compile and install dependencies more, you can search the Internet on how to install. [Unfortunately howto author was given a wall, I will FQ howto download the pdf document, after reading you will have a better understanding of the Linux file system. Download: http: //pan.baidu.com/s/1kT1ETVp].
Finally, I hope that colleagues little friends can remember this article event, happy to knock the code, never wrong -
