Occupied! Test server
One of the company's test servers went down today.
At the beginning, the front-end said that it could not be released, and gitlab could not be opened; then the back-end said that the test environment mysql was down.
Check the service processes first, and they are indeed gone. Start mysql and report an error. I thought I had gone through the battlefield, and used [top -ic] [free -h] [df -h] to explode in three bursts. . . -0 -0 -0 Three shots all empty! My eyes sharpened suddenly, this is not a normal problem.
crisis! cd command fails
Seeing that mysql startup reports an error: snapd.service unit has failed. Using this clue to check, it turns out that many ubuntu installation software errors. Not right. When I wanted to cd to the log folder to view the detailed log, a terrible scene appeared! When I input cd and press tab completion, I get an error [Cannot create temporary file for immediate document: read-only file system]. A crisis that has never been encountered before, the file system has a problem!
Which disk is the problem? Although I probably guessed the answer, I still input mount to check
/dev/std2 ro relatime errors=remount-ro
is the disk corresponding to the root directory. . . Try to modify his read and write attributes: mount -o remount,rw /dev/std2
[It is write-protected.] (is write-protected) The crappy Sinicization error message is displayed, and I can only find another way.
restore! Restart and disk repair
Enter dmesg according to the solution found, and a lot of disk errors popped up.
But at this moment, ssh lost the connection. . .
Sure enough, our technology minister couldn't help it, and finally pressed the restart button. . .
I found this machine in the computer room. Dozens of people's daily git operations and constant release tests may overwhelm its mechanical hard drive.
Can't get into the system. . The minister replugged all the cables of the hard drive and restarted, but still couldn't get in. .
At this time, I remembered the fsck disk repair command I saw when I just checked the problem. I tried to enter fsck /dev/std2 and
pressed a bunch of y and a, and successfully entered the system. . This feels similar to the Windows blue screen self-test.
Then I went back to the ssh connection and saw that everything was normal, no read-only.
to sum up
In the face of mystery problem, a restart trick can occasionally work!