Linux file deletion, disk space but not released after df
Linux disk space is always alert, to be found in large files, then delete, df see the disk space is not released.
Find a system under discovery has been rm alias, because Linux is no mechanism to delete the recycle bin, rm operations to custom, to delete files moved to / tmp directory.
And on / temp delete disk shock but did not find release
Execute lsof | grep deleted just found a large number of deleted files process, kill off the process (or restart the process) OK
Reference material
Operation and maintenance of real case files have been deleted, but space does not resolve the question of release
1. Symptom
Operation and maintenance of the monitoring system sent a notification, reporting a server space is full, login to the server view, the root partition does not have the space, as shown below:
Here first explain some of the deleted policy server, because Linux is not the Recycle Bin function, our online servers you want to delete all files will first move to the system / tmp directory, and then periodically clear the data in the / tmp directory. The strategy itself is no problem, but through the inspection found that server system partition and no separate partition data in the / tmp partition, so / tmp is actually the space occupied by the root partition. Now find a problem, then delete the / tmp directory to some big data, execute the following command, the three largest data file check / tmp, as shown below:
[root@localhost~ ]# du -s /tmp/*|sort -nr|head -3
69206016 /tmp/access_log
36 /tmp/hsperfdata_root
36 /tmp/hsperfdata_mapred
Command output file access_log found a 66G size in the / tmp directory, this file should be apache access log file generated from the log size point of view, it should not be a long time to clean up apache log files, this file is the basic cause determination full of root space, after confirming that the file can be deleted, do the following deletions:
[root@localhost ~]# rm /tmp/access_log
The system then looks at the root partition space is released, as shown below:
From the output you can see, the root partition space is still not released, how is this going?
2 Solutions
In general the situation after deleting the file space does not release does not appear, but there are exceptions, such as the file is locked process, or have been in the process of writing data to the file, and so on, to understand this problem, we need to know the file under Linux the storage mechanism and storage structures.
A file stored in the file system is divided into two parts: a pointer portion and a data portion, the file pointer is located in the meta-data system, the data is deleted, the pointer is cleared from the meta-data in the data storage section in the disk, the data corresponding to the pointer removed from the meta-data, the data portion of the space occupied by the file can be overwritten and write new content, delete access_log file space not release the reason why is because the httpd process also has written to the file content, resulting in while deleting the access_log file, but the file pointer corresponding locking portion due process, are not cleared from the meta-data, rather because the pointer is not deleted, the system kernel file is considered has not been removed, so the query does not release the space is not surprising that by the df command.
3, Troubleshooting
Now that you have ideas to solve the problem, then the next and see if there has been a process to write data to a file in acess.log here need to use lsof command under Linux, you can get through a command has been deleted but is still being applied the program takes the list of files, the command execution as shown below:
You can see from the output, / tmp / acess.log httpd process file is locked, and the httpd process has also been written to the log data to this file, you can see from the seventh row, the log file size of only 70G, and total system root partition size only 100G, can be seen, the file system root partition space that led to the depletion of the culprit, the "deleted" status of the last one, indicating that the log file has been deleted, but the process still has to write to this file data space not released.
4, solve the problem
Basic troubleshooting the problem here is clear, to solve this kind of problem There are many ways, the easiest way is to shut down or restart the httpd process, of course, you can reboot the operating system, but this is not the best way to treat this kept on file write process log files to free disk space, the best way is to empty the file online can be done through the following command:
[root@localhost ~]# echo " " >/tmp/acess.log
In this way, not only the disk space can be released immediately, as well as protecting the process to continue to write to the log files, this method is often used to clean up the online log files Apache, Tomcat, Nginx and other Web services generated.