【Abstract】 How to delete a large number of small files in Linux? This article describes two methods.
【Author】Zhao Jingyu
environment:
RHEL 6.5 + Oracle 11.2.0.4
need:
Use df -i to inspect and find that Inodes usage is too high, and you need to clean and delete files to solve the problem. If the Inodes are full, the directory will not be writable, even if df -h checks that there is remaining space.
1. Problem phenomenon
Oracle's adump records sys login audit information, which is characterized by a large number of small files. It is often encountered that the rm -rf * command cannot be used to delete them, and the error is -bash: /bin/rm: Argument list too long.
This is because the wildcard character * will be replaced by a specific file name during execution, such as rm -rf file1 file2 file3 .... If there are too many files, this error will easily occur.
For example, in the following environment, the number of files in the adump directory has reached 1.14 million+, and this error will be reported when executing the rm -rf * command:
[oracle@jystdrac2 adump]$ pwd/opt/app/oracle/admin/crmdb/adump[oracle@jystdrac2 adump]$ ls|wc -l1149787[oracle@jystdrac2 adump]$ rm -rf *-bash: /bin/rm: Argument list too long[oracle@jystdrac2 adump]$ du -sh4.4G
2.Solution
After understanding the problem, the solution is to remove the rm -rf * command. What other methods are available? If you search on the Internet, you may find a way to execute rm in combination with the find command, but in fact the efficiency is very poor. The specific writing method is not listed here, because we usually don’t handle it this way. So how to delete a large number of small files more efficiently? Combining network experience and actual measurement verification, we finally summarized two common solutions, both of which are efficient in terms of efficiency.
Option 1: Cleverly use rsync to achieve the purpose of deletion
Create an empty folder and use the rsync --delete-before -d <empty folder> <directory where small files need to be cleaned and deleted> command to ultimately delete a large number of small files. The specific operations are demonstrated below:
[oracle@jystdrac2 adump]$ mkdir /data/null[oracle@jystdrac2 adump]$ ls -l /data/nulltotal 0[oracle@jystdrac2 ~]$ nohup rsync --delete-before -d /data/null/ /opt/app/oracle/admin/crmdb/adump/ &
Use man rsync to view the parameters related to the rsync command as follows:
-d, --dirs transfer directories without recursing--delete-before receiver deletes before transfer (default)
Option 2: Use the delete parameter of the find command
Use the find <directory where small files need to be cleaned and deleted> -type f -delete command to directly delete a large number of small files.
Use man find to view the parameters related to the find command as follows:
-type cFile is of type c:b block (buffered) specialc character (unbuffered) speciald directoryp named pipe (FIFO)f regular filel symbolic link; this is never true if the -L option or the -follow option is in effect, unless the symbolic link is broken. If you want tosearch for symbolic links when -L is in effect, use -xtype.s socketD door (Solaris)-deleteDelete files; true if removal succeeded. If the removal failed, an error message is issued. If -delete fails, find’s exit status will be nonzero(when it eventually exits). Use of -delete automatically turns on the ‘-depth’ option.Warnings: Don’t forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everythingbelow the starting points you specified. When testing a find command line that you later intend to use with -delete, you should explicitly spec-ify -depth in order to avoid later surprises. Because -delete implies -depth, you cannot usefully use -prune and -delete together.
The specific operations are demonstrated below:
[oracle@jystdrac1 adump]$ nohup find /opt/app/oracle/admin/crmdb/adump/ -type f -delete &You can refer to the following command to simply monitor changes in Inodes usage during the deletion process:
while true; do df -i /; sleep 10; doneFor example, the find method used by the node jystdrac1 here and the rsync method used by the node jystdrac2 are actually observed. The difference in Inodes release speed is not big:
# 使用的find方法，观察Inodes释放速度：[oracle@jystdrac1 ~]$ while true; do df -i /; sleep 10; doneFilesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1519124 287772 85% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1519015 287881 85% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1513880 293016 84% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1511132 295764 84% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1502434 304462 84% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1494583 312313 83% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1489111 317785 83% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 1487629 319267 83% /# 使用的rsync方法，观察Inodes释放速度：[oracle@jystdrac2 ~]$ while true; do df -i /; sleep 10; doneFilesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 963029 843867 54% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 955037 851859 53% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 953088 853808 53% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 950523 856373 53% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 948754 858142 53% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 944613 862283 53% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 942619 864277 53% /Filesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/vg_linuxbase-lv_root 1806896 938510 868386 52% /
Since the difference between the two methods is not big, just choose according to your needs or personal habits. I prefer option 2, because it does not require the creation of an empty directory and the operation is more intuitive.
Finally, let’s summarize how to delete a large number of small files:
# 方案一：mkdir <空文件夹>rsync --delete-before -d <空文件夹> <需要清理删除小文件的目录># 方案二：find <需要清理删除小文件的目录> -type f -delete
Relatively speaking, both methods are more efficient, but since there are a lot of small files overall, you can actually choose nohup to execute in the background.
Original title: How to delete a large number of small files in Linux