Linux servers appear: No space left on device solutions _ _DRACOYU operation and maintenance of the blog -CSDN blog
https://blog.csdn.net/chanbo8205/article/details/85229453

Linux servers appear: No space left on device solutions

Error message:

2016-06-02 23:56:26 [com.thinkive.server.logger.TradeLogger]-[ERROR]

java.io.IOException: No space left on device

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(FileOutputStream.java:282)

at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)

at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)

at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)

at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)

at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)

at com.thinkive.server.logger.TradeLogger$LoggerThread.flushLogFile(TradeLogger.java:404)

at com.thinkive.server.logger.TradeLogger$LoggerThread.run(TradeLogger.java:322)

Inadequate server disk, df -h look, var Normally, there is a surplus, the problem is not found, and later displayed a lot of No space left on device:

/ Var obviously there is a lot of space, why prompt "not enough space" of it to use the results of the node df -i command to view disk found in the following figure?:

100% of the nodes.

This time the user can look at whether the log file too if log files take up a lot of inodes expired at cleaning up the log can be temporarily released.

The following are checking inodes full one situation, for reference only:

View the original sentence crontab inside the regular implementation There is no added> / dev / null 2> & 1, program the system cron execution of the output contents, output contents will be sent cron users by mail, and sendmail does not start so we had a a lot of fragmented files:

cd / var / spool / clientmqueue / into this directory, delete these useless files, use ls to see what the result is no response, too many files, so he uses this command: ls | xargs rm -rf in batches the process of deleting files

Operation to this step, according to the occupancy of large directory files, temporary files to take up part of the transfer disk smaller directory.

The last two days for a server, the findings suggest "No space left on device" when using wget to download files, and even use the error will prompt completion when the tab key.

Once I encountered this problem before, because of disk space was already occupied, making it impossible to create a new file. Under normal circumstances, delete some files to free up space, you can solve the problem.

When I use the df command to view the partition, the results are as follows:

# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 29G 29G 0 100% /

udev 10M 0 10M 0% /dev

tmpfs 101M 232K 100M 1% /run

tmpfs 5.0M 0 5.0M 0% /run/lock

tmpfs 405M 0 405M 0% /run/shm

See here, I really think that the disk is 100% occupied, so he viewed the occupation of each directory:

# You -sh / *

8.8M / bin

30M /boot

0 /dev

5.3M /etc

24K /home

0 /initrd.img

205M /lib

4.0K /lib64

16K /lost+found

8.0K / media

4.0K / mnt

4.0K /opt

0 /proc

2.5G /root

232K /run

5.2M /sbin

8.0K /srv

0 /sys

4.0K /tmp

2.6G / usr

1.8G / was

0 /vmlinuz

Obviously, the total disk usage is less than completely 10G, disk theory has not been filled.

In this case, it is likely to be too small files, leading to a sharp increase inode, inode area of space consumed .

If so, even if the remaining disk space, but can not create new inode to store the meta-information file, it can not create new files.

So, I verify with df command:

# Df -ih

Filesystem Inodes IUsed IFree IUse% Mounted on

/ Dev / vda1 1.9M 299K 1.6M 17% /

udev 123K 299 123K 1% /dev

tmpfs 126K 249 125K 1% /run

tmpfs 126K 4 126K 1% /run/lock

tmpfs 126K 2 126K 1% /run/shm

You can see, inode area occupied only a small part, there are a lot of unused space, so the inode is not a problem area was already occupied.

To here, my heart is very depressed. This problem led directly to the Apache, MySql and some other services are not started, the server has basically can not use, but the reason was confusing.

Finally, the server provider told me that another possible case, delete some files Shihai is being used by another process, then the file is not actually deleted, just marked as deleted, and only after the end of the process will actually file from disk Clear.

So I see the files being used in the process by lsop command:

# lsof | grep deleted

mysqld 1952 2982 mysql 5u REG 254,1 0 127 /tmp/ibzMEe4z (deleted)

mysqld 1952 2982 mysql 6u REG 254,1 0 146 /tmp/ibq6ZFge (deleted)

mysqld 1952 2982 mysql 10u REG 254,1 0 150 /tmp/ibyNHH8y (deleted)

apache2 2869 root 9u REG 254,1 0 168 /tmp/.ZendSem.2w14iv (deleted)

apache2 2869 root 10w REG 0,16 0 11077 /run/lock/apache2/rewrite-map.2869 (deleted)

...

python 3102 root 1w REG 254,1 22412342132 264070 /var/log/nohup.out (deleted)

Finally found the culprit turned out to be a Python script running in the background of a steady flow of /var/log/nohup.out save the output to a file, the file size actually reached 20G +!

After running in the background before the stage script, no longer control over it. My estimate is deleted nothup.out Python files during operation, because the file is occupied, it can only be marked as deleted, but not actually deleted, leading to disk full.

The lesson to remind me, the task can not be simply put back on the laissez-faire attitude, especially when using the nohup command, all output will be continuously added to the same file, if the process does not terminate, you might as causes the output file occupies the entire disk.

=============================================================================

A simple understanding of Linux inode and block - hard to force the operation and maintenance - blog Park
https://www.cnblogs.com/diantong/p/10504513.html

A simple understanding of Linux inode and block

Linux common file system type: ext3 (CentOS5), ext4 (CentOS6), xfs (CentOS7)

Windows common file system types: FAT32, NTFS

(1) .inode content

1) inode file that contains meta information, specifically, have the following:

Number of bytes in the file

User ID file owner

Group ID documents

Read files, write, and execute permissions

Timestamp file, there are three: ctime inode refers to the time of the last change file attributes, for example: chmod + x a.sh; mtime refers to the last time the file content changes, such as: echo aa >> a.sh or vim a.sh; atime value of the file's last open time, for example: cat a.sh.

The number of links that point to the number of file names inode

The location of the file data block

2) We can use the stat command to view a file's inode information:

[root@xuexi ~]# stat /etc/passwd

文件："/etc/passwd"

大小：2257 块：8 IO 块：4096 普通文件

设备：803h/2051d Inode：17324457 硬链接：1

权限：(0644/-rw-r--r--) Uid：( 0/ root) Gid：( 0/ root)

环境：system_u:object_r:passwd_file_t:s0

最近访问：2019-03-08 11:00:01.672759577 +0800

最近更改：2019-02-13 16:19:23.303395879 +0800

最近改动：2019-02-13 16:19:23.305395782 +0800

创建时间：-

[root@xuexi ~]# echo $LANG //查看当前语言

zh_CN.UTF-8

[root@xuexi ~]# LANG="en_US.UTF-8" //语言临时转为英文

[root@xuexi ~]# stat /etc/passwd

File: ‘/etc/passwd’

Size: 2257 Blocks: 8 IO Block: 4096 regular file

Device: 803h/2051d Inode: 17324457 Links: 1

Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)

Context: system_u:object_r:passwd_file_t:s0

Access: 2019-03-08 11:00:01.672759577 +0800

Modify: 2019-02-13 16:19:23.303395879 +0800

Change: 2019-02-13 16:19:23.305395782 +0800

Birth: -

[root@xuexi ~]# ll /etc/passwd //ll其实就是查看文件的inode信息

-rw-r--r--. 1 root root 2257 Feb 13 16:19 /etc/passwd //ll查看到的时间是ctime时间

3) test ctime, mtime and atime

[root@xuexi ~]# touch a.txt

[root@xuexi ~]# stat a.txt

File: ‘a.txt’

Size: 0 Blocks: 0 IO Block: 4096 regular empty file

Device: 803h/2051d Inode: 34042450 Links: 1

Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)

Context: unconfined_u:object_r:admin_home_t:s0

Access: 2019-03-08 14:24:17.272886924 +0800

Modify: 2019-03-08 14:24:17.272886924 +0800

Change: 2019-03-08 14:24:17.272886924 +0800

Birth: -

[root@xuexi ~]# date -s "2019-03-07 14:29:49" //修改系统时间

Thu Mar 7 14:29:49 CST 2019

[root@xuexi ~]# chmod +x a.txt

[root@xuexi ~]# stat a.txt //可以看到’ chmod +x a.txt’修改了ctime

File: ‘a.txt’

Size: 0 Blocks: 0 IO Block: 4096 regular empty file

Device: 803h/2051d Inode: 34042450 Links: 1

Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Context: unconfined_u:object_r:admin_home_t:s0

Access: 2019-03-08 14:24:17.272886924 +0800

Modify: 2019-03-08 14:24:17.272886924 +0800

Change: 2019-03-07 14:30:32.246910417 +0800

Birth: -

[root@xuexi ~]# echo aa >> a.txt

[root@xuexi ~]# stat a.txt //可以看到’ echo aa >> a.txt’修改了mtime和ctime

File: ‘a.txt’

Size: 3 Blocks: 8 IO Block: 4096 regular file

Device: 803h/2051d Inode: 34042450 Links: 1

Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Context: unconfined_u:object_r:admin_home_t:s0

Access: 2019-03-08 14:24:17.272886924 +0800

Modify: 2019-03-07 14:31:19.164728344 +0800

Change: 2019-03-07 14:31:19.164728344 +0800

Birth: -

[root@xuexi ~]# vim a.txt

//添加了一行bbb

[root@xuexi ~]# stat a.txt //可以看到’ vim a.txt’修改了atime，ctime和mtime

File: ‘a.txt’

Size: 7 Blocks: 8 IO Block: 4096 regular file

Device: 803h/2051d Inode: 34729170 Links: 1

Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Context: unconfined_u:object_r:admin_home_t:s0

Access: 2019-03-07 14:32:42.604862498 +0800

Modify: 2019-03-07 14:32:42.604862498 +0800

Change: 2019-03-07 14:32:42.605945804 +0800

Birth: -

[root@xuexi ~]# cat a.txt

aa

bbb

[root@xuexi ~]# stat a.txt //可以看到’ cat a.txt’修改了atime

File: ‘a.txt’

Size: 7 Blocks: 8 IO Block: 4096 regular file

Device: 803h/2051d Inode: 34729170 Links: 1

Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Context: unconfined_u:object_r:admin_home_t:s0

Access: 2019-03-07 14:34:27.525217232 +0800

Modify: 2019-03-07 14:32:42.604862498 +0800

Change: 2019-03-07 14:32:42.605945804 +0800

Birth: -

　　由此可见，黑客是可以通过修改时间，再植入木马，防止被find找到。

4）inode的大小

Inode也会消耗硬盘空间，所以硬盘格式化时，操作系统自动将文件分成两个区域。一个是数据区，用于存放文件数据；另一个是inode区，用于存放inode所包含的信息。

每个inode节点的大小一般是128字节或256字节。Inode节点的总数在格式化时就给定，一般是1KB或2KB或4KB就设置一个inode。如果一块1GB的硬盘，每个inode节点的大小为128字节，每1KB就设置一个inode，那么inode区的大小就会达到128MB，占整个硬盘的12.8%。（所以block可以设置的大一点）

5）inode号

每个inode都有一个号码，操作系统用inode号来识别不同的文件。

Unix/Linux系统内部不是用文件名，而使用inode号来识别文件。对于系统来说，文件名只是iode号便于识别的别名。表面上用户是通过文件名打开文件，实际上系统内部分成三个步骤：首先，系统找到这个文件名对应的inode号；其次通过inode号获取inode信息；最后根据inode信息找到文件数据所在的block，读出数据。

可以使用’ls -i [filename]’快速查看文件的inode号：

1 2	`[root@xuexi ~]# ls -i a.txt` `34729170 a.txt`

　　在Unix/Liunx系统中，目录也是一种文件。目录文件的结构非常简单，即使一系列目录项的列表。每个目录项由所包含文件的文件名以及该文件名对应的inode号组成。

　　可以使用’ls -id [directory]’快速查看目录的inode号：

1 2	`[root@xuexi ~]# ls -id /etc` `16777281 /etc`

　　另外可以使用’df -i’查看每个磁盘分区的inode总数和已经使用的数量

[root@xuexi ~]# df -i

文件系统 Inode 已用(I) 可用(I) 已用(I)% 挂载点

/dev/sda3 8912384 193509 8718875 3% /

devtmpfs 249583 408 249175 1% /dev

tmpfs 253514 1 253513 1% /dev/shm

tmpfs 253514 966 252548 1% /run

tmpfs 253514 16 253498 1% /sys/fs/cgroup

/dev/sda1 524288 359 523929 1% /boot

/dev/sr0 0 0 0 - /mnt

tmpfs 253514 6 253508 1% /run/user/42

tmpfs 253514 17 253497 1% /run/user/1000

tmpfs 253514 1 253513 1% /run/user/0

　　注意：由于每个文件都必须有一个inode号，因此有可能发生inode已用光，但硬盘未存满的情况。这时就无法在硬盘上创建新的文件。

6）inode的特殊作用

　　有时文件名有特殊字符或乱码，无法正常修改或删除，可以通过inode号来进行操作。实例：

[xf@xuexi ~]$ mkdir Dir

[xf@xuexi ~]$ cd Dir/

[xf@xuexi Dir]$ touch 学习　　\\创建一个中文的文件名

[xf@xuexi Dir]$ ls

学习

[xf@xuexi Dir]$ LANG="en_US.gbk"　　\\临时调整一下编码，使得文件名变为乱码

[xf@xuexi Dir]$ ls

??????

[xf@xuexi Dir]$ ls -i　　\\看一下该文件的inode号

407062 ??????<br>\\删除还可以使用"find . -inum 407062 -delete"（没有提示）以及"find . -inum 407062 | xargs -i rm {}"<br>\\修改类似

[xf@xuexi Dir]$ find . -inum 407062 -exec rm -i {} \;

rm: remove regular empty file './\345\255\246\344\271\240'? y

[xf@xuexi Dir]$ ls

　　移动或重命名文件，只改变文件名，inode没影响。

　　使用vim修改文件内容后，inode会发生改变。实例：

[xf@xuexi ~]$ touch File

[xf@xuexi ~]$ ls -i File

52016344 File

[xf@xuexi ~]$ vim File<br>//增加一行aaaaa

[xf@xuexi ~]$ ls -i File　　//inode号改变了

52016202 File

[xf@xuexi ~]$ echo bbbbb>>File　　//追加一行bbbbb

[xf@xuexi ~]$ ls -i File　　//inode号没变

52016202 File

[xf@xuexi ~]$ cat File

aaaaa

bbbbb

[xf@xuexi ~]$ ls -i File

52016202 File

[xf@xuexi ~]$ cat > File <<EOF　　//重写File

> ccccc

> ddddd

> EOF

[xf@xuexi ~]$ ls -i File　　//inode号还是没变

52016202 File

　　我一开始还怀疑会不会是我字节没有写满一个block，但我用cat写了20416个字节，结果inode号还是没变。我估计echo追加和cat重写都不改变inode号。

(2).block的内容

block是真正存储数据的地方。block是文件系统中的最小存储单位，扇区是磁盘中的最小存储单位。

注意：Linux下叫block，Windows下叫簇。

1） Windows如何修改簇的大小

　　可以右键一个分区，点击格式化会出现如下窗口：

这里的分配单元大小就是设置簇的大小。

2） Block或簇的大小对系统的影响

　　簇或block调大时，节约了寻址时间，速度变快，但浪费空间；簇和block调小时，节约空间，但寻址时间变长，速度变慢。

　　说明：为什么簇或block调大会浪费空间？这是因为一个文件会占用多个簇或block来存放。当前一个簇或block放不下时，就会占用下一个簇或block，到最后如果产生不足以占用一个完整的簇或block时，仍然会占用一个完整的簇或block，就会浪费这个簇或block剩下的空间。

如果有一个2T的硬盘，可以前1.5T使用4K的簇或block，后0.5G使用64K的簇或block，这样可以改善机械硬盘越到最后速度越慢的问题。

3） Linux下查看block的大小以及修改方式

　　查看block的大小

[root@xuexi ~]# file -s /dev/sdb1 //把block或设备当做普通文件看待

/dev/sdb1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

[root@xuexi ~]# xfs_info /dev/sdb1

xfs_info: /dev/sdb1 is not a mounted XFS filesystem //居然需要先挂载

[root@xuexi ~]# ls /

bin dev etc lib media opt root sbin srv tmp var

boot dump home lib64 mnt proc run sdb1 sys usr

[root@xuexi ~]# mkdir /sdb1

mkdir: 无法创建目录"/sdb1": 文件已存在

[root@xuexi ~]# mount /sdb1 /dev/sdb1

mount: /sdb1 不是一个块设备

[root@xuexi ~]# mount /dev/sdb1 /sdb1/

[root@xuexi ~]# xfs_info /dev/sdb1

meta-data=/dev/sdb1 isize=512 agcount=4, agsize=65536 blks

= sectsz=512 attr=2, projid32bit=1

= crc=1 finobt=0 spinodes=0

data = bsize=4096 blocks=262144, imaxpct=25

= sunit=0 swidth=0 blks

naming =version 2 bsize=4096 ascii-ci=0 ftype=1

log =internal bsize=4096 blocks=2560, version=2

= sectsz=512 sunit=0 blks, lazy-count=1

realtime =none extsz=4096 blocks=0, rtextents=0

　　注意：xfs_info只能用于xfs的文件系统，如果是ext4或ext3请使用tune2fs

　　修改block的大小，就需要用到mkfs.xfs格式化命令，需要格式化分区

[root@xuexi ~]# umount /sdb1

[root@xuexi ~]# mkfs -t xfs -b size=2048 /dev/sdb1

mkfs.xfs: /dev/sdb1 appears to contain an existing filesystem (xfs).

mkfs.xfs: Use the -f option to force overwrite.

[root@xuexi ~]# mkfs -t xfs -b size=2048 -f /dev/sdb1

meta-data=/dev/sdb1 isize=512 agcount=4, agsize=131072 blks

= sectsz=512 attr=2, projid32bit=1

= crc=1 finobt=0, sparse=0

data = bsize=2048 blocks=524288, imaxpct=25

= sunit=0 swidth=0 blks

naming =version 2 bsize=4096 ascii-ci=0 ftype=1

log =internal log bsize=2048 blocks=5120, version=2

= sectsz=512 sunit=0 blks, lazy-count=1

realtime =none extsz=4096 blocks=0, rtextents=0

　　注意：mkfs.xfs就是mkfs –t xfs的简写。

guyue35

发布了103 篇原创文章 · 获赞 127 · 访问量 107万+

他的留言板关注