Deep understanding of Linux file system and log files
block与inode
Overview of inodes and blocks
- File data contains meta-information and actual data, meta-information is stored in inode, and actual data is stored in block
- Files are stored on the hard disk, the smallest storage unit of the hard disk is "sector", each sector stores 512 bytes
- The inode does not contain the file name. The file name is stored in the directory. Everything in a Linux system is a file, so a directory is also a kind of file.
- Each inode has a number, and the operating system uses the inode number to identify different files. The file name is not used internally in the Linux system, but the inode number is used to identify the file. For the system, the file name is just another name for the easy identification of the inode number. The file name and the inode number have a one-to-one correspondence, and each inode number corresponds to a file name.
- Therefore, when a user tries to access a file in the Linux system, the system will first find its corresponding inode number based on the file name. Obtain the inode information through the inode number, and according to the inode information, see if the user has the permission to access the file; if so, point to the corresponding data block and read the data
- block
- Eight consecutive sectors form a block (4k)
- It is the smallest unit of file access. When the
operating system reads the hard disk, it reads multiple sectors continuously at one time, that is, read block by block
- inode (index node)
- Chinese translation is "index node", also known as "i node"
- Used to store file meta information
inode content
- inode contains the meta information of the file
- The number of bytes of the file
- User ID of the file owner
- Group ID of the file
- File read, write, and execute permissions
- File timestamp
- But does not include the file name
- Use the stat command to view the inode information of a file,
for example: stat qz.txt - Three main time attributes of Linux system files
- ctime(change time)
last time the file or directory ( attribute ) was changed - atime (access time) the
last time the file or directory was accessed - mtime (modify time)
last time the file or directory ( content ) was modified
- ctime(change time)
- Each indoe has a number, and the operating system uses the inode number to identify different files
- The Linux system does not use the file name, but uses the Indoe number to identify the file
- For users, the file name is just another name for the indoe number for easy identification
- Directory is also a kind of file
inode number
- When the user opens the file by the file name, the internal process of the system
- The system first finds the inode number corresponding to this file name
- Get inode information through indoe number
- According to the inode information, find the block where the file data is located, and read the data
- How to check the inode number
- ls -i (check the inode number corresponding to the file name)
For example: ls -i qz.txt - stat (check the inode number in the file inode information)
For example: stat qz.txt
- ls -i (check the inode number corresponding to the file name)
inode size
- Inode also consumes hard disk space, the size of each inode is generally 128 bytes or 256 bytes
- The total number of inodes will be determined when formatting the file system
- When formatting, the operating system will automatically divide the hard disk into two areas. One is the data area, which stores file data; the other is the inode area, which stores the information contained in the inode
- df -i (you can view the total number of inodes of each hard disk partition and the number that has been used)
Special role of inode
- Due to the separation of the inode number and the file name, some Unix/Linux systems have the following phenomena:
1. When the file name contains special characters, the file may not be deleted normally. In this case, delete the inode directly, which can play the role of deleting the file.
2. Move or When renaming a file, only the file name is changed, and the inode number is not affected.
3. After opening a file, the system recognizes the file by the inode number, regardless of the file name
. 4. After the file data is modified, a new inode number will be generated - Use the find command to delete the inode number
find ./ -inum 52201242 -exec rm -i {};
find ./-inum 55156844 -delete
Inode node exhaustion fault handling
fdisk /dev/sdb
Use fdisk to create the partition /dev/sdb1. For the experiment, the partition size is 30M.
mkfs.ext4 /dev/sdb1
format (use ext4 file type, because it is more accurate)
mkdir /test
create directory
mount / dev/sdb1 /mnt
mount
df -i
view
for ((i=1; i<=7680; i++));do touch /test/file$i;done
Use the for command to create the directory
touch {1... 7680}.txt
or use the touch command to create a directory
df -i
df -hT to
see how much space is left
rm -rf /test/* After
deleting the file, you can restore it
df -i
df -hT
view
Schematic diagram of the process of accessing files
Hard link and soft link
- Linked file classification
| Soft link |
Hard link | |
|---|---|---|
| After deleting the original file | Invalidation | Still available |
| Scope of use | Suitable for files or directories | Can only be used for files |
| Save location | Can be located in a different file system from the original file | Must be in the same file system as the original file (like a Linux partition) |
- Hard link
- ln source file destination
- Soft link
- ln [-s] Source file or directory... Link file or target location
Recover accidentally deleted files
EXT file recovery
- extundelete is an open source Linux data recovery tool that supports ext3 and ext4 (can only be recovered in the CentOS6 version) file system.
Steps
fdisk /dev/sdc
使用fdisk创建分区/dev/sdc1
mkfs.ext3 /dev/sdc1
格式化
mkdir /test
创建
mount /dev/sdc1 /test
挂载
df -hT
查看
yum -y install e2fsprogs-devel e2fsprogs-libs
通过yum安装依赖包
extundelete
编译安装
cd /test
wget 使用wget通过该网站下载进行安装
tar jxvf extundelete-0.2.4.tar.bz2
解压
cd extundelete-0.2.4/
./configure - -prefix=usr/local/extundelete && make && install
使用源码编译进行安装
ln -s /usr/local/extundelete/bin* /usr/bin/
模拟删除并恢复操作
cd /test
echo a>a
echo a>b
echo a>c
echo a>d
ls
extundelete /dev/sdc1 - -inode2
查看文件系统/dev/sdc1下存在哪些文件，i节点是从2开始的，2代表该文件系统最开始的目录
rm -rf a b
extundelete /dev/sdc1 - -inode 2
cd~
umount /test
解挂
extundelete /dev/sdc1 - -restore-all
恢复/dev/sdc1 文件系统下的所有内容。然后会在当前目录下出现一个RECOVERED_FILES/目录，里面包含了已经恢复的文件
ls RECOVERED_FILES/
查看
xfs 类型文件备份和恢复
- CentOS7 系统默认采用xfs类型的文件，xfs类型的文件可使用xfsdump与xfsrestore工具进行备份恢复
- xfsdump的备份级别有两种
0表示完全备份
1-9表示增量备份
xfsdump的备份默认级别为0 - xfsdump的命令格式
| 命令 | 说明 |
|---|---|
| -f | 指定备份文件目录 |
| -L | 指定标签session label |
| -M | 指定设备标签 media label |
| -s | 备份单个文件，-s后面不能直接跟路径 |
- xfsdump使用限制
1.只能备份已挂载的文件系统
2.必须使用root的权限才能操作
3.只能备份XFS文件系统
4.备份后的数据只能让xfsrestore解析
5.不能备份两个具有相同UUID的文件系统（可用blkid命令进行查看）
操作步骤
fdisk /dev/sdv
使用fdisk创建分区
partprobe /dev/sdb
刷新
mkfs.xfs -f /dev/sdb1
使用-f进行强制格式化
mkdir /data
mount /dev/sdb1 /data/
cd /data
cp /etc/passwd ./
mkdir test
touch test/a
使用xfsdump命令备份整个分区
rpm -qa | grep xfsdump
yum install -y xfsdump
xfsdump -f /opt/dump_sdb1 /dev/sdb1 [-L dump_sdb1 -M sdb1]
模拟数据丢失并使用xfsrestore命令恢复文件
cd /data/
rm -re ./*
ls
xfsrestore -f /opt/dump_sdb1 /data/
日志文件
日志文件概述
-
日志的功能
- 用于记录系统、程序运行中发生的各种事件
- 通过阅读日志，有助于诊断和解决系统故障
-
日志的分类
- 内核系统日志
由系统服务rsyslog统一进行管理，日志格式基本相似
主要配置文件/etc/rsyslog.conf - 用户日志
记录系统用户登录及退出系统的相关信息 - 程序日志
由各种应用程序独立管理的日志文件，记录格式不统一
- 内核系统日志
-
系统日志默认保存在：/var/log目录下
-
系统日志主要程序：/sbin/rsyslogd
-
系统日志配置文件：/etc/rsyslog.conf
-
系统日志软件包：rsyslog-7.4.7-16.el7.x86_64
Linux主要包含的日志文件
内核及公共消息日志
- /var/log/messages:记录Linux内核消息及各种应用程序的公共日志信息，包括启动、IO错误、网络错误、程序故障等。对于未使用独立日志文件的应用程序或服务，一般都可以从该日志文件中获得相关的事件记录信息。
计划任务日志
- /var/log/cron:记录crond计划任务产生的事件信息。
系统引导日志
- /var/log/dmesg: 记录Linux系统在引导过程中的各种事件信息。
邮件系统日志
- /var/log/maillog:记录进入或发出系统的电子邮件活动。
用户登录日志
- /var/log/secure:记录用户认证相关的安全事件信息。
- /var/log/lastlog:记录每个用户最近的登录事件。二进制格式
- /var/log/wtmp:记录每个用户登录、注销及系统启动和停机事件。二进制格式
- /var/run/btmp:记录失败的、错误的登录尝试及验证事件。二进制格式
vim /etc/rsyslog.conf
查看reyslog.conf配置文件
例如：
Linux系统的日志消息级别
- Linux系统内核日志消息的优先级别（数字等级越小，优先级越高，消息越重要）
| 级别 | 词组 | 说明 |
|---|---|---|
| 0 | EMERG（紧急） | 会导致主机系统不可用的情况 |
| 1 | ALERT（警告） | 必须马上才去措施解决的问题 |
| 2 | CRIT（严重） | 比较严重的情况 |
| 3 | ERR（错误） | 运行出现错误 |
| 4 | WARNING（提醒） | 可能影响系统功能，需要提醒用户的重要事件 |
| 5 | NOTICE（注意） | 不会影响正常功能，但是需要注意的事件 |
| 6 | INFO（信息） | 一般信息 |
| 7 | DEBUG（调试） | 程序或系统调试信息等 |
Linux系统中用户日志的查询命令及日志记录格式
- users、who、w、last、lastb
- last命令用于查询成功登陆到系统的用于记录
- lastb命令用于查询登陆失败的用户记录
日志记录的一般格式
程序日志分析
- Web服务：/var/log/httpd/
access_log 记录客户访问事件
error_log 记录错误事件 - 代理服务：/var/log/squid/
access.log 记录客户访问事件
cache.log 记录缓存事件 - 分析工具
文本查看、grep过滤检索、Webmin管理套件中查看
awk、sed等文本过滤、格式化编辑工具
Webalizer、Awstats等专用日志分析工具
一般通过Windows和Linux挂载的方法，从Linux系统中导出日志查看，或者在Windows中使用WinSCP查看
日志管理策略
- 及时做好备份和归档
- 延长日志保存期限
- 控制日志访问权限，因为日志中可能会包含各类敏感信息，如账号、口令等
chmod 640 （主成员能读、能写，自己组成员能读，其他用户没有任何权限） - 集中管理日志
将服务器的日志文件发到同一的日志文件服务器
便于日志信息的同一收集、整理和分析
杜绝日志信息的意外丢失、恶意篡改或删除