CentOS 7 file system and log analysis Detailed

Linux file system

In dealing with a variety of Linux systems failures occur, the symptoms are most likely to find fault, but the cause of this failure is the key to the ultimate failure of exclusion. Familiar with common Linux system log files, to understand the general failure analysis and solutions will help administrators quickly locate the point of failure, "right", and promptly resolve system problems.

Detailed block and inode

• Then the file storage drive, a hard disk storage unit is the smallest "sectors", each sector stores 512 bytes

• File data includes meta-information and the actual data

• Meta information file contains attribute information, the content file is opened for the actual data input

• Block (Block)

  • Eight successive sectors of a block (4k)
  • It is the smallest unit of file access
• the inode (inode)
  • Chinese translation of "inode", also i-node
  • Used to store the files meta information

inode content

inode contains meta-information file

• Number of bytes in the file
• The file has a User ID
• Group ID documents
• Read files, write, and execute permissions
• Timestamp file
• ……

Use stat command to view the inode information

[Root @ localhost mnt] # stat abc.txt // View abc file inode information
  File: "abc.txt"
  Size: 12: 8 IO blocks: an ordinary file 4096
Equipment: 802h / 2050d Inode: 18123362 hard link: 1
Permissions: (0644 / -rw-r - r--) Uid :( 0 / root) Gid :( 0 / root)
环境：unconfined_u:object_r:mnt_t:s0
Last Visit: 2019-09-0204: 12: 18.210370997 +0800
Recent changes: 2019-09-0204: 12: 18.210370997 +0800
Recent changes: 2019-09-0204: 12: 18.210370997 +0800
Created: -

Linux file system attributes three main time

• ctime (change time): Last change a file or directory (attributes) of time
• The last time a file or directory: atime (access time)
• mtime (modify time): last modified files or directories (content) of time

Directory file structure

• It is the file directory
• Directory file structure

• Each has an inode number, inode number with the operating system to identify different files
• Internal Linux system does not use file names, use the inode number to identify the file command
• For the user, the file name inode number just another name for easy identification

inode numbers

• When a user opens a file by file name, the internal system process

  • The system finds the inode number corresponding to the file name
  • By inode number, get inode information
  • According inode information, where to find the data block file, read data
• View inode number of methods
  • ls -i command: View inde number corresponding to the file name

[Root @ localhost mnt] # ls -i abc.txt // node view file information 
18123362 abc.txt

inode size

• inode will consume disk space, the size of each inode, typically 128 bytes or 256 bytes

• Inode when determining the total formatted file system

• The total number of inode using df -i command to view each hard disk partition and the number already in use

[Root @ localhost mnt] # df -i // view the partition node information
Inode file system is used (I) can be used (I) has a mount point (I)%
/dev/sda2      10485760  121203 10364557       2% /
devtmpfs         229708     376   229332       1% /dev
tmpfs            233381       1   233380       1% /dev/shm
tmpfs            233381     560   232821       1% /run
tmpfs            233381      16   233365       1% /sys/fs/cgroup
/dev/sda5       5241856     146  5241710       1% /home
/dev/sda1       3145728     328  3145400       1% /boot
tmpfs            233381       6   233375       1% /run/user/42
tmpfs            233381      17   233364       1% /run/user/0
/dev/sr0              0       0        0        - /run/media/root/ 7 x86_64

The special role of the inode

• Since the inode number and file name separated, resulting in some of the Linux system has the following phenomenon
  • When the file name replacement special characters may not delete the file normal, delete inode, you can also delete files
  • By node number to delete the file command: find ./* -inum [Node No.] -delete

[root@localhost mnt]# ls -i
16777284 ac1.txt  18123362 ac2.txt  18123366 ac3.txt  18123370 ac4.txt  18123371 ac5.txt
[root@localhost mnt]# find ./* -inum 16777284 -delete
[root@localhost mnt]# ls -i
18123362 ac2.txt  18123366 ac3.txt  18123370 ac4.txt  18123371 ac5.txt

• Move or rename a file, just change the file name does not affect the inode number

[root@localhost mnt]# ls -i
18123362 ac2.txt  18123366 ac3.txt  18123370 ac4.txt  18123371 ac5.txt
[root@localhost mnt]# mv ac2.txt /opt
[root@localhost mnt]# ls -i /opt
18123362 ac2.txt   1389838 rh

• After opening a file, the file system is identified by the inode number, the file name is no longer considered

Simple process to access files

xfs types of file backup and restore

Xfs file recovery we need to first back up files to another hard disk when the file is damaged can be restored through the backup file (there are backup files are backed by compressed files make fixed-format).

• xfs types of files can be used with xfsrestore xfsdump tools for backup and recovery. If the xfsrestore xfsdump tool system is not installed, the tool can be installed by yum.

[root@localhost mnt]# yum install xfsdump -y
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: centos.ustc.edu.cn
 * extras: centos.ustc.edu.cn
 * updates: centos.ustc.edu.cn
Dependencies are resolved
-> Checking Services
---> packages will be upgraded xfsdump.x86_64.0.3.1.4-1.el7
---> packages will be updated xfsdump.x86_64.0.3.1.7-1.el7
... // ... omitted part

• Command Format

  Path xfsdump -f backup storage location or device files to be backed up

• There are two levels of backup xfsdump

  • 0: full backup (default is 0)
  • 1 to 9: represents incremental backup
• Common backup parameters
  • -f: Specifies the backup file directory
  • -L: Specifies the session label label
  • -M: Specifies the device label media label
  • -s: back up individual files, not directly with the path behind -s.

[Root @ localhost ~] # fdisk / dev / sdb // Create a new disk
Welcome to the fdisk (util-linux 2.23.2).

Change will stay in memory until you decide to write the changes to disk.
Please think twice before using the write command.

Device does not contain a recognized partition table
Use disk identifier 0x57599cea create a new DOS disk label.

Command (input m to get help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
Starting sector (2048-41943039, the default is 2048):
We will use the default value 2048
Last sector, sector + or + size {K, M, G} (2048-41943039, default 41943039):
We will use the default value 41,943,039
Partition 1 is set to Linux type, size to 20 GiB

Command (input m to get help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[Root @ localhost ~] # mkfs.xfs / dev / sdb1 // format the disk
meta-data = / dev / sdb1 help = 512 agcount = 4, agsize = 1310656 blks
         =                       sectsz=512   attr=2, projid32bit=1
         Crc finobt = = 1 = 0, sparse = 0
data     =                       bsize=4096   blocks=5242624, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[Root @ localhost ~] # mkdir / opt / data // create the mount point
[Root @ localhost ~] # mount / dev / sdb1 / opt / data // mount disks created
[Root @ localhost ~] # cp / etc / passwd / opt / data // copy passwdd directory to mount the disk
[Root @ localhost ~] # ls / opt / data // View Files
passwd
[Root @ localhost ~] # xfsdump -f / mnt / xfs_dump / dev / sdb1 // backup file using the command
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 3.1.7 (dump format 3.0) - type ^C for status and control

 ============================= dump label dialog ==============================

please enter label for this dump session (timeout in 300 sec)
 -> xfs_dump // Enter the backup file name
session label entered: "xfs_dump"

 --------------------------------- end dialog ---------------------------------

xfsdump: level 0 dump of localhost.localdomain:/opt/data
xfsdump: dump date: Mon Sep  2 05:27:29 2019
xfsdump: session id: 96c17954-e969-4041-9d9e-4bd59746ddab
xfsdump: session label: "xfs_dump"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: skipping (only one dump stream)
xfsdump: ino map construction complete
xfsdump: estimated dump size: 25216 bytes
xfsdump: /var/lib/xfsdump/inventory created

 ============================= media label dialog =============================

please enter label for media in drive 0 (timeout in 300 sec)
 -> sdb1 // Enter the backup device
media label entered: "sdb1"

 --------------------------------- end dialog ---------------------------------

xfsdump: creating dump session media file 0 (media 0, file 0)
xfsdump: dumping ino map
xfsdump: dumping directories
xfsdump: dumping non-directory files
xfsdump: ending media file
xfsdump: media file size 24264 bytes
xfsdump: dump size (non-dir files) : 2592 bytes
xfsdump: dump complete: 13 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /mnt/xfs_dump OK (success)
xfsdump: Dump Status: SUCCESS // backup success
[Root @ localhost ~] # rm -rf / opt / data / * // delete disk files
Whether [root @ localhost ~] # ls / opt / data // View Files deleted
[Root @ localhost ~] # xfsrestore -f / mnt / xfs_dump / opt / data // execute command to restore files
xfsrestore: using file dump (drive_simple) strategy
xfsrestore: version 3.1.7 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: examining media file 0
xfsrestore: dump description: 
xfsrestore: hostname: localhost.localdomain
xfsrestore: mount point: /opt/data
xfsrestore: volume: /dev/sdb1
xfsrestore: session time: Mon Sep  2 05:27:29 2019
xfsrestore: level: 0
xfsrestore: session label: "xfs_dump"
xfsrestore: media label: "sdb1"
xfsrestore: file system id: 2e3df4e8-dd92-46f7-84ca-f254c4e579bd
xfsrestore: session id: 96c17954-e969-4041-9d9e-4bd59746ddab
xfsrestore: media id: d27aafc8-a405-4a8f-b7f0-1a018a7026ee
xfsrestore: using online session inventory
xfsrestore: searching media for directory dump
xfsrestore: reading directories
xfsrestore: 1 directories and 1 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: restore complete: 0 seconds elapsed
xfsrestore: Restore Summary:
xfsrestore:   stream 0 /mnt/xfs_dump OK (success)
xfsrestore: Restore Status: SUCCESS
Whether [root @ localhost ~] # ls / opt / data // View File Recovery
passwd // file recovery success

• Limit the use of xfsdump
  • Only backup mounted file system
  • You must use root privileges to operate
  • XFS file system can only back up
  • Data backup can only be resolved so that xfsrestore
  • Two file systems can not back up with the same UUID

Log Files

Log function

• A recording system, a variety of events running program
• By reading log to help diagnose and resolve system failures

Classification of the log file

• The kernel and system log

  • Unified management by the system service syslog, log format is similar to the basic

• Users log

  • Related information recording system user login and exit system
• Log
  • Log files by the independent management of various applications, recording format is not uniform

Log save location

• Default is located at / var / log directory

The main log file Introduction

• Core and the common message log: / var / log / messages

• Scheduled Tasks log: / var / log / cron

• System boot log: / var / log / dmesg

• Mail system log: / var / log / maillog

• User login log: / var / log / lastlog, / var / log / secure, / var / log / wtmp, / var / run / btmp

Log Management

• By a unified management system services rsyslogd
  • Package: reyelog-7.4.7-16.el7.x86_64
  • The main programs: / sbin / rsyslogd
  • Profile: /etc/rsyslog.conf

[Root @ localhost ~] # vim /etc/rsyslog.conf // view the log file configuration information

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
... // ... omitted part
[Root @ localhost ~] # cd / var / log // view the log file directory
[root@localhost log]# ls
anaconda  dmesg               libvirt   rhsm               tallylog                Xorg.0.log
audit     dmesg.old           maillog   sa                 tuned                   Xorg.0.log.old
boot.log  firewalld           messages  samba              vmware-vgauthsvc.log.0  Xorg.1.log
btmp      gdm                 ntpstats  secure             vmware-vmsvc.log        Xorg.9.log
chrony    glusterfs           pluto     speech-dispatcher  vmware-vmusr.log        yum.log
cron      grubby_prune_debug  ppp       spooler            wpa_supplicant.log
cups      lastlog             qemu-ga   sssd               wtmp

• Check the system log file

  [Root @ localhost log] # vim messages // view the system log file
  
  Aug 10 03:53:40 localhost journal: Runtime journal is using 8.0M (max allowed 91.1M, trying to leave 136.7M free of 903.6M available → current limit 91.1M).
  Aug 10 03:53:40 localhost kernel: Initializing cgroup subsys cpuset
  Aug 10 03:53:40 localhost kernel: Initializing cgroup subsys cpu
  Aug 10 03:53:40 localhost kernel: Initializing cgroup subsys cpuacct
  Aug 10 03:53:40 localhost kernel: Linux version 3.10.0-693.el7.x86_64 ([email protected]) (gcc  version 4.8.5 20150623 ( 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017
  Aug 10 03:53:40 localhost kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64   root=UUID=729c9a26-dfdc-40f9-ae91-1ade55be51bb ro crashkernel=auto rhgb quiet LANG=zh_CN.UTF-8
  Aug 10 03:53:40 localhost kernel: Disabled fast string operations
  Aug 10 03:53:40 localhost kernel: e820: BIOS-provided physical RAM map:
  Aug 10 03:53:40 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable
  Aug 10 03:53:40 localhost kernel: BIOS-e820: [mem 0x000000000009ec00-0x000000000009ffff] reserved
  Aug 10 03:53:40 localhost kernel: BIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reserved
  ... // ... omitted part

• last command to view the user login log

[root@localhost log]# last
root     pts/0        192.168.144.1    Mon Sep  2 05:17   still logged in   
reboot   system boot  3.10.0-693.el7.x Mon Sep  2 05:17 - 05:58  (00:40)    
root     pts/0        192.168.144.1    Mon Sep  2 04:11 - crash  (01:05)    
root     :0           :0               Mon Sep  2 04:11 - crash  (01:05)    
reboot   system boot  3.10.0-693.el7.x Mon Sep  2 04:10 - 05:58  (01:47)    
root     pts/0        :0               Sun Aug 25 01:10 - 01:10  (00:00)    
root     :0           :0               Sun Aug 25 01:10 - crash (8+03:00)   
... // ... omitted part

• lastb View user logins log

[root@localhost log]# lastb
root     :0           :0               Sun Aug 25 01:10 - 01:10  (00:00)    
root     :1           :1               Sat Aug 10 06:26 - 06:26  (00:00)    

btmp begins Sat Aug 10 06:26:22 2019

• View Log File

  1, install httpd service, build Apache Web Services; then turn off the firewall, so that the host can access

[Root @ localhost ~] # yum install httpd -y // install httpd service
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: centos.ustc.edu.cn
 * extras: centos.ustc.edu.cn
 * updates: centos.ustc.edu.cn
Dependencies are resolved
-> Checking Services
---> packages will be installed httpd.x86_64.0.2.4.6-89.el7.centos.1
... // ... omitted part
[Root @ localhost ~] # systemctl start httpd // start the service
[Root @ localhost ~] # systemctl stop firewalld.service // turn off the firewall
[root@localhost ~]# setenforce 0                       
[Root @ localhost ~] # cd / var / log // view the log file directory to see if the smell generated httpd log file directory
[root@localhost log]# ls
anaconda  dmesg               lastlog   qemu-ga            sssd                    wtmp
audit     dmesg.old           libvirt   rhsm               tallylog                Xorg.0.log
boot.log  firewalld           maillog   sa                 tuned                   Xorg.0.log.old
btmp      gdm                 messages  samba              vmware-vgauthsvc.log.0  Xorg.1.log
chrony    glusterfs           ntpstats  secure             vmware-vmsvc.log        Xorg.9.log
cron      grubby_prune_debug  pluto     speech-dispatcher  vmware-vmusr.log        yum.log
cups      httpd               ppp       spooler            wpa_supplicant.log

2, host access through the building site to see the system's log file

[Root @ localhost log] # cd httpd // httpd program into the directory
[root@localhost httpd]# ls 
access_log  error_log
[Root @ localhost httpd] # vim access_log // view the log file

192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /noindex/css/bootstrap.min.css HTTP/1.1" 200 19341 "http://192.168.144.133/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /noindex/css/open-sans.css HTTP/1.1" 200 5081 "http://192.168.144.133/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /images/apache_pb.gif HTTP/1.1" 200 2326 "http://192.168.144.133/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /images/poweredby.png HTTP/1.1" 200 3956 "http://192.168.144.133/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /noindex/css/fonts/Light/OpenSans-Light.woff HTTP/1.1" 404 241 "http://192.168.144.133/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /noindex/css/fonts/Bold/OpenSans-Bold.woff HTTP/1.1" 404 239 "http://192.168.144.133/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
192.168.144.1 - - [02/Sep/2019:06:12:48 +0800] "GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1" 404 238 "http://192.168.144.133/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) 

Level log messages

• 0 EMERG (emergency): can lead to a host system unavailable
• 1 ALERT (warning): You must take immediate measures to solve the problem
• 2 CRIT (serious): more serious cases
• 3 ERR (error): An error occurs running
• 4 WARNING (remind): events that may affect the function of the system
• 5 NOTICE (Note): It is noteworthy that the system will not affect
• 6 INFO (Information): General Information
• 7 DEBUG (debug): programmers debug information

Log Management Policy

• Backup and archiving in a timely manner
• Extend the shelf life log
• Access control log
• Your logs may contain all kinds of sensitive information, such as accounts, passwords, etc.

Centralized log management

• The server's log file sent to the unified server log files

• Easy to collect the same log information, collation and analysis
• To prevent accidental loss of log information, tampering or delete