Analyze and store logs
Describe the system log structure
System logging
Processes and the operating system kernel log events as they occur, which can be used for system auditing and troubleshooting of problems.
Generally stored in /var/logthe directory in text form
The systemd-journald service is the core of the operating system event log architecture. Collect event messages from many sources and write them into an indexed, structured system log. By default, the log is stored on a file system that is not retained across system reboots.
The rsyslog service will read systemd-journald received syslog messages from the log. After that, the syslog events will be processed, recorded to log files, and forwarded to other services according to their configuration. rsyslog will sort syslog messages and write them to log files that will not be retained after restart. The rsyslog service sorts log messages into specific log files based on the type of program or device sending each message and the priority of each syslog message.
View system log files
Many programs use the syslog protocol to log time to the system, with each log message classified according to device and priority. The rsyslog.conf man page explains the available functionality.
The rsyslog service uses the log message's device and priority to determine how to handle it. Configuration rules are located in files with a .conf extension in the /etc/rsyslog.confand /etc/rsyslog.ddirectory.
The left side of each line represents the device and severity of the syslog messages that match the rule, and the right side represents the file to which the log messages are to be saved.
Monitoring one or more log files for events can help reproduce the problem. tail -f /path/to/file outputs the last 10 lines of the specified file.
Monitor a failed login attempt by running the tail command on one terminal and running ssh as root on the other terminal
[root@workstation ~]# ssh root@localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:iVGX9mvT0cLZlAtI8EK2LJGJR8NpIvjkJYCdg//tZB4.
ECDSA key fingerprint is MD5:81:21:db:98:dd:a2:c4:d5:39:72:a2:6e:57:2c:16:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
root@localhost's password:
Permission denied, please try again.
[root@workstation ~]# tail -f /var/log/secure
Jul 24 02:01:31 workstation sshd[7120]: Server listening on 0.0.0.0 port 22.
Jul 24 02:01:31 workstation sshd[7120]: Server listening on :: port 22.
Jul 24 02:04:35 workstation sshd[7389]: Accepted password for root from 192.168.182.1 port 55567 ssh2
Jul 24 02:04:35 workstation sshd[7389]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 02:04:35 workstation sshd[7393]: Accepted password for root from 192.168.182.1 port 55569 ssh2
Jul 24 02:04:36 workstation sshd[7393]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 02:12:36 workstation sshd[7417]: Accepted password for root from 192.168.182.1 port 55669 ssh2
Jul 24 02:12:36 workstation sshd[7417]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 02:12:36 workstation sshd[7421]: Accepted password for root from 192.168.182.1 port 55670 ssh2
Jul 24 02:12:36 workstation sshd[7421]: pam_unix(sshd:session): session opened for user root by (uid=0)
loJul 24 02:13:36 workstation unix_chkpwd[7451]: password check failed for user (root)
Jul 24 02:13:36 workstation sshd[7449]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=root
Jul 24 02:13:36 workstation sshd[7449]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 24 02:13:38 workstation sshd[7449]: Failed password for root from ::1 port 39398 ssh2
Jul 24 02:14:24 workstation sshd[7449]: Connection closed by ::1 port 39398 [preauth]
Send syslog messages manually
The logger command can send messages to the rsyslog service. By default, messages with priority notice are sent to the user device, unless the -p parameter specifies otherwise.
[root@workstation ~]# logger -p local7.notice "Log entry created on host"
[root@workstation ~]# tail -f -n 5 /var/log/boot.log
Starting Wait for Plymouth Boot Screen to Quit...
[ OK ] Started Command Scheduler.
Starting Terminate Plymouth Boot Screen...
[ OK ] Started NTP client/server.
[ OK ] Started Load/Save RF Kill Switch Status of rfkill0.
Jul 24 02:20:14 workstation root: Log entry created on host
test
Configure rsyslog by adding the rsyslog configuration file /etc/rayslog.d/debug.conf to log all messages with a service priority of debug or above to a new /var/log/messages-debuglog file.
[root@workstation rsyslog.d]# pwd
/etc/rsyslog.d
[root@workstation rsyslog.d]# vi debug.conf
[root@workstation rsyslog.d]# cat debug.conf
*.debug /var/log/messages-debug
[root@workstation ~]# touch /var/log/messages-debug
[root@workstation ~]# systemctl restart rsyslog
Verify that all log messages with priority debug appear in the /var/log/messages-debug file
[root@workstation ~]# logger -p user.debug "Debug Message Test"
[root@workstation ~]# logger -p local7.notice "Log entry created on host"
[root@workstation ~]# tail -f -n 5 /var/log/messages-debug
Jul 24 02:29:26 workstation systemd: Stopped System Logging Service.
Jul 24 02:29:26 workstation systemd: Starting System Logging Service...
Jul 24 02:29:31 workstation rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-34.el7" x-pid="7470" x-info="http://www.rsyslog.com"] start
Jul 24 02:29:31 workstation polkitd[6137]: Unregistered Authentication Agent for unix-process:7463:169671 (system bus name :1.24, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jul 24 02:29:31 workstation systemd: Started System Logging Service.
Jul 24 02:31:44 workstation root: Debug Message Test
View system log entries
Find events
The systemd-journald service stores log data in an indexed, structured binary file called a journal.
To retrieve log messages from the log, use the journalctl command. Use this command to view all messages in the log and search for specific events based on various selections and criteria.
[root@workstation ~]# journalctl
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 02:31:44 EDT. --
Jul 24 02:01:10 servera systemd-journal[86]: Runtime journal is using 6.0M (max allowed 48.6M, trying to leave 72.9M free of
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpuset
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpu
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpuacct
Jul 24 02:01:10 servera kernel: Linux version 3.10.0-957.el7.x86_64 ([email protected]) (gcc version 4.8.5
Jul 24 02:01:10 servera kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos_servera-root
Jul 24 02:01:10 servera kernel: Disabled fast string operations
Jul 24 02:01:10 servera kernel: e820: BIOS-provided physical RAM map:
Jul 24 02:01:10 servera kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable
The journalctl command highlights important log messages: messages with a priority of notice or warning appear in bold text, while messages with a priority of error or above appear in red text.
Show last 5 log messages
[root@workstation ~]# journalctl -n 5
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 02:31:44 EDT. --
Jul 24 02:29:26 workstation systemd[1]: Starting System Logging Service...
Jul 24 02:29:31 workstation rsyslogd[7470]: [origin software="rsyslogd" swVersion="8.24.0-34.el7" x-pid="7470" x-info="http
Jul 24 02:29:31 workstation polkitd[6137]: Unregistered Authentication Agent for unix-process:7463:169671 (system bus name :
Jul 24 02:29:31 workstation systemd[1]: Started System Logging Service.
Jul 24 02:31:44 workstation root[7475]: Debug Message Test
Show log entries with priority err or above
[root@workstation ~]# journalctl -p err
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 02:31:44 EDT. --
Jul 24 02:01:15 servera kernel: sd 2:0:0:0: [sda] Assuming drive cache: write through
Jul 24 02:01:15 servera kernel: sd 2:0:1:0: [sdb] Assuming drive cache: write through
Jul 24 02:01:18 workstation kernel: piix4_smbus 0000:00:07.3: SMBus Host Controller not enabled!
Jul 24 02:01:31 workstation systemd[1]: Failed to start Crash recovery kernel arming.
Show last 5 log entries of today
[root@workstation ~]# journalctl --since today -n 5
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 02:31:44 EDT. --
Jul 24 02:01:10 servera systemd-journal[86]: Runtime journal is using 6.0M (max allowed 48.6M, trying to leave 72.9M free of
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpuset
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpu
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpuacct
Jul 24 02:01:10 servera kernel: Linux version 3.10.0-957.el7.x86_64 ([email protected]) (gcc version 4.8.5
The latest 10 log messages from 2023-7-23 12:00:00 to 2023-7-24 12:00:00
[root@workstation ~]# journalctl --since "2023-7-23 12:00:00" --until "2023-7-24 12:00:00" -n 10
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 02:31:44 EDT. --
Jul 24 02:01:10 servera systemd-journal[86]: Runtime journal is using 6.0M (max allowed 48.6M, trying to leave 72.9M free of
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpuset
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpu
Jul 24 02:01:10 servera kernel: Initializing cgroup subsys cpuacct
Jul 24 02:01:10 servera kernel: Linux version 3.10.0-957.el7.x86_64 ([email protected]) (gcc version 4.8.5
Jul 24 02:01:10 servera kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/centos_servera-root
Jul 24 02:01:10 servera kernel: Disabled fast string operations
Jul 24 02:01:10 servera kernel: e820: BIOS-provided physical RAM map:
Jul 24 02:01:10 servera kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable
Jul 24 02:01:10 servera kernel: BIOS-e820: [mem 0x000000000009ec00-0x000000000009ffff] reserved
Keep system logs
Permanently store system logs
By default, system logs are stored in /run/log/journaldirectories, meaning they are purged when the system calls them. You can /etc/systemd/journal.confchange the configuration in so that the logs are retained across system reboots.
/etc/systemd/journal.confThe Storage parameter in the file determines whether the system log is stored in a volatile manner or persistently.
- volatile: volatile
/run/log/journaldirectory - persistent: stored in
/var/log/journal, can be persisted after the system restarts. - auto: rsyslog decides whether to use persistent storage or volatile storage. If the /var/log/journal directory exists, rsyslog will use persistent storage, otherwise it will use volatile storage.
The advantage of persistent system logs is that historical data can be used after the system is started. However, even with persistent logs, not all data can be persisted. The log size cannot exceed 10% of the file system, nor can it cause the file system free space to fall below 15%. /etc/systemd/journald.confThese values can be adjusted for runtime and persistent logs.
[root@workstation ~]# journalctl | grep -E 'Runtime|System journal'
Jul 24 02:01:10 servera systemd-journal[86]: Runtime journal is using 6.0M (max allowed 48.6M, trying to leave 72.9M free of 480.1M available → current limit 48.6M).
Jul 24 02:01:17 workstation systemd-journal[3147]: Runtime journal is using 6.0M (max allowed 48.6M, trying to leave 72.9M free of 480.1M available → current limit 48.6M).
Jul 24 02:01:23 workstation systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
Jul 24 02:01:23 workstation systemd[1]: Started Tell Plymouth To Write Out Runtime Data.
Configure persistent system logs
/etc/systemd/journald.confSet storage policy
[root@workstation ~]# vi /etc/systemd/journald.conf
[root@workstation ~]# cat /etc/systemd/journald.conf | grep 'Storage'
Storage=persistent
Restart service
[root@workstation ~]# systemctl restart systemd-journald
After restarting, you can see that the /var/log/journal directory has been created, containing one or more subdirectories. The subdirectory names contain hexadecimal characters and contain *.journal files (binary files that store indexed structured log entries) .
[root@workstation ~]# cd /var/log/journal/
[root@workstation journal]# ls
70088e734c8348c2b09d5c6c14125c21
[root@workstation journal]# ls /var/log/journal/70088e734c8348c2b09d5c6c14125c21/
system.journal
Since the system log is retained across reboots, I get a large number of entries in the journalctl output, limiting the system to specific output. The -b option is used with the journalctl command to retrieve entries for the first system startup.
[root@workstation ~]# journalctl -b 1 -n 5
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 03:12:43 EDT. --
Jul 24 03:11:17 workstation systemd[1]: Shutting down.
Jul 24 03:11:17 workstation systemd-shutdown[1]: Syncing filesystems and block devices.
Jul 24 03:11:17 workstation lvm[7775]: 2 logical volume(s) in volume group "centos_servera" unmonitored
Jul 24 03:11:17 workstation systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jul 24 03:11:17 workstation systemd-journal[7539]: Journal stopped
#检索第二次系统启动条目
[root@workstation ~]# journalctl -b 2 -n 5
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 03:12:43 EDT. --
Jul 24 03:12:42 workstation sshd[7386]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 03:12:42 workstation sshd[7390]: Accepted password for root from 192.168.182.1 port 56904 ssh2
Jul 24 03:12:43 workstation systemd[1]: Started Session 2 of user root.
Jul 24 03:12:43 workstation sshd[7390]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 03:12:43 workstation systemd-logind[6231]: New session 2 of user root.
Retrieve current system startup entries
[root@workstation ~]# journalctl -b -n 5
-- Logs begin at Mon 2023-07-24 02:01:10 EDT, end at Mon 2023-07-24 03:12:43 EDT. --
Jul 24 03:12:42 workstation sshd[7386]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 03:12:42 workstation sshd[7390]: Accepted password for root from 192.168.182.1 port 56904 ssh2
Jul 24 03:12:43 workstation systemd[1]: Started Session 2 of user root.
Jul 24 03:12:43 workstation sshd[7390]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jul 24 03:12:43 workstation systemd-logind[6231]: New session 2 of user root.
Maintain accurate time
Set local clock and time zone
Correctly synchronizing system times is critical for analyzing log files across multiple systems.
Displays the current time and related system settings such as current time, time zone and NTP synchronization settings.
[root@workstation ~]# timedatectl
Local time: Mon 2023-07-24 03:20:52 EDT
Universal time: Mon 2023-07-24 07:20:52 UTC
RTC time: Mon 2023-07-24 07:20:52
Time zone: America/New_York (EDT, -0400)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: yes
Last DST change: DST began at
Sun 2023-03-12 01:59:59 EST
Sun 2023-03-12 03:00:00 EDT
Next DST change: DST ends (the clock jumps one hour backwards) at
Sun 2023-11-05 01:59:59 EDT
Sun 2023-11-05 01:00:00 EST
The system provides a database containing time zones
[root@workstation ~]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau
...
Set time zone to Asia/Shanghai
[root@workstation ~]# timedatectl set-timezone Asia/Shanghai
[root@workstation ~]# timedatectl
Local time: Mon 2023-07-24 15:26:28 CST
Universal time: Mon 2023-07-24 07:26:28 UTC
RTC time: Mon 2023-07-24 07:26:28
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a
Set time
[root@workstation ~]# timedatectl set-time 9:00:00
[root@workstation ~]# timedatectl
Local time: Mon 2023-07-24 09:00:05 CST
Universal time: Mon 2023-07-24 01:00:05 UTC
RTC time: Mon 2023-07-24 01:00:06
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: no
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
Set up NTP synchronization and automatically adjust time
[root@workstation ~]# timedatectl set-ntp true
[root@workstation ~]# timedatectl
Local time: Mon 2023-07-24 09:02:11 CST
Universal time: Mon 2023-07-24 01:02:11 UTC
RTC time: Mon 2023-07-24 01:02:11
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
Configuring and monitoring CHRONYD
The chronyd service keeps the inaccurate local hardware clock (RTC) running correctly by synchronizing with the configured ntp service.
Point chronyd to the local time source servera
[root@workstation ~]# vi /etc/chrony.conf
[root@workstation ~]# cat /etc/chrony.conf | grep 'server'
# Use public servers from the pool.ntp.org project.
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
server servera iburst
Restart the service
[root@workstation ~]# systemctl restart chronyd
[root@workstation ~]# chronyc sources -v
210 Number of sources = 5
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ time.cloudflare.com 3 6 77 20 +19ms[ +19ms] +/- 118ms
^* stratum2-1.ntp.mow01.ru.> 2 6 337 24 -26ms[ -56ms] +/- 109ms
^- a.chl.la 2 6 7 25 +40ms[ +10ms] +/- 109ms
^- tick.ntp.infomaniak.ch 1 6 357 20 -4898us[-4898us] +/- 159ms
^? servera 0 8 0 - +0ns[ +0ns] +/- 0ns
Open two terminals to check whether the server and workstation time are synchronized.
[root@workstation ~]# date
Mon Jul 24 15:47:50 CST 2023
[root@servera ~]# date
Mon Jul 24 03:47:50 EDT 2023
The time has been synchronized and servra is the ntp time source of the workstation.