Detailed explanation of Linux Kdump kernel crash dump deployment

Recently, I was reviewing the kdump kernel crash dump technology, just to sort out the relevant knowledge points, and systematically explain the deployment process and principles of Kdump.

The kdump kernel crash dump technology can keep the memory image (including function stack, memory, CPU and other information) at the moment of the crash when dealing with abnormal problems such as the crash of the Linux kernel, so that developers can analyze the cause of the failure. Similar to the coredump mechanism of the application layer.

The process of deploying Kdump requires several knowledge points:

1. How to start the kernel compilation process?
2. kexec mechanism?
3. How to adapt to system startup (depending on conditions)?
4. How to parse the dump file?

---

Kernel compilation opens kdump

The following config parameters need to be enabled and compiled:

1. 在"Processor type and features.“下使能"kexec system call”。 CONFIG_KEXEC=y
2. 在"Filesystem" -> “Pseudo filesystems.“下使能"sysfs file system support”。 CONFIG_SYSFS=y
3. 在"Kernel hacking.“下使能"Compile the kernel with debug info” 。 CONFIG_DEBUG_INFO=y
4. 在"Processor type and features"下使能"kernel crash dumps”。 CONFIG_CRASH_DUMP=y
5. Enable "/proc/vmcore support" under "Filesystems" -> "Pseudo filesystems". CONFIG_PROC_VMCORE=y

---

kexec mechanism

The core of the implementation of the kudmp mechanism is kexec. Kexec can transfer the second kernel (capture kernel) to the specified memory, and when the first kernel crashes, it switches to the second kernel (capture kernel) to run.
Kexec comes from the "kexec-tools" component, compile the kexec-tools tool package to get the kexec tool (centos can download the kexec-tools package and install it directly). To enable kexec in the kernel, you need to enable the CONFIG_KEXEC option as described above

The kexec command call syntax is as follows:

1. Load the capture kernel (the capture kernel can be the same image file as the first kernel, generally the same one is used by default)

kexec -l < kernel-image > --initrd=< initramfsxxx.img > --append="< command-line-options >"

For example: if you want to start the image /boot/bzImage, the content of /proc/cmdline is "root=/dev/hda1", then the command to load the kernel will be:
kexec -l /boot/bzImage -append="root=/ dev/hda1" (that is, inform the kexec entry)

• kernel-image: is the kernel file you want to restart.
• initramfsxxx.img: Capture the root file system of the kernel
• command-line-options: The command line cmdline parameters that need to be passed to the capture kernel. Usually the content parameter of /proc/cmdline can be passed.

2. Manually switch to capture kernel operation.

kexec -e system automatically switches to capture kernel operation (requires the above specified capture kernel entry)

3. In order to automatically jump to the capture kernel to run after the kernel crashes, the following -p parameter is used.

kexec -p < kernel-image > --initrd=< initramfsxxx.img > --append="< command-line-options >"

The above knows: kexec needs to capture kernel files, root file system files. When a kernel crash occurs, the memory image of the first kernel will be saved in /proc/vmcore

---

System boot adaptation capture kernel

1.
In order to automatically adapt the capture kernel every time the crashkernel is turned on, specify the memory space of the capture kernel under the cmdline of the first kernel, that is, add parameters: crashkernel
crashkernel memory selection can refer to: Documentation\kdump\kdump.txt file. Of course, you can also directly use crashkernel=auto, and the system will automatically assign it.

While the “crashkernel=size[@offset]

1. if the RAM is smaller than 512M, then don’t reserve anything (this is the “rescue” case)
2. if the RAM size is between 512M and 2G (exclusive), then reserve 64M
3. if the RAM size is larger than 2G, then reserve 128M

2. initramfsxxx.img
initramfsxxx.img root file system file, after switching to the capture kernel to start, we can make some artificial changes to the content in initramfsxxx.img, such as the need to mount partitions, run some programs or install some drivers during startup .

For example, in centos, the kdump service can automatically generate initramfsxxx.img. There is an important script file of kdump-initxxx.sh in the image file, which is used to compress the /proc/vmcore dump file (makedumpfile tool) and disk Mount the partition and copy files to the specified path (copy to /var/log/crash by default).

Of course, the img file can also be manually generated by the mkinitramfs tool.

3. kexec -p xxx
needs to execute kexec -p xxx to specify the capture kernel entry every time it starts.

In centos, through the kdump service, booting automatically completes the operation of kexec setting the capture kernel. If the image file (xxxkdump.img) of the root file system of the kernel is not captured under /boot, it will also be automatically generated.

centos为例，服务所在脚本里 /usr/bin/kdumpctl实际kexec装载命令如下：

/sbin/kexec -p --command-line="BOOT_IMAGE=/vmlinuz-3.10.0-123.el7.x86_64 root=UUID=79704805-e306-420b-827a-52849e1376c1 ro vconsole.keymap=us vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_US.UTF-8 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug disable_cpu_apicid=0"   --initrd=/boot/initramfs-3.10.0-123.el7.x86_64kdump.img /boot/vmlinuz-3.10.0-123.el7.x86_64

Cmdline under the first kernel

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-123.el7.x86_64 root=UUID=79704805-e306-420b-827a-52849e1376c1 ro vconsole.keymap=us crashkernel=auto vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_US.UTF-8

In the /etc/sysconfig/kdump configuration file, an additional supplement to capture kernel cmdline input has been added

KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug"

crash and vmlinux

The kernel panic can be triggered by echo c> /pros/sysrq-trigger. Then generate a dump file vmcore.

After the dump file vmcore is generated, it needs to be parsed and analyzed by the crash tool, which relies on a kernel debugging file: vmlinux .

1. vmlinux The vmlinux
mentioned here is a file based on the ELF format, which is just an uncompressed version of the kernel image that can be used for debugging . And zImage or bzImage is a compressed version of the kernel image, usually used for booting.

When compiling the kernel image file vmlinux, the -g parameter must be specified, that is, with debugging information, it can be generated after compilation (the centos package can be found in the kernel-debuginfo-xxx.rpm kernel package).

2. The crash
crash startup analysis command is as follows:

crash vmlinux vmcore

If crash does not specify vmcore, the current real-time system memory is used by default.

After entering the crash, you need to use the crash related commands (the commands are omitted) to obtain the information when the first kernel is down (such as: each cpu call stack, memory margin, process information, etc.), and then you can analyze the cause of the kernel crash The specific reason.