In the process of our project development, many times there will be a situation where the mobile phone system crashes and restarts due to some reason (restart is divided into Android restart and kernel restart, and we only discuss the situation of kernel restart, that is, kernel panic), crash restart Basically, it is the most serious system problem. There are stable recurrences and probabilities. The difficulty of solving the problems is also very different. After a problem occurs, we usually get kernel log information like this (the following log is only used to call BUG( ) as an example, there will be some differences in the crash log information caused by other abnormalities):
[ 2.052157] <2>-(2)[1:swapper/0]------------[ cut here ]------------
[ 2.052163] <2>-(2)[1:swapper/0]Kernel BUG at c04289dc [verbose debug info unavailable]
[ 2.052169] <2>-(2)[1:swapper/0]Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[ 2.052178] <2>-(2)[1:swapper/0]disable aee kernel api[ 3.052192] <2>-(2)[1:swapper/0]Non-crashing CPUs did not react to IPI
[ 3.052204] <2>-(2)[1:swapper/0]CPU: 2 PID: 1 Comm: swapper/0 Tainted: G W 3.18.35+ #3
[ 3.052211] <2>-(2)[1:swapper/0]task: df060000 ti: df04a000 task.ti: df04a000
[ 3.052227] <2>-(2)[1:swapper/0]PC is at ltr553_i2c_probe+0x94/0x9c
[ 3.052233] <2>-(2)[1:swapper/0]LR is at 0x0
[ 3.052242] <2>-(2)[1:swapper/0]pc : [<c04289dc>] lr : [<00000000>] psr: a0000113
[ 3.052242] <2>sp : df04bd30 ip : 00000000 fp : df04bd4c
[ 3.052249] <2>-(2)[1:swapper/0]r10: 00000003 r9 : de348fc0 r8 : c0428948
[ 3.052255] <2>-(2)[1:swapper/0]r7 : dea1bc00 r6 : dea1bc04 r5 : dea1bc20 r4 : c0b53358
[ 3.052262] <2>-(2)[1:swapper/0]r3 : c115ef4c r2 : 00000000 r1 : 00000000 r0 : de366a00
[ 4.354655] <2>-(2)[1:swapper/0] oops_end, 1, 11
[ 4.354740] <2>-(2)[1:swapper/0]Kernel panic - not syncing: Fatal exceptionThis is the relevant important information output by the linux kernel before it crashes, including PC pointers, call stacks, etc., which are very important clues for debugging. For example, we can use the GUN tools (add2Line) tool combined with the kernel symbol mapping table vmlinux to locate the current The specific line number of the code where the PC pointer is located (locating the error code line does not mean that the root cause of the problem has been found and the exception has been fixed, this depends on the complexity of the exception). An in-depth understanding of the meaning and mechanism of these key print log information is very helpful for us to locate and analyze such crash problems (there are limitations in the analysis of problems caused by memory being stepped on and hardware instability), which is what we need to in-depth The original intention of learning the kernel exception process.
Here we have to figure out a few questions:
- How did the key register information left before the crash come from, what is the use, and what is the specific meaning?
- How to use these remaining clues to find out which file and line the problematic code is in?
- What is the overall process from a fatal exception to a crash in the kernel, and how to analyze problems like a crash?
For this reason, this article takes the most common active trigger BUG() as an example to analyze the above questions and analyze the entire kernel panic process.
What is BUG()?
People who have experience in driver debugging must know this thing. The BUG here is not the same as the "software defect" we generally think of. The BUG() mentioned here is actually used in the linux kernel to intercept kernel programs that exceed expectations. Behavior is a mechanism for software to actively report exceptions. There is a question here, when will it be used? Generally speaking, there are two situations in which it is used. One is during the software development process. If a fatal fault is found in the code logic, you can call BUG() to make the kernel die (similar to assert). This is convenient for locating the problem and correcting it. Code execution logic; Another situation is that we need to use it when the system enters kernel panic due to some special reason (usually ramdump is needed for debugging).
BUG() and BUG_ON(1) are essentially the same thing. The latter is just a simple encapsulation based on the former. The essence of BUG() is to embed an undefined instruction: 0xe7f001f2, triggering ARM to initiate Undefined Instruction exception (PS: ARM has 10 types of exceptions, you can review the ARM exception model chapter for details).
<kernel-3.18/arch/arm/include/asm/bug.h>
#define BUG_INSTR_VALUE 0xe7f001f2
#define BUG_INSTR(__value) __inst_arm(__value)
#define BUG() _BUG(__FILE__, __LINE__, BUG_INSTR_VALUE)
#define _BUG(file, line, value) __BUG(file, line, value)
#define __BUG(__file, __line, __value) \
do { \
asm volatile(BUG_INSTR(__value) "\n"); \
unreachable(); \
} while (0)BUG() process analysis
The general flow chart from BUG() to system restart:
Calling BUG() will send an undefined instruction to the CPU, triggering ARM to initiate an undefined instruction exception, and then enter the kernel exception handling process to go through Oops, die(), __die() and other processes to output key clues for debugging and analysis, and finally Enter panic() to end the process of rebirth. This is the basic flow of the whole process. Let’s first look at what die() specifically does.
die() process
source code:
void die(const char *str, struct pt_regs *regs, int err)
{
enum bug_trap_type bug_type = BUG_TRAP_TYPE_NONE;
unsigned long flags = oops_begin();
int sig = SIGSEGV;
if (!user_mode(regs))
bug_type = report_bug(regs->ARM_pc, regs);
if (bug_type != BUG_TRAP_TYPE_NONE)
str = "Oops - BUG";
if (__die(str, err, regs))
sig = 0;
oops_end(flags, regs, sig);
}
The overall process is roughly as follows:
Generally speaking, the code analysis process combined with the kernel log will give a deeper understanding. If it is an exception caused by BUG()/BUG_ON(1), then you can see the following iconic log when you go to report_bug:
enum bug_trap_type report_bug(unsigned long bugaddr, struct pt_regs *regs)
{
...
if (!is_valid_bugaddr(bugaddr))
return BUG_TRAP_TYPE_NONE;
...
printk(KERN_DEFAULT "------------[ cut here ]------------\n");
if (file)
pr_crit("kernel BUG at %s:%u!\n", file, line);
else
pr_crit("Kernel BUG at %p [verbose debug info unavailable]\n",
(void *)bugaddr);
===>
[ 2.052157] <2>-(2)[1:swapper/0]------------[ cut here ]------------
[ 2.052163] <2>-(2)[1:swapper/0]Kernel BUG at c04289dc [verbose debug info unavailable]
[ 2.052169] <2>-(2)[1:swapper/0]Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
Therefore, if you see this "[ cut here ]" information in the log, you can infer that the system restarted due to a fatal fault in the software and actively called BUG(), and you can try to locate, analyze and repair the exception based on the relevant information. Here It should be noted that there is another __WARN() situation that will also print out the iconic log of "[ cut here ]", but the kernel will not hang up, which is easy to cause misleading:
#define __WARN() warn_slowpath_null(__FILE__, __LINE__)
#define __WARN_printf(arg...) warn_slowpath_fmt(__FILE__, __LINE__, arg)
void warn_slowpath_fmt(const char *file, int line, const char *fmt, ...)
{
...
warn_slowpath_common(file, line, __builtin_return_address(0),
TAINT_WARN, &args);
static void warn_slowpath_common(const char *file, int line, void *caller,
unsigned taint, struct slowpath_args *args)
{
...
pr_warn("------------[ cut here ]------------\n");
pr_warn("WARNING: CPU: %d PID: %d at %s:%d %pS()\n",
raw_smp_processor_id(), current->pid, file, line, caller);
===>
[ 1.106219] <2>-(2)[1:swapper/0]------------[ cut here ]------------
[ 1.107018] <2>-(2)[1:swapper/0]WARNING: CPU: 2 PID: 1 at /home/android/work/prj/n-6580/kernel-3.18/kernel/irq/manage.c:454 __enable_irq+0x50/0x8c()
So in fact, it is easy to distinguish the two situations from the appearance. If it is BUG()/BUG_ON(1), the kernel will definitely hang up and restart, and __WARN() will only dump_stack() without crashing. From the source code and log information It is also easy to distinguish the two situations. If it is BUG()/BUG_ON(1), there must be a log output similar to the following, just search for the keyword: "Internal error: Oops".
[ 2.052163] <2>-(2)[1:swapper/0]Kernel BUG at c04289dc [verbose debug info unavailable]
[ 2.052169] <2>-(2)[1:swapper/0]Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARMInformation through train: Linux kernel source code technology learning route + video tutorial kernel source code
Learning through train: Linux kernel source code memory tuning file system process management device driver/network protocol stack
__die() process analysis
The log information output from the above is not enough to locate the specific problematic code location, including the most critical PC pointer, call stack and other clues that are crucial for debugging are in __die() in the output.
flow chart:
Print out the iconic log information:
static int __die(const char *str, int err, struct pt_regs *regs)
{
...
printk(KERN_EMERG "Internal error: %s: %x [#%d]" S_PREEMPT S_SMP
S_ISA "\n", str, err, ++die_counter);
===>
[ 2.052169] <2>-(2)[1:swapper/0]Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
The log shows that the abnormal str is Oops - BUG, the error-code is 0, and the number of die counters: 1
The original meaning of Oops is a slang term for "oops", which means that the kernel has an accident and does not know how to deal with it.
notify_die() will notify modules that are interested in Oops to execute related callbacks. For example, the aee exception engine module of mtk is registered to the die_chain notification chain.
int notrace notify_die(enum die_val val, const char *str,
struct pt_regs *regs, long err, int trap, int sig)
{
struct die_args args = {
.regs = regs,
.str = str,
.err = err,
.trapnr = trap,
.signr = sig,
};
return atomic_notifier_call_chain(&die_chain, val, &args);
}
The aee exception engine of mtk will register to the die_chain notification chain when the kernel is initialized, and we can see that the panic notification chain is actually registered.
int __init aee_ipanic_init(void)
{
spin_lock_init(&ipanic_lock);
mrdump_init();
atomic_notifier_chain_register(&panic_notifier_list, &panic_blk);
register_die_notifier(&die_blk);
register_ipanic_ops(&ipanic_oops_ops);
ipanic_log_temp_init();
ipanic_msdc_init();
LOGI("ipanic: startup, partition assgined %s\n", AEE_IPANIC_PLABEL);
return 0;
}
The key information useful for our debugging and tracking is printed in __show_regs():
void __show_regs(struct pt_regs *regs)
{
unsigned long flags;
char buf[64];
show_regs_print_info(KERN_DEFAULT);
print_symbol("PC is at %s\n", instruction_pointer(regs));
print_symbol("LR is at %s\n", regs->ARM_lr);
printk("pc : [<%08lx>] lr : [<%08lx>] psr: %08lx\n"
"sp : %08lx ip : %08lx fp : %08lx\n",
regs->ARM_pc, regs->ARM_lr, regs->ARM_cpsr,
regs->ARM_sp, regs->ARM_ip, regs->ARM_fp);
printk("r10: %08lx r9 : %08lx r8 : %08lx\n",
regs->ARM_r10, regs->ARM_r9,
regs->ARM_r8);
printk("r7 : %08lx r6 : %08lx r5 : %08lx r4 : %08lx\n",
regs->ARM_r7, regs->ARM_r6,
regs->ARM_r5, regs->ARM_r4);
printk("r3 : %08lx r2 : %08lx r1 : %08lx r0 : %08lx\n",
regs->ARM_r3, regs->ARM_r2,
regs->ARM_r1, regs->ARM_r0);
flags = regs->ARM_cpsr;
buf[0] = flags & PSR_N_BIT ? 'N' : 'n';
buf[1] = flags & PSR_Z_BIT ? 'Z' : 'z';
buf[2] = flags & PSR_C_BIT ? 'C' : 'c';
buf[3] = flags & PSR_V_BIT ? 'V' : 'v';
buf[4] = '\0';
printk("Flags: %s IRQs o%s FIQs o%s Mode %s ISA %s Segment %s\n",
buf, interrupts_enabled(regs) ? "n" : "ff",
fast_interrupts_enabled(regs) ? "n" : "ff",
processor_modes[processor_mode(regs)],
isa_modes[isa_mode(regs)],
get_fs() == get_ds() ? "kernel" : "user");
show_extra_register_data(regs, 128);
}
void dump_stack_print_info(const char *log_lvl)
{
printk("%sCPU: %d PID: %d Comm: %.20s %s %s %.*s\n",
log_lvl, raw_smp_processor_id(), current->pid, current->comm,
print_tainted(), init_utsname()->release,
(int)strcspn(init_utsname()->version, " "),
init_utsname()->version);
...
This prints out important information such as where the PC stopped, related register information, whether a user or kernel exception occurred, the cpu where the exception occurred, and the process pid.
Next dump_mem() is used to dump the memory information of the current thread:
dump_mem(KERN_EMERG, "Stack: ", regs->ARM_sp,
THREAD_SIZE + (unsigned long)task_stack_page(tsk));
Use dump_backtrace(regs, tsk) to print out the most intuitive call stack information for debugging:
[ 3.056363] <2>-(2)[1:swapper/0]Backtrace: -(2)[1:swapper/0]
[ 3.056386] <2>-(2)[1:swapper/0][<c010badc>] (dump_backtrace) from [<c010bc7c>] (show_stack+0x18/0x1c) [ 3.056393] <2>-(2)[1:swapper/0] r6:c103d790-(2)[1:swapper/0] r5:ffffffff-(2)[1:swapper/0] r4:00000000-(2)[1:swapper/0] r3:00000000-(2)[1:swapper/0] [ 3.056426] <2>-(2)[1:swapper/0][<c010bc64>] (show_stack) from [<c0a91e64>] (dump_stack+0x90/0xa4) [ 3.056439] <2>-(2)[1:swapper/0][<c0a91dd4>] (dump_stack) from [<c072d264>] (ipan6503] <2>-(2)[1:swapper/0][<c013e6bc>] (notifier_call_chain) from [<c013eb84>] (atomic_notifier_call_chain+0x3c/0x50) [ 3.056509] <2>-(2)[1:swapper/0] r10:c10efec4-(2)[1:swapper/0] r9:df060000-(2)[1:swapper/0] r8:df04a020-(2)[1:swapper/0] r7:c0caaaf0-(2)[1:swapper/0] r6:c10f0c88-(2)[1:swapper/0] r5:00000001 [ 3.056539] <2>-(2)[1:swapper/0] r4:df04bb74-(2)[1:swapper/0] [ 3.056554] <2>-(2)[1:swapper/0][<c013eb48>] (atomic_notifier_call_chain) from [<c013f244>] (notify_die+0x44/0x4c) [ 3.056560] <2>-(2)[1:swapper/0] r6:df04bce8-(2)[1:swapper/0] r5:00000000-(2)[1:swapper/0] r4:00000001-(2)[1:swapper/0] [ 3.056585] <2>-(2)[1:swapper/0][<c013f200>] (notify_die) from [<c010bd94>] (die+0x114/0x41c) [ 3.056590] <2>-(2)[1:swapper/0] r4:c102826c-(2)[1:swapper/0] [ 3.056607] <2>-(2)[1:swapper/0][<c010bc80>] (die) from [<c010c0c0>] (arm_notify_die+0x24/0x5c) [ 3.056612] <2>-(2)[1:swapper/0] r10:df04a000-(2)[1:swapper/0] r9:00000000-(2)[1:swapper/0] r8:df04bce8-(2)[1:swapper/0] r7:e7f001f2-(2)[1:swapper/0] r6:df04a000-(2)[1:swapper/0] r5:c04289dc [ 3.056642] <2>-(2)[1:swapper/0] r4:00000004-(2)[1:swapper/0] [ 3.056658] <2>-(2)[1:swapper/0][<c010c09c>] (arm_notify_die) from [<c01001cc>] (do_undefinstr+0x1a4/0x1ec) [ 3.056670] <2>-(2)[1:swapper/0][<c0100028>] (do_undefinstr) from [<c010c98c>] (__und_svc_finish+0x0/0x34) [ 3.056676] <2>-(2)[1:swapper/0]Exception stack(0xdf04bce8 to 0xdf04bd30) [ 3.056687] <2>-(2)[1:swapper/0]bce0: de366a00 00000000 00000000 c115ef4c c0b53358 dea1bc20 [ 3.056698] <2>-(2)[1:swapper/0]bd00: dea1bc04 dea1bc00 c0428948 de348fc0 00000003 df04bd4c 00000000 df04bd30 [ 3.056706] <2>-(2)[1:swapper/0]bd20: 00000000 c04289dc a0000113 ffffffff [ 3.056711] <2>-(2)[1:swapper/0] r9:c010c98c-(2)[1:swapper/0] r8:e7100000-(2)[1:swapper/0] r7:00000000-(2)[1:swapper/0] r6:c010cd98-(2)[1:swapper/0] r5:00000000-(2)[1:swapper/0] r4:c04289e0 [ 3.056750] <2>-(2)[1:swapper/0][<c0428948>] (ltr553_i2c_probe) from [<c07d88d8>] (i2c_device_probe+0xd0/0x12c) [ 3.056756] <2>-(2)[1:swapper/0] r5:dea1bc20-(2)[1:swapper/0] r4:c0b53358-(2)[1:swapper/0] [ 3.056778] <2>-(2)[1:swapper/0][<c07d8808>] (i2c_device_probe) from [<c03c298c>] (driver_probe_device+0x160/0x43c)
Through the above call stack information combined with GUN Tools (add2Line), you can basically locate the specific code location where the exception occurred.
Finally, the pc pointer and the first 4 instructions will be printed out through dump_instr(KERN_EMERG, regs):
static void dump_instr(const char *lvl, struct pt_regs *regs)
{
...
for (i = -4; i < 1 + !!thumb; i++) {
unsigned int val, bad;
if (thumb)
bad = __get_user(val, &((u16 *)addr)[i]);
else
bad = __get_user(val, &((u32 *)addr)[i]);
if (!bad)
p += sprintf(p, i == 0 ? "(%0*x) " : "%0*x ", width, val);
else {
p += sprintf(p, "bad PC value");
break;
}
}
printk("%sCode: %s\n", lvl, str);
..
===>
[ 3.226706] <2>-(2)[1:swapper/0][<c0a8ae74>] (/0]Code: e89da830 e30e3f4c e34c3115 e5830000 (e7f001f2)
Have you seen this e7f001f2, does it look familiar? This is the undefined instruction embedded in BUG()!
At this point, most of the key information has been output. You can use the add2Line tool to locate the specific line number of the dead code, and roughly see what happened. If it is an exception caused by BUG(), then you can consider analyzing and repairing it. Abnormal, because BUG() is an active reporting exception, generally speaking, the difficulty of debugging is much easier than other passive reporting methods.
For example:
Know the address where the PC died from the above log, and use the add2Line tool combined with the kernel symbol mapping table vmlinux to locate the line number of the file where the specific code is located:
arm-linux-androideabi-addr2line -e out/target/product/$project/obj/KERNEL_OBJ/vmlinux -f -C c04289dc
ltr553_i2c_probe
/aosp/kernel-3.18/drivers/misc/mediatek/alsps/ltr553/ltr553.c:3278
After locating the specific code line number, you can further analyze the code log to find out the cause of the problem and fix the exception (generally speaking, the exception caused by BUG() is easier to solve, and the difficulty of other cases is very different..). So what will the kernel do next? After the important information has been output, it will go directly to the kernel panic process.
panic process
The original meaning of panic is "panic". Here it means that the kernel has a fatal error and cannot continue to run.
flow chart:
Finally, attach the overall timing diagram:
