Description: On the ARM server, oops abnormal structure, this should generate panic, flow into the dump, and reboot the system, but the system does not restart, but there has been stuck in the serial print cycle will from time to time call stack information . As follows
linux-fATqUY login: [ME] Fault detect start!
[ME] Fault detect start!
[ 254.202183] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 254.211111] Mem abort info:
[ 254.213939] ESR = 0x96000044
[ 254.217042] Exception class = DABT (current EL), IL = 32 bits
[ 254.223054] SET = 0, FnV = 0
[ 254.226154] EA = 0, S1PTW = 0
[ 254.229335] Data abort info:
[ 254.232261] ISV = 0, ISS = 0x00000044
[ 254.236155] CM = 0, WnR = 1
[ 254.239167] user pgtable: 4k pages, 48-bit VAs, pgdp = 00000000509fb5ee
[ 254.245883] [0000000000000000] pgd=0000000000000000
[ 254.250838] Internal error: Oops: 96000044 [#1] SMP
[ 254.255785] CPU: 0 PID: 58147 Comm: gen_seri_oops/0 Kdump: loaded Tainted:
[ 254.273985] pstate: 60000005 (nZCv daif -PAN -UAO)
[ 254.278851] pc : gen_seri_oops+0x28/0x38 [kpgen_kbox]
[ 254.283971] lr : gen_seri_oops+0x1c/0x38 [kpgen_kbox]
[ 254.289089] sp : ffff000033ea3e50
[ 254.292443] x29: ffff000033ea3e50 x28: 0000000000000000
[ 254.297831] x27: ffff000035fdbb08 x26: ffff803f713412b8
[ 254.303218] x25: ffff000000fb2370 x24: 0000000000000000
[ 254.308605] x23: ffff0000091bcb10 x22: ffff80df64dc2e80
[ 254.380350] Process gen_seri_oops/0 (pid: 58147, stack limit = 0x0000000088aed8c0)
[ 254.391639] Call trace:
[ 254.397760] gen_seri_oops+0x28/0x38 [kpgen_kbox]
[ 254.406069] kthread+0x108/0x138
[ 254.412721] ret_from_fork+0x10/0x18
[ 254.419733] Code: 95c66ac7 d2800001 52800c22 52800000 (39000022)
[ 254.429390] die event detected
[ 254.436533] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
[ 254.448966] Mem abort info:
[ 254.455502] ESR = 0x96000004
[ 254.462278] Exception class = DABT (current EL), IL = 32 bits
[ 254.472023] SET = 0, FnV = 0
[ 254.478845] EA = 0, S1PTW = 0
[ 254.485645] Data abort info:
[ 254.492130] ISV = 0, ISS = 0x00000004
[ 254.499576] CM = 0, WnR = 0
[ 254.505933] user pgtable: 4k pages, 48-bit VAs, pgdp = 00000000509fb5ee
[254.515960] [0000000000000008] PGD = 0000000000000000
360.462046] watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [docker-containe:29506] [ 360.473208] Modules linked in: pt_MASQUERADE(E) iptable_nat(E) …… [ 360.565132] CPU: 35 PID: 29506 Comm: docker-containe Kdump: loaded Tainted: G OEL 4.19.5-1.1.29.aarch64 #1[ 360.595535] pstate: 80000005 (Nzcv daif -PAN -UAO) [ 360.604905] pc : smp_call_function_many+0x308/0x370 [ 360.614105] lr : smp_call_function_many+0x2c4/0x370 [ 360.623278] sp : ffff00002f4739e0 [ 360.630878] x29: ffff00002f4739e0 x28: ffff000009592184 [ 360.640673] x27: ffff80bffbeb2f08 x26: 0000000000000000 [ 360.650447] x25: ffff00000818a790 x24: 0000000000000001 [ 360.660310] x23: 0000000000001000 x22: ffff000009592184 [ 360.670118] x21: ffff000009589000 x20: ffff80bffbeb2d08 [ 360.680004] x19: ffff80bffbeb2d00 x18: 000000000bf5d62e [ 360.689922] x17: 000000bf5d62e000 x16: 000000bf5d62f000 [ 360.699700] x15: 0000000000000008 x14: 0000000000000000 [ 360.709468] x13: 0000000000000001 x12: ffff803f72c8b530 [ 360.719251] x11: 0000000000000000 x10: 0000000000000b80 [ 360.728983] x9 : 0000000000000000 x8 : ffff80bffbeb3108 [ 360.738859] x7 : 0000000000000000 x6 : 000000000000003f [ 360.748533] x5 : 0000000000000000 x4 : fffffff7ffffffff [ 360.758230] x3 : 0000000000000000 x2 : ffff803ffbe69638 [ 360.767909] x1 : 0000000000000003 x0 : 0000000000000000 [ 360.777504] Call trace: [ 360.784282] smp_call_function_many+0x308/0x370 [ 360.793075] kick_all_cpus_sync+0x30/0x38 [ 360.801443] sync_icache_aliases+0x74/0x98 [ 360.809852] __sync_icache_dcache+0x94/0xc8 [ 360.818371] alloc_set_pte+0x460/0x570 [ 360.826435] filemap_map_pages+0x3e0/0x400 [ 360.834510] __handle_mm_fault+0xb78/0x10f0 [ 360.842743] handle_mm_fault+0xf4/0x1c0 [ 360.850561] do_page_fault+0x230/0x488 [ 360.858095] do_translation_fault+0x74/0x84 [ 360.865948] do_mem_abort+0x6c/0x130 [ 360.873255] do_el0_ia_bp_hardening+0x64/0xa8 [ 360.881020] el0_ia+0x18/0x1c [ 364.514043] watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [vm_io_monitor.p:10065] 下面的内容同上
[ 464.446045] watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [kill:2031]
[ 516.538048] watchdog: BUG: soft lockup - CPU#58 stuck for 23s! [libvirtd:29334]
[ 576.358044] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [logrotate:2045]
Cause of the problem: the flow into the dump, dump-related components such as the kernel box appeared null pointer access, there has been oops nesting. Each time after the oops occurs, the kernel exception handling process, perform the die function, which will acquire the lock, raw_spin_lock_irqsave ($ die_lock, flags) , and then call the kernel box will register the callback function in the notification chain, the callback process , due to the callback function and there are internal null pointer access, appeared oops, and exception handling into the flow into the die function, then it is you want to get a lock, you get less, I have been waiting for the lock.
The question is:
1, why each cpu print the call stack is stuck 20s about it?
2, if there was a deadlock cpu, cpu why other softlockup it?
3, why the call stack for each cpu are in smp_call_function_many
Now we look at the first question, 20s time exactly softlockup time, then what is softlockup does, in fact, plainly, is to seize the time to be closed, resulting in process can not dispatch. System for each cpu core registered a general kernel thread, which regularly calls watchdog function that when the clock interrupt the update, kernel threads will be to get running. When the kernel threads in the watchdog function of a variable watchdog_touch_ts 20s have not been updated recently, it would mean that this thread is not scheduled in the 20s, it is very likely to seize was closed on one cpu core. Therefore, the scheduler performs scheduling not. Then the corresponding is hardlockup, is a more serious problem caused by interruption is long closed.
So why a deadlock appears on the cpu, cpu others will also appear softlockup it, let us analyze the call stack softlockup, and when unclear calltrace functions are doing, and why other cpu analysis will It appears softlockup is a difficult task. After my multiple searches, almost sort out ideas. Two key functions in the call stack is kick_all_cpus_sync, sync_icache_aliases, from the name of the function would be able to see that these functions are used for cache synchronization between the cpu. We are about 64 on the server cpu, in order to ensure cache consistency, there should be a mechanism to synchronize all cpu. So how to synchronize search from a patch article to https://patchwork.kernel.org/patch/10325093/ , can be found in some of the relevant information, the following:
kick_all_cpus_sync() forces all CPUs to sync caches by sending broadcast IPI. If CPU is in extended quiescent state (idle task or nohz_full userspace), this work may be done at the exit of this state.
That broadcast by IPI interrupt to force the cpu to synchronize all cache. Specific IPI interrupt related presentations can refer http://www.voidcn.com/article/p-auwhbhgz-oh.html . IPI interrupt, it is a way of inter-core communication synchronization.
This blog https://segmentfault.com/a/1190000017238912 , in fact, like with my problems, summarize here some excerpts:
On the one hand, in order to avoid competition, when the refresh thread local tlb, it will seize stopped. This leads to a result: the other threads, including of course the watchdog thread, there is no way to be scheduled for execution (soft lockup). On the other hand, in order to ask other cpu refresh synchronously tlb, the current thread will use ipi and other cpu synchronization progress, until the other cpu also completed refreshed. Other cpu if the delay does not cooperate, then the current thread will die and so on.
We look at the rearmost smp_call_function_many this function is waiting.
void smp_call_function_many(const struct cpumask *mask, smp_call_func_t func, void *info, bool wait) { …… if (wait) { for_each_cpu(cpu, cfd->cpumask) { struct call_single_data *csd; csd = per_cpu_ptr(cfd->csd, cpu); csd_lock_wait(csd); } } }
Then when appeared on cpu 0 after oops, appeared behind the deadlock, when you enter oops, the first thing to do is to disable interrupts. This is very easy to understand, oops logical thing to do is preserve the scene, it certainly does not want to interrupt the scene undermine issue at this time.
Analysis here, the problem is clear, when other cpu routinely sent me IPI interrupt, CPU0 can not respond to the emergence of a deadlock, so the other cpu in death and so on, which led to other cpu have had softlockup.
appendix:
do_mem_abort in the arm architecture, a page fault occurs when an exception or interrupt exception handling arm after visiting a null pointer. __handle_mm_fault This function is the handler when dealing with large pages missing pages. Specific reference http://www.leviathan.vip/2019/03/03/Linux%E5%86%85%E6%A0%B8%E6%BA%90%E7%A0%81%E5%88%86 % E6% 9E% 90-% E5 % 86% 85% E5% AD% 98% E8% AF% B7% E9% A1% B5% E6% 9C% BA% E5% 88% B6 /
Core processing flow after the null pointer access https://blog.csdn.net/rikeyone/article/details/80021720 , https://blog.csdn.net/zangdongming/article/details/38543059