参考

https://www.kernel.org/doc/html/latest/RCU/stallwarn.html

简介

这两个内核日志的原因是一样的，原因就是CPU被某些进程：不管是Kernel内部的还是应用层的，占用的CPU非常高，而且对应的rcu_sched和 khugepaged两个线程得不到足够的CPU来处理内核事务。就会打印这个trace出来

日志实例

Jul 27 11:32:28 -a kernel: [ 4953.392255] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Jul 27 11:32:28 -a kernel: [ 4953.394479] (detected by 0, t=60004 jiffies, g=283629, q=3698)
Jul 27 11:32:28 -a kernel: [ 4953.396400] rcu: All QSes seen, last rcu_sched kthread activity 59914 (4299620612-4299560698), jiffies_till_next_fqs=3, root ->qsmask 0x0
Jul 27 11:32:28 -a kernel: [ 4953.400032] rcu: rcu_sched kthread starved for 59917 jiffies! g283629 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3
Jul 27 11:32:28 -a kernel: [ 4953.402585] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
Jul 27 11:32:28 -a kernel: [ 4953.404946] rcu: RCU grace-period kthread stack dump:
Jul 27 11:32:28 -a kernel: [ 4953.406728] task:rcu_sched state:R running task stack: 0 pid: 11 ppid: 2 flags:0x80004080
Jul 27 11:32:28 -a kernel: [ 4953.409229] Call Trace:
Jul 27 11:32:28 -a kernel: [ 4953.410643] __schedule+0x2bd/0x760
Jul 27 11:32:28 -a kernel: [ 4953.412203] schedule+0x37/0xa0
Jul 27 11:32:28 -a kernel: [ 4953.413684] schedule_timeout+0x197/0x300
Jul 27 11:32:28 -a kernel: [ 4953.415207] ? __next_timer_interrupt+0xf0/0xf0
Jul 27 11:32:28 -a kernel: [ 4953.416754] rcu_gp_kthread+0x48a/0xa50
Jul 27 11:32:28 -a kernel: [ 4953.418404] ? call_rcu+0x550/0x550
Jul 27 11:32:28 -a kernel: [ 4953.419974] kthread+0x116/0x130
Jul 27 11:32:28 -a kernel: [ 4953.421488] ? kthread_flush_work_fn+0x10/0x10
Jul 27 11:32:28 -a kernel: [ 4953.423124] ret_from_fork+0x1f/0x40
Jul 27 11:50:54 -a kernel: [ 6063.170731] INFO: task khugepaged:47 blocked for more than 120 seconds.
Jul 27 11:50:54 -a kernel: [ 6063.173029] Tainted: G OE --------- - - 4.18.0-348.20.1.el8_5.x86_64 #1
Jul 27 11:50:54 -a kernel: [ 6063.175283] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
Jul 27 11:50:54 -a kernel: [ 6063.177464] task:khugepaged state:D stack: 0 pid: 47 ppid: 2 flags:0x80004080
Jul 27 11:50:54 -a kernel: [ 6063.179734] Call Trace:
Jul 27 11:50:54 -a kernel: [ 6063.181142] __schedule+0x2bd/0x760
Jul 27 11:50:54 -a kernel: [ 6063.182667] schedule+0x37/0xa0
Jul 27 11:50:54 -a kernel: [ 6063.184131] schedule_timeout+0x274/0x300
Jul 27 11:50:54 -a kernel: [ 6063.185685] ? enqueue_entity+0xf1/0x6e0
Jul 27 11:50:54 -a kernel: [ 6063.187223] ? check_preempt_wakeup+0x113/0x230
Jul 27 11:50:54 -a kernel: [ 6063.188881] wait_for_completion+0x97/0x100
Jul 27 11:50:54 -a kernel: [ 6063.190462] flush_work+0x11d/0x1c0
Jul 27 11:50:54 -a kernel: [ 6063.191916] ? worker_attach_to_pool+0xa0/0xa0
Jul 27 11:50:54 -a kernel: [ 6063.193497] ? lru_add_drain+0x30/0x30
Jul 27 11:50:54 -a kernel: [ 6063.194960] lru_add_drain_all+0x13f/0x190
Jul 27 11:50:54 -a kernel: [ 6063.196468] khugepaged+0x66/0x11d0
Jul 27 11:50:54 -a kernel: [ 6063.197854] ? finish_wait+0x80/0x80
Jul 27 11:50:54 -a kernel: [ 6063.199255] ? collapse_pte_mapped_thp+0x430/0x430
Jul 27 11:50:54 -a kernel: [ 6063.200801] kthread+0x116/0x130
Jul 27 11:50:54 -a kernel: [ 6063.202130] ? kthread_flush_work_fn+0x10/0x10
Jul 27 11:50:54 -a kernel: [ 6063.203596] ret_from_fork+0x1f/0x40

如果实时应用程序占用CPU非常高

除了会导致内核日志的打印，同时会影响，softirqd内核进程的正常运行；间接影响中断，定时器（pthread_condition相关功能，接收数据的软中断处理）
[root@10 ~]# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm | grep soft
12 12 TS - 0 19 0 0.0 S - ksoftirqd/0

[root@10 proc]# cat softirqs
CPU0
HI: 2
TIMER: 19307
NET_TX: 106
NET_RX: 638
BLOCK: 8008
IRQ_POLL: 0
TASKLET: 72
SCHED: 0
HRTIMER: 0
RCU: 37279
[root@10 proc]#

Kernel: “rcu_sched detected stalls“ 与 “task khugepaged:47 blocked for more than 120 seconds”

文章目录

参考

简介

日志实例

最近遇到了一例

如果实时应用程序占用CPU非常高

猜你喜欢