Der ES5-Knoten hat die Animation angehalten und das Kernel-Protokoll meldete INFO: Aufgabe für mehr als 120 Sekunden blockiert.

Language 2023-09-16 19:21:12 views: null

Es gibt eine Maschine im es-Cluster. Plötzlich steigt die CPU-Auslastung auf etwa 21 (8-Kern-CPU), aber die CPU-Auslastung wird 0, und gleichzeitig werden die E/A- und andere Nutzungen alle 0. Dieser Zustand kann nicht sein wird von selbst wiederhergestellt, sofern kein Neustart erfolgt. Es wurde nicht festgestellt, dass ein Problem mit dem Knoten vorlag, der gesamte Cluster nicht zugänglich war und alle Vorgänge, einschließlich Schnittstellen wie Cat-Cluster-Informationen, abgelaufen waren, bis der Computer neu gestartet wurde. In diesem Fall ist der Knoten tot, aber der Knoten kann immer noch auf die von seinem Master gesendete Transport-Ping-Nachricht antworten und der Master wird ihn nicht aus dem Cluster entfernen. Eingabeaufforderung im Protokoll

monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
还有
Failed to connect to nodexxxxtimeout

Aus Sicht der Überwachung gibt es keinen plötzlichen Anstieg des Datenverkehrs und die Festplatten-IO-Operationen nehmen nicht plötzlich zu.

Verwenden Sie zu diesem Zeitpunkt kill -9, um den es-Prozess abzubrechen. Wenn er nicht abgebrochen werden kann, wird der ursprüngliche Prozess zu einem Zombie-Prozess. Die Last sinkt nie.

Als der Zombie starb, gab dmsg das Protokoll aus:

[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] kthreadd        D ffffffffffffffff     0     2      0 0x00000000
[Wed Jun 14 18:03:48 2017]  ffff881fd2a234e0 0000000000000046 ffff881fd2998b80 ffff881fd2a23fd8
[Wed Jun 14 18:03:48 2017]  ffff881fd2a23fd8 ffff881fd2a23fd8 ffff881fd2998b80 ffff881fd2a23628
[Wed Jun 14 18:03:48 2017]  ffff881fd2a23630 7fffffffffffffff ffff881fd2998b80 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff81631ada>] ? __slab_free+0x10e/0x277
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a666>] kernel_thread+0x26/0x30
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a65f2>] kthreadd+0x2b2/0x2f0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6340>] ? kthread_create_on_cpu+0x60/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6340>] ? kthread_create_on_cpu+0x60/0x60
[Wed Jun 14 18:03:48 2017] INFO: task kswapd0:244 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] kswapd0         D ffffffffffffffff     0   244      2 0x00000000
[Wed Jun 14 18:03:48 2017]  ffff881fcda7f6f0 0000000000000046 ffff881fcd9ba280 ffff881fcda7ffd8
[Wed Jun 14 18:03:48 2017]  ffff881fcda7ffd8 ffff881fcda7ffd8 ffff881fcd9ba280 ffff881fcda7f838
[Wed Jun 14 18:03:48 2017]  ffff881fcda7f840 7fffffffffffffff ffff881fcd9ba280 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bd008>] ? __enqueue_entity+0x78/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c34b7>] ? enqueue_entity+0x237/0x890
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5eb7>] ? vmpressure+0x87/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81180131>] balance_pgdat+0x4b1/0x5e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff811803d3>] kswapd+0x173/0x450
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6ae0>] ? wake_up_atomic_t+0x30/0x30
[Wed Jun 14 18:03:48 2017]  [<ffffffff81180260>] ? balance_pgdat+0x5e0/0x5e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017] INFO: task xfsaild/sdb1:781 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] xfsaild/sdb1    D ffffffffffffffff     0   781      2 0x00000000
[Wed Jun 14 18:03:48 2017]  ffff881fcf1afb20 0000000000000046 ffff881fcfbe7300 ffff881fcf1affd8
[Wed Jun 14 18:03:48 2017]  ffff881fcf1affd8 ffff881fcf1affd8 ffff881fcfbe7300 ffff881fcf1afc68
[Wed Jun 14 18:03:48 2017]  ffff881fcf1afc70 7fffffffffffffff ffff881fcfbe7300 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8108d6fe>] ? try_to_del_timer_sync+0x5e/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4720>] _xfs_log_force+0x70/0x290 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8108bf50>] ? internal_add_timer+0x70/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4966>] xfs_log_force+0x26/0x80 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02cf470>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02cf5c1>] xfsaild+0x151/0x5e0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02cf470>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1594 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1594   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fcf91f5a0 0000000000000086 ffff881fccdaf300 ffff881fcf91ffd8
[Wed Jun 14 18:03:48 2017]  ffff881fcf91ffd8 ffff881fcf91ffd8 ffff881fccdaf300 ffff881fcf91f6e8
[Wed Jun 14 18:03:48 2017]  ffff881fcf91f6f0 7fffffffffffffff ffff881fccdaf300 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c4a74>] ? find_busiest_group+0x114/0x910
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1595 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1595   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fceb935a0 0000000000000086 ffff881fccdadc00 ffff881fceb93fd8
[Wed Jun 14 18:03:48 2017]  ffff881fceb93fd8 ffff881fceb93fd8 ffff881fccdadc00 ffff881fceb936e8
[Wed Jun 14 18:03:48 2017]  ffff881fceb936f0 7fffffffffffffff ffff881fccdadc00 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff81631ada>] ? __slab_free+0x10e/0x277
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1598 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1598   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fbec535a0 0000000000000086 ffff881fbec21700 ffff881fbec53fd8
[Wed Jun 14 18:03:48 2017]  ffff881fbec53fd8 ffff881fbec53fd8 ffff881fbec21700 ffff881fbec536e8
[Wed Jun 14 18:03:48 2017]  ffff881fbec536f0 7fffffffffffffff ffff881fbec21700 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c4a74>] ? find_busiest_group+0x114/0x910
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1601 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1601   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fcef035a0 0000000000000082 ffff881fbec23980 ffff881fcef03fd8
[Wed Jun 14 18:03:48 2017]  ffff881fcef03fd8 ffff881fcef03fd8 ffff881fbec23980 ffff881fcef036e8
[Wed Jun 14 18:03:48 2017]  ffff881fcef036f0 7fffffffffffffff ffff881fbec23980 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task zabbix_agentd:1609 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] zabbix_agentd   D ffffffffffffffff     0  1609   1572 0x00000084
[Wed Jun 14 18:03:48 2017]  ffff881fbee335a0 0000000000000082 ffff881fbec27300 ffff881fbee33fd8
[Wed Jun 14 18:03:48 2017]  ffff881fbee33fd8 ffff881fbee33fd8 ffff881fbec27300 ffff881fbee336e8
[Wed Jun 14 18:03:48 2017]  ffff881fbee336f0 7fffffffffffffff ffff881fbec27300 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f4d3d>] ? d_free+0x4d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff811f6e80>] ? shrink_dentry_list+0x250/0x480
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff81285ea6>] ? security_file_alloc+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e07de>] ? alloc_file+0x1e/0xf0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff811fcad7>] ? __fd_install+0x47/0x60
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
[Wed Jun 14 18:03:48 2017] INFO: task java:9265 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] java            D ffffffffffffffff     0  9265      1 0x00000180
[Wed Jun 14 18:03:48 2017]  ffff880f94f075a0 0000000000000086 ffff880fe98bdc00 ffff880f94f07fd8
[Wed Jun 14 18:03:48 2017]  ffff880f94f07fd8 ffff880f94f07fd8 ffff880fe98bdc00 ffff880f94f076e8
[Wed Jun 14 18:03:48 2017]  ffff880f94f076f0 7fffffffffffffff ffff880fe98bdc00 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5dc9>] ? ttwu_do_wakeup+0x19/0xd0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa0295e47>] ? xfs_iext_bno_to_ext+0xa7/0x1a0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811717eb>] ? free_pcppages_bulk+0x34b/0x3a0
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bd008>] ? __enqueue_entity+0x78/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff810c34b7>] ? enqueue_entity+0x237/0x890
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4d2e>] xfs_log_force_lsn+0x2e/0x90 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b34b7>] __xfs_iunpin_wait+0xa7/0x150 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02b6fc9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ab84c>] xfs_reclaim_inode+0x8c/0x350 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02abd77>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8bd3>] ? wake_up_process+0x23/0x40
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02ac923>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02bb895>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff811e0c28>] prune_super+0xe8/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117c4c5>] shrink_slab+0x165/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff811d5e51>] ? vmpressure+0x21/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f642>] do_try_to_free_pages+0x3c2/0x4e0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117f85c>] try_to_free_pages+0xfc/0x180
[Wed Jun 14 18:03:48 2017]  [<ffffffff8117355d>] __alloc_pages_nodemask+0x7fd/0xb90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81078d73>] copy_process.part.25+0x163/0x1610
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a401>] do_fork+0xe1/0x320
[Wed Jun 14 18:03:48 2017]  [<ffffffff81122b33>] ? __secure_computing+0x73/0x240
[Wed Jun 14 18:03:48 2017]  [<ffffffff8110b796>] ? __audit_syscall_exit+0x1e6/0x280
[Wed Jun 14 18:03:48 2017]  [<ffffffff8107a6c6>] SyS_clone+0x16/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645c59>] stub_clone+0x69/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645b12>] ? tracesys+0xdd/0xe2
[Wed Jun 14 18:03:48 2017] INFO: task java:28683 blocked for more than 120 seconds.
[Wed Jun 14 18:03:48 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 14 18:03:48 2017] java            D ffffffffffffffff     0 28683      1 0x00000180
[Wed Jun 14 18:03:48 2017]  ffff880f6d8ebc20 0000000000000086 ffff881fc8ffdc00 ffff880f6d8ebfd8
[Wed Jun 14 18:03:48 2017]  ffff880f6d8ebfd8 ffff880f6d8ebfd8 ffff881fc8ffdc00 ffff880f6d8ebd68
[Wed Jun 14 18:03:48 2017]  ffff880f6d8ebd70 7fffffffffffffff ffff881fc8ffdc00 ffffffffffffffff
[Wed Jun 14 18:03:48 2017] Call Trace:
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163a909>] schedule+0x29/0x70
[Wed Jun 14 18:03:48 2017]  [<ffffffff816385f9>] schedule_timeout+0x209/0x2d0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd15>] ? native_sched_clock+0x35/0x80
[Wed Jun 14 18:03:48 2017]  [<ffffffff8101cd69>] ? sched_clock+0x9/0x10
[Wed Jun 14 18:03:48 2017]  [<ffffffff810bb685>] ? sched_clock_cpu+0x85/0xc0
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8a66>] ? try_to_wake_up+0x1b6/0x300
[Wed Jun 14 18:03:48 2017]  [<ffffffff8163acd6>] wait_for_completion+0x116/0x170
[Wed Jun 14 18:03:48 2017]  [<ffffffff810b8c10>] ? wake_up_state+0x20/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109e7ac>] flush_work+0xfc/0x1c0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8109a7e0>] ? move_linked_works+0x90/0x90
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c643a>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02c4a7e>] _xfs_log_force_lsn+0x6e/0x2f0 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff81639b12>] ? down_read+0x12/0x30
[Wed Jun 14 18:03:48 2017]  [<ffffffffa02a74d0>] xfs_file_fsync+0x1b0/0x200 [xfs]
[Wed Jun 14 18:03:48 2017]  [<ffffffff8120f975>] do_fsync+0x65/0xa0
[Wed Jun 14 18:03:48 2017]  [<ffffffff8120fc63>] SyS_fdatasync+0x13/0x20
[Wed Jun 14 18:03:48 2017]  [<ffffffff81645b12>] tracesys+0xdd/0xe2

Bei der Online-Suche nach Informationen gibt es zwei Szenarien. Einer davon liegt darin, dass der Kernel die Linux-NUMA-Optimierung standardmäßig aktiviert. Schalten Sie sie einfach aus. Der zweite Grund ist, dass ich damals xfs verwendet habe, also einfach wieder auf ext4 umsteigen. Die Grundursache für die zweite Situation konnte zu diesem Zeitpunkt nicht gefunden werden. Der Grund lag lediglich darin, dass es bei der Ausführung auf ext4 stabil gelaufen war, also wechselte ich zurück. Liegt es an einem Fehler zwischen XFS und dem Linux-Kernel?

Prüfen Sie, ob das Deaktivieren von NUMA Ihr Problem lösen kann. Wenn nicht, überlegen Sie, ob es sich um ein XFS-Problem handelt.

So deaktivieren Sie NUMA:

Fügen Sie die Konfiguration vm.zone_reclaim_mode = 0 in /etc/sysctl.conf hinzu und führen Sie dann sysctl -p aus, damit die Konfiguration wirksam wird.

Supongo que te gusta

Origin blog.csdn.net/h952520296/article/details/129423638
Recomendado
Clasificación
Diario
Más
2024-09-19(0)
2024-09-18(0)
2024-09-17(0)
2024-09-16(0)
2024-09-15(0)
2024-09-14(0)
2024-09-13(0)
2024-09-12(0)
2024-09-11(0)
2024-09-10(0)

Code World

© 2018 codetd.com