一、描述
Hadoop 集群经常能看到system cpu利用率超过30%，这种一般都在数据量大的机器中出现，操作系统CentOS6。
由文章1所述:
Symptom: top and other system monitoring tools show a large percentage of the CPU usage classified as "system CPU". If system CPU usage is 30% or more of the total CPU usage, your system may be experiencing this issue.
由文章2所述:
There can be two types of HugePages in the system: Explicit Huge Pages which are allocated explicitly by vm.nr_hugepages sysctl parameter and Transparent Huge Pages which are allocated automatically by the kernel.
类型1：Explicit Huge Pages
通过以下命令判断Explicit Huge Pages是关闭的
#grep -i HugePages_Total /proc/meminfo
HugePages_Total: 0
类型2：Transparent Huge Pages（THP）
通过以下命令和堆栈判断THP是打开的
验证是否关闭了透明大页：
[root@littlelawson ~]# grep Huge /proc/meminfo
AnonHugePages: 83968 kB
HugePages_Total: 0 #success
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@littlelawson ~]# cat /proc/sys/vm/nr_hugepages
0 #success
#cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
#sudo perf record -a -g -F 1000 sleep 60
19.00% 18.73% java [kernel.kallsyms] [k] compaction_alloc
---compaction_alloc
migrate_pages
compact_zone
compact_zone_order
try_to_compact_pages
__alloc_pages_direct_compact
__alloc_pages_nodemask
alloc_pages_vma
do_huge_pmd_anonymous_page
handle_mm_fault
__do_page_fault
do_page_fault
page_fault
二、解决方法
1.直接生效：
sudo bash -c "echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled"
sudo bash -c "never > /sys/kernel/mm/redhat_transparent_hugepage/defrag"
2.机器重启生效:
sudo /sbin/grubby --update-kernel=ALL --args='transparent_hugepage=never'
三、小知识点
1：从RedHat 6, OEL 6, SLES 11 and UEK2 kernels 开始，系统缺省会启用 Transparent HugePages ：用来提高内存管理的性能透明大页（Transparent HugePages ）和之前版本中的大页功能上类似。主要的区别是：Transparent HugePages 可以实时配置，不需要重启才能生效配置；
2：Transparent Huge Pages在32位的RHEL 6中是不支持的。
Transparent Huge Pages are not available on the 32-bit version of RHEL 6.
3： ORACLE官方不建议我们使用RedHat 6, OEL 6, SLES 11 and UEK2 kernels 时的开启透明大页（Transparent HugePages ）， 因为透明大页（Transparent HugePages ） 存在一些问题：
1.在RAC环境下 透明大页（Transparent HugePages ）会导致异常节点重启，和性能问题；
2.在单机环境中，透明大页（Transparent HugePages ） 也会导致一些异常的性能问题；
Transparent HugePages memory is enabled by default with Red Hat Enterprise Linux 6, SUSE Linux Enterprise Server 11, and Oracle Linux 6 with earlier releases of Oracle Linux Unbreakable Enterprise Kernel 2 (UEK2) kernels. Transparent HugePages memory is disabled in later releases of Oracle Linux UEK2 kernels.Transparent HugePages can cause memory allocation delays during runtime. To avoid performance issues, Oracle recommends that you disable Transparent HugePages on all Oracle Database servers. Oracle recommends that you instead use standard HugePages for enhanced performance.Transparent HugePages memory differs from standard HugePages memory because the kernel khugepaged thread allocates memory dynamically during runtime. Standard HugePages memory is pre-allocated at startup, and does not change during runtime.
Starting with RedHat 6, OEL 6, SLES 11 and UEK2 kernels, Transparent HugePages are implemented and enabled (default) in an attempt to improve the memory management. Transparent HugePages are similar to the HugePages that have been available in previous Linux releases. The main difference is that the Transparent HugePages are set up dynamically at run time by the khugepaged thread in kernel while the regular HugePages had to be preallocated at the boot up time. Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle.