The Linux system uses the special file system of hugetlbfs to add support for 2MB and 1GB huge page memory. For configuration, first check whether the CPU supports it:
# cat /proc/cpuinfo |grep --color pse
# cat /proc/cpuinfo |grep --color pdpe1gb
Check if the kernel supports:
# grep -i hugetlb /boot/config-4.18.0-193.el8.x86_64
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_CGROUP_HUGETLB=y
# 以下两种都为y则标识支持
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
The cpu function list includes the pse flag that supports 2MB memory huge pages, and includes pdpe1gb that supports 1GB memory huge pages.
View huge page memory usage
# grep Huge /proc/meminfo
AnonHugePages: 4089856 kB
ShmemHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
Here it says:
A page size is: Hugepagesize: 2048 kB
The number of huge pages in memory is: HugePages_Total: 0
View of NUMA architecture:
[root@localhost ~]# cat /sys/devices/system/node/node*/meminfo|fgrep Huge
Node 0 AnonHugePages: 153600 kB
Node 0 HugePages_Total: 0
Node 0 HugePages_Free: 0
Node 0 HugePages_Surp: 0
Node 1 AnonHugePages: 30720 kB
Node 1 HugePages_Total: 0
Node 1 HugePages_Free: 0
Node 1 HugePages_Surp: 0
It can be seen that no large page memory is allocated on this NUMA machine:
HugePages_Total: 0
If it is relatively simple to allocate 2MB of large page memory, you can replace it with the command:
echo 1024> /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
A resolution of 1024 2MB huge pages means copying 2GB of huge page memory.
In the case of NUMA architectures, in each of the previously attached additions:
echo 1024> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 1024> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
If you want to allocate more than 1GB of huge page memory, you need to set and mount it in the Linux startup item.
1) install
yum install libhugetlbfs
2) Allocation change startup file, add:
transparent_hugepage=never
default_hugepagesz=1G
hugepagesz=1G
hugepages=4
Allocate 4 large pages of memory, each of which is 1G. In centos6, modify
/etc/grub.conf
centos7 to modify the English /etc/grub2.cfg file. 3) Mounting maps huge page memory to an empty directory:
# mkdir /mnt/myhuge
# mount -t hugetlbfs nodev /mnt/myhuge
If you want to set it automatically at power-on:
# vim /etc/fstab
nodev /mnt/myhuge hugetlbfs pagesize=1GB 0 0
There are special setting tools such as DPDK, which can be set immediately when starting up to prevent insufficient memory. Huge page memory requires continuous space.
program use
This is based on research, but I haven't tried it.
HUGETLB_MORECORE=yes
LD_PRELOAD=libhugetlbfs.so
./your_program
Advantages and disadvantages of four pages
advantage:
1) The page table of large page memory is much smaller and can be placed in the TLB. There are few page fault interruptions, and the memory access performance is better. For programs that occupy a large amount of memory, the performance improvement is obvious, and some can even be improved. to around 50%.
2) The memory pages of the huge page memory will not be swapped to disk.
shortcoming:
1) It must be used in a specific way, and it must be mapped by mmap or specified in the above way.
2) The program uses a small amount of memory, but applies for a large page of memory, which will cause memory waste, because the smallest unit of memory allocation is a page.
3) If used improperly, memory may be wasted, and other programs cannot use large page memory.
The tragedy caused by the five large page memory
After talking for a long time, I haven't talked about the tragedy caused by the large page memory. It is like this. On a machine, the memory is very small, only 4GB of memory.
Use the Top command to check that the program memory is less than 1G occupied. At that time, the remaining memory was only 300MB. Where did the remaining memory go?
Analyze memory allocation:
# cat /proc/meminfo|grep Huge
HugePages_Total: 1035
HugePages_Free: 1033
HugePages_Rsvd: 62
HugePages_Surp: 0
Hugepagesize: 2048 kB
A better tool is to check with atop and find that the huge page memory occupies up to 2GB of memory, but it is not used much.
Solution
## 禁用大页缓存
# vi /etc/sysctl.conf
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
## 生效
sysctl -p
After the command is executed, it has an immediate effect, releasing 2GB of memory at once, which accounts for half of the system's memory, so whether to use the large page memory should be considered, otherwise the memory will be wasted in vain.
If the memory usage of the program is large, but there are many misses in the TLB, it can be used. Specifically, how to check the number of misses in the TLB can be analyzed through performance tools:
perf record -e dTLB-loads -e faults -p pid
perf report