Linux performance tuning - memory

How Linux memory works

The concept of memory mapping

  The main memory used in most computers is dynamic random access memory (DRAM), and only the kernel can directly access physical memory. The Linux kernel provides an independent virtual address space for each process, and this address space is continuous. In this way, the process can easily access the memory (Virtual Memory)。

  The interior of the virtual address space is divided into two parts: kernel space and user space. The range of the address space of processors with different word lengths is different. The 32-bit system kernel space occupies 1G and the user space occupies 3G. The kernel space and user space of 64-bit systems are both 128T, occupying the highest and lowest parts of the memory space respectively, and the middle part is undefined.

  Not all virtual memory is allocated physical memory, only the one actually used. The allocated physical memory is managed through memory mapping. In order to complete the memory mapping, the kernel maintains apage table, record the mapping relationship between virtual address and physical address. The page table is actually stored in the CPU's memory management unit MMU, and the processor can directly find out the memory to be accessed through hardware.

  When the virtual address accessed by the process cannot be found in the page table, the system will generate a page fault exception, enter the kernel space to allocate physical memory, update the process page table, and performpage replacement, and then return to user space to resume the running of the process.

  The MMU manages memory in units of pages, with a page size of 4KB. In order to solve the problem of too many page table entries, Linux providesmulti-level page tableandHugePageMechanisms.

  Linux also uses a portion of memory to store caches of file systems and block devices to speed up access to files and block devices. this is calledfile system cacheorPage cache。

  Linux can swap infrequently used memory pages to diskswap partition(swap partition) to release physical memory for use by other processes. The use of swap space is a means when physical memory is insufficient, but excessive use of swap will reduce performance.

Virtual memory space distribution

  User space memory is divided into five different memory segments from low to high:

  Read-only segment Data segment such as code and constants Heap such as global variables Dynamically allocated memory, growing upward from the low address File mapping Dynamic library, shared memory, etc., growing downward from the high address The stack includes local variables and the context of function calls, etc. , the stack size is fixed. Normally 8MB
  
  
  
  

Memory allocation and recycling

distribute

  For small blocks of memory (<128K), allocate by moving the top position of the heap. The memory is not returned immediately after it is released, but is cached.
  For large blocks of memory (>128K), allocate directly using memory mapping, that is, find a free memory allocation in the file mapping segment.

  The former's cache can reduce the occurrence of page fault exceptions and improve memory access efficiency. However, because the memory is not returned to the system, frequent memory allocation/release will cause memory fragmentation when the memory is busy.

  The latter is returned directly to the system when released, so a page fault exception will occur every time. When memory work is busy, frequent memory allocation will cause a large number of page fault exceptions, increasing the kernel management burden.

Recycle

  When memory is tight, the system reclaims memory in the following ways:

  · Recycle cache: LRU algorithm recycles the least recently used memory pages;
  · Recycle infrequently accessed memory: write infrequently used memory to disk through swap partition
  · Kill process

Memory viewing and analysis

View memory usage

Command: free

  Enter the following command:

free -h

  The output is as follows:

              总计         已用        空闲      共享    缓冲/缓存    可用
内存：        62Gi       9.2Gi        22Gi       1.4Gi        31Gi        51Gi
交换：       2.0Gi       1.9Gi        62Mi

• Total : Total capacity of physical memory. In this example, the system has a total of 62 GB of physical memory.

• Used : The amount of memory currently used by the system and processes. In this example, 9.2 GB of memory has been used.

• Free : The amount of memory that is currently unused. In this example, 22 GB of memory is free.

• Shared : The amount of memory shared by multiple processes. In this example, there are 1.4 GB of memory shared by multiple processes.

• Buffer/Cache : The amount of memory used for file system caching and disk I/O buffering. In this example, 31 GB of memory is used for buffering and caching.

• Available : The system estimates the amount of memory available for new processes, including memory that may be freed in the future. In this example, 51 GB of memory is estimated to be available.

  About swap space:

• Total Swap : The total capacity of swap space in the system. In this example, the system has a total of 2.0 GB of swap space.

• Used Swap : The amount currently used in swap space. In this example, 1.9 GB of swap space has been used.

• Free Swap : The amount of remaining unused swap space. In this example, 62 MB of swap space is free.

  Overall, the system's physical memory usage appears to be relatively adequate, as most of the memory is free and there is a fair amount of buffering and cache available. Swap space is also only lightly used, which is a good sign since overusing swap space can impact performance. If system performance is still good, current memory and swap space usage may be acceptable. However, if your system is experiencing performance issues, you may need to further analyze the memory usage of processes and services to determine if further actions are needed, such as optimization or additional memory.

Command: vmstat

  Enter the following command to count every 5 seconds:

vmstat 5

  The output is as follows:

  Result Description

• r 表示运行队列(就是说多少个进程真的分配到CPU)，我测试的服务器目前CPU比较空闲，没什么程序在跑，当这个值超过了CPU数目，就会出现CPU瓶颈了。这个也和top的负载有关系，一般负载超过了3就比较高，超过了5就高，超过了10就不正常了，服务器的状态很危险。top的负载类似每秒的运行队列。如果运行队列过大，表示你的CPU很繁忙，一般会造成CPU使用率很高。

• b 表示阻塞的进程,这个不多说，进程阻塞，大家懂的。

• swpd 虚拟内存已使用的大小，如果大于0，表示你的机器物理内存不足了，如果不是程序内存泄露的原因，那么你该升级内存了或者把耗内存的任务迁移到其他机器。

• free 空闲的物理内存的大小。

• buff Linux/Unix系统是用来存储，目录里面有什么内容，权限等的缓存

• cache cache直接用来记忆我们打开的文件,给文件做缓冲，这里是Linux/Unix的聪明之处，把空闲的物理内存的一部分拿来做文件和目录的缓存，是为了提高 程序执行的性能，当程序使用内存时，buffer/cached会很快地被使用。

• si 每秒从磁盘读入虚拟内存的大小，如果这个值大于0，表示物理内存不够用或者内存泄露了，要查找耗内存进程解决掉。我的机器内存充裕，一切正常。

• so 每秒虚拟内存写入磁盘的大小，如果这个值大于0，同上。

• bi 块设备每秒接收的块数量，这里的块设备是指系统上所有的磁盘和其他块设备，默认块大小是1024byte，我本机上没什么IO操作，所以一直是0，但是我曾在处理拷贝大量数据(2-3T)的机器上看过可以达到140000/s，磁盘写入速度差不多140M每秒

• bo 块设备每秒发送的块数量，例如我们读取文件，bo就要大于0。bi和bo一般都要接近0，不然就是IO过于频繁，需要调整。

• in 每秒CPU的中断次数，包括时间中断

• cs 每秒上下文切换次数，例如我们调用系统函数，就要进行上下文切换，线程的切换，也要进程上下文切换，这个值要越小越好，太大了，要考虑调低线程或者进程的数目,例如在apache和nginx这种web服务器中，我们一般做性能测试时会进行几千并发甚至几万并发的测试，选择web服务器的进程可以由进程或者线程的峰值一直下调，压测，直到cs到一个比较小的值，这个进程和线程数就是比较合适的值了。系统调用也是，每次调用系统函数，我们的代码就会进入内核空间，导致上下文切换，这个是很耗资源，也要尽量避免频繁调用系统函数。上下文切换次数过多表示你的CPU大部分浪费在上下文切换，导致CPU干正经事的时间少了，CPU没有充分利用，是不可取的。

• us 用户CPU时间，我曾经在一个做加密解密很频繁的服务器上，可以看到us接近100,r运行队列达到80(机器在做压力测试，性能表现不佳)。

• sy 系统CPU时间，如果太高，表示系统调用时间长，例如是IO操作频繁。

• id 空闲CPU时间，一般来说，id + us + sy = 100,一般我认为id是空闲CPU使用率，us是用户CPU使用率，sy是系统CPU使用率。

• wt 等待IO CPU时间

命令：top

  输入以下命令，随后再输入M表示按照内存占用率排序：

top

  输出如下：

  从上图可以发现，内存占用率较高的是jsvc和mysql。jsvc可能并不知道是个什么命令，所以我们进一步分析单个进程。

分析单个进程

命令：ps -p

  查看进程1180的运行信息：

ps -p 1180 -o pid,ppid,%cpu,%mem,cmd

  输出如下：

  cmd表示查看这个进程的执行命令，最终锁定1180进程是tomcat的守护进程。