Linux Memory Buffer Cache and understanding

On Linux systems, we often use the free command to view the system memory use. RHEL6 on a system, the command displays the contents free of something like a state:

1. [root@tencent64 ~]# free
2. total used free shared buffers cached
3. Mem: 132256952 72571772 59685180 0 1762632 53034704
4. -/+ buffers/cache: 17774436 114482516
5. Swap: 2101192 508 2100684

Here is the default display units kb, my server is 128G memory, so the numbers appear to be relatively large. This command is almost every person must be used Linux commands, but more so the command, the less it seems really understand people (I mean the less proportion). In general, the understanding of this command's output can be divided into several levels:

1. Do not understand. The first reaction is that such people: God, with a lot of memory, more than 70 G, but I have almost no big running ah? Why is this so? Linux accounted for a good memory!
2. I think very understanding. Such people are generally assessed will say: ah, my professional eye to see it, only with a memory about 17G, there are a lot of free memory available. buffers / cache occupied large, indicating that the system has the process been used to read and write files, but it does not matter, this memory when idle to use.
3. Really understand. Such people's reaction but people feel most do not understand Linux, their reaction was: is this free show, well I know. What? You ask me these memory enough, I certainly do not know it! I know how special it how to write your program?

According to the contents of the current technical documentation on the Web, I believe most people should understand that Linux is in the second level. It was widely felt, buffers and cached memory occupied space can be released as a free space in the larger memory pressure when used. But really the case? Prior to demonstrate this problem, we first briefly describe buffers and cached what it means:

 

What is the buffer / cache?

buffer cache and there will be two different meanings in computer technology is used in terms of abuse, placed under different contexts. In the Linux memory management, buffer here refers to the Linux memory: Buffer cache. Cache here refers to the Linux memory: Page cache. Translated into Chinese can be called the buffer cache and page cache . Historically, they are a (buffer) is used as a cache to write io equipment, while the other (cache) is used as read caching devices io, io equipment here, mainly referring to the block device file and ordinary file on the file system. But now, they have not the same meaning. In the current kernel, the name suggests is for the cache page cache memory page, it means that if there is a memory page allocation management, you can use the page cache as a cache to manage its use. Of course, not all memory is managed page to page, and many are managed for the block block, this memory If you want to use the cache function, are concentrated in the buffer cache to use. (From this point of view, is not called a block cache buffer cache renamed better?) However, not all blocks block has a fixed length, the length of the main block of the system is determined based on block devices used, while pages in length X86 whether the 32-bit or 64-bit is 4k.

Understand the difference between these two sets of system cache, we can understand what exactly they can be used to do.

 

What is the page cache

Page cache is mainly used as a cache file data on the file system to use, especially when the process there is a time for file read / write operations. If you think about it, you can map the file into memory as a system call: mmap is not very natural should also use page cache? In the current system implementation where, as a page cache is also another type of file caching device to use, so in fact the cache page cache is also responsible for most of the work block device file.

 

What is the buffer cache

Buffer cache is mainly used when the system is designed to block read and write device, the block data cache system to use. This means that some of the operations will use the buffer cache block cache, such as when we are in the format of the file system. Under normal circumstances, together with two cache system is used, such as when we write to a file, the contents page cache will be changed, and the buffer cache can be used to mark different page buffers, and which is a record buffer was changed. In this way, the kernel when you write the writeback subsequent execution of dirty data, you do not write back the whole page, and just need to write back the modified part can be.

 

How to recycle cache?

Linux kernel when the memory will be depleted, triggering garbage collection work in order to release the much-needed memory to the memory used by the process. In general, this operation comes from the main memory to the release of the release buffer / cache is. Especially by using more cache space. Since it is mainly used for caching, just speed up the process of file read and write speeds at the time of memory enough, then in larger memory pressure, of course, it is necessary to empty the release of cache, and give the relevant process used as a free space. So under normal circumstances, we believe buffer / cache space can be released, this understanding is correct.

But this clear the cache work is not without cost. Understand the cache is doing can understand the clear cache must be consistent with the data in the cache data corresponding to the file, in order for the cache to be released. So along with the cache cleared of behavior are generally system IO skyrocketed. Because data on whether the kernel cache in comparison to the corresponding data and hard disk file, and if not need to write back before you can recover.

Except when memory is exhausted outside can clear the cache, we can use the following file system cache clearing operation to manually trigger:

1. [root@tencent64 ~]# cat /proc/sys/vm/drop_caches
2. 1

the way is:

1. echo 1 > /proc/sys/vm/drop_caches

Of course, the value of this file can be set 1,2,3, respectively. Meaning that they represent are:

echo 1> / proc / sys / vm / drop_caches: clears the page cache.

echo 2> / proc / sys / vm / drop_caches: represents scavenged slab allocator object (including a cache directory entry and the inode cache). The slab allocator is a mechanism in the kernel memory management, cache data to achieve many of which are used in page cache.

echo 3> / proc / sys / vm / drop_caches: clears the page cache and a cache object slab dispenser.

 

cache can be recycled it?

We analyzed the situation cache can be recovered, then there can not be recovered cache it? Of course. Let's look at the first case:

 

tmpfs

We all know that Linux provides a "temporary" file system called tmpfs, it can be part of the memory space used to as a file system, the memory space can be used as a catalog file. Now the vast majority of Linux systems have called / dev / shm tmpfs the directory is such a presence. Of course, we can also manually create your own tmpfs, as follows:

1. [root@tencent64 ~]# mkdir /tmp/tmpfs
2. [root@tencent64 ~]# mount -t tmpfs -o size=20G none /tmp/tmpfs/
3.  
4. [root@tencent64 ~]# df
5. Filesystem 1K-blocks Used Available Use% Mounted on
6. /dev/sda1 10325000 3529604 6270916 37% /
7. /dev/sda3 20646064 9595940 10001360 49% /usr/local
8. /dev/mapper/vg-data 103212320 26244284 71725156 27% /data
9. tmpfs 66128476 14709004 51419472 23% /dev/shm
10. none 20971520 0 20971520 0% /tmp/tmpfs

So we created a new tmpfs, space is 20G, we can create within a 20G file in / tmp / tmpfs in. If the file we created the actual space occupied by the memory, then what these data should be part of the memory space occupied it? It can be understood from the page cache function to achieve, since it is some kind of file system, so naturally the use of space to manage the page cache. We try is not it?

1. [root@tencent64 ~]# free -g
2. total used free shared buffers cached
3. Mem: 126 36 89 0 1 19
4. -/+ buffers/cache: 15 111
5. Swap: 2 0 2
6. [root@tencent64 ~]# dd if=/dev/zero of=/tmp/tmpfs/testfile bs=1G count=13
7. 13+0 records in
8. 13+0 records out
9. 13958643712 bytes (14 GB) copied, 9.49858 s, 1.5 GB/s
10. [root@tencent64 ~]#
11. [root@tencent64 ~]# free -g
12. total used free shared buffers cached
13. Mem: 126 49 76 0 1 32
14. -/+ buffers/cache: 15 110
15. Swap: 2 0 2

We create a file in a directory tmpfs 13G and found by comparing before and after the free command, cached grew 13G, explain this document does on the memory and the kernel is used as cache memory. Look at the metrics we care about: - / + buffers / cache that line. We found that in this case the free command prompt us still have 110G memory available, but really so what? We can manually trigger garbage collection is now in the end to see how much memory can be recovered:

1. [root@tencent64 ~]# echo 3 > /proc/sys/vm/drop_caches
2. [root@tencent64 ~]# free -g
3. total used free shared buffers cached
4. Mem: 126 43 82 0 0 29
5. -/+ buffers/cache: 14 111
6. Swap: 2 0 2

We can see, cached space occupied, and not as we thought fully released, which 13G of space is still occupied / tmp / tmpfs files. Of course, my system as well as other non-release of cache memory space occupied by the rest of 16G. So tmpfs occupied cache space when it will be released? When it was deleted files. If you do not delete the file, no matter to what extent out of memory, the kernel will not help you to automatically delete files in tmpfs to free cache space.

1. [root@tencent64 ~]# rm /tmp/tmpfs/testfile
2. [root@tencent64 ~]# free -g
3. total used free shared buffers cached
4. Mem: 126 30 95 0 0 16
5. -/+ buffers/cache: 14 111
6. Swap: 2 0 2

This is the case of the first cache of our analysis can not be recycled. There are other cases, such as:

 

Shared memory

Shared memory is a system available to us between a common interprocess communication (IPC) way, but this means of communication can not apply for and use the shell, so we need a simple test program code is as follows:

1. [root@tencent64 ~]# cat shm.c
2.  
3. #include <stdio.h>
4. #include <stdlib.h>
5. #include <unistd.h>
6. #include <sys/ipc.h>
7. #include <sys/shm.h>
8. #include <string.h>
9.  
10. #define MEMSIZE 2048*1024*1023
11.  
12. int
13. main()
14. {
15. int shmid;
16. char *ptr;
17. pid_t pid;
18. struct shmid_ds buf;
19. int ret;
20.  
21. shmid = shmget(IPC_PRIVATE, MEMSIZE, 0600);
22. if (shmid<0) {
23. perror("shmget()");
24. exit(1);
25. }
26.  
27. ret = shmctl(shmid, IPC_STAT, &buf);
28. if (ret < 0) {
29. perror("shmctl()");
30. exit(1);
31. }
32.  
33. printf("shmid: %d\n", shmid);
34. printf("shmsize: %d\n", buf.shm_segsz);
35.  
36. buf.shm_segsz *= 2;
37.  
38. ret = shmctl(shmid, IPC_SET, &buf);
39. if (ret < 0) {
40. perror("shmctl()");
41. exit(1);
42. }
43.  
44. ret = shmctl(shmid, IPC_SET, &buf);
45. if (ret < 0) {
46. perror("shmctl()");
47. exit(1);
48. }
49.  
50. printf("shmid: %d\n", shmid);
51. printf("shmsize: %d\n", buf.shm_segsz);
52.  
53.  
54. pid = fork();
55. if (pid<0) {
56. perror("fork()");
57. exit(1);
58. }
59. if (pid==0) {
60. ptr = shmat(shmid, NULL, 0);
61. if (ptr==(void*)-1) {
62. perror("shmat()");
63. exit(1);
64. }
65. bzero(ptr, MEMSIZE);
66. strcpy(ptr, "Hello!");
67. exit(0);
68. } else {
69. wait(NULL);
70. ptr = shmat(shmid, NULL, 0);
71. if (ptr==(void*)-1) {
72. perror("shmat()");
73. exit(1);
74. }
75. puts(ptr);
76. exit(0);
77. }
78. }

The program features a very simple, it is to apply for a period of less than 2G shared memory, and then open a child process to make this a shared memory initialization operation, the output after completion of the initialization process of the child and other parent about the contents of shared memory, then exit. But before quitting did not remove the shared memory. Let's look at memory usage before and after implementation of this program:

1. [root@tencent64 ~]# free -g
2. total used free shared buffers cached
3. Mem: 126 30 95 0 0 16
4. -/+ buffers/cache: 14 111
5. Swap: 2 0 2
6. [root@tencent64 ~]# ./shm
7. shmid: 294918
8. shmsize: 2145386496
9. shmid: 294918
10. shmsize: -4194304
11. Hello!
12. [root@tencent64 ~]# free -g
13. total used free shared buffers cached
14. Mem: 126 32 93 0 0 18
15. -/+ buffers/cache: 14 111
16. Swap: 2 0 2

cached by the 16G space rose to 18G. Then this cache can be recovered it? Continue to test:

1. [root@tencent64 ~]# echo 3 > /proc/sys/vm/drop_caches
2. [root@tencent64 ~]# free -g
3. total used free shared buffers cached
4. Mem: 126 32 93 0 0 18
5. -/+ buffers/cache: 14 111
6. Swap: 2 0 2

The result is still not recovered. We can observe this even if no one is using the shared memory will still be long-term storage in the cache until it is removed. Delete There are two ways, one is used in the program shmctl () to IPC_RMID, the other is to use ipcrm command. We try to delete:

1. [root@tencent64 ~]# ipcs -m
2.  
3. ------ Shared Memory Segments --------
4. key shmid owner perms bytes nattch status
5. 0x00005feb 0 root 666 12000 4
6. 0x00005fe7 32769 root 666 524288 2
7. 0x00005fe8 65538 root 666 2097152 2
8. 0x00038c0e 131075 root 777 2072 1
9. 0x00038c14 163844 root 777 5603392 0
10. 0x00038c09 196613 root 777 221248 0
11. 0x00000000 294918 root 600 2145386496 0
12.  
13. [root@tencent64 ~]# ipcrm -m 294918
14. [root@tencent64 ~]# ipcs -m
15.  
16. ------ Shared Memory Segments --------
17. key shmid owner perms bytes nattch status
18. 0x00005feb 0 root 666 12000 4
19. 0x00005fe7 32769 root 666 524288 2
20. 0x00005fe8 65538 root 666 2097152 2
21. 0x00038c0e 131075 root 777 2072 1
22. 0x00038c14 163844 root 777 5603392 0
23. 0x00038c09 196613 root 777 221248 0
24.  
25. [root@tencent64 ~]# free -g
26. total used free shared buffers cached
27. Mem: 126 30 95 0 0 16
28. -/+ buffers/cache: 14 111
29. Swap: 2 0 2

After deleting shared memory, cache is normally released. This behavior is similar to the logic tmpfs. Kernel underlayer shared memory (SHM), the message queue (msg) and semaphore array (SEM) These POSIX: XSI IPC mechanism when the storage memory, are used tmpfs. This is also the reason why the shared memory and logic operations tmpfs similar. Of course, under normal circumstances it is shm take up more memory, so we emphasize the use of shared memory at this point. When it comes to shared memory, Linux also provides us another way to shared memory, that is:

 

mmap

mmap () is a very important system calls this function only from the description of the mmap itself is not apparent. Literally, mmap a file is mapped into the process's virtual memory address, then you can manipulate the contents of a file by manipulating the way memory. But in fact the purpose of the call is very extensive. When the application malloc memory, small pieces of memory used by the kernel sbrk process, while a large segment of the memory will use mmap. When the system call exec family of functions to perform, because it is essentially an executable file is loaded into memory to perform, so it is natural to the core can be processed using mmap way. Here we only consider a situation in which the use of shared memory mmap apply, will not follow shmget () also uses the same cache?

Similarly, we also need a simple test program:

1. [root@tencent64 ~]# cat mmap.c
2. #include <stdlib.h>
3. #include <stdio.h>
4. #include <strings.h>
5. #include <sys/mman.h>
6. #include <sys/stat.h>
7. #include <sys/types.h>
8. #include <fcntl.h>
9. #include <unistd.h>
10.  
11. #define MEMSIZE 1024*1024*1023*2
12. #define MPFILE "./mmapfile"
13.  
14. int main()
15. {
16. void *ptr;
17. int fd;
18.  
19. fd = open(MPFILE, O_RDWR);
20. if (fd < 0) {
21. perror("open()");
22. exit(1);
23. }
24.  
25. ptr = mmap(NULL, MEMSIZE, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANON, fd, 0);
26. if (ptr == NULL) {
27. perror("malloc()");
28. exit(1);
29. }
30.  
31. printf("%p\n", ptr);
32. bzero(ptr, MEMSIZE);
33.  
34. sleep(100);
35.  
36. munmap(ptr, MEMSIZE);
37. close(fd);
38.  
39. exit(1);
40. }

This time we simply do not have a parent and child the way, a process to apply for a period of 2G mmap shared memory, and then initialize this space after waiting 100 seconds, then lifted insinuate so we need to check it in our sleep these 100 seconds system memory usage and see what it is used for space? Of course, before that first create a file ./mmapfile of 2G. The results are as follows:

1. [root@tencent64 ~]# dd if=/dev/zero of=mmapfile bs=1G count=2
2. [root@tencent64 ~]# echo 3 > /proc/sys/vm/drop_caches
3. [root@tencent64 ~]# free -g
4. total used free shared buffers cached
5. Mem: 126 30 95 0 0 16
6. -/+ buffers/cache: 14 111
7. Swap: 2 0 2

Then run the test program:

1. [root@tencent64 ~]# ./mmap &
2. [1] 19157
3. 0x7f1ae3635000
4. [root@tencent64 ~]# free -g
5. total used free shared buffers cached
6. Mem: 126 32 93 0 0 18
7. -/+ buffers/cache: 14 111
8. Swap: 2 0 2
9.  
10. [root@tencent64 ~]# echo 3 > /proc/sys/vm/drop_caches
11. [root@tencent64 ~]# free -g
12. total used free shared buffers cached
13. Mem: 126 32 93 0 0 18
14. -/+ buffers/cache: 14 111
15. Swap: 2 0 2

We can see, during program execution, cached has been 18G, than before rose 2G, and at this time this cache still can not be recovered. Then we wait for 100 seconds after the program ends.

1. [root@tencent64 ~]#
2. [1]+ Exit 1 ./mmap
3. [root@tencent64 ~]#
4. [root@tencent64 ~]# free -g
5. total used free shared buffers cached
6. Mem: 126 30 95 0 0 16
7. -/+ buffers/cache: 14 111
8. Swap: 2 0 2

After the program exits, cached occupied space is freed. In this way we can see, using mmap request flag state for the cache MAP_SHARED memory, the kernel is also used for storing. Before the process related memory is not released, this cache is not normally released. In fact, the memory mmap MAP_SHARED way of application, by the tmpfs in the kernel is implemented. From this we can speculate, because the read-only portion of the shared libraries are mmap way to MAP_SHARED is managed in memory, in fact, they are to take up cache and can not be released.

 

At last

We tested three examples, find  Linux system memory cache can not be released in all cases as a free space used. And it has also made clear, even if you can release the cache, not without cost for the system . Summarize the main points, we should remember these points:

1. When the cache is released as a file cache it will lead to increased IO, which is the cost of cache to speed up file access speed to be paid.
2. tmpfs files stored in the cache take up space, delete files unless otherwise the cache will not be automatically released.
3. Shmget apply for the use of shared memory cache will occupy the space, unless the use of shared memory is ipcrm or shmctl to IPC_RMID, or otherwise the cache space will not be automatically released.
4. Memory cache will occupy space using mmap method of application MAP_SHARED mark, unless this memory process will munmap, or otherwise the cache space will not be automatically released.
5. Indeed shmget, mmap shared memory, tmpfs implemented at the kernel layer, with tmpfs are implemented cache memory are passed.

When the time to understand these, I hope you understand that we can achieve the free command said third level. We should understand that memory usage is not a simple concept, cache also not really be used as a free space. If we really want a deep understanding of the reasonableness of memory on your system to use in the end, is the need to understand much more clearly the details of knowledge, and to do more to implement the details related business judgment. Our current experimental scene is Centos environment 6, free real state of different versions of Linux may not be the same, you can go find a different reason.

Of course, this is not the case where all the cache can not be released. So, in your scenario, there are those who can not be released cache of scenario?

 

Reprinted from: https: //blog.csdn.net/lqglqglqg/article/details/82313966