Keywords: meminfo, slabinfo, Top, pthread_join, the Thread Stack , and so on.
Recording a positioning process on the thread of memory leaks, as well as during the harvest.
1. Initial Positioning
Memory leaks: think of a memory leak, first check the / proc / meminfo, via / proc / meminfo memory can be seen in the overall decline. Determining a memory leak exists. top memory can be displayed in various forms, and thus can determine the kind of leak. For example vss / rss / pss and so on.
Determine which process memory leak: to see the process through which top the leak. So far almost certain to which process.
Determine the process leaks memory types: then view the process of / proc / <pid> / maps , you can see the type of memory leaks by maps (heap, stack, anonymous memory, etc.), and sometimes luck can directly determine the leak.
If slab: via / proc / slabinfo, we can see the dynamic changes in the process. If it is determined which one slab, you can find the call site directly in / sys / kernel / slab / < slab name> / alloc_calls and free_calls in. Of course, seeing is a function of the kernel space.
Use mcheck (): You can check leaks malloc / free result, detailed reference " 2. the mtrace / muntrace / MALLOC_TRACE (repeat release, leak) "
, Then every time we crawl content Beyond Compare by the following script. Each cycle grab some information related to memory consumption.
#!/bin/bash echo > mem_log.txt while true do cat /proc/meminfo >>mem_log.txt cat /proc/<pid>/maps >>mem_log.txt cat /proc/slabinfo >>mem_log.txt sleep 240 done
Of course, there are other tools gcc Sanitier, Valgrind, etc., is limited due to the embedded environment of use.
2. In-depth positioning
Synchronization view meminfo, maps, slabinfo, find the process virtual memory loss quickly, faster than the system MemFree loss. And slabinfo no loss of synchronization and maps.
So the question focus on examination of maps problem.
00010000-00083000 r-xp 00000000 b3:11 22 /heop/package/AiApp/AiApp 00092000-00099000 rwxp 00072000 b3:11 22 /heop/package/AiApp/AiApp 00099000-00b25000 rwxp 00000000 00:00 0 [heap] 00b51000-00b52000 ---p 00000000 00:00 0 00b52000-01351000 rwxp 00000000 00:00 0 [stack:30451] 01351000-01352000 ---p 00000000 00:00 0 01352000-01b51000 rwxp 00000000 00:00 0 01b51000-01b52000 ---p 00000000 00:00 0 01b52000-02351000 rwxp 00000000 00:00 0 [stack:30432] 02351000-02352000 ---p 00000000 00:00 0 02352000-02b51000 rwxp 00000000 00:00 0 02b51000-02b52000 ---p 00000000 00:00 0 ... 64f55000-65754000 rwxp 00000000 00:00 0 [stack:28646] 65754000-65755000 ---p 00000000 00:00 0 65755000-65f54000 rwxp 00000000 00:00 0 [stack:28645] 65f54000-65f55000 ---p 00000000 00:00 0 65f55000-66754000 rwxp 00000000 00:00 0 [stack:28642] 66754000-6675a000 r-xp 00000000 00:02 5000324 /usr/lib/AiApp/gstreamer-1.0/libgsticcsink.so 6675a000-66769000 ---p 00000000 00:00 0 ... 6699f000-669a0000 rwxp 00000000 00:02 4999516 /usr/lib/AiApp/gstreamer-1.0/libgstapp.so 669a0000-66a2e000 rwxp 00000000 00:02 4999517 /usr/lib/AiApp/gstreamer-1.0/libgstlive555src.so 66a2e000-66a3e000 ---p 00000000 00:00 0 66a3e000-66a44000 rwxp 0008e000 00:02 4999517 /usr/lib/AiApp/gstreamer-1.0/libgstlive555src.so 66a44000-66a45000 rwxp 00000000 00:00 0 66a45000-66a46000 ---p 00000000 00:00 0 66a46000-67245000 rwxp 00000000 00:00 0 [stack:28631] 67245000-67246000 ---p 00000000 00:00 0 67246000-67a45000 rwxp 00000000 00:00 0 [stack:28630] ... 6b245000-6b246000 ---p 00000000 00:00 0 6b246000-6ba45000 rwxp 00000000 00:00 0 [stack:28613] 6ba45000-6ba46000 ---p 00000000 00:00 0 6ba46000-6c245000 rwxp 00000000 00:00 0 [stack:28610] 6c245000-71066000 rwxs 00000000 00:01 196614 /SYSV5553fc99 (deleted) 71066000-71067000 ---p 00000000 00:00 0 71067000-71866000 rwxp 00000000 00:00 0 [stack:28609] 71866000-71867000 ---p 00000000 00:00 0 71867000-72066000 rwxp 00000000 00:00 0 [stack:28608] 72066000-72228000 rwxs e3dc4000 00:02 6918 /dev/mmz_userdev 72228000-725ac000 rwxs e3a40000 00:02 6918 /dev/mmz_userdev 725ac000-75cac000 rwxs 00000000 00:01 131076 /SYSV6702121c (deleted) 75cac000-75e8a000 rwxs 00000000 00:01 98307 /SYSV6602121c (deleted) 75e8a000-7608e000 rwxp 00000000 00:00 0... 76eeb000-76efb000 ---p 00000000 00:00 0 76efb000-76eff000 r-xp 000ce000 00:02 1234 /lib/libstdc++.so.6.0.20 76eff000-76f01000 rwxp 000d2000 00:02 1234 /lib/libstdc++.so.6.0.20 76f01000-76f08000 rwxp 00000000 00:00 0 76f08000-76f0f000 r-xp 00000000 00:02 1235 /lib/ld-uClibc-0.9.33.2.so 76f1a000-76f1e000 rwxp 00000000 00:00 0 76f1e000-76f1f000 rwxp 00006000 00:02 1235 /lib/ld-uClibc-0.9.33.2.so 76f1f000-76f20000 ---p 00000000 00:00 0 76f20000-7771f000 rwxp 00000000 00:00 0 7771f000-77720000 ---p 00000000 00:00 0 77720000-77f1f000 rwxp 00000000 00:00 0 [stack:30470] 77f1f000-77f20000 ---p 00000000 00:00 0 77f20000-7871f000 rwxp 00000000 00:00 0 7871f000-78720000 ---p 00000000 00:00 0 78720000-78f1f000 rwxp 00000000 00:00 0 [stack:30489] 78f1f000-78f20000 ---p 00000000 00:00 0 78f20000-7971f000 rwxp 00000000 00:00 0 7971f000-79720000 ---p 00000000 00:00 0 79720000-79f1f000 rwxp 00000000 00:00 0 [stack:30508] 79f1f000-79f20000 ---p 00000000 00:00 0 79f20000-7a71f000 rwxp 00000000 00:00 0 7a71f000-7a720000 ---p 00000000 00:00 0 7a720000-7af1f000 rwxp 00000000 00:00 0 [stack:30539] 7af1f000-7af20000 ---p 00000000 00:00 0 7af20000-7b71f000 rwxp 00000000 00:00 0 7b71f000-7b720000 ---p 00000000 00:00 0 7b720000-7bf1f000 rwxp 00000000 00:00 0 [stack:30556] 7bf1f000-7bf20000 ---p 00000000 00:00 0 7bf20000-7c71f000 rwxp 00000000 00:00 0 7c71f000-7c720000 ---p 00000000 00:00 0 7c720000-7cf1f000 rwxp 00000000 00:00 0 [stack:30574] 7cf1f000-7cf20000 ---p 00000000 00:00 0 7cf20000-7e121000 rwxp 00000000 00:00 0 [stack:30575] 7eef7000-7ef18000 rwxp 00000000 00:00 0 [stack] 7efb7000-7efb8000 r-xp 00000000 00:00 0 [sigpage] ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]
By comparison of several maps, can be found in [stack: xxxxx] anonymous type memory and a memory constantly increasing memory consumption.
Wherein [stack: xxxxx] type of memory for the relevant code in the kernel is not explicitly corresponding property. Initially determined that thread's stack, xxxxxx represents the thread id number.
So there should be a thread leak.
2.1 thread stack leak (Joinable thread stack)
A lead thread stack leak may be due for a Joinable thread, the thread will be created proprietary stack, threand ID, thread end status.
If this thread is not pthread_join (), then the system will not be recovered more information. This may result in leakage of thread stack and so on.
The method of determining the thread stack leak is: / task by ls / proc / <pid> | wc -l determine the number of the next thread in the process. Then check [stack: xxxxx] in the number of maps. If the two are inconsistent, Joinable thread does not call pthread_join () caused by the leakage exists.
If no maps [stack: xxxxx], can pmap <pid> | grep <stack size> | wc -l, i.e. to determine the number of stacks by checking the number vma stack size.
About thread memory leak Reference: " Avoiding Memory leaks in the POSIX Programming the Thread "
3. The root of the problem
By examining the stack consumption and the actual number of threads in the thread, found that the number of both agreement. So the thread does not exit. That is not due to a memory leak unused pthread_join () caused.
Then according to the maps [stack: xxxxx] The pid number, cat / proc / <pid> / comm found to be kept the same to create a thread. But not released.
In fact, by top -H -p <pid> and maps can be found in the issue, in the middle take a detour.
So the root of the problem is that the process to create a non-stop but did not quit cause memory exhausted.
4. Harvest
There are two gains, one memory handling under pthread threads Join created two states and Detach difference; second is to show the thread stack [stack: xxxxx] In the process maps more conducive to debug.
[Stack: xxxxx] is not a standard function provided by the kernel, this research needs to be.