This article is shared from Huawei Cloud Community " Solving Memory Leak Problems from the Source: Comprehensive Analysis of Memory Leak Detection and Repair Technology " by Lion Long.
1. Background: What is memory leak detection?
1.1. Causes of memory leaks
Memory leaks are a common problem in programming languages without automatic gc; because there is no gc, the allocated memory needs to be released by the program itself. The core is that the call allocation and release do not comply with the opening and closing principle and are not paired, forming a pointer that is allocated but not released, resulting in a memory leak.
For example:
void func(size_t s1)
{
void p1=malloc(s1);
void p2=malloc(s1);
// ...
free(p1);
}
The above code segment allocates two memory blocks of size s1, pointed to by p1 and p2. After the code block is executed, p1 is released, but p2 is not released. A pointer that is allocated but not released is formed, resulting in a memory leak.
1.2. Consequences caused by memory leaks
As the amount of engineering code increases, troubleshooting memory leaks becomes extremely troublesome. The virtual memory of a program keeps growing, and it is impossible to accurately determine whether it is a program need or a memory leak. If there is allocation but no release, the memory of the process heap will naturally become less and less until it is exhausted . This will cause subsequent running code to fail to allocate memory successfully. It may even cause the program to crash.
1.3. How to solve memory leak?
Memory leaks are caused by programming languages without automatic gc. Solution one is to introduce gc. This is the best solution to cure memory leaks. However, such a solution loses the advantages of the c/c++ language. Option 2: When a memory leak occurs, you can accurately locate which line of code caused it. This is also the core implementation requirement for memory leak detection.
(1) Able to detect memory leaks.
(2) Be able to determine which line of code caused the memory leak.
A program's virtual memory keeps growing, and it is impossible to accurately determine whether it is a program need or a memory leak; if it is a memory leak, we do not know which line of code it occurs in.
2. Convert address into symbolic information
2.1. addr2line tool
Convert addresses to filenames and line numbers.
addr2line [-a|--addresses]
[-b bfdname|--target=bfdname]
[-C|--demangle[=style]]
[-e filename|--exe=filename]
[-f|--functions] [-s|--basename]
[-i|--inlines]
[-p|--pretty-print]
[-j|--section=name]
[-H|--help] [-V|--version]
[addr addr ...]
describe:
addr2line converts addresses into filenames and line numbers. Given an address in an executable file or an offset in a relocatable object section, it uses debugging information to determine the file name and line number associated with it.
The executable or relocatable object to be used is specified with the -e option. Defaults to file a.out. The section in the relocatable object to be used is specified with the -j option.
addr2line has two modes of operation.
- In the first command line, hexadecimal addresses are specified on the command line and addr2line displays the file name and line number for each address.
- In the second command, addr2line reads hexadecimal addresses from standard input and prints the file name and line number of each address to standard output. In this mode, addr2line can be used in a pipeline to translate dynamically selected addresses.
Notice:
addr2line converts the address into a file number, and the file is saved on the disk. The address where the program runs is in the virtual memory (code segment). In higher versions of Linux, it may not be possible to parse where the address is in the file. addr2line can only look at the address of the virtual area.
2.2. dladdr1() function
Convert addresses into symbolic information. Function prototype:
#define _GNU_SOURCE #include <dlfcn.h> int dladdr(void *addr, Dl_info *info); int dladdr1(void *addr, Dl_info *info, void **extra_info, int flags); // Link with -ldl.
describe:
Function dladdr（）determines whether the address specified in addr is located in a shared object loaded by the calling application. If so, dladdr（）information about shared objects and symbols that overlap addr is returned. This information Dl_infois returned as a structure:
typedef struct {
const char *dli_fname; /* Pathname of shared object that contains address */
void *dli_fbase; /* Base address at which shared object is loaded */
const char *dli_sname; /* Name of symbol whose definition overlaps addr */
void *dli_saddr; /* Exact address of symbol named in dli_sname */
} Dl_info;
Function dladdr1()is similar dladdr()but extra_inforeturns additional information via parameters. The information returned depends on the value specified in the flag, which can have one of the following values:
(1) RTLD_DL_LINKMAP. Gets a pointer to the link map of the matching file. The extra_info parameter is a pointer to <link.h>a link_map structure (ie) defined in struct link_map**.
struct link_map {
ElfW(Addr) l_addr; /* Difference between the address in the ELF file and the address in memory */
char *l_name; /* Absolute pathname where object was found */
ElfW(Dyn) *l_ld; /* Dynamic section of the shared object */
struct link_map *l_next, *l_prev;
/* Chain of loaded objects */
/* Plus additional fields private to the implementation */
};
(2) RTLD_DL_SYMENT. Gets a pointer to the ELF symbol table entry for the matching symbol. extra_infoThe argument is a pointer to a symbol pointer: const ElfW（Sym）**. ElfW()A macro definition converts its argument to the name of an ELF data type appropriate for the hardware architecture. For example, on 64-bit platforms, ElfW(Sym)the data type name is generated Elf64_Sym, which is <elf.h>defined in:
typedef struct {
Elf64_Word st_name; /* Symbol name */
unsigned char st_info; /* Symbol type and binding */
unsigned char st_other; /* Symbol visibility */
Elf64_Section st_shndx; /* Section index */
Elf64_Addr st_value; /* Symbol value */
Elf64_Xword st_size; /* Symbol size */
} Elf64_Sym;
Package:
void * ConvertToElf(void *addr)
{
Dl_info info;
struct link_map *link;
dladdr1(addr, &info, (void **)&link, RTLD_DL_LINKMAP);
// 偏差纠正
return (void *)((size_t)addr - link->l_addr);
}
3. Implementation of memory leak detection
Memory leaks are caused by the mismatch between memory allocation and memory release. "Hijack" hooks for memory allocation functions malloc/calloc/realloc and memory release free. It can count the memory allocation location and memory release location to determine whether there is a match.
3.1. Method 1: Use mtrace
mtrace() and muntrace() functions
mtrace trace log. Function prototype:
#include <mcheck.h> void mtrace(void); void muntrace(void);
describe:
The mtrace() function installs hook functions [malloc(), realloc(), memalign(), free()] for the memory allocation function. These hook functions record tracking information about memory allocation and deallocation. Trace information can be used to discover memory leaks and attempt to free unallocated memory in the program.
muntrace()Function disables mtrace()installed hook functions so that trace information is no longer logged for memory allocation functions. If mtrace()no hook function is successfully installed, muntrace()no action is performed.
When called mtrace(), it checks MALLOC_TRACEthe value of an environment variable, which should contain the pathname of the file to which trace information is to be logged. If the pathname is opened successfully, its length will be truncated to zero.
If not set MALLOC_TRACE, or if the pathname it specifies is invalid or unwritable, the hook function will not be installed and will mtrace()have no effect. In set-user-ID and programs, it is ignored and has no effect. set-group-IDMALLOC_TRACEmtrace()
setenv() and unsetenv() functions
Change or add environment variables. Function prototype:
#include <stdlib.h>
int setenv(const char *name, const char *value, int overwrite);
int unsetenv(const char *name);
/*
Feature Test Macro Requirements for glibc (see feature_test_macros(7)):
setenv(), unsetenv():
_BSD_SOURCE || _POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600
*/
describe:
If the name does not exist, the setenv() function adds the variable name to the environment with value value. If name does exist in the environment and overwrite is non-zero, its value will be changed to value; if overwrite is zero, the value of name will not change (setenv() returns a success status). This function copies the string pointed to by name and value (as opposed to putenv(3)).
The unsetenv() function is used to remove a variable name from the environment. If the name does not exist in the environment, the function succeeds and the environment is unchanged.
return value:
The setenv() function returns zero on success and -1 on error, and sets errno to indicate the cause of the error.
The unsetenv() function returns zero on success and -1 on error, and sets errno to indicate the cause of the error.
mistake:
| error code | meaning |
|---|---|
| SINGLE CHOICE | name is NULL, pointing to a string of length 0, or containing the "=" character. |
| ENOMEM | Insufficient memory to add new variables to the environment. |
Steps for usage
(1) Call mtrace() before calling the memory allocation function;
(2) Call muntrace() at the end of the program or where there is no need to track memory leaks;
(3) Set the value of the environment variable MALLOC_TRACE (setenv function or export command);
(4) Bring the -g parameter when compiling.
(5) When a memory leak occurs, use the addr2line tool to locate the location of the memory leak.
$ addr2line -f -e memleak -a 0x4006b8
In the example, memleak is the program name, and 0x4006b8 is the address of the memory leak.
For example:
$ cc -g t_mtrace.c -o t_mtrace $ export MALLOC_TRACE=/tmp/t $ ./t_mtrace $ mtrace ./t_mtrace $MALLOC_TRACE
Sample code
#include <stdio.h>
#include <stdlib.h>
#include <mcheck.h>
int main(int argc,char **argv)
{
setenv("MALLOC_TRACE", "./mem.txt", 1);
mtrace();
void *p1 = malloc(10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free(p3);
free(p2);
muntrace();
unsetenv("MALLOC_TRACE");
return 0;
}
Memory leak detection file content:
$ cat mem.txt = Start @ ./memleak:[0x4006b8] + 0x1886580 0xa @ ./memleak:[0x4006c6] + 0x18865a0 0x14 @ ./memleak:[0x4006d4] + 0x18865c0 0x1e @ ./memleak:[0x4006e4] - 0x18865c0 @ ./memleak:[0x4006f0] - 0x18865a0 = End
Locate the memory leak location:
$ addr2line -f -e memleak -a 0x4006b8 0x00000000004006b8 main memleak.c:13
3.2. Method 2: Use macro definition
There are two macros __FILE__, __func__and in Linux __LINE__, which respectively indicate the current file name, function name and line number. Macro definitions are used to encapsulate the memory allocation function and release function.
#define malloc(size) _malloc(size,__FILE__,__LINE__) #define free(p) _free(p,__FILE__,__LINE__)
Call the malloc function and free function in the _malloc function and _free function yourself, and do some operations.
Prerequisite: Macros must be defined before memory allocation, so that the pre-compilation phase will replace malloc with our own implemented _mallocsum _free.
Sample code:
#include <stdio.h>
#include <stdlib.h>
void *_malloc(size_t size,const char*filename,int line)
{
void *p = malloc(size);
printf("[+] %s : %d, %p\n", filename, line,p);
return p;
}
void _free(void *p, const char*filename, int line)
{
printf("[-] %s : %d, %p\n", filename, line,p);
return free(p);
}
#define malloc(size) _malloc(size,__FILE__,__LINE__)
#define free(p) _free(p,__FILE__,__LINE__)
int main(int argc,char **argv)
{
void *p1 = malloc(10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free(p3);
free(p2);
return 0;
}
Advantages and disadvantages of using macro definition method:
(1) Advantages: simple implementation.
(2) Disadvantage: It is only suitable for a single file, and the macro definition must be placed at the front of the file.
Print using file replacement:
When the program is running, unnecessary information is always printed, which affects efficiency and is unsightly. You can create and delete files in a folder to count memory leaks.
Use the pointer value as the file name, allocate memory to create the file, release the memory to delete the file, and record the file name and line number of the allocated memory in the file.
If there are files in the folder, there is a memory leak. If there are no files, there is no memory leak.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define LEAK_FILE_PATH "./mem/%p.mem"
void *_malloc(size_t size,const char*filename,int line)
{
void *p = malloc(size);
//printf("[+] %s : %d, %p\n", filename, line,p);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, p);
FILE *fp = fopen(buff, "w");
fprintf(fp, "[+] %s : %d, addr: %p, size: %ld\n", filename, line, p, size);
fflush(fp);//刷新数据到文件中
fclose(fp);
return p;
}
void _free(void *p, const char*filename, int line)
{
//printf("[-] %s : %d, %p\n", filename, line,p);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, p);
if (unlink(buff) < 0)
{
printf("double free %p\n", p);
return;
}
return free(p);
}
#define malloc(size) _malloc(size,__FILE__,__LINE__)
#define free(p) _free(p,__FILE__,__LINE__)
int main(int argc,char **argv)
{
void *p1 = malloc(10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free(p3);
free(p2);
return 0;
}
Note that the tool can only speed up the analysis and cannot 100% determine the memory leak, because the situation of complex systems is more complicated.
Detecting memory leaks is not added at the beginning. It is usually turned on when needed through "hot update", that is, there is a flag in the configuration file that turns on memory leak detection. Only turn it on when needed, so it does not affect program efficiency.
3.3. Method three: hook (hook)
Hook usage steps:
(1) Define function pointers.
typedef void *(*malloc_t)(size_t size); typedef void(*free_t)(void *p); malloc_t malloc_f = NULL; free_t free_f = NULL;
(2) Function implementation, the function name is consistent with the target function name.
void *malloc(size_t size)
{
// ...
}
void free(void *ptr)
{
// ...
}
(3) Initialize the hook and call dlsym().
static init_hook()
{
if (malloc_f == NULL)
malloc_f = (malloc_t)dlsym(RTLD_NEXT, "malloc");
if (free_f == NULL)
free_f = (malloc_t)dlsym(RTLD_NEXT, "free");
}
Notice:
When hooking, you must consider that other functions also use the hooked function. For example, malloc is also called in the printf() function, so you need to prevent internal recursion from entering an infinite loop.
For example:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <dlfcn.h>
typedef void *(*malloc_t)(size_t size);
typedef void(*free_t)(void *p);
malloc_t malloc_f = NULL;
free_t free_f = NULL;
void *malloc(size_t size)
{
printf("malloc size: %ld", size);
return NULL;
}
void free(void *ptr)
{
printf("free: %p\n",ptr);
}
static int init_hook()
{
if (malloc_f == NULL)
malloc_f = (malloc_t)dlsym(RTLD_NEXT, "malloc");
if (free_f == NULL)
free_f = (free_t)dlsym(RTLD_NEXT, "free");
return 0;
}
int main(int argc,char **argv)
{
init_hook();
void *p1 = malloc(10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free(p3);
free(p2);
return 0;
}
The above code will cause a segfault. If you use gdb to debug, you will find that the printf() call in the malloc function has entered infinite recursion; a stack overflow.
The solution is to add a logo.
gcc built-in function: void *__builtin_return_address (unsigned integer level)
This function returns the return address of the current function or one of its callers. The argument is the number of frames to scan up the call stack. The value yields the return address of the current function, the value yields the return address of the caller of the current function, and so on. When inlining the expected behavior, the function returns the address of the returned function. To work around this problem, use function attributes.
level：
This parameter must be a constant integer.
On some computers, the return address of any function other than the current function may not be determined; in this case, or when the top of the stack is reached, this function returns an unspecified value. Additionally, can be used to determine if the top of the stack has been reached.
Sample code:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define LEAK_FILE_PATH "./mem/%p.mem"
#include <dlfcn.h>
static int enable_malloc_hook = 1;
static int enable_free_hook = 1;
typedef void *(*malloc_t)(size_t size);
typedef void(*free_t)(void *p);
malloc_t malloc_f = NULL;
free_t free_f = NULL;
void *malloc(size_t size)
{
void * p;
if (enable_malloc_hook)
{
enable_malloc_hook = 0;
p = malloc_f(size);
printf("malloc size: %ld,p=%p\n", size,p);
// Get the location where malloc is called from the previous layer Address, this address is used by the addr2line tool to convert it into a line number
void *caller = __builtin_return_address(0);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, p);
FILE *fp = fopen(buff, "w");
fprintf(fp, "[+] %p , addr: %p, size: %ld\n", caller, p, size); fflush
(fp);//Refresh data to the file
fclose( fp);
enable_malloc_hook = 1;
}
else
p = malloc_f(size);
return p;
}
void free(void *p)
{
if (enable_free_hook)
{
enable_free_hook = 0;
//printf("free: %p\n", p);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, p);
if (unlink(buff) < 0)
{
printf("double free %p\n", p);
//enable_free_hook = 1;
free_f(p);
return;
}
free_f(p);
enable_free_hook = 1;
}
else
free_f(p);
}
static int init_hook()
{
if (malloc_f == NULL)
malloc_f = (malloc_t)dlsym(RTLD_NEXT, " malloc");
if (free_f == NULL)
free_f = (free_t)dlsym(RTLD_NEXT, "free");
return 0;
}
int main(int argc,char **argv)
{
init_hook();
void *p1 = malloc (10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free(p3);
free(p2);
return 0;
}
The address obtained __builtin_return_address(0)needs the addr2line tool to convert it into a file line number to locate the location of the memory leak.
3.4. Method 4: Use __libc_malloc and __libc_free
The idea is the same as that of hook, because the underlying calls of malloc and free are also __libc_mallocand __libc_free.
Sample code:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
// Remember to create a mem folder manually
#define LEAK_FILE_PATH "./mem/%p.mem"
extern void * __libc_malloc(size_t size);
extern void __libc_free(void *p);
static int enable_malloc_hook = 1;
void *malloc(size_t size)
{
void *p;
if (enable_malloc_hook)
{
enable_malloc_hook = 0;
p = __libc_malloc(size);
printf ("malloc size: %ld,p=%p\n", size, p); //
Get the address of the place where malloc is called in the previous layer. This address is used by the addr2line tool to convert it into a line number
void *caller = __builtin_return_address (0);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, p);
FILE *fp = fopen(buff, "w");
fprintf(fp, "[+] %p , addr: % p, size: %ld\n", caller, p, size);
fflush(fp);//Refresh data to the file
fclose(fp);
enable_malloc_hook = 1;
}
else
p = __libc_malloc(size);
return p;
}
void free(void *p)
{
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, p);
if (unlink(buff) < 0)
{
printf("double free %p\n", p );
}
__libc_free(p);
}
int main(int argc,char **argv)
{
void *p1 = malloc(10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free( p3);
free(p2);
return 0;
}
3.5. Method 5: __malloc_hook (not recommended)
This method is suitable for older Linux versions, belongs to the API of the old version, __malloc_hookis a pointer method, and is a fixed value. It is essentially a hook technology.
Function prototype:
#include <malloc.h> void *(*__malloc_hook)(size_t size, const void *caller); void *(*__realloc_hook)(void *ptr, size_t size, const void *caller); void *(*__memalign_hook)(size_t alignment, size_t size, const void *caller); void (*__free_hook)(void *ptr, const void *caller); void (*__malloc_initialize_hook)(void); void (*__after_morecore_hook)(void);
describe:
The GNUC library allows you to modify the behavior of malloc, realloc and free by specifying the appropriate hook functions. For example, you can use these hooks to help debug programs that use dynamic memory allocation.
The variable __malloc_initialize_hook points to a function that is called once when initializing the malloc implementation. This is a weak variable, so it can be overridden in your application using a definition like this:
void（*__malloc_initialize_hook）（void）=my_init_hook();
Now the function my_init_hook() can initialize all hooks.
__malloc_hookThe prototypes of the four functions pointed to by , __realloc_hooks, __memalign_hooke, __free_hookyare respectively aligned with the functions malloc, realloc and memalign.
plan:
Exchange method, customize function pointers, implement specific functions, and exchange the functions you implement with the __malloc_hook provided by the system.
Sample code:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
// 要记得手动创建一个mem文件夹
#define LEAK_FILE_PATH "./mem/%p.mem"
#include <malloc.h>
/*
typedef void *(*malloc_t)(size_t size);
typedef void(*free_t)(void *p);
malloc_t malloc_f = NULL;
free_t free_f = NULL;
*/
static int enable_malloc_hook = 1;
static void my_init_hook(void);
static void *my_malloc_hook(size_t, const void *);
static void my_free_hook(void *, const void *);
/* Variables to save original hooks. */
static void *(*old_malloc_hook)(size_t, const void *);
static void(*old_free_hook)(void *, const void *);
/* Override initializing hook from the C library. */
void(*__malloc_initialize_hook) (void) = my_init_hook;
static void
my_init_hook(void)
{
old_malloc_hook = __malloc_hook;
__malloc_hook = my_malloc_hook;
old_free_hook = __free_hook;
__free_hook = my_free_hook;
}
static void *
my_malloc_hook(size_t size, const void *caller)
{
void *result;
/* Restore all old hooks */
__malloc_hook = old_malloc_hook;
/* Call recursively */
//result = malloc(size);
if (enable_malloc_hook)
{
enable_malloc_hook = 0;
result = malloc(size);
/* printf() might call malloc(), so protect it too. */
printf("malloc(%u) called from %p returns %p\n",
(unsigned int)size, caller, result);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, result);
FILE *fp = fopen(buff, "w");
fprintf(fp, "[+] %p , addr: %p, size: %ld\n", caller, result, size);
fflush(fp);//刷新数据到文件中
fclose(fp);
enable_malloc_hook = 1;
}
else
result = malloc(size);
/* Save underlying hooks */
old_malloc_hook = __malloc_hook;
/* Restore our own hooks */
__malloc_hook = my_malloc_hook;
return result;
}
static void my_free_hook(void *ptr, const void *caller)
{
__free_hook = old_free_hook;
free(ptr);
old_free_hook = __free_hook;
/* printf() might call malloc(), so protect it too. */
printf("free(%p) called from %p\n",
ptr, caller);
char buff[128] = { 0 };
sprintf(buff, LEAK_FILE_PATH, ptr);
if (unlink(buff) < 0)
{
printf("double free %p\n", ptr);
}
/* Restore our own hooks */
__free_hook = my_free_hook;
}
int main(int argc,char **argv)
{
my_init_hook();
void *p1 = malloc(10);
void *p2 = malloc(20);
void *p3 = malloc(30);
free(p3);
free(p2);
return 0;
}
A warning will appear during compilation, and the system does not recommend this method.
4. Complete sample code
The code is relatively long. In order to avoid being too long and difficult to read, I have not posted it here. If necessary, you can contact the blogger, or follow the WeChat public account "Lion" to obtain it.
Summarize
- The core of memory leak detection is to know whether there is a memory leak and where the memory leak has occurred.
- Ways to detect memory leaks include: mtrace, hook, macro definition, libc_malloc, __malloc_hook. Among them, mtrace needs to set the MALLOC_TRACE environment variable and needs to be restarted; macro definition is suitable for single files; __malloc_hook has been eliminated.
- Adding -g when compiling the program can use the addr2line tool to locate the location of the memory leak in the file.
- In order to improve program efficiency, the release program uses a "hot update" method to set the configuration file identifier for memory leak detection when needed.
Click to follow and learn about Huawei Cloud’s new technologies as soon as possible~
Lei Jun: The official version of Xiaomi's new operating system ThePaper OS has been packaged. The pop-up window on the lottery page of Gome App insults its founder. Ubuntu 23.10 is officially released. You might as well take advantage of Friday to upgrade! Ubuntu 23.10 release episode: The ISO image was urgently "recalled" due to containing hate speech. A 23-year-old PhD student fixed the 22-year-old "ghost bug" in Firefox. RustDesk remote desktop 1.2.3 was released, enhanced Wayland to support TiDB 7.4 Release: Official Compatible with MySQL 8.0. After unplugging the Logitech USB receiver, the Linux kernel crashed. The master used Scratch to rub the RISC-V simulator and successfully ran the Linux kernel. JetBrains launched Writerside, a tool for creating technical documents.