In summary these days I do, put three years ago, wrote a positioning case turned out. Recall the scene when the positioning of this leadership reminder tight grip, on their own are not familiar with a lot of things, so the pressure faced at the time still very large. In retrospect still a lot of emotion, we encounter any problems, we must not give up. Remember when doing product line development, a number of projects acquired knowledge, any bug, he will one day explode. Any questions, one day to find a good solution. When we tasted all can try the program, locate solutions to ideas, often the problem will be solved.
Things look on the engineering scale, in fact, many things in life are the same, any problem has a solution, but this approach is not out there to us with, he is hiding somewhere, waiting for us to dig of. So hard work, effort, do not be discouraged, find the right direction is very important.
Well, we get down to business, referring to the next step on my memory problems. First we look at the company's programming specifications often say: There is a saying,
Structure pointer initial value required before use.
This is a very simple rule, many times, we will feel in trouble, or when we use this specific structure, and we have the members of the structure can be assigned, or in the back when we use, only our sense interested members of the assignment just fine, we do not care about the other. But Now I come to tell you that this is not enough, the following question will tell you, programming specifications are written with the blood of the lessons. Comply with it, we can avoid a lot of unnecessary debuging.
Here we look at such an approach is correct:
struct msghdr msg; msg.msg_iov = &iov; msg.msg_iovlen = iovlen; ret = recvmsg(sockfd, &msg, int flags);
In function, we define a msg structure, but the structure did not give initial values, variables, only two of which we are concerned to do the assignment. So whether this will bring the problem?
Before project delivery, A students need to complete the stress test on the S system, repeatedly restart the virtual machine above, repeated kill a large number of service processes, do cpu, memory pressure. Such stress tests probably perform day-to-day or two, there will be abnormal, C service there a lot of coredump.
The problem is how to produce, how to locate the problem, look at the following specific targeting steps:
1、coredump
Applications for a variety of bug cause abnormal or abnormal exit or abort during operation, and will be produced under certain conditions called core file. Normally, core files contain the memory when the program is running, the status register, stack pointer, memory management information and various function call stack information, we can understand the current state of the work program is to generate the first file storage, many programs will generate an error when a core file by analyzing the file, we can locate the call stack and other information corresponding to the program quit unexpectedly when, identify the problems and timely resolution.
Applications can generate core files when a crash, need to do some settings:
1) Modify ulimit -c
As follows, ulimit -c set core file size, generally set to unlimited, so that core files without size restrictions. If you set the current session, then the application is valid only for the current session, that program only for the current session of the crash will have a core file is generated. If you want the system are valid, we need to set / etc / profile document, adding ulimit -c unlimited in file and source / etc / profile, so that configuration to take effect. If you still can not generate a core file, you can add ulimit -c unlimited in the process of pulling up the script in.
2) core configuration file generating path
core files default storage location corresponding executable in the same directory, a file called core, often crashes, and we can not find the core files are stored in what position, you can specify the storage location of the core file as follows.
echo "/home/xx/core-%e-%p-%u-%g-%t" > /proc/sys/kernel/core_pattern
Definition: %%: equivalent% % P: corresponds to <PID> % U: equivalent <UID> % G: corresponds to <GID> % S: digital signal corresponding to lead the dump % T: corresponds to the dump time % E: the equivalent of the name of the executable file % h: hostname equivalent
Coredump generated several possible scenarios:
1 ) Memory access violation a) the use of the wrong index, resulting array access out of bounds. b) When a search string, to rely on end of the string is determined whether the end of the string, the string is not normally used but terminator. c) using strcpy, strcat, sprintf, strcmp, strcasecmp string manipulation functions, etc., of the target string read / write burst. You should be used strncpy, strlcpy, strncat, strlcat, snprintf, strncmp, strncasecmp the like to read and write functions to prevent cross-border. 2 ) the use of multithreaded programs thread-safe function. 3 data) to read and write multi-threaded unlocked protection. For global data will be accessed by multiple threads at the same time, it should be noted that locking protection, or is likely to cause coredump 4 ) Illegal pointer a) the use of a null pointer b) Feel free to use pointer conversion. A pointer pointing to a section of memory, unless this memory previously allocated to a certain type or structure, or the structure or type of the array, do not convert it to this type of structure or pointer to determine, but this should be copied to a memory structure or this type, and then access the configuration or type. It is very easy because of bus error and core dump when the start because if this memory address is not aligned According to this structure or type, then access it. 5 ) Stack Overflow Do not use large local variables (because local variables are allocated on the stack), so likely to cause a stack overflow, stack and heap structural damage system, leading to inexplicable error occurs
2, Gdb process coredump positioning
Come to look at the specific core file, generally using gdb way to view, in order to see more information, you need to compile C at the time of service of the debug version, you can take the corresponding codes, add the -g option to compile, or on the obs debuginfo taken to the corresponding installation package. Glibc also need to install the debuginfo package.
Then the command gdb C (binary executable file for the debug version of the C c services) core.C-pid print out the core information. bt to view the call stack.
program terminated with signal 11, Segmentation fault. #0 strcmp () at ../sysdeps/x86_64/strcmp.s:135 135 cmpb [%rsi], %al (gdb) bt #0 strcmp () at ../sysdeps/x86_64/strcmp.s:135 #1 0x00007fc63177a345 in _dl_name_watch_p (name=0x400d56 "libc.so.6", map=0x7fc631909000) ad dl-misc.c:305 #2……
gdb common commands: are often used with a l (list), b (break), p (print), r (run), c (continue), q (quit), ir (see time register information), info local (see The current variable information), frame 4 (handover), disassemble + address (disassembly analysis)
By command p * (struct link_map *) + content to see the map of an address problems, as follows:
(gdb) p *(struct link_map*)0x7fae515fe000 $1 = {l_addr = 16, l_name = 0x7fab00000001<Address 0x7fab0000001 out of bonds>},……}
You can see the structure in l_name address is obviously wrong to view the glibc code, the code below, strcmp mistakes occurred when comparing the name. L_name address is 0x7fab00000001, four bytes suspected low position step.
int internal_function _dl_name_match_p (const char *name, const struct link_map *map) { if (strcmp (name, map->l_name) == 0) return 1; struct libname_list *runp = map->l_libname; while (runp != NULL) if (strcmp (name, runp->name) == 0) return 1; else runp = runp->next; return 0; }
Further in the map by the next pointer, the value can be found l_name sequentially linked list of all nodes, as follows:
link_map->l_name "" link_map->l_name "" link_map->l_name /usr/lib64/libsecurec.so link_map->l_name 0x7fab00000001 link_map->l_name /usr/lib64/libalarm.so link_map->l_name /usr/lib64/libc.so.6 link_map->l_name /usr/lib64//lib64/ld-linux-x86-64.so.2
View C dependent dynamic library:
:/home # ldd C
linux-vdso.so.1 => (0x00007ffff27e3000) libsecurec.so => /usr/lib64/libsecurec.so (0x00007fea37fb2000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fea37d95000) libalarm.so => /usr/lib64/libalarm.so (0x00007fea37b92000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007fea377d2000) /lib64/ld-linux-x86-64.so.2 (0x00007fea381c5000)
L_name know the corresponding value should be /usr/lib64/libpthread.so.
Here we have a little knowledge spots:
3, a dynamic library symbol resolution
The problem for the time being there is no idea, you start with the call stack to start it, this family function, _dl_runtime_resolve -> _ dl_fixup -> _ dl_lookup_symbol_x-> do_lookup_x -> _ dl_name_match_p-> strcmp is associated with dynamic analytic functions, online and read the relevant information , it simply involves two aspects:
1, dynamic loading, is when the module is running when you need to run the module was only mapped into virtual memory space, such as a module in operation to use mylib.so in myget function, but there is no call mylib .so this module before other functions, this module is not loaded into your program (that is, memory mapping).
2, dynamic resolution, that is, when the function to be called is called, we will go to this function in the start address of the virtual memory space parsed, and then write the address stored in the special calling module, as previously You say you have called the myget, so mylib.so module must have been mapped to the virtual program into memory, and if you recall mylib.so in myput function, then it calls the function address in time will be parsed.
Call stack is the address resolution related things, then that is not a function call before being parsed, that in this process is the first call. segfault is generated when calling prctl, the function will first fork a child process, then call prctl function in the child process, while the prctl is precisely this function will only be called in the child process, bringing this issue to some phenomena can be explained clearly: a field related link_map structure of the dynamic library is stepped on, fork a child process, although it will get a copy of all the information from the parent process there, but the parent did not call off prctl this function, it will not when this function is to resolve the address information, the call prctl function in the child process, it will flow into the dynamically resolved, mistakes occur during strcmp compare. Because the parent process does not segfault, the parent process continues to run, will then continue to fork child process, so the core file will produce more of the same size, but the process id, timestamp different core file.
4, positioning means step on memory issues
1)gdb watch
Code investigation, found a global variable to store link_map of _rtld_global , in the C module main function, the daemon pull, joined delay (so that we have time to find the address of the memory is always being stepped according to a global variable) recompile the code, replace the bin file, then ps -aux | grep C, see the daemon the above mentioned id , then gdb -p PID , enter the daemon gdb debugging, first the p-_rtld_global , then find the first global variable variable , the p-* (struct link_map *) the address, this is the link_map address of the first member of the list, and then follow the next address in order to find the store pthread.so information link_map address structure, and then Watch * ( int * ) this address . If you have an application rewrite this address, then when the time is written, gdb will live off, and prints the call stack information.
But, unfortunately, it has happened segfault, but did not call stack information recording. In gdb, if you watch less, most likely overwrite kernel memory.
2) step is set to read-only memory
This address will be read-only, then when there is a program to rewrite the memory, it will generate the corresponding core files, through core, we can know the call stack, it is clear who is to rewrite the memory.
void set_page_ro(void) { char *p = NULL; p = find_link_map("/usr/lib64/libpthread.so.0"); mprotect((void*)((unsigned long)p & 0xfffffffffffff000UL), 4096, PROT_READ); }
Code, first find the corresponding address libpthread.so, and then set the address where the page is read-only. But after practice, I found that this approach does not work, because you can only set up to 4k page read-only, so a lot of the normal written in the 4k range will produce core files.
So since it is suspected to rewrite the kernel in memory, then we begin to start it from the kernel.
3) to get the call stack kernel memory by stepping system calls
Need to compile a ko, then the ismod ko, then the main function of the C module added syscall (SYS_afs_syscall, p); acquired pthread.so p is the corresponding address. Then the compiled executable file, replace the bin, start C service, when p This address is rewritten by dmesg you can see the call stack.
[ 6322.163418] Call Trace: [ 6322.166046] <#DB> [<ffffffff816379e4>] dump_stack+0x19/0x1b [ 6322.171993] [<ffffffffa049402a>] sample_hbp_handler+0x2a/0x30 [a] [ 6322.178361] [<ffffffff81160d31>] __perf_event_overflow+0xa1/0x250 [ 6322.184733] [<ffffffff81161081>] perf_swevent_overflow+0x51/0xe0 [ 6322.191013] [<ffffffff8116117f>] perf_swevent_event+0x6f/0x90 [ 6322.197031] [<ffffffff81161b49>] perf_bp_event+0x99/0xc0 [ 6322.202618] [<ffffffff816412b0>] hw_breakpoint_exceptions_notify+0x120/0x150 [ 6322.209944] [<ffffffff81643a1c>] notifier_call_chain+0x4c/0x70 [ 6322.216025] [<ffffffff81643ac5>] notify_die+0x45/0x60 [ 6322.221351] [<ffffffff816404ef>] do_debug+0xaf/0x230 [ 6322.226589] [<ffffffff8163f92b>] debug+0x2b/0x40 [ 6322.231484] [<ffffffff8151b2c7>] ? kfree_skbmem+0x37/0x90 [ 6322.237163] [<ffffffff8130323b>] ? copy_user_enhanced_fast_string+0xb/0x20 [ 6322.244278] <<EOE>> [<ffffffff81513062>] ? move_addr_to_user+0xb2/0xd0 [ 6322.251210] [<ffffffff81514c9d>] ___sys_recvmsg+0x14d/0x2b0 [ 6322.257052] [<ffffffff8163cfb9>] ? schedule+0x29/0x70 [ 6322.262383] [<ffffffff8119912c>] ? handle_mm_fault+0x65c/0xf50 [ 6322.268493] [<ffffffff81515821>] __sys_recvmsg+0x51/0x90 [ 6322.274052] [<ffffffff81515872>] SyS_recvmsg+0x12/0x20 [ 6322.279465] [<ffffffff81648049>] system_call_fastpath+0x16/0x1b [ 6322.285796] --- 14678, 00007f73450a3000 value is changed
Using this method, you need to explain more content, please see my other blog
Later investigation by the code, find out all the places recvmsg calls, combined with modules of problems, and finally found the msg structure is not initialized relationship. Let's put this issue from the start and then analyze the cause again.
5, the cause of the problem
1) When C service just started, the new space is allocated, so should the value of the memory is cleared. Pull a thread, a local variable MSG application for the thread, although not to initialize, but this time the value of the entire memory sheet is 0.
[2016-08-09:10:23:04]C[690]: C starting up [2016-08-09:10:23:04]C[690]: --- nlh[0x23750e0]: typ[32676] seq[677353752]; msg[0x7fa424236d10]: name[(nil)] len[(nil)] control[(nil)] clen[0] flags[0]
2) started a long steady test case, see C logs found in 1s probably record 40 monitors, and will recvmsg error, the error code is 105, meaning No buffer space available.
[2016-08-10:07:59:59]C[690]: recvmsg from NETLINK_SIGNO socket failed [105]
After 3) signal monitoring thread exits, the daemon and then pull the thread to monitor the signal, then the time of LWP process space is not new, it will not be all zeros up. You can see the value of msg, name has not 0 a.
[2016-08-10:07:59:59]C[690]: C starting up [2016-08-10:07:59:59]C[690]: --- nlh[0x2376fd0]: typ[32676] seq[677353560]; msg[0x7fa424236d10]: name[0x7fa428e50000] len[0x7fa428c3ae6f] control[0x5] clen[0] flags[1]
So you can see the address 0x7fa428e50000, then the address is the address space of the process corresponding link in the library.
Resulting in the emergence of this issue, in fact, developers programming practices and norms, we recvmsg when calling function, passing the parameters, there are several variables no initial value.
Return it to recvmsg, the following function prototype:
ssize_t recvmsg(int sockfd, struct msghdr *msg, int flags);
Msghdr herein are defined as follows:
struct msghdr { void *msg_name; /* protocol address */ socklen_t msg_namelen; /* sieze of protocol address */ struct iovec *msg_iov; /* scatter/gather array */ int msg_iovlen; /* # elements in msg_iov */ void *msg_control; /* ancillary data ( cmsghdr struct) */ socklen_t msg_conntrollen; /* length of ancillary data */ int msg_flags; /* flags returned by recvmsg() */ }
When we use this function to msg_iov msghdr of this structure, msg_iovlen do the assignment, the other variables are not dealt with, there is no initial value. When the service starts C, C service process space is newly allocated, this space will be initialized to 0 (because this space is a piece of physical memory allocated by the kernel, assigned to the user mode, must be cleared, or users when the kernel will read the content), process C service after creating a thread, the thread is a shared process space, so the space thread is cleared, we define a variable of type struct msghdr in this thread, this memory is full zero, even if we have no initial value, under normal circumstances, not a problem.
However, unusual circumstances, such as we now stress test, cpu, memory pressure in the case of anything unusual are likely to occur. When an exception occurs, the thread appeared abnormal exit, the daemon C services will then pull the thread, then pull up the thread again, would not be so coincidentally, is assigned to a space zero. In the address using the msg value, the code only two members of the assignment, other value is unknown, just msg variable to store the dynamic library had been visited before, which still remained before using this address local variables, and this value is just l_name address link_map structure of the pthread.so. In this way, when we call recvmsg, put the parameters passed to the kernel, that we msg_name address passed to it, so it gives us return the socket name, that is the kernel to write the address on the wrong data. Well, go down after leading to the occurrence of the problem segfault.
Now let's look back at the company's programming specifications, it is not such a simple request can be avoided behind the debugging work we have done so much of it.