2.5. Real Debugging Examples
The best way to learn about any tool is to roll up your sleeves and try to solve some real problems on your own. This section includes some examples to get you started.
了解任何工具的最好方法是卷起袖子, 尝试自己解决一些真正的问题。本节包含一些示例以使您开始尝试。
2.5.1. Reducing Start Up Time by Fixing LD_LIBRARY_PATH
The LD_LIBRARY_PATH environment variable is used by the run time linker to find the depended libraries for an executable or library. The ldd command can be used to find the dependent libraries:
运行时linker使用 LD_LIBRARY_PATH 环境变量查找可执行文件或库的依赖库。ldd 命令可用于查找从属库:
ion 201% ldd /bin/ls
librt.so.1 => /lib/librt.so.1 (0x40024000)
libacl.so.1 => /lib/libacl.so.1 (0x40035000)
libc.so.6 => /lib/libc.so.6 (0x4003b000)
libpthread.so.0 => /lib/libpthread.so.0 (0x40159000)
libattr.so.1 => /lib/libattr.so.1 (0x4016e000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
When a program is first run, the run time linker must locate and load all of these libraries before the program can execute. The run time linker runs inside the process itself, and any interactions with the operating system can be traced with strace. Before getting into details, first let me apologize in advance for the long strace output. This is a good example of a poor LD_LIBRARY_PATH, but unfortunately the output is very long.
第一次运行程序时, 运行时linker必须找到并加载所有这些库, 然后才能执行程序。运行时链接器在进程本身内运行, 与操作系统的任何交互都可以用 strace 跟踪。在详细介绍之前, 首先让我为过长的strace的输出道歉。这是一个很好的例子, 一个可怜的 LD_LIBRARY_PATH, 但不幸的是, 输出很长。
Code View: Scroll / Show All
ion 685% echo $LD_LIBRARY_PATH
/usr/lib:/home/wilding/sqllib/lib:/usr/java/lib:/usr/ucblib:/opt/IBMcset/lib
With this LD_LIBRARY_PATH, the run time linker will have to search /usr/lib first, then /home/wilding/sqllib/lib, then /usr/java/lib, and so on. The strace tool will show just how much work is involved:
使用此 LD_LIBRARY_PATH, 运行时linker将必须先搜索/usr/lib, 然后/home/wilding/sqllib/lib, 然后/usr/java/lib, 等等。strace 工具将显示所涉及的工作量:
Code View: Scroll / Show All
ion 206% strace telnet
execve("/usr/bin/telnet", ["telnet", "foo", "136"], [/* 64 vars */]) = 0
uname({sys="Linux", node="ion", ...}) = 0 brk(0) = 0x8066308
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -
1, 0) = 0x40013000
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/lib/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/lib/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib", {st_mode=S_IFDIR0755, st_size=32768, ...})
= 0
open("/home/wilding/sqllib/lib/i686/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/home/wilding/sqllib/lib/i686/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/home/wilding/sqllib/lib/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/home/wilding/sqllib/lib/i686", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/home/wilding/sqllib/lib/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/home/wilding/sqllib/lib/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/home/wilding/sqllib/lib/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/home/wilding/sqllib/lib",
{st_mode=S_IFDIRS_ISGID0755, st_size=12288, ...}) = 0
open("/usr/java/lib/i686/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/lib/i686/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/java/lib/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/lib/i686", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/java/lib/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/lib/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/java/lib/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/java/lib", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/ucblib/i686/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT No such file or directory)
stat64("/usr/ucblib/i686/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/ucblib/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/ucblib/i686", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/ucblib/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT(No such file or directory)
stat64("/usr/ucblib/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/usr/ucblib/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/ucblib", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/opt/IBMcset/lib/i686/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/opt/IBMcset/lib/i686/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/opt/IBMcset/lib/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/opt/IBMcset/lib/i686", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/opt/IBMcset/lib/mmx/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/opt/IBMcset/lib/mmx", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/opt/IBMcset/lib/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/opt/IBMcset/lib", 0xbfffe59c) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=65169, ...}) = 0
mmap2(NULL, 65169, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40014000 close(3) = 0
open("/lib/libncurses.so.5", O_RDONLY) = 3 read(3,
"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\357\0"..., 1024) = 1024
The strace output shows 20 failed attempts to find the libncurses.so.5 library. In fact, 40 of the 52 lines of this strace deal with the failed attempts to find the curses library. The LD_LIBRARY_PATH includes too many paths (starting from the beginning) that do not contain libncurses.so.5. A better LD_LIBRARY_PATH would contain /lib (where libncurses.so.5 was eventually found) near the beginning of the path list:
strace 输出显示20次失败尝试找到 libncurses. 5 库。事实上, 这个 strace 的52行中有40行是用来处理失败的尝试找到curses库的。LD_LIBRARY_PATH 包含的路径太多 (从开头开始) 不包含 libncurses. so. 5。一个更好的 LD_LIBRARY_PATH 应该将/lib (在那里 libncurses.so.5, 最终找到了) 放在路径列表的开头:
ion 689% echo $LD_LIBRARY_PATH
/lib:/usr/lib:/home/wilding/sqllib/lib:/usr/java/lib
The strace shows that this LD_LIBRARY_PATH is much more efficient:
strace 表明, 这种 LD_LIBRARY_PATH 效率更高:
Code View: Scroll / Show All
ion 701% strace telnet | & head -15
execve("/usr/bin/telnet", ["telnet"], [/* 77 vars */]) = 0
uname({sys="Linux", node="ion", ...}) = 0
brk(0) = 0x8066308
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -
1, 0) = 0x40013000
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/i686/29/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/29", 0xbfffe41c) = -1 ENOENT (No such file or directory)
open("/lib/i686/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", 0xbfffe41c) = -1 ENOENT (No such file or directory)
open("/lib/29/libncurses.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/29", 0xbfffe41c) = -1 ENOENT (No such file or directory)
open("/lib/libncurses.so.5", O_RDONLY) = 3
A bad LD_LIBRARY_PATH environment variable is not usually a problem for everyday human-driven command line activity; however, it can really affect the performance of scripts and Common Gateway Interface (CGI) programs. It is always worth testing CGI programs and scripts to ensure that the library path picks up the libraries quickly and with the least amount of failures.
一个糟糕的 LD_LIBRARY_PATH 环境变量通常不是日常人为驱动的命令行活动的问题;但是, 它确实会影响脚本和通用网关接口 (CGI) 程序的性能。它始终值得测试 CGI 程序和脚本, 以确保库路径快速和以最少的失败数量获取库。
2.5.2. The PATH Environment Variable
Tracing a shell can also reveal a bad PATH environment variable. From a different shell, run strace -fp <shell pid>, where <shell pid> is the process ID of the target shell. Next, run the program of your choice and look for exec in the strace output. In the following example, there are only two failed searches for the program called main.
跟踪 shell 还可以显示错误的路径环境变量。从另一个 shell 中, 运行 strace -fp <shell pid>, 其中<shell pid>是目标 shell 的进程 ID。接下来, 运行您选择的程序并在 strace 输出中查找 exec。在下面的示例中, 只有两个名为 main 的程序失败的搜索.
Code View: Scroll / Show All
[pid 27187] execve("main", ["main"], [/* 64 vars */]) = -1 ENOENT (No such file or directory)
[pid 27187] execve("/usr/sbin/main", ["main"], [/* 64 vars */]) = -1 ENOENT (No such file or directory)
[pid 27187] execve("/home/wilding/bin/main", ["main"], [/* 64 vars */]) = 0
A bad PATH environment variable can cause many failed executions of a script or tool. This too can impact the startup costs of a new program.
错误的路径环境变量可能导致许多脚本或工具执行失败。这也会提高新程序的启动成本。
2.5.3. stracing inetd or xinetd (the Super Server)
Most Linux systems are connected to a network and can accept remote connections via TCP. Some common examples include telnet and Web communications. For the most part, everything usually works as expected, although what if the software that is driven by a remote connection encounters a problem? For example, what if a remote user cannot connect to a system and log in using a telnet client? Is it a problem with the user’s shell? Does the user’s shell get hung up on a path mounted by a problematic NFS server?
大多数 Linux 系统都连接到网络, 通过 TCP 接受远程连接。一些常见的例子包括 telnet 和 Web 通信。在大多数情况下, 所有的内容通常都按预期方式工作, 但如果远程连接驱动的软件遇到问题怎么办？例如, 如果远程用户无法使用 telnet 客户端连接到系统, 该怎么办？用户的shell有问题吗？用户的 shell 是否停在由故障 NFS 服务器安装的路径上？
Here is an example of how strace can be used to examine an incoming telnet connection. This example requires root access.
下面是一个示例, 说明如何使用 strace 来检查传入的 telnet 连接。此示例需要root权限。
The first step is to find and strace the inetd daemon on the telnet server (as root):
第一步是查找并 跟踪 telnet 服务器上的 inetd 守护进程 (作为 root 用户):
ion 200# ps -fea | grep inetd | grep -v grep
root 986 1 0 Jan27 ? 00:00:00 /usr/sbin/inetd
ion 201# strace -o inetd.strace -f -p 986
Then, on a remote system use the telnet client to log in to the telnet server:
然后, 在远程系统上, 使用 telnet 客户端登录到 telnet 服务器:
sunfish % telnet ion
After logging in to the telnet server, hit control-C to break out of the strace command and examine the strace output:
登录到 telnet 服务器后, 按ctrl + C 以跳出 strace 命令并检查 strace 输出:
| Note: lines with ... represent strace output that was removed for simplicity. 注意: 以...开始的行 表示为了简单起见而删除的 strace 输出。 |
Code View: Scroll / Show All
ion 203# less inetd.strace
986 select(13, [4 5 6 7 8 9 10 11 12], NULL, NULL, NULL) = 1(in [5])
986 accept(5, 0, NULL) = 3
986 getpeername(3, {sin_family=AF_INET, sin_port=htons(63117), sin_addr=inet_addr("9.26.48.230")}}, [16]) = 0
...
986 fork() = 27202
...
986 close(3 <unfinished ...>
27202 socket(PF_UNIX, SOCK_STREAM, 0 <unfinished ...>
986 <... close resumed> ) = 0
27202 <... socket resumed> ) = 13
The select system call waits for a new connection to come in. The new connection comes in on the inetd’s socket descriptor 5. The accept system call creates a new socket descriptor that is directly connected to the remote telnet client. Immediately after accepting the new connection, the inetd gathers information about the source of the connection, including the remote port and IP address. The inetd then forks and closes the socket description. The strace output is continued as follows:
select系统调用等待新连接进入。新连接出现在 inetd 的套接字描述符5上。accept系统调用创建一个与远程 telnet 客户端直接连接的新的套接字描述符。在接受新连接后, inetd 会立即收集有关连接源的信息, 包括远程端口和 IP 地址。然后, inetd 复制自身进程处理telnet连接并关闭套接字描述。strace 输出按如下方式继续:
Code View: Scroll / Show All
27202 connect(13, {sin_family=AF_UNIX, path="/var/run/.nscd_socket"}, 110) = 0
...
27202 dup2(3, 0) = 0
27202 close(3) = 0
27202 dup2(0, 1) = 1
27202 dup2(0, 2) = 2
27202 close(1022) = -1 EBADF (Bad file descriptor)
27202 close(1021) = -1 EBADF (Bad file descriptor)
...
27202 close(4) = 0
27202 close(3) = -1 EBADF (Bad file descriptor)
27202 rt_sigaction(SIGPIPE, {SIG_DFL}, NULL, 8) = 0
27202 execve("/usr/sbin/tcpd", ["in.telnetd"], [/* 16 vars */])=0
The strace output here shows a connection to the name service cache daemon (see man page for nscd). Next, the file descriptors for stdin (0), stdout (1), and stderr (2) are created by duplicating the socket descriptor. The full list of socket descriptors are then closed (from 1022 to 3) to ensure that none of the file descriptors from the inetd are inherited by the eventual shell. Last, the forked inetd changes itself into the access control program.
此处的 strace 输出显示了到名称服务缓存守护进程的连接 (参见 nscd 的帮助手册)。接下来, 将通过复制套接字描述符来创建 stdin (0)、标准输出 (1) 和 stderr (2) 的文件描述符。然后关闭套接字描述符的完整列表 (从1022到 3), 以确保 inetd 中的文件描述符都不会被最终的 shell 继承。最后, 复制的inetd 将自身转变为访问控制程序。
Code View: Scroll / Show All
...
27202 open("/etc/hosts.allow", O_RDONLY) = 3
...
27202 execve("/usr/sbin/in.telnetd", ["in.telnetd"], [/* 16 vars */]) = 0
...
27202 open("/dev/ptmx", O_RDWR) = 3
...
27202 ioctl(3, TIOCGPTN, [123]) = 0
27202 stat64("/dev/pts/123", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 123), ...}) = 0
...
27202 open("/dev/pts/123", O_RDWR|O_NOCTTY) = 4
...
27203 execve("/bin/login", ["/bin/login", "-h", "sunfish.torolab.ibm.com", "-p"], [/* 3 vars */] <unfinished ...>
The access control program checks the hosts.allow file to ensure the remote client is allowed to connect via telnet to this server. After confirming that the remote client is allowed to connect, the process turns itself into the actual telnet daemon, which establishes a new pseudo terminal (number 123). After opening the pseudo terminal, the process changes itself into the login process.
访问控制程序检查hosts.allow以确保允许远程客户端通过 telnet 连接到此服务器。确认远程客户端被允许连接后, 该进程将自身转变为实际的 telnet 守护进程, 从而建立一个新的伪终端 (数字 123)。打开伪终端后, 进程将自己更改为登录过程。
Code View: Scroll / Show All
...
27202 open("/dev/tty", O_RDWR <unfinished ...
...
27203 readlink("/proc/self/fd/0", "/dev/pts/123", 4095) = 12
...
27203 open("/dev/pts/123", O_RDWR|O_NONBLOCK|O_LARGEFILE) = 3
...
27203 open("/etc/shadow", O_RDONLY) = 3
...
27203 access("/var/log/wtmpx", F_OK) = -1 ENOENT (No such file or directory)
27203 open("/var/log/wtmp", O_WRONLY) = 3
...
27203 open("/var/log/lastlog", O_RDWR|O_LARGEFILE) = 3
...
27204 chdir("/home/wilding" <unfinished ...>
...
27204 execve("/bin/tcsh", ["-tcsh"], [/* 9 vars */]) = 0
...
27202 select(4, [0 3], [], [0], NULL <unfinished ...>
Lastly, the login process goes through the login steps, confirms the user’s password by checking the /etc/shadow file, records the user’s login (lastlog, for example), and changes the directory to the user’s home directory. The final step is the process changing itself into the shell (tcsh in this example) and the shell waiting for input from stdin. When the remote user types any character, the select call will wake up, and the shell will handle the character as appropriate.
最后, 登录过程进入登录步骤, 通过检查/etc/shadow文件确认用户的密码, 记录用户的登录 (例如 lastlog), 并将目录更改为用户的主目录。最后一步是进程将自身转换为 shell (本例中为 tcsh) , 以等待 stdin 输入。当远程用户键入字符时, select 调用将会唤醒, shell 将根据需要处理该字符。
2.5.4. Communication Errors
The following example shows how strace can be used to provide more detail about a telnet connection failure.
下面的示例演示如何使用 strace 提供telnet 连接失败的更多详细信息。
Code View: Scroll / Show All
ion 203% strace -o strace.out telnet foo 136
Trying 9.26.78.114...
telnet: connect to address 9.26.78.114: Connection refused
ion 204% less strace.out
...
open("/etc/resolv.conf", O_RDONLY) = 3
...
open("/etc/nsswitch.conf", O_RDONLY) = 3
...
open("/etc/hosts", O_RDONLY) = 3
...
open("/etc/services", O_RDONLY) = 3
...
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_IP, IP_TOS, [16], 4) = 0
connect(3, {sin_family=AF_INET, sin_port=htons(136), sin_addr=inet_addr("9.26.78.114")}}, 16) = -1 ECONNREFUSED
Connection refused)
Most of the strace output is not included here for the sake of simplicity. The only interesting system call is the last one where the connect fails. The IP address and the port number are clearly shown in the strace output. This could be useful if the host name had several IP addresses or when the problem is more complex. The man page for the connect system call will have a clear description of ECONNREFUSED or any other error that may be returned for connect.
为了简单起见，大部分strace 输出不包括在这里。唯一有趣的系统调用是连接失败的最后一个。IP 地址和端口号清楚地显示在 strace 输出中。如果主机名有多个 IP 地址, 或者问题比较复杂时, 这可能很有用。connect系统调用的帮助手册将对 ECONNREFUSED 或其他可能的返回错误有清楚的描述。
2.5.5. Investigating a Hang Using strace
If the problem is occurring now or can be reproduced, use the strace with the -ttt switch to get a system call trace with timestamps. If the hang is in user code, the last line of the strace will show a completed system call. If the hang is in a system call, the last line of the strace will show an incomplete system call (that is, one with no return value). Here is a simple problem to show the difference:
如果问题现在正在发生或可以重现, 请使用带 -ttt 参数的 strace 来获取带有时间戳的系统调用跟踪。如果程序停在了用户代码, strace 的最后一行将显示已完成的系统调用。如果程序停在系统调用里, 则 strace 的最后一行将显示一个不完整的系统调用 (即没有返回值的一个)。下面是一个简单的例子来显示差异:
Code View: Scroll / Show All
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <string.h>
int main( int argc, char *argv[] )
{
getpid( ) ; // a system call to show that we've entered this code
if ( argc < 2 )
{
printf( "hang (user|system)" ) ;
return 1 ;
}
if ( !strcmp( argv[1], "user" ) )
{
while ( 1 ) ;
}
else if ( !strcmp( argv[1], "system" ) )
{
sleep( 5000 ) ;
}
return 0 ;
}
Here is an example of a “user hang” using the hang tool. The strace shows that the system call getpid() completed and no other system calls were executed.
下面是使用挂起工具的 "用户挂起" 的示例。strace 显示系统调用 getpid () 已完成, 并且未执行其他系统调用。
ion 191% g++ hang.C -o hang
ion 192% strace -ttt hang user
...
1093627399.734539 munmap(0x400c7000, 65169) = 0
1093627399.735341 brk(0) = 0x8049660
1093627399.735678 brk(0x8049688) = 0x8049688
1093627399.736061 brk(0x804a000) = 0x804a000
1093627399.736571 getpid() = 18406
Since the last system call has completed, the hang must be in the user code somewhere. Be careful about using a screen pager like “more” or “less” because the buffered I/O may not show the last system call. Let strace run until it hangs in your terminal.
由于上次系统调用已完成, hang必定发生在用户代码中的某个位置。请注意使用像 "more" 或 "less" 这样的屏幕分页工具, 因为缓冲 I/O 可能不会显示上一次系统调用。让 strace 跑, 直到它hang在你的终端。
If the strace tool traces an application that hangs in a system call, the last system call will be incomplete as in the following sample output:
如果 strace 工具跟踪一个hang在系统调用中的应用程序, 则最后一个系统调用将不完整, 如下面的示例输出所示:
ion 193% strace -ttt hang system
1093627447.115573 brk(0x8049688) = 0x8049688
1093627447.115611 brk(0x804a000) = 0x804a000
1093627447.115830 getpid() = 18408
1093627447.115887 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
1093627447.115970 rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
1093627447.116026 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
1093627447.116072 nanosleep({5000, 0},
Notice that nanosleep() does not have a return code because it has not yet completed. A hang in a system call can tell you a lot about the type of hang. If the hang was in a system call such as read, it may have been waiting on a socket or reading from a file. The investigation from this point on will depend on what you find in the strace output and from gdb. A hang in a system call also can tell you a lot about the cause of the hang. If you are not familiar with the system call, read the man page for it and try to understand under what circumstances it can hang.
请注意, nanosleep () 没有返回代码, 因为它尚未完成。在系统调用中hang可以告诉您很多关于hang类型的信息。如果hang发生在系统调用 (如读取), 则可能是在等待套接字或从文件读取。从这一点上进行的调查将取决于您在 strace 输出和 gdb 中发现的内容。hang在系统调用也可以告诉你很多关于挂起的原因。如果您不熟悉系统调用, 请阅读帮助手册, 并尝试了解在什么情况下它会hang。
The arguments to the system call can give you additional hints about the hang. If the hang is on a read system call, the first argument will be the file description. With the file descriptor, you can use the lsof tool to understand which file or socket the read system call is hung on.
系统调用的参数可以为您提供有关hang的其他提示。如果hang发生在read系统调用, 则第一个参数将是文件描述。使用文件描述符, 您可以用 lsof 工具了解read系统调用的hang发生的文件或套接字。
ion 1000% strace -p 23735
read(16,
The read system call in the preceding example has a file descriptor of 16. This could be a file or socket, and running lsof will tell you which file or socket it is. If the hang symptom is affecting a network client, try to break the problem down into a client side hang, network hang, or server side hang.
前面示例中的 read 系统调用的文件描述符为16。这可能是一个文件或套接字, 运行 lsof 将告诉您它是哪个文件或套接字。如果hang正在影响网络客户端, 请尝试将问题分解为客户端hang、网络hang或服务器端hang。
2.5.6. Reverse Engineering (How the strace Tool Itself Works)
The strace tool can also be used to understand how something works. Of course, this example of reverse engineering is illustrated here for educational purposes only. In this example, we’ll use strace to see how it works.
strace 工具也可以用来理解某些东西是如何工作的。当然, 这一逆向工程的例子仅用于教育目的。在这个例子中, 我们将使用 strace 来查看它是如何工作的。
First, let’s strace the strace tool as it traces the /bin/ls program.
首先, 让我们 strace 跟踪strace 工具, 正如它跟踪/bin/ls 程序。
| Note: Uninteresting strace output has been replaced by a single line with ... 注意: 无趣的 strace 输出已用…开头的行替换..。 |
Code View: Scroll / Show All
ion 226% strace -o strace.out strace /bin/ls
ion 227% less strace.out
execve("/usr/bin/strace", ["strace", "/bin/ls"], [/* 75 vars */])= 0
...
fork() = 16474
...
wait4(-1, [WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP], 0x40000000, NULL) = 16474
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
ptrace(PTRACE_SYSCALL, 16474, 0x1, SIG_0) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(-1, [WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP], 0x40000000, NULL) = 16474
— SIGCHLD (Child exited) —
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*ORIG_EAX, [0x7a]) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*EAX, [0xffffffda]) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*EBX, [0xbfffed2c]) = 0
write(2, "uname(", 6) = 6
ptrace(PTRACE_SYSCALL, 16474, 0x1, SIG_0) = 0
The first system call of interest is the fork to create another process. This fork call is required to eventually spawn the /bin/ls program. The next interesting system call is the wait4 call. This call waits for any child processes of the strace program to change state. The process that stops is the child process of the previous fork call (pid: 16474). Shortly after the wait call, there is a call to ptrace with the PTRACE_SYSCALL value for the first argument. The man page for ptrace states:
第一个系统调用的兴趣是复制另一个进程。此fork调用最终生成/bin/ls 程序。下一个有趣的系统调用是 wait4 调用。此调用等待 strace 程序的任何子进程更改状态。停止的过程是上一个fork调用的子进程 (pid: 16474)。wait调用后不久, 就会调用 ptrace 的 PTRACE_SYSCALL 值作为第一个参数。ptrace 的帮助手册:
TRACE_SYSCALL, PTRACE_SINGLESTEP
Restarts the stopped child as for PTRACE_CONT, but arranges for the child to be stopped at the next entry to or exit from a system call, or after execution of a single instruction, respectively. (The child will also, as usual, be stopped upon receipt of a signal.) From the parent’s perspective, the child will appear to have been stopped by receipt of a SIGTRAP. So, for PTRACE_SYSCALL, for example, the idea is to inspect the arguments to the system call at the first stop, then do another PTRACE_SYSCALL and inspect the return value of the system call at the second stop. (addr is ignored.)
重新启动已停止的子进程作为 PTRACE_CONT, 但安排子进程停在下一入口或从系统调用中退出或在执行单个指令后停止该子项。(子进程也会像往常一样, 在收到信号后停止)。从父进程的角度来看, 该子进程似乎已通过接收 SIGTRAP 停止。因此, 对于 PTRACE_SYSCALL, 例如, 想法是在第一站检查系统调用的参数, 然后在第二站做另一个 PTRACE_SYSCALL 和检查系统调用的返回值。(地址被忽略)。
So the strace tool is waiting for the child process to stop and then starts it in such a way that it will stop on the next entry or exit of a system call. After the second call to wait4, there are a number of calls to ptrace with PTRACE_PEEKUSER as the first argument. According to the ptrace man page, this argument does the following:
因此, strace 工具正在等待子进程停止, 然后以这样的方式启动它, 它将在系统调用的下一入口或在系统调用退出时停止。在第二次调用wait4之后, 多次以PTRACE_PEEKUSER 作为第一个参数调用 ptrace。根据 ptrace 帮助手册, 此参数执行以下操作:
PTRACE_PEEKUSR
Reads a word at offset addr in the child’s USER area, which holds the registers and other information about the process (see <linux/user.h> and <sys/user.h>). The word is returned as the result of the ptrace call. Typically the offset must be word-aligned, though this might vary by architecture. (Data is ignored.)
在子进程用户区域中读取位于偏移地址的字, 其中包含有关该进程的寄存器和其他信息 (请参见<linux ser.h>和<sys ser.h>)。该字作为 ptrace 调用的结果返回。通常, 偏移量必须是字对齐的, 尽管这可能因体系结构而异。(数据被忽略.)
From this information, it appears that strace is reading information from the user area of the child process. In particular, it can be used to get the registers for the process. The registers are used for the arguments to system calls as per the calling conventions for system calls. Notice the second to last system call that writes the strace output to the terminal. The /bin/ls program just called the uname system call, and the strace output printed the information about that system call to the terminal.
从这些信息中, 似乎 strace 正在从子进程的用户区域读取信息。特别是, 它可用于获取进程的寄存器。根据系统调用的调用约定, 寄存器用于系统调用的参数。请注意将 strace 输出写入终端的第二个系统调用。/bin/ls 程序刚刚调用了 uname 系统调用, strace 将该系统调用的信息打印到终端。
The strace output continues:
Code View: Scroll / Show All
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(-1, [WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP], 0x40000000, NULL) = 16474
— SIGCHLD (Child exited) —
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT PIPE TERM], NULL, 8) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*ORIG_EAX, [0x5]) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*EAX, [0xffffffda]) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*EBX, [0xbfffe3e8]) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*ECX, [0]) = 0
ptrace(PTRACE_PEEKUSER, 16474, 4*EDX, [0]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe3e8, [0x7273752f]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe3ec, [0x62696c2f]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe3f0, [0x3836692f]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe3f4, [0x39322f36]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe3f8, [0x62696c2f]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe3fc, [0x732e7472]) = 0
ptrace(PTRACE_PEEKDATA, 16474, 0xbfffe400, [0x312e6f]) = 0
write(2, "open(\"/usr/lib/i686/29/librt.so."..., 44) = 44
This strace output shows the processing of another system call. However, in this snippet of strace output, there are several calls to ptrace with the PTRACE_PEEKDATA value as the first argument. The ptrace man page has the following information on this value:
此 strace 输出显示另一个系统调用的处理。但是, 在 strace 输出的这段代码中, 有几个调用以 PTRACE_PEEKDATA 值作为ptrace的第一个参数。ptrace 帮助手册有以下有关此值的信息:
PTRACE_PEEKTEXT, PTRACE_PEEKDATA
Reads a word at the location addr in the child’s memory, returning the word as the result of the ptrace call. Linux does not have separate text and data address spaces, so the two requests are currently equivalent. (The argument data is ignored.)
在子进程内存中的位置地址读一个字, 将该字作为 ptrace 调用的结果返回。Linux 没有单独的文本和数据地址空间, 因此这两个请求当前是等效的。(参数数据被忽略)。
The strace utility was retrieving information from the process’ address space. The last system call listed provides a clue as to why strace needed to read from the address space. According to what the strace utility was printing to the terminal, the system call that was being processed was open(). The calling convention for a system call uses the registers, but the argument to the open system call is an address in the process’ address space ... the file name that is to be opened. In other words, the register for the first argument of the open system call is the address for the file name, and strace had to read the file name from the process’ address space.
strace 实用程序正在从进程的地址空间中检索信息。列出的最后一个系统调用提供了有关为什么 strace 需要从地址空间读取的线索。根据 strace 实用程序向终端打印的内容, 正在处理的系统调用是open ()。系统调用的调用约定使用寄存器, 但打开的系统调用的参数是进程 "地址空间" 中的地址... 要打开的文件名。换言之, 打开系统调用的第一个参数的寄存器是文件名的地址, strace 必须从进程的地址空间读取文件名。
Now there is still one missing piece of information about how strace works. Remember, we straced the strace utility without the -f switch, which means that we did not follow the forked strace process. For the sake of completeness, let’s see what that reveals:
现在还有一条关于 strace 如何工作的信息丢失了。请记住, 我们 straced 的 strace 实用程序没有-f 开关, 这意味着我们没有遵循复制 strace 过程。为了完整起见, 让我们来看看这揭示了什么:
Code View: Scroll / Show All
ion 220% strace -f strace /bin/ls | & less
...
fork() = 18918
[pid 18914] write(2, "execve(\"/bin/ls\", [\"/bin/ls\"],
["..., 52execve("/bin/ls", ["/bin/ls"], [/* 64 vars */]
) = 0
<unfinished ...>
[pid 18918] ptrace(PTRACE_TRACEME, 0, 0x1, 0 <unfinished ...>
[pid 18914] <... write resumed> ) = 52
[pid 18918] <... ptrace resumed> ) = -1 EPERM (Operation not permitted)
The EPERM error occurs because the kernel only allows one process (at a time) to trace a specific process. Since we traced with the -f switch, both strace commands were trying to strace the /bin/ls process, which caused the EPERM error (the one directly and the other because of the -f switch).
发生 EPERM 错误的原因是内核只允许一个进程 (一次) 跟踪特定进程。由于我们的strace启用了 -f 开关, 两个 strace 命令都试图跟踪/bin/ls 进程, 这导致 EPERM 错误 (一个直接的, 另一个是因为-f 开关)。
The strace utility forks off a process, which immediately tries to call the ptrace system call with PTRACE_TRACEME as the first argument. The man page for ptrace states the following:
strace 实用程序将一个进程复制, 它立即尝试用 PTRACE_TRACEME 作为第一个参数调用 ptrace 系统调用。ptrace 的帮助手册声明如下:
PTRACE_TRACEME
Indicates that this process is to be traced by its parent. Any signal (except SIGKILL) delivered to this process will cause it to stop and its parent to be notified via wait. Also, all subsequent calls to exec by this process will cause a SIGTRAP to be sent to it, giving the parent a chance to gain control before the new program begins execution. A process probably shouldn’t make this request if its parent isn’t expecting to trace it. (pid, addr, and data are ignored.)
指示此进程将由其父进程跟踪。发送到此进程的任何信号 (SIGKILL) 将导致其停止, 并通过wait通知其父进程。此外, 此进程对 exec 的所有后续调用都将导致 SIGTRAP 发送到它, 从而使父进程有机会在新程序开始执行之前获得控制权。如果某个进程的父进程不希望跟踪它, 则可能不应发出此请求。(pid、地址和数据被忽略)
When tracing a process off of the command line, the strace output should contain all the system calls. Without this ptrace feature, the strace utility may or may not capture the initial system calls because the child process would be running unhampered calling system calls at will. With this feature, the child process (which will eventually be /bin/ls in this example) will stop on any system call and wait for the parent process (the strace process) to process the system call.
在从命令行中跟踪进程时, strace 输出应包含所有系统调用。如果没有此 ptrace 功能, strace 实用程序可能会或可能无法捕获初始系统调用, 因为子进程将运行不受阻碍地调用系统调用。使用此功能, 子进程 (此示例中最终为/bin/ls) 将在任何系统调用上停止, 并等待父进程 (strace 进程) 处理系统调用。
