Original Park Young Min Linux Reading Field2019-12-18
Introduction to this article :
Kernel deadlock problems are generally caused by read-write locks (rw_semaphore) and mutex locks (mutex). This article mainly talks about how to analyze such deadlock problems through the ramdump+crash tool.
About the author :
Piao Yingmin, currently works in a domestic mobile phone research and development company, as a senior system engineer, mainly responsible for the debugging of the Android system.
0, background knowledge points
Ramdump is a memory dump mechanism. We can dump the system memory to a file at a certain moment, and then import it together with symbol information (vmlinux) into memory analysis tools such as trace32 or crash for offline analysis. It is an important debugging method for analyzing kernel problems such as crashes, deadlocks, and memory leaks.
Crash is an open source tool ( http://people.redhat.com/anderson/) for analyzing ramdump . It is a command-line interactive mode and provides many powerful debugging commands. It is a powerful tool for analyzing and locating complex kernel problems .
Deadlock refers to a phenomenon of blockage caused by competition for lock resources during the execution of two or more execution streams. As shown:
1. Problem description
The interface stuck when running monkey in Android7.1 system:
1) Without any refresh, all input events are invalid, including the power button
2) Watchdog does not restart system_server
3) Adb can be connected, but debugging commands such as ps are stuck
2. Preliminary analysis
Since it is not possible to debug directly with adb, long press the power button to enter the dump mode and export the ramdump file, and then use the crash tool to load the randump file to start offline analysis.
Generally, it may be that the core thread is in the UNINTERRUPTIBLE state when it is stuck, so first use the ps command in the crash environment to view the threads in the UNINTERRUPTIBLE state of the phone. The parameter -u can filter out the kernel threads:
The bt command can view the call stack of a thread. Let’s take a look at the most critical watchdog thread in the UN state above: 
You can see from the call stack that the proc_pid_cmdline_read() function is blocked. The corresponding code is: 
here is to get the The mmap_sem lock of a thread mm, and this lock is held by another thread.
3. Derivation of read-write lock
To know which thread holds the lock, we must first use assembly to deduce the specific value of the lock. You can use the dis command to look at the assembly code of proc_pid_cmdline_read(): 
0xffffff99a680aaa0 is where down_read() is called, and its first parameter x0 is the sem lock, such as: 
x0 and x28 registers store the value of sem, so x21 is naturally The address of mm_struct, because the offset of the mmap_sem member of mm_struct is 104 (0x68), and the declaration of the structure can be viewed with the whatis command, such as: 
so we only need to know x21 or x28 to know the value of mm and mmap_sem lock.
When the function is called, the called function will save the register to be modified in its own stack frame, so we can find these two registers in down_read() and the function calls after it: In
other words, in the following functions, As long as x21 or x28 is found, these registers will be saved in its stack frame. 
Start looking for down_read() at the bottom:
Obviously it does not use x21 or x28, continue to look at the assembly code of rwsem_down_read_failed(): 
find x21 in this function, and it is stored at the offset 32 bytes of the rwsem_down_read_failed stack frame.
The sp of rwsem_down_read_failed() is 0xffffffd6d9e4bcb0 
sp + 32 =0xffffffd6d9e4bcd0, use the rd command to view the value of x21 stored in the address 0xffffffd6d9e4bcd0: 
use the struct command to view this mm_struct: the address of the sownertask 
is the mm_struct of the thread of mm_ffd for 0x349ff: 
+0x68= 0xffffffd76e349a68, so: 
at this point, we know that the watchdog thread is blocked when reading the proc node of the 1651 thread. The reason is that the mmap_sem lock of this process is held by other threads. What is it? Who holds this lock?
4. Thread holding read-write lock
With the question, let’s continue the analysis. First, traverse the wait_list through the list command to see how many threads are waiting for this read-write lock: 
From the above output, we can see that there are 2 writers and 17 readers waiting, 19 All threads are in the UNINTERRUPTIBLE state.
Let's review all the threads in the UNINTERRUPTIBLE state in the current system: 
19 threads except for the 5 threads marked in red are the threads waiting for the read-write lock mentioned above. When the lock-holding thread is the writer, we can find the lock-holding thread through the owner of the rw_semaphore structure. Unfortunately, here the owner is 0, which means that the lock holder is the reader thread, so we cannot find the lock thread through the owner. In this case, the current lock can be searched from the stack space of all threads in the system through the search command and the -t parameter: the 
general lock value will be stored in the register, and the register will be stored in the stack during the sub-function call . So as long as the current lock value (0xffffffd76e349a68) is found in the stack space, then this thread is likely to hold the lock or wait for the lock
thread. 19 of the 20 threads found here are the aforementioned wait for lock threads, and the remaining 1 One is likely to be the lock-holding thread: 
check the call stack of this thread: 
Since the address of the lock stored in the 2124 thread is 0xffffffd6d396b8b0, this is within the stack frame of handle_mm_fault(), so it can be inferred that the lock-holding function should be in handle_mm_fault ()prior to.
Let's take a look at the do_page_fault function first: 
there is indeed a place in the code that holds mmap_sem, and it is the reader, so it can be determined that the read-write lock held by 2124 is blocking 19 threads including watchdog.
Next, we need to look at why the 2124 thread will hold the lock and not release it, but before that, let's look at the reasons for the blocking of several threads in the UNINTERRUPTIBLE state of system_server.
5. Other blocked threads (derivation of mutex lock)
First look at the ActivityManager thread: through the call stack you can see that it was suspended at binder_alloc_new_buf, we have to find out the address of this lock first.
First start with the mutex_lock() function:
From its declaration, you can see that its parameter is only one, which is the mutex structure pointer.
Look at the implementation of mutex_lock function:
The first parameter x0 of mutex lock is the struct mutex we are looking for. It is stored in the x19 register at 0xffffff99a74e1648, and then _mutex_lock slowpath() is called at 0xffffff99a74e1664 , so we can find x19 in _mutex_lock_slowpath(): 
because of _ The sp of _mutex_lock_slowpath() is 0xffffffd75ca379a0: 
Therefore, the value of x19 is stored at 0xffffffd75ca379a0+ 16 = 0xffffffd75ca379b0 
The mutex we are looking for is 0xffffffd6dfa02200: the 
owner is the task_struct pointer of the thread that holds the institute. Its pid is:
View the call stack of this thread:
This 3337 thread is one of the 19 threads locked by the read-write lock mentioned earlier.
In the same way, you can find that the audioserver's 1643 thread, and the system_server's 1909 and 2650 threads are also blocked by the mutex lock held by this 3337 thread.
In summary:
1) A total of 4 threads are waiting for the same mutex lock, and the lock is 3337 threads.
2) 19 threads including 3337 are waiting for the same read-write lock, and the lock is 2124 threads.
In other words, most threads are directly or indirectly blocked by 2124 threads.
6. Deadlock
The last thread in the UNINTERRUPTIBLE state is the 2767 (sdcard) thread: it 
can be seen that the 2124 thread is waiting for the processing result of the fuse, and we know that the request of the fuse is processed by the sdcard.
It is easy to think that the suspension of 2124 may be related to the 2767 (sdcard) thread, but the 2124 thread is making read requests, and the 2767 thread is suspended while processing the open request.
That is to say, the sdcard thread is not processing the request of the 2124 thread, but even in this case, the sdcard thread can still block the 2124 thread. Because for an APP process, there will only be a specific sdcard thread to serve it, if the same process multi-threaded sdcard access request, the sdcard thread will be processed serially.
If the previous request is not processed, all subsequent requests will be blocked. The same as the derivation method of the mutex lock before, the mutex lock that the 2767 thread is waiting for is 0xffffffd6948f4090, and
its owner's task and pid are: 
first use the bt command to find the stack range of 2124 for 0xffffffd6d396b4b0～0xffffffd6d396be70: 
mutex can be found from the stack: 
mutex The value is found at the address ffffffd6d396bc40, which is in the stack frame of _generic_file_write_iter. 
It is certain that the lock was held before _ _generic_file_write_iter, and it is probably in ext4_file_write_iter. Check the source code:
Now it becomes clear that 2124 is waiting for 2767 to process the fuse request, and 2767 is locked by the mutex lock held by the 2124 thread, which means that the two threads are interlocked.
This article is only limited to introducing how to locate the deadlock problem. As for how to solve the specific implementation involving the module, I will not repeat it here due to the space.
(Finish)