There is a relatively rare pattern of outages where seemingly completely unrelated machines are down at the same time. To deal with the problem of this pattern, we need to find the conditions that can trigger the problem on these machines at the same time.
Often, these machines either had problems at about the same time, or started at a certain point in time, one after the other. For the former case, it is more common that the failure of the physical machine causes all the virtual machines running on it to go down, or a remote management software kills key processes in all the managed systems at the same time; for the latter case In the case, one possible reason is that the user deploys the same problematic module (software, driver) on all instances.
Instances are widely hacked, which is another common reason. When the WannaCry virus is rampant, it often happens that some companies or some departments' machines are all blue-screened. Today I will share with you an example of such a problem.
broken kernel stack
Using the crash tool sys command, we can see some basic information about the system and the direct cause of the downtime. For this issue, the immediate cause of the outage is "Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffffa02987eb". Regarding this message, we must read it word for word. "Kernel panic - not syncing:" This part of the content is output in the kernel function panic. Whenever the panic function is called, this part of the output must be output, so this part of the content is not directly related to the problem. And "stack-protector: Kernel stack is corrupted in:", in the kernel function __stack_chk_fail, this function is a stack check function, it will check the stack, and call the panic function to generate a core dump to report the problem when a problem is found . And the problem it reports is stack corruption. Regarding this function, we will follow up with further analysis.
The address ffffffffa02987eb is the return value of the function __builtin_return_address(0). When the parameter of this function is 0, the output value of this function is the return address of the function that called it. This sentence is a bit confusing now, but it will be clear after analyzing the call stack later.
function call stack
The core of analyzing the downtime problem is to analyze the call stack of panic. The call stack below, at first glance, system_call_fastpath calls __stack_chk_fail, and then __stack_chk_fail calls panic, reporting stack corruption. But a little comparison with similar stacks shows that it's not that simple.
Below is a similar call stack that starts with the system_call_fastpath function. I wonder if you can see the difference between this call stack and the above call stack. In fact, the call stack starting with the system_call_fastpath function indicates that this is the kernel call stack of a system call. The call stack in the figure below represents the user mode process, there is an epoll system call, and then this call enters the kernel mode. The call stack in the above figure is obviously problematic, because even if we search all the documents, we will not find a system call, which corresponds to the kernel __stack_chk_fail function.
Hint: This leads to another problem that needs to be paid attention to when analyzing the core dump, that is, the call stack printed by bt is sometimes wrong.
Raw stack
The so-called call stack is not actually a data structure. The call stack printed with bt is actually reconstructed from the real data structure, the thread (kernel) stack, according to a certain algorithm. This refactoring process is actually a reverse engineering of the function calling process. I believe everyone knows the characteristics of the stack, first in, last out. For function calls and the use of stacks, you can refer to the picture below. As you can see, each function call will allocate a certain amount of space on the stack. When the CPU executes each function call instruction (call), it will push the next instruction of the call instruction on the stack by the way. These "next instructions" are the so-called function return addresses.
这个时候，我们再回头看Panic的直接原因那一部分，函数__builtin_return_address(0)的返回值。这个返回值，其实就是调用__stack_chk_fail的call指令的下一条指令，这条指令属于调用者函数。这条指令地址被记录为ffffffffa02987eb。我们用sym命令查看这个地址临近的函数名，显然这个地址不属于函数system_call_fastpath，也不属于内核任何函数。这也再次验证了，panic调用栈是错误的这个结论。
关于raw stack，我们可以用bt -r命令来查看。因为raw stack往往有几个页面，这里只截图和__stack_chk_fail相关的这一部分内容。
这部分内容，有三个重点数据需要注意，panic调用__crash_kexec函数的返回值，这个值是panic函数的一条指令的地址；__stack_chk_fail调用panic函数的返回值，同样的，它是__stack_chk_fail函数的一条指令的地址；ffffffffa02987eb这个指令地址，属于另外一个未知函数，这个函数调用了__stack_chk_fail。
Syscall number & Syscall table
因为带有system_call_fastpath函数的调用栈，对应着一次系统调用，而panic的调用栈是坏的，所以这个时候我们自然而然会疑问，到底这个调用栈对应的是什么系统调用。在linux操作系统实现中，系统调用被实现为异常。而操作系统通过这次异常，把系统调用相关的参数，通过寄存器传递到内核。在我们使用bt命令打印出调用栈的时候，我们同时会输出，发生在这个调用栈上的异常上下文，也就是保存下来的，异常发生的时候，寄存器的值。对于系统调用（异常），关键的寄存器是RAX，它保存的是系统调用号。我们先找一个正常的调用栈验证一下这个结论。0xe8是十进制的232。
使用crash工具，sys -c命令可以查看内核系统调用表。我们可以看到，232对应的系统调用号，就是epoll。
这个时候我们再回头看“函数调用栈”这节的插图，我们会发现异常上下文中RAX是0。正常情况下这个系统调用号对应read函数，如下图。
如下图，有问题的系统调用表显然是被修改过的。修改系统调用表（system call table）这种事情，常见的有两种代码会做（这个相当辩证），一种是杀毒软件，而另外一种是hack程序。当然还有另外一种情况，就是某个蹩脚的内核驱动，无意识的改写了系统调用表。
另外我们可以看到，被改写过的函数的地址，显然和最初被__stack_chk_fail函数报出来的地址，是非常邻近的。这也可以证明，系统调用确实是走进了错误的read函数，最终踩到了__stack_chk_fail函数。
Raw data
基于上边的数据，来下结论，总归还是有点经验主义。更何况，我们甚至不能区分，问题是由杀毒软件导致的，还是木马导致的。这个时候我们花费了比较多的时间，尝试从core dump里挖掘出ffffffffa02987eb这个地址更多的信息。有一些最基本的尝试，比如尝试找出这个地址对应的内核模块等，但是都无功而返。这个地址既不属于任何内核模块，也不被已知的内核函数所引用。这个时候，我们做了一件事情，就是把这个地址前后连续的，所有已经落实（到物理页面）的页面，用rd命令打印出来，然后看看有没有什么奇怪的字符串可以用来作为特征串定位问题。
就这样，我们在邻近地址发现了下边这些字符串。很明显这些字符串应该是函数名。我们可以看到hack_open和hack_read这两个函数，对应被hacked的0和2号系统调用。还有函数像disable_write_protection等等。这些函数名，显然说明这是一段“不平凡”的代码。公网搜索这些函数会发现很多rootkit的示例。
后记
The core dump analysis of the downtime problem requires us to be very patient. One of my personal experiences is: every bit matters, that is, don't let go of any bit of information. Because of the mechanism itself and some random factors in the generation process, core dump will inevitably have inconsistent data, so many times, a small conclusion needs to be verified from different angles.
