Author: eight strange (Gao) also in database technology experts
Original: https://www.jianshu.com/p/93b69955344b
1. The source of the problem
The problem comes from a friend, as follows: The mysql client cannot log in, and no high load information is found when checking the server load. Viewing thread stack information through pstack, no exception information was found.
2. Problem diagnosis and resolution
Generally speaking, when this happens, we will use pstack to see what function the newly created thread is stuck on, and then it is easy to find the reason. But after this problem, when mysql initiates a connection and gets stuck, use pstack to view the process of the mysqld server and find that there is no thread to interact with it at all, so mysqld suspects that there is something wrong with the listening thread, so the mysql client The connection process performed pstack found as follows:
#0 0x00007f262e7889c0 in __connect_nocancel () from /lib64/libpthread.so.0
#1 0x0000000000435123 in inline_mysql_socket_connect (src_file=0x504490 "../../mysql-8.0.20/vio/viosocket.cc", src_line=1054, len=110, addr=0x7fffb38daa20, mysql_socket=...) at ../../mysql-8.0.20/include/mysql/psi/mysql_socket.h:647
#2 vio_socket_connect (vio=0x110d2c0, addr=addr@entry=0x7fffb38daa20, len=len@entry=110, nonblocking=<optimized out>, timeout=-1, connect_done=connect_done@entry=0x7fffb38da9ef) at ../../mysql-8.0.20/vio/viosocket.cc:1054
#3 0x0000000000422ad6 in csm_begin_connect (ctx=0x7fffb38daad0) at ../../mysql-8.0.20/sql-common/client.cc:250
#4 0x000000000041fe99 in mysql_real_connect (mysql=mysql@entry=0xaf16a0 <mysql>, host=host@entry=0x0, user=user@entry=0x10defc0 "root", passwd=passwd@entry=0x10deff0 "xsh0923", db=db@entry=0x0, port=<optimized out>, unix_socket=0x10e0860 "/data/mysql3306/tmp/mysql.sock", client_flag=66560) at ../../mysql-8.0.20/sql-common/client.cc:5600
#5 0x000000000040ddb6 in sql_real_connect (silent=0, password=0x10deff0 "xsh0923", user=0x10defc0 "root", database=0x0, host=0x0) at ../../mysql-8.0.20/client/mysql.cc:4515
#6 sql_connect (host=0x0, database=0x0, user=0x10defc0 "root", password=0x10deff0 "xsh0923", silent=<optimized out>) at ../../mysql-8.0.20/client/mysql.cc:4670
#7 0x000000000040968b in main () at ../../mysql-8.0.20/client/mysql.cc:1326
#8 0x00007f262ce3e495 in __libc_start_main () from /lib64/libc.so.6
#9 0x000000000040a4e1 in _start () at ../../mysql-8.0.20/include/my_alloc.h:86
In fact, mysql stuck the connect function above as follows:
Therefore, neither the mysql client nor the mysqld server has established a connection. The netstat view is as follows:
unix 2 [ ACC ] STREAM LISTENING 8584712 29676/mysqld /tmp/mysqlx.sock
unix 7 [ ACC ] STREAM LISTENING 8584713 29676/mysqld /data/mysql3306/tmp/mysql.sock
unix 2 [ ] STREAM CONNECTING 0 - /data/mysql3306/tmp/mysql.sock
unix 2 [ ] STREAM CONNECTING 0 - /data/mysql3306/tmp/mysql.sock
unix 2 [ ] STREAM CONNECTING 0 - /data/mysql3306/tmp/mysql.sock
大量connectint状态的连接
Then I did the operation of the listening thread of strace mysqld, and the results obtained are as follows: Under normal circumstances, this should be poll and accept and then start a new (or take one from the cache thread) thread to process the interactive information. But here we can clearly see the SIGSTOP signal. Immediately check the status of the positive mysqld thread as follows:
all threads are in the T state, which is caused by the message SIGSTOP. So we send a SIGCONT signal to the mysqld process as follows:
kill -18 29676
Three, about the signal
Below are some notes when I was learning signals.
In Linux, a signal is a soft interrupt mechanism handled by the kernel. It satisfies the characteristics of simplicity, cannot carry a large amount of information, and must meet certain conditions before sending. The signal will go through the following process:
Generation-->Blocking signal set-->Pending signal set-->Signal delivery-->Signal processing method
First, the signal can be generated in many ways. For example, the kill command we often use is a means of initiating a signal as follows:
[root@mgr4 8277]# kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
63) SIGRTMAX-1 64) SIGRTMAX
Our regular keystrokes can also produce
Ctrl+c 2)SIGINT
Ctrl+\ 3)SIGQUIT
Ctrl+z 4)SIGTSTP
Of course, there are many other triggering methods such as hardware exception, raise function, abort function, alarm function and so on. The second is the blocking signal set. The blocking signal set can shield the signals that you want to exclude (other than the 9/19 signal). If the signal will not reach the delivery state after shielding , it will not be processed. The blocking signal set is set through the sigprocmask function, but the sigset_t set must be set before, and it is set through functions such as sigaddset, sigdelset, sigemptyset, and sigfillset.
The pending signal set cannot be manipulated and can only be obtained through the sigpending function, but it and the blocking signal set can control the delivery of the signal.
After the signal is delivered, the signal needs to be processed. The default behavior is listed above, but the signal (other than the 9/19 signal) can be captured to change its processing method. We can customize the function as a signal processing method, This can be captured and processed through the signal function and the sigaction function. The sigaction function is relatively complex and requires a struct sigaction structure variable, which contains the sa_handler\sa_mask\sa_flags\sa_siaction member. You can check the Linux man page without explaining it here.
The above is the signal processing method in a single process. In multithreading, the processing method is shared between threads, but there can be different signal shielding sets. In multithreading, a unified signal shielding word and signal processing method are generally used to set the pthread_mask function. Inherit to each thread, and use functions such as sigwait/sigwaitinfo to set up a separate signal processing thread for unified processing. This is how MySQL handles it. MySQL actually has a signal processing thread,
| 36 | 1927 | sql/signal_handler | NULL | BACKGROUND | NULL | NULL |
If you need to understand, you can refer to the following article: Processing of signals in MySQL (SIGTERM, SIGQUIT, SIGHUP, etc.), http://blog.itpub.net/7728585/viewspace-2142060/
Four, pstack and SIGSTOP
Here is a brief mention that pstack will trigger a SIGSTOP signal when it grabs the thread stack, because pstack actually calls the following commands of gdb:
/usr/bin/gdb --quiet -nx /proc/8277/exe 8277
thread apply all bt
输出所有线程信息
gdb usually initiates the SIGSTOP signal to stop the process. Therefore, we must be careful when executing the pstack command online. pstack may cause your MySQL to stop running for a while, especially when there are many high-pressure threads with high load. It may take a long time, pstack should be used as the last resort for the diagnosis of important online databases .
As follows, we use pstack to get the mysqld stack, and close the window at the beginning of the operation. At this time, our mysqld is already in a stopped state as follows:
Therefore, you must be very careful when using pstack. Generally speaking, if there are not many library threads, the pressure is not important and the importance is not high. You can use pstack for information collection. Of course, you can also manually initiate the SIGSTOP signal to achieve the test effect.
Five, capture all the signals
The following small code can capture all signals for testing:
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include "unistd.h"
void handler(int sig)
{
printf("get signal %d\n",sig);
}
void test()
{
;
}
int main(void)
{
struct sigaction s;
int* a=NULL;
int i=0;
a=(int*)calloc(32,sizeof(int));
for(i=0;i<32;i++)
{
*(a+i) = (i+1);
}
s.sa_handler=handler;
for(i=0;i<32;i++)
{
sigaction(*(a+i),&s,NULL);
}
while(1)
{
test();
sleep(1);
}
return 0;
}
The full text is over.
Enjoy MySQL :)
Scan code to add author WeChat
Teacher Ye's "MySQL Core Optimization" class has been upgraded to MySQL 8.0, scan the code to start the journey of MySQL 8.0 practice