I. Introduction
First of all, I wish you all a happy Double Festival, peace and joy! It’s been a long time since I wrote an article, mainly because I’m still settling down. I always feel that sharing for learning is for sharing. I may not remember the details for a few days, so I haven’t written any more. This time I met a more interesting one. Bug, a bug in multiplexing, in this field, although I have studied it myself, I have never written code exercises. I will practice it together at this opportunity. It may be a common problem for experts. But it took us a day or so to troubleshoot the problem.
2 Problem Description and Troubleshooting Steps
We have a system developed in c that has been running for a long time. During the test of the new version, we found that there is always a core, and the position of the core is erratic, and the core is a bit inexplicable. The core should not be a core at all. From the beginning, it can be seen from the phenomenon It seems to be a problem caused by multi-threading, but after checking it, I didn't find the problem.
Due to the large amount of code, our troubleshooting steps are:
Using the ascan library to locate the location of the core, we start to turn off related functions according to the location of the core.
After reducing the number of cores, the core will still be used next. The location of the core is in the creation of a unix socket communication thread. This thread should have been created a long time ago, but why does it start to be created after running for 5-10 minutes? There is no thread creation. Do the monitoring of the parent-child process, so there is no possibility of restarting and if the thread hangs up, it is impossible to recreate it, because the thread hangs up, it will inevitably cause the process to hang up, and as a result, other threads in the whole process are still normal In progress. (This is still unsolved)
Because it is a thread creation problem, my colleague noticed that the new function of writing kafka in this process caused too many threads, so I commented out these functions in the code and continued to investigate.
Since this thread is mainly used to execute some program interaction commands, I used the client tool to connect to the test, and found that it often failed to connect, and sometimes the core and ascan would report an error message:
ASAN:SIGSEGV
=================================================================
==316088== ERROR: AddressSanitizer: SEGV on unknown address 0x00000366650b (pc 0x00000366650b sp 0x7f6e7db81fa0 bp 0x7f6e7db82820 T236)
AddressSanitizer can not provide additional info.
#0 0x366650a (+0xbd650a)From the error message, use the add2line command to find the specific stack. This command has been discussed in previous articles, and it is executed as follows:
addr2line -a -C -e bin/可执行程序 pc对应的地址If after debugging with gdb, it is found that the core is on the return of the unix socket processing function, that is to say, the stack information has been destroyed.
When connecting normally, the client process is stuck, and the system calls of the client are tracked through strace, as follows:
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="./run/xxxx.socket"}, 33) = 0
ioctl(3, FIONBIO, [1]) = 0
poll([{fd=3, events=POLLOUT}], 1, 10000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "{\"version\": \"0.2\"}", 18, 0, NULL, 0) = 18
select(4, [3], [], [], {tv_sec=600, tv_usec=0}) = 1 (in [3], left {tv_sec=599, tv_usec=999794})
poll([{fd=3, events=POLLIN}], 1, 10000) = 1 ([{fd=3, revents=POLLIN}])
recvfrom(3, "{\"return\":\"OK\"}\n", 1024, 0, NULL, NULL) = 16
poll([{fd=3, events=POLLOUT}], 1, 10000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "{\"command\": \"command-list\"}\n", 28, 0, NULL, 0) = 28
select(4, [3], [], [], {tv_sec=600, tv_usec=0}Print information through the log of the client. After sending the command-list command, the server does not return. The sendto command is successful and returns 28. Let’s see what the server says:
[23331] 9/9/2022 -- 22:01:47 - (xxx.c:403) <Info> (xx) -- Unix socket: recv msg: {"version": "0.2"}
[23331] 9/9/2022 -- 22:01:47 - (xxx.c:449) <Info> (xx) -- Unix socket: send to client:(null)
[23331] 9/9/2022 -- 22:01:47 - (xxx.c:343) <Info> (xx) -- Unix socket: send content:{"return":"OK"}
[23331] 9/9/2022 -- 22:01:47 - (xxx.c:345) <Info> (xx) -- Unix socket:sent message of size 16 to client socket 1118According to the log on the server side, only the initial version information was received, and the subsequent command-list commands were not received. This is very strange.
Puzzled, is it a kernel bug? No problem was found through gdb debugging, and then the connection number of the socket file was checked through lsof. When we connected through the client, the number of connections increased, which is no problem, as shown in the figure below:
[root@localhost xx]# lsof ./run/command.socket
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
xxx 30894 root 101u unix 0xffff8810e7e99800 0t0 300947 ./run/command.socket
xxx 30894 root 1172u unix 0xffff8802b42a4000 0t0 1446065 ./run/command.socketI didn’t notice this 1172 at the beginning. What’s special about this file descriptor? I also know that when select is used for multiplexing, it has certain limitations. It can only handle 1024 connections. I was thinking that we only have one connection. Exceeding the limit of 1024, maybe some friends know the reason, that is, 1172 exceeds 1024, that is to say, the number of selected FDs cannot exceed 1024, and the size cannot exceed, so it is so simple, continue to practice.
The behavior of these macros is undefined if a descriptor value is less than zero or
greater than or equal to FD_SETSIZE, which is normally at least equal to the maximum num-
ber of descriptors supported by the system.Three multiplexing
On high-performance servers, multiplexing technology is often used. Multiplexing is actually multiple connections. Multiplexing means multiplexing the server process. So why multiplexing together means using one process to process multiple connections. .
For the server, the open port is waiting for the client to connect, and the method of multi-process or multi-thread programming is used more often, that is, each connection is processed by a separate process or thread, but each computer can be opened due to resource limitations such as memory. The number of processes or threads is limited, and too many threads will lead to a series of problems such as excessive cost of thread switching and cache failure. It is impossible to handle 100,000 or millions of connections on a single machine.
If non-blocking is used, what about polling in the user process? This will take up a lot of cpu resources, so multiplexing technology was developed later, that is, using one process to handle multiple connections, how to handle multiple connections with one reference, it is impossible to use the blocking method, once blocked in a connection On the IO, there is no way to deal with events coming from other connections. You can only poll to see if there are readable and writable messages on each connection, so as to achieve the purpose of multiplexing. The Linux kernel provides select, poll, and epoll. A multiplexing mechanism.
3.1 select mechanism for multiplexing
3.1.1 Basic instructions
/* According to POSIX.1-2001 */
#include <sys/select.h>
int select(int nfds, fd_set *readfds, fd_set *writefds,
fd_set *exceptfds, struct timeval *timeout);
void FD_CLR(int fd, fd_set *set); // 从fdset中删除fd
int FD_ISSET(int fd, fd_set *set); // 判断fd是否已存在fdset
void FD_SET(int fd, fd_set *set); // 将fd添加到fdset
void FD_ZERO(fd_set *set); // fdset所有位清01. nfds means the maximum descriptor to be tested + 1 among the monitored file descriptors. 2. readfds: monitors the set of file descriptors that have read data arriving. 3. writefds: Monitor the collection of file descriptors with write data arriving. 4. exceptfds: Monitor the collection of file descriptors with exceptions. These three collections must be passed in every time, and will be copied every time the event to be monitored occurs. 5. If timeout is set to NULL, select will block until the event occurs; if it is not NULL and the value is not 0, it will wait for a fixed time, and if there is no monitoring event for this event, it will still return; if it is not NULL, And the value is 0, then return immediately without waiting.
The following four are macros, the meaning of which is as the note below, and the implementation in the Linux kernel is as follows (the implementation of different versions is slightly different):
#define __NFDBITS (8 * sizeof(unsigned long)) // 每个ulong型可以表示多少个bit,
#define __FD_SETSIZE 1024 // socket最大取值为1024
#define __FDSET_LONGS (__FD_SETSIZE/__NFDBITS) // bitmap一共有1024个bit，共需要多少个ulong
#define __FD_ELT(d) ((d) / __NFDBITS)
#define __FD_MASK(d) ((__fd_mask) 1 << ((d) % __NFDBITS))
typedef struct {
unsigned long fds_bits [__FDSET_LONGS]; //用ulong数组来表示bitmap
} __kernel_fd_set;
typedef __kernel_fd_set fd_set;
#define __FD_SET(d, set) \
((void) (__FDS_BITS (set)[__FD_ELT (d)] |= __FD_MASK (d)))
#define __FD_CLR(d, set) \
((void) (__FDS_BITS (set)[__FD_ELT (d)] &= ~__FD_MASK (d)))
#define __FD_ISSET(d, set) \
((__FDS_BITS (set)[__FD_ELT (d)] & __FD_MASK (d)) != 0)fd_set is a bitmap composed of unsigned long type groups. The FD_SET operation is to find which bit of unsigned long, and use ((__fd_mask) 1 << ((d) % __NFDBITS)) to locate the specific bit information. Bit is set to 1, negated is set to 0.
The problem lies in FD_SET, that is, in __FD_ELT(d) ((d) / __NFDBITS) If the value of d is greater than 1024, then fds_bits is out of bounds, and the stack data will be destroyed, resulting in an exception being returned.
Simplify our program and write it as follows:
#服务器端
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/un.h>
#include <unistd.h>
#define CLIENT_SIZE 100
#define SOCK_FILE "command.socket"
#define TOO_MANY "Too many client."
typedef struct unix_socket_infos_ {
int socket;
int select_max;
struct sockaddr_un client_addr;
int clients[CLIENT_SIZE];
} unix_socket_infos_t;
static int create_unix_socket(unix_socket_infos_t *this) {
struct sockaddr_un addr;
addr.sun_family = AF_UNIX;
strncpy(addr.sun_path, SOCK_FILE, sizeof(addr.sun_path));
addr.sun_path[sizeof(addr.sun_path) - 1] = 0;
int len = strlen(addr.sun_path) + sizeof(addr.sun_family) + 1;
int listen_socket = socket(AF_UNIX, SOCK_STREAM, 0);
if (listen_socket == -1) {
perror("create socket error.\n");
return -1;
}
// fcntl (socket, F_SETFL,SOCK_NONBLOCK) ;
int on = 1;
/* set reuse option */
int ret = setsockopt(listen_socket, SOL_SOCKET, SO_REUSEADDR, (char *)&on,
sizeof(on));
unlink(SOCK_FILE);
/* bind socket */
ret = bind(listen_socket, (struct sockaddr *)&addr, len);
if (ret == -1) {
perror("bind error.\n");
return -1;
}
printf("start to listen\n");
ret = listen(listen_socket, 1);
if (ret == -1) {
perror("listen error\n");
return -1;
}
ret = chmod(SOCK_FILE, S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP);
if (ret == -1) {
perror("chmod error\n");
return -1;
}
this->socket = listen_socket;
this->select_max = listen_socket;
return 1;
}
static int set_max(unix_socket_infos_t *this) {
for (int i = 0; i < CLIENT_SIZE; i++) {
if (this->clients[i] >= this->select_max) {
this->select_max = this->clients[i];
}
if (this->clients[i] < 0) {
break;
}
}
fprintf(stderr, "max is:%d\n", this->select_max);
return 0;
}
static int close_client(unix_socket_infos_t *this, int index) {
int client = this->clients[index];
close(client);
this->clients[index] = -1;
set_max(this);
}
static int deal_client(unix_socket_infos_t *this, int index) {
char buffer[1024] = {0};
int ret = recv(this->clients[index], buffer, sizeof(buffer) - 1, 0);
if (ret <= 0) {
if (ret == 0) {
printf("lost connect.\n");
} else {
printf("recv error:%s \n", strerror(errno));
}
close_client(this, index);
return -1;
}
if (ret < sizeof(buffer)-2) {
buffer[ret] = '\n';
buffer[ret+1] = 0;
}
fprintf(stderr, "client[%d]:%s", this->clients[index],buffer);
ret = send(this->clients[index], buffer, strlen(buffer), MSG_NOSIGNAL);
if (ret < 0) {
perror("send error:");
} else {
fprintf(stderr, "server:%s", buffer);
}
return 1;
}
static int accept_client(unix_socket_infos_t *this) {
socklen_t len = sizeof(this->client_addr);
char buffer[1024] = {0};
int client = accept(this->socket, (struct sockaddr *)&(this->client_addr), &len);
printf("client to comming:%d\n", client);
if (client < 0) {
perror("accept error\n");
return -1;
}
memset(buffer,0x0,1024);
int ret = recv(client, buffer, sizeof(buffer) - 1, 0);
if (ret < 0) {
perror("recv error\n");
return -1;
}
if (ret < sizeof(buffer)-2) {
buffer[ret] = '\n';
buffer[ret+1] = 0;
}
fprintf(stderr, "client[%d][first]:%s",client,buffer);
ret = send(client, buffer, strlen(buffer), MSG_NOSIGNAL);
if (ret < 0) {
perror("send error\n");
} else {
fprintf(stderr, "server[first]:%s", buffer);
}
int is_set = 0;
for (int i = 0; i < CLIENT_SIZE; i++) {
if (this->clients[i] < 0) {
this->clients[i] = client;
is_set = 1;
break;
}
}
set_max(this);
if (is_set == 0) {
fputs(TOO_MANY, stdout);
close(client);
return -1;
}
return 1;
}
static int run_select(unix_socket_infos_t *this) {
struct timeval tv;
int ret;
fd_set select_set;
FD_ZERO(&select_set);
FD_SET(this->socket, &select_set);
for (int i = 0; i < CLIENT_SIZE; i++) {
if (this->clients[i] > 0) {
FD_SET(this->clients[i], &select_set);
} else {
break;
}
}
tv.tv_sec = 0;
tv.tv_usec = 200 * 1000;
int select_max = this->select_max + 1;
ret = select(select_max, &select_set, NULL, NULL, &tv);
if (ret == -1) {
if (errno == EINTR) {
return 1;
}
return -1;
}
if (ret == 0) {
return 1;
}
if (FD_ISSET(this->socket, &select_set)) {
accept_client(this);
}
for (int i = 0; i < CLIENT_SIZE; i++) {
if (this->clients[i] <= 0) {
break;
}
if (FD_ISSET(this->clients[i], &select_set)) {
deal_client(this, i);
}
}
return 1;
}
int main(int argc, char **argv) {
unix_socket_infos_t unix_socket_infos;
for (int i = 0; i < CLIENT_SIZE; i++) {
unix_socket_infos.clients[i] = -1;
}
int ret = create_unix_socket(&unix_socket_infos);
printf("start to loop\n");
while (1) {
int run_ret = run_select(&unix_socket_infos);
if (run_ret == -1) {
break;
}
}
return 0;
}Client connection code:
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/un.h>
#include <unistd.h>
#include <string.h>
#define SOCK_FILE "command.socket"
int main(int argc, char **argv) {
struct sockaddr_un un;
int sock_fd;
char buffer[1024] = "hello unix socket server";
char recv_buffer[1024];
un.sun_family = AF_UNIX;
strcpy(un.sun_path, SOCK_FILE);
sock_fd = socket(AF_UNIX, SOCK_STREAM, 0);
if (sock_fd < 0) {
perror("socket error.\n");
return -1;
}
if (connect(sock_fd, (struct sockaddr *)&un, sizeof(un)) < 0) {
perror("connect error.\n");
return -1;
}
while (1) {
memset(recv_buffer,0,1024);
memset(buffer,0,1024);
fprintf(stderr,"\nmy[%d]:",sock_fd);
fgets(buffer,sizeof(buffer)-1,stdin);
if (strncmp(buffer, "quit", 4) == 0) {
break;
}
int ret = send(sock_fd, buffer, strlen(buffer) - 1, 0);
if (ret == -1) {
perror("send error.\n");
} else {
ret = recv(sock_fd, recv_buffer, sizeof(recv_buffer) - 1, 0);
if (ret <= 0) {
perror("recv error.\n");
}
recv_buffer[ret - 1] = 0;
fprintf(stderr,"server:%s",recv_buffer);
}
}
close(sock_fd);
return 0;
}Practice the code, the writing is quite frustrating, the client connects to the server through the unix socket, then receives user input and sends it to the server, and the server sends back a message until the user enters quit to exit. Suggest the following effect:
root@ubuntu-lab:/home/miao/c-test/select# ./a.out
start to listen
start to loop
client to comming:4
client[4][first]:123
server[first]:123
max is:4
client[4]:456
server:456
client[4]:abc
server:abc
lost connect.
max is:4
.....The client shows:
root@ubuntu-lab:/home/miao/c-test/select# ./client
my[3]:123
server:123
my[3]:456
server:456
my[3]:abc
server:abc
my[3]:quit3.1.2 core simulation
Add the following code at the beginning of main:
int files[1800] = {0};
char fileName[256] = {0};
for (int i = 0; i < 1800; i++) {
memset(fileName, 0x0, sizeof(fileName));
sprintf(fileName, "test_%d", i);
files[i] = open(fileName, O_RDWR | O_CREAT);
if (files[i] < 0) {
close(files[i]);
}
}You will find that the program will automatically exit or core, and occasionally succeed, and in other cases, the command sent is not replied, that is, it is not monitored.
3.2 Disadvantages of select
Although select also supports IO multiplexing, there are the following problems:
After each select returns, the monitored collection needs to be reset, which is troublesome.
The limit is 1024 connections. If you want to break through connections in the application, you can use malloc and other dynamic memory application methods, but it is best to use poll or epoll.
The monitored file descriptor must be copied to the kernel space every time. When an event occurs, it needs to be copied from the kernel space to the user space, which consumes more CPU resources. The performance comparison of several mechanisms is as follows
To be continued.....