It's all dry goods~
Article Directory
foreword
In the previous article, we implemented a Tcp server, but in order to demonstrate the effect of multi-process and multi-thread, we wrote the communication between the server and the client as an infinite loop, so that as long as one user has not communicated with the server, other users cannot communicate with the server.
1. Tcp server multi-process version
The serviceIO interface we wrote in the previous article is an infinite loop, so multiple users cannot communicate, so how to use multi-process to solve this problem? In fact, it is very simple, we only need to create a child process, because the child process will inherit the file descriptor of the parent process, so they must be able to point to the same sock file, and then let the child process call serviceIO, our parent process will block and wait for the child process.
pid_t id = fork();
if (id == 0)
{
// 子进程
// 子进程不需要listensock,既然不需要我们就关闭
close(_listensock);
serviceID(sock);
close(sock);
exit(0);
}
// 父进程
pid_t ret = waitpid(id, nullptr, 0);
if (ret > 0)
{
cout << "waitsuccess: " << ret << endl;
}First of all, our child process will also inherit the listensock file descriptor of the parent process, but this file descriptor is used by the parent process to monitor. The work of our child process is only to communicate with the client, so we should close unused file descriptors. This is like an old driver, who checks whether the tires and other equipment are in good condition every time before going out. When the client exits, our child process closes the sock file descriptor, and then exits the child process, because we let the parent process wait for the child process, so there is no need to be afraid of the child process exiting and becoming an orphan process, but such code always feels wrong, is there any problem? That's right! Once we let the parent process wait for the child process, isn't this code still serial? If we want to achieve multi-user communication, the parent process must create multiple child processes. Because start is an infinite loop, the parent process will create a child process to communicate with the user every time. If we let the parent process wait for the child process, it will be the same as before. Only when the child process finishes processing the communication of one user, another user can communicate with the server. So how to solve this problem? Look at the code below:
pid_t id = fork();
if (id == 0)
{
// 子进程
// 子进程不需要listensock,既然不需要我们就关闭
close(_listensock);
// 子进程创建孙进程，如果成功将子进程关闭让孙进程处理任务，由于孙进程的父进程退出所以变成孤儿进程最终会被
// 操作系统领养，不需要进程等待
if (fork() > 0)
{
exit(0);
}
serviceID(sock);
close(sock);
exit(0);
}
// 父进程
pid_t ret = waitpid(id, nullptr, 0);
if (ret > 0)
{
cout << "waitsuccess: " << ret << endl;
}After we let the child process close the unused file descriptor, we immediately let the child process create another child process. Once the creation is successful, we let the original child process exit, and let the child process of the original child process execute the code that communicates with the client. The advantage of this is that we don’t have to wait for the child process of the original child process. Because the child process of the original child process has exited, this process becomes an orphan process. We all know that once a process becomes an orphan process, it will be adopted by the operating system, so we don’t have to worry about the exit of this orphan process. Only when a certain client exits, the grandson process will exit and be adopted by the operating system. This will solve our problem just now, so let’s run it and see:
It can be seen that there is no problem, no matter which client exits this time, it will not affect other users. The printing of our log above has been modified, we will talk about it later. We can see that the file descriptors are 4 and 5, now let's restart to see if the file descriptors are closed correctly:
It can be seen that the file descriptors are still 4 and 5 after we restart, which means that we closed the file descriptors correctly before. If it starts from other numbers, then the previous file descriptors must have leaked.
Of course, it is definitely not good for us to frequently create child processes above, so we have a second method: signal ignore version:
void start()
{
//忽略17号信号
signal(SIGCHLD,SIG_IGN);
for (;;)
{
//4.server获取新链接 未来真正使用的是accept返回的文件描述符
struct sockaddr_in peer;
socklen_t len = sizeof(peer);
// sock是和client通信的fd
int sock = accept(_listensock,(struct sockaddr*)&peer,&len);
//accept失败也无所谓，继续让accept去获取新链接
if (sock<0)
{
logMessage(ERROR,"accept error,next");
continue;
}
logMessage(NORMAL,"accept a new link success");
cout<<"sock: "<<sock<<endl;
pid_t id = fork();
if (id==0)
{
close(_listensock);
serviceID(sock);
close(sock);
exit(0);
}
//父进程
//已经对17号信号做忽略，父进程不用等待子进程，子进程会自动退出，但是需要父进程关闭文件描述符
close(sock);
}
}First of all, we ignore the No. 17 signal SIGCHLD. What is the No. 17 signal? When our child process exits, we will send a signal No. 17 to the parent process to tell the parent process that it is about to exit, and if we ignore this signal, the parent process will not wait for the child process, and then we let the child process close unnecessary file descriptors to execute the code that communicates with the client. If the client does not exit, the child process will not close the file descriptor, so will the parent process close the file descriptor not affect the child process? Actually not, it’s like reference counting, both the child process and the parent process point to a file descriptor, then the reference count of this file descriptor is 2, and only when the reference count is reduced to 0 will it be really closed, so if we don’t close the file descriptor by the parent process, then the child process will only reduce the reference count to 1, and the file descriptor will never be closed, causing the file descriptor to leak, so the parent process needs to close the file descriptor.
We can see that the file descriptors are all 4. This is because our cpu’s operation speed is too fast. We just applied for the No. 4 file descriptor in the accept interface, and then created a child process (the child process inherits the No. 4 descriptor), and then the parent process directly closed the sock.
Two, Tcp server multi-threaded version
Since the workload of creating a process is very large, we use multithreading to provide services to users. The principle of multi-threading is the same as that of multi-process. We only need to create a thread and let this new thread execute the code to communicate with the client, but the file descriptor and serviceIO method must be used to communicate with the client, and our serviceIO is a member function. The callback function of our multi-threaded execution must be static, so we can write a class, which stores the this pointer and sock, and then write a static member function, and call the callback method in the function :
class TcpServer;
struct ThreadData
{
ThreadData(TcpServer* self,int sock)
:_self(self)
,_sock(sock)
{
}
TcpServer* _self;
int _sock;
};
static void* threadRoutine(void *args)
{
pthread_detach(pthread_self());
ThreadData *td = static_cast<ThreadData*>(args);
td->_self->serviceID(td->_sock);
close(td->_sock);
delete td;
return nullptr;
}
As can be seen in the figure above, first we created a thread, and then implemented a struct ThreadData class. The members in the class include pointers to Tcpserver and sock file descriptors. Then we created a pointer to ThreadData, initialized with this and sock, and passed this pointer to the callback function when creating a thread. Entering the callback function, the thread is first separated. Just close the file descriptor, and then release the pointer to null. Why does multi-threading not need to let the main thread close the file descriptor here? Because all threads will share the file descriptor of the process, unlike multi-process, the child process will point to the file descriptor of the parent process. Once pointed to, the reference count of the file descriptor will be +1, but the file descriptor seen by the multi-thread is the one opened in the process, so we let the thread close the file descriptor, which means that the file descriptor in the process is also closed. Let's run it below (remember to add the -pthread option to the server in the makefile before running, otherwise it will not compile):
Let's verify that the file descriptor is closed correctly and reopen the server:
After clearing the screen, we reopened and found that the file descriptor still starts from 4, which means that our file descriptor has not been leaked.
3. Tcp server thread pool version
Remember the thread pool we wrote before, the advantage of the thread pool is that it can create multiple threads at a time and perform tasks, but our task today is to communicate with the client, so we take the previous code of the thread pool:
#include <pthread.h>
#include <iostream>
#include <vector>
#include <queue>
#include <unistd.h>
#include <mutex>
#include "lockguard.hpp"
#include "log.hpp"
using namespace std;
const int gnum = 5;
template <class T>
class ThreadPool
{
public:
static ThreadPool<T>* getInstance()
{
if (_tp == nullptr)
{
_mtx.lock();
if (_tp == nullptr)
{
_tp = new ThreadPool<T>();
}
_mtx.unlock();
}
return _tp;
}
static void* handerTask(void* args)
{
ThreadPool<T>* threadpool = static_cast<ThreadPool<T>*>(args);
while (true)
{
T t;
{
// threadpool->lockQueue();
LockGuard lock(threadpool->getMutex());
while (threadpool->IsQueueEmpty())
{
threadpool->condwaitQueue();
}
// 获取任务队列中的任务
t = threadpool->popQueue();
}
t();
}
return nullptr;
}
void Push(const T& in)
{
LockGuard lock(&_mutex);
//pthread_mutex_lock(&_mutex);
_task_queue.push(in);
pthread_cond_signal(&_cond);
//pthread_mutex_unlock(&_mutex);
}
void start()
{
for (const auto& t: _threads)
{
pthread_create(t,nullptr,handerTask,this);
logMessage(DEBUG,"线程%p创建成功",t);
}
}
~ThreadPool()
{
pthread_mutex_destroy(&_mutex);
pthread_cond_destroy(&_cond);
for (auto& t: _threads)
{
delete t;
}
}
public:
void lockQueue()
{
pthread_mutex_lock(&_mutex);
}
void unlockQueue()
{
pthread_mutex_unlock(&_mutex);
}
void condwaitQueue()
{
pthread_cond_wait(&_cond,&_mutex);
}
bool IsQueueEmpty()
{
return _task_queue.empty();
}
T popQueue()
{
T t = _task_queue.front();
_task_queue.pop();
return t;
}
pthread_mutex_t* getMutex()
{
return &_mutex;
}
private:
ThreadPool(const int &num = gnum)
: _num(num)
{
pthread_mutex_init(&_mutex,nullptr);
pthread_cond_init(&_cond,nullptr);
for (int i = 0;i<_num;i++)
{
_threads.push_back(new pthread_t);
}
}
ThreadPool(const ThreadPool<T>& tp) = delete;
ThreadPool<T>& operator=(const ThreadPool<T>& tp) = delete;
int _num;
vector<pthread_t *> _threads;
queue<T> _task_queue;
pthread_mutex_t _mutex;
pthread_cond_t _cond;
static ThreadPool<T>* _tp;
static mutex _mtx;
};
template <class T>
ThreadPool<T>* ThreadPool<T>::_tp = nullptr;
template <class T>
mutex ThreadPool<T>::_mtx;In addition to a thread pool in singleton mode, we also have a lock:
#include <iostream>
#include <pthread.h>
class Mutex //自己不维护锁，有外部传入
{
public:
Mutex(pthread_mutex_t *mutex)
:_pmutex(mutex)
{
}
void lock()
{
pthread_mutex_lock(_pmutex);
}
void unlock()
{
pthread_mutex_unlock(_pmutex);
}
~Mutex()
{
}
private:
pthread_mutex_t *_pmutex;
};
class LockGuard //自己不维护锁，由外部传入
{
public:
LockGuard(pthread_mutex_t *mutex)
:_mutex(mutex)
{
_mutex.lock();
}
~LockGuard()
{
_mutex.unlock();
}
private:
Mutex _mutex;
};Then we start writing the thread pool version, first we need to start the thread pool, because we communicate in the start interface, so we start the thread pool in the start interface:
Of course, we still need to write a task. This task is also very simple, which is the code we communicated before:
void start()
{
// 4.线程池初始化
ThreadPool<Task>::getInstance()->start();
logMessage(NORMAL,"ThreadPool init success");
for (;;)
{
//4.server获取新链接 未来真正使用的是accept返回的文件描述符
struct sockaddr_in peer;
socklen_t len = sizeof(peer);
// sock是和client通信的fd
int sock = accept(_listensock,(struct sockaddr*)&peer,&len);
//accept失败也无所谓，继续让accept去获取新链接
if (sock<0)
{
logMessage(ERROR,"accept error,next");
continue;
}
logMessage(NORMAL,"accept a new link success,get new sock: %d",sock);
//5.用sock和客户端通信,面向字节流的，后续全部都是文件操作
//serviceID(sock);
//对于一个已经使用完毕的sock，我们要关闭这个sock,要不然会导致文件描述符泄漏
//close(sock);
// 4.线程池版本
ThreadPool<Task>::getInstance()->Push(Task(sock,serviceID));
}
}Because we changed the thread pool to a singleton mode last time, we started it as a singleton, created multiple threads for us after startup, and then we wrote a task:
#include <iostream>
class Task
{
using func_t=std::function<void(int)>;
public:
Task()
{
}
Task(int sock,func_t func)
:_sock(sock)
,_callback(func)
{
}
void operator()()
{
_callback(_sock);
}
~Task()
{
}
private:
int _sock;
func_t _callback;
};This task only needs to know the callback method and the file descriptor to be called by the thread, and then we write a functor overloaded () symbol.
With the task, we directly construct an anonymous object and then push a task. In the thread pool, the serviceIO function will be called directly through the functor, as shown in the following figure:
Next we run the thread pool:
It can be seen that there is no problem. The above are the three versions of our Tcp server. Let's explain how to add more interesting functions to the log, such as printing data as I demonstrated.
4. Improvement of Tcp server log
We add the variable parameter list on the basis of the original log code, so how to extract the variable parameters?
To use variable parameters, we first need to know what va_last is, and then how to use va_last. To use va_last, we need to use three macros, va_start(), va_arg(), va_end().
As shown in the figure above, va_last should point to the above parameters 3.14, 10, and 'c'. For example, now va_last points to the first parameter 3.14. To point to the second parameter 10, you only need to offset the va_last pointer by a certain number of bytes. So how to make va_last point to the first parameter? Directly use va_start(start) to make va_last point to the first parameter. va_arg() can make the pointer move backward by a specific type. For example, if you just want to point from 3.14 to 10, then you only need va_arg(start, int), and va_end() is to make the start pointer become nullptr.
Let's use the vsprintf interface to demonstrate:
void logMessage(int level,const char* format, ...)
{
//[日志等级][时间戳/时间][pid][message]
//std::cout<<message<<std::endl;
char logprefix[1024]; //日志前缀
snprintf(logprefix,sizeof(logprefix),"[%s][%ld][pid:%d]",to_levelstr(level),(long int)time(nullptr),getpid());
char logcontent[1024]; //日志内容
va_list arg;
va_start(arg,format);
vsprintf(logcontent,format,arg);
std::cout<<logprefix<<logcontent<<std::endl;
}First of all, our log has a fixed format. The prefix of the log we write must be [log level][][][], and the log content is the string that appears in our log, so we need two buffers, prefix represents the prefix, and content represents the log content. In the prefix, we need to print out the level, time, and pid, and message is our log content. We need to define va_last, and then let the arg pointer point to the position of format. vsprintf can read the string in the parameter and the parameter to be printed into the buffer, and then we splice the two buffers into a string to complete the printing with parameters in the log. For example, in the above demonstration, we directly print the number of file descriptors created in the log, which is realized by using variable parameters.
5. Daemonize the Tcp server
The tcp server we are writing now is affected by the xshell client. Once we exit the xshell client, our server will also exit. In fact, a server cannot be affected. Let’s first talk about the principle of the linux foreground and background, and realize a daemon process.
As shown in the figure above, first, after we log in to xshell, Linux will give us a session, which includes a foreground process and multiple background processes. Note: No matter what time it is, there can only be one foreground process. And the command line we enter the command is bash, and the command to view the background process is jobs:
For example, when we create a sleep 10000 task, it will give us a serial number 1, which means that this is job No. 1, and we can also add a few more:
Adding an & symbol after a program means putting the process in the background:
First of all, we see that the ppids of the sleep we created are all 29324, because we started them on the command line, so their parent processes are all bash, and then we observe the PGID. The sleep processes we wrote together with I have the same PGID. The same PGID means that they are in the same process group, and the same process group needs to complete a job, just like the No. 2 and No. 3 jobs just now, and the first process with the same PGID is the leader of this process group. SID represents the session ID, and the session ID also means that these processes are all in one session.
At this time, we put No. 1 on the front desk to let everyone see the phenomenon:
The fg command stands for bringing a task to the foreground. After we put the sleep task in the foreground, we found that bash does not work, which verifies that each session can only have one foreground task, as shown in the figure below:
So how do I switch bash back? Just ctrl + z, ctrl + z can suspend a task, once the task is suspended, it will be put in the background:
How to make the process continue to run after the pause? Use the bg command:
After understanding the above principles, we can implement a daemon process.
We can see from the demonstration just now that once the task is suspended, it will be switched to the background, and whether our server is in the foreground or in the background, once someone logs in to xshell, bash will be switched to the foreground process by default. At this time, our server may be affected by user login, and if our process forms a session by itself, and we are the leader of our own process group, then we will not be affected by what we said, as shown in the figure below:
Let's start implementing the daemon:
Please remember the above interface. This interface can turn a process that is not the leader of a process group into an independent session. Note: it must not be the leader of a process group, it can only be an ordinary team member. Later we will have a way to make a team leader become not a team leader.
There are 3 steps to implement the daemon process:
1. Let the calling process ignore the abnormal signal
For example, in our server, if the client has closed the file descriptor and the server is still writing to the file descriptor, the operating system will send a SIGPIP signal to the process at this time (indicating abnormal writing to the pipeline), because we cannot let the process exit due to the influence of the operating system, so we ignore this signal.
2. Make yourself not the leader of this process, setsid()
This step is actually very simple. We only need to create a subprocess, and once the creation is successful, the original process will exit. The principle is: if we are the group leader, the created child process must be a group member. Once the process that was originally the group leader exits, the new group leader will be the next process of the original process group leader. At this time, we setsid() the created child process, and this process will become a process group by itself, and the PID, PGID and SID will be the same.
3. Because the daemon process is separated from the terminal, even if we close xshell, as long as the remote server is not shut down, our daemon process will continue to run unless we use kill -9 to kill the process. Since it is separated from the terminal, we must close the three file descriptors opened by default. So we can redirect to this path.
4. (Optional) Our process will open a cwd command by default, which will record the current path of our process, which can also prove why the file created by default is in the current path when we do not specify the path, and we can actually make changes to this path. For example, our daemon process does not want to be placed in the current path, but can be placed in other paths.
#include <unistd.h>
#include <signal.h>
#include <cstdlib>
#include <cassert>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#define DEV "/dev/null"
void daemonSelf(const char* currPath = nullptr)
{
//1.让调用进程忽略掉异常的信号
signal(SIGPIPE,SIG_IGN);
//2.如何让自己不是组长，setsid
if (fork()>0)
{
exit(0);
}
//只剩子进程
pid_t n = setsid();
assert(n != -1);
//3.守护进程是脱离终端的，关闭或者重定向以前进程默认打开的文件
int fd = open(DEV,O_RDWR);
if (fd>0)
{
dup2(fd,0);
dup2(fd,1);
dup2(fd,2);
close(fd);
}
else
{
close(0);
close(1);
close(2);
}
//4.可选：进程执行路径发生更改
if (currPath) chdir(currPath);
}If the dev is successfully opened, we will redirect it, and if it fails, we will close the 0,1,2 file descriptors. As we have said about the redirection function, the first parameter is old, and the second parameter is new. We want to redirect the 0, 1, and 2 descriptors to dev/null, so old is the file descriptor where dev/null is located. Because this excuse is very convoluted, we said at the time: Just remember that the first parameter is the destination of the redirection. After the redirection is complete, we close the previous file descriptor to prevent file descriptor leaks.
chdir is the interface to modify the default path. Let's demonstrate:
First include the header file. After the server is initialized, we turn the server into a daemon process. Next, we run it:
As shown in the figure above, the PID and PGID are the same as the SID and have been daemonized. Now let's try to send a message to the server:
It can be seen that there is no problem, as long as the message is echoed, it means that the server is running.
Because once our server is daemonized, the log messages originally written in the file descriptor opened by default will be redirected to dev/null, so we can only see the log information of server initialization, and we will not see it once it is started. Next, we will modify the log and print the log information directly to two files for us:
#pragma once
#include <iostream>
#include <string>
#include <stdarg.h>
#include <ctime>
#include <unistd.h>
#define DEBUG 0
#define NORMAL 1
#define WARNING 2
#define ERROR 3
#define FATAL 4
#define LOG_ERR "log.error"
#define LOG_NORMAL "log.txt"
const char* to_levelstr(int level)
{
switch(level)
{
case DEBUG:return "DEBUG";
case NORMAL:return "NORMAL";
case WARNING:return " WARNING";
case ERROR:return "ERROR";
case FATAL:return "FATAL";
default : return nullptr;
}
}
void logMessage(int level,const char* format, ...)
{
//[日志等级][时间戳/时间][pid][message]
char logprefix[1024]; //日志前缀
snprintf(logprefix,sizeof(logprefix),"[%s][%ld][pid:%d]",to_levelstr(level),(long int)time(nullptr),getpid());
char logcontent[1024]; //日志内容
va_list arg;
va_start(arg,format);
vsprintf(logcontent,format,arg);
//文件版
FILE* log = fopen(LOG_NORMAL,"a");
FILE* err = fopen(LOG_ERR,"a");
if (log!=nullptr && err!=nullptr)
{
FILE* tep = nullptr;
if (level==DEBUG || level==NORMAL || level==WARNING)
{
tep = log;
}
else
{
tep = err;
}
if (tep)
{
fprintf(tep,"%s%s\n",logprefix,logcontent);
}
fclose(log);
fclose(err);
}
}First of all, we classify the logs, a log storage level of 0,1,2, a log storage level of 4,5, and then we open these two files by reading. If they are all open, we will define a file pointer. When the log level is 0,1,2, we will let the new file pointer point to the file log.txt. After confirming which file is we to write, we are like this file to write our previous log prefix+log content. After writing, turn off these two files.
Let's take a look at the results:
It can be seen that there is no problem, as long as a new user logs in, logs will be uploaded.
Summarize
The most important thing in this article is the integration of network knowledge and system knowledge. For example, in the multi-process version and the multi-thread version, the multi-process needs to close the file descriptor twice, while the multi-thread only needs to be closed once. To understand these concepts, you must know the concept of process and thread, so learning the network is a test of the basic skills of the system.