(Study Notes-Process Management) What are the communication methods between processes?

The user address space of each process is independent and generally cannot access each other, but the kernel space is shared by each process, so the communication between processes must pass through the kernel

pipeline

In the Linux command [ | ] the vertical bar is a pipe .

$ ps auxf | grep mysql

Its function is to use the output of the previous command (ps auxf) as the input of the latter command (grep mysql). From this function description, it can be seen that the pipeline transmits data in one direction. If you want to communicate with each other, you need to create two Only a pipeline will do.

At the same time, the above pipe has no name, so the pipe represented by [ | ] is called an anonymous pipe , and it will be destroyed when it is used up.

Another type of pipe is a named pipe , which needs to be created by the mkfifo command first, and the pipe name is specified:

$ mkfifo myPipe

myPipe is the name of the pipeline. Based on the idea that everything in Linux is a file, the pipeline also exists in the form of a file. You can use ls to see that the file type is p, which means pipe:

$ ls -l
prw-r--r--. 1 root    root         0 Jul 17 02:45 myPipe

Write data to the pipe myPipe:

$ echo "hello" > myPipe  // 将数据写进管道
                         // 停住了 ...

After the operation, the command will stop after execution, because the content in the pipeline has not been read, and the command can exit normally only after the data in the pipeline has been read.

So, we execute another command to read the data in this pipeline:

$ cat < myPipe  // 读取管道里的数据
hello

It can be seen that the content in the pipeline is read and printed on the terminal. On the other hand, the echo command also exits normally.

The communication method of pipeline is inefficient and not suitable for frequent data exchange between processes . Of course, its advantage is simplicity, and it is easy to know whether the data in the pipeline has been read by another process.

The principle of pipeline

The creation of anonymous pipes requires the following system call:

int pipe(int fd[2])

Here, an anonymous pipe is created and two descriptors are returned, one is the read-end descriptor fd [0] of the pipe , and the other is the write-end descriptor fd[1] of the pipe . Note that this anonymous pipe is a special file that only exists in memory and does not exist in the file system.

In fact, the so-called pipeline is a string of caches in the kernel . The data written from one section of the pipeline is actually cached in the kernel, and read from the other end, that is, the data is read from the kernel. In addition, piped data is an unformatted stream with a limited size.

We can use fork to create a child process, and the created child process will copy the file descriptor of the parent process , so that there are two [ fd[0] and fd[1] ] for each of the two processes, and the two processes can pass The respective fd writes and reads the same pipeline file to realize cross-process communication.

The pipe can only be written at one end and read at the other end, so the above pattern is easy to cause confusion, because both the parent process and the child process can both write and read at the same time. Well, to avoid this, the usual approach is:

The parent process closes the read fd[0] and only keeps the written fd[1]
The child process closes the written fd[1] and only keeps the read fd[0]

So, if two-way communication is required, two pipes should be created.

So far, it has only explained the use of pipes to communicate between the parent process and the child process, but this is not the case in the shell.

When the A | B command is executed in the shell, the A process and the B process are both processes created by the shell. There is no parent-child relationship between A and B, and their parent processes are both shells.

Therefore, connecting multiple commands together through [ | ] anonymous pipes in the shell actually creates multiple sub-processes, so when we write shell scripts, we should not use multiple pipes if we can use one pipe. This reduces the overhead of creating subprocesses.

To sum up, for the anonymous pipe, its communication scope is the process with parent-child relationship . Because the pipeline has no entity, that is, there is no pipeline file, the fd file descriptor of the parent process can only be copied through fork to achieve the purpose of communication.

For named pipes, it can communicate with each other between unrelated processes . Because of the command pipeline, a device file of type pipe is created in advance. As long as the device file is used in the process, it can communicate with each other.

Regardless of whether it is an anonymous pipe or a named pipe, the data written by a process is cached in the kernel. When another process reads data, it is naturally obtained from the kernel. At the same time, the communication data follows the first-in-first-out principle .

message queue

As mentioned earlier, the communication method of pipelines is inefficient, so pipelines are not suitable for frequent data exchange between processes.

For this problem, the communication module of the message queue can solve it. For example, if process A wants to send a message to process B, process A can return the data after putting the data in the corresponding message queue, and process B can read the data when it needs it. In the same way, the same is true for the B process to send a message to the A process.

The message queue is a message linked list stored in the kernel . When sending data, it will be divided into independent data units, that is, the message body (data block). The message body is a user-defined data type, and the sender and receiver of the message It is necessary to agree on the data type of the message body, so each message body is a fixed-size storage block, unlike the unformatted byte stream data of the pipeline. If the process reads the message body from the message queue, the kernel will delete the message body.

The message queue lifecycle varies with the kernel . If the message queue is not released or the operating system is not shut down, the message queue will always exist, and the life cycle of the aforementioned anonymous pipe is established with the creation of the process and destroyed with the end of the process .

The message queue is not suitable for the transmission of relatively large data , because each message body has a maximum length limit in the kernel, and the total length of all message bodies contained in all queues is also limited. In the Linux kernel, there are two macro definitions MSGMAX and MSGMNB, which define the maximum length of a message and the maximum length of a queue in bytes.

In the message queue communication process, there is a data copy overhead between the user state and the kernel state , because when a process writes data to the message queue in the kernel, the process of copying from the user state to the kernel state will occur. Similarly, another process reads When fetching message data in the kernel, the process of copying data from the kernel state to the user state will occur.

Shared memory

During the reading and writing process of the message queue, there will be a message copy process between the user state and the kernel state. The way of shared memory solves this problem very well.

Modern operating systems use virtual memory technology for memory management, that is, each process has its own independent virtual memory space, and the virtual memory of different processes is mapped to different physical memory. Therefore, even if the virtual addresses of process A and process B are the same, they actually access different physical memory addresses, and the addition, deletion, checking, and modification of data do not affect each other.

The mechanism of shared memory is to take out a virtual address space and map it to the same physical memory . In this way, the things written by this process can be seen by another process immediately, without copying and transmission, which greatly improves the speed of inter-process communication.

amount of signal

Using the memory sharing communication method will bring new problems: if multiple processes modify the same shared memory at the same time, it is likely to conflict. For example, if two processes write an address at the same time, the process that writes first will find that the content is overwritten by others.

In order to prevent data confusion caused by multiple processes competing for shared resources, a protection mechanism is required so that shared resources can only be accessed by one process at any time. Just in time, the semaphore realizes this protection mechanism.

A semaphore is actually an integer counter, which is mainly used to realize mutual exclusion and synchronization between processes, rather than to cache data for inter-process communication .

The semaphore represents the number of resources, and there are two atomic operations to control the semaphore:

One is the P operation , which will subtract 1 from the semaphore. If the semaphore < 0 after the subtraction, it indicates that the resource has been occupied and the process needs to block and wait; if the semaphore > = 0 after the subtraction, it indicates that there are still Resources are available and the process can continue executing normally.
The other is the V operation . This operation will add 1 to the semaphore. If the semaphore <= 0 after the addition, it indicates that there is currently a blocked process, so the process will be woken up to run; after the addition, if the semaphore > 0 , it indicates that there is no currently blocked process;

The P operation is used before entering the shared resource, and the V operation is used after leaving the shared resource. These two operations must appear in pairs.

For example, if we want two processes to mutually exclusive access to shared memory, we can initialize the semaphore to 1 .

The specific process is as follows:

Process A executes the P operation before accessing the shared memory. Since the initial value of the semaphore is 1, the semaphore becomes 0 after the process A executes the P operation, indicating that the shared resource is available, so process A can access the shared memory.
If at this time, process B also wants to access the shared memory and executes the P operation, and the semaphore becomes -1, which means that critical resources have been occupied, so process B is blocked.
The V operation will not be executed until the process A finishes accessing the shared memory, so that the semaphore returns to 0, and then the blocked thread B will be woken up, so that the process B can access the shared memory, and finally the V operation will be executed after the access to the shared memory is completed. , to restore the semaphore to its initial value of 1.

It can be found that the signal is initialized to 1 , which represents a mutual exclusion semaphore , which can ensure that only one process is accessing the shared memory at any time, which protects the shared memory very well.

In addition, in multi-processes, each process is not necessarily executed sequentially. They basically move forward independently and at unpredictable speeds, but sometimes we hope that multiple processes can cooperate closely to achieve a common task.

For example, process A is responsible for producing data, while process B is responsible for reading data. These two processes cooperate and depend on each other. Process A must first produce data before process B can read the data, so execution is limited. sequentially.

Then at this time, we can use the semaphore to realize multi-process synchronization, and we can initialize the semaphore to 0 .

Specific process:

If process B is executed before process A, then when P operation is executed, since the initial value of the semaphore is 0, the semaphore will become -1, indicating that process A has not produced data yet, so process B blocks and waits;
Then, after the current process A finishes producing the data, it executes the V operation, which will make the semaphore become 0, and then wake up the process B blocked in the P operation;
Finally, after process B is woken up, it means that process A has generated data, so process B can read data normally.

It can be found that the signal is initialized to 0 , which means that this is a synchronous semaphore , which can ensure that process A should be executed before process B.

Signal

The inter-process communication mentioned above is the normal working mode. For the working mode under abnormal conditions, it is necessary to use the signal to notify the process .

In the Linux operating system, in order to respond to various events, dozens of signals are provided, each representing a different meaning. We can view all signals through the kill -l command:

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

For the process running in the shell terminal, we can directly enter some key combinations to send signals to the process through the keyboard. For example:

Ctrl + C generates a SIGINT signal, indicating that the process is terminated
Ctrl + Z generates a SIGTSTP signal, which means that the process is stopped, but it is not over

If the process is running in the background, you can send a signal to the process through the kill command, but you need to know the PID number of the running process in advance, for example:

kill -9 1050, which means sending a SIGKILL signal to the process with PID 1050 to end the process immediately;

Therefore, the sources of signal events mainly include hardware sources (such as keyboard Ctrl + C) and software sources (such as kill commands)

Signal is the only asynchronous communication mechanism in the inter-process communication mechanism , because a signal can be sent to a certain program at any time. Once a signal is generated, we have the following ways of processing the signal by the user process:

Perform the default action . Linux specifies default actions for each signal. For example, the SIGTERM signal in the above list means to terminate the process.
Capture the signal . We can define a signal handler function for a signal. When a signal occurs, we execute the corresponding signal processing function.
Ignore the signal . When we don't want to process certain signals, we can ignore the signal and do nothing. There are two signals that cannot be caught and ignored by the application process, namely SIGKILL and SIGSTOP , which are used to interrupt or end a certain process at any time.

Socket

The aforementioned pipelines, message queues, shared memory, semaphores, and signals all perform inter-process communication on the same host. If you want to communicate with processes on different hosts across the network, you need Socket communication.

In fact, Socket communication can not only communicate with processes on different hosts across the network, but also communicate between processes on the same host.

System call to create socket:

int socket(int domain , int type ,int protocal)

The three parameters represent:

The domain parameter is used to specify the protocol family, such as AF_INET for IPV4, AF_INET6 for IPV6, and AF_LOCAL/AF_UNIX for this machine
The type parameter is used to specify the communication characteristics. For example, SOCK_STREAM represents a byte stream, corresponding to TCP, SOCK_DGRAM represents a datagram, and corresponds to UDP, SOCK_RAW represents a raw socket;
The protocol parameter was originally used to specify the communication protocol, but it is basically obsolete now. Because the protocol has been specified by the first two parameters, the protocol is generally written as 0 at present.

Depending on the type of socket created, the communication method is different:

Realize TCP byte stream communication: socket type is AF_INET and SOCK_STREAM;
Realize UDP datagram communication: the socket type is AF_INET and SOCK_DGRAM;
Realize local inter-process communication: "local byte stream socket" type is AF_LOCAL and SOCK_STREAM, "local datagram socket" type is AF_LOCAL and SOCK_DGRAM. In addition, AF_UNIX and AF_LOCAL are equivalent, so AF_UNIX also belongs to the local socket

Socket programming model for TCP protocol communication

The server and client initialize the socket and get the file descriptor;
The server calls bind to bind the IP address and port
The server calls listen to monitor
The server calls accept and waits for the client to connect
The client calls connect to initiate a connection request to the address and port of the server
The server accept returns the file descriptor of the socket used for transmission
The client calls write to write data; the server calls read to read data
When the client disconnects, it will call close , and when the server reads data, it will read EOF. After the data is processed, the server will call close to indicate that the connection is closed.

It should be noted that when the server calls accpet, if the connection is successful, it will return a socket that has completed the connection, which will be used to transmit data later

Therefore, the listening socket and the real socket used to transmit data are two sockets, one is called the listening socket , and the other is called the completed connection socket .

After the connection is successfully established, the two parties start to read and write data through the read and write functions, just like writing something into a file stream

Socket programming model for UDP protocol communication

UDP is not connected, so there is no need for a three-way handshake, and there is no need to call listen and connect like TCP, but the interaction of UDP still requires an ip address and port number, so bind is also required.

For UDP, there is no need to maintain the connection, so there is no so-called sender and receiver, and there is no concept of client and server. As long as there is one socket, multiple machines can communicate arbitrarily, so each UDP Socket needs to bind.

In addition, when calling sendto and recvfrom each time, the IP address and port of the target host must be passed in.

A socket programming model for local interprocess communication

Local sockets are used for inter-process communication on the same host :

The programming interface of the local socket is consistent with the programming interface of IPV4 and IPV6 socket, and can support two protocols of [byte stream] and [datagram];
The implementation efficiency of local socket is much higher than that of byte stream and datagram socket of IPV4 and IPV6;

For local byte stream sockets, the socket types are AF_LOCAL and SOCK_STREAM

For local datagram sockets, the socket types are AF_LOCAL and SOCK_DGRAM

When the local byte stream socket and the local datagram socket are bound, unlike TCP and UDP, which bind the IP address and port number, they bind a local file , which is the biggest difference between them.

Summarize

Since the user space of each process is independent and cannot access each other, it is necessary to use the kernel space to achieve inter-process communication. The reason is very simple, each process shares the same kernel space

The Linux kernel provides many inter-process communication methods, the simplest of which is the pipe, which is divided into anonymous pipes and named pipes.

Anonymous pipe : It has no name identification. Anonymous pipe is a special file that only exists in memory and does not exist in the file system. The [ | ] vertical line in the shell command is an anonymous pipe. The communication data is an unformatted stream and its size is limited. , the communication method is one- way, and data can only flow in one direction. If two-way communication is required, two pipelines need to be created. Then anonymous pipelines can only be used for inter-process communication with parent-child relationships . The life cycle of anonymous pipelines Created with the process and disappear with the process terminated.

Named pipes : break through the limitation that anonymous pipes can only communicate between related processes, because the premise of using named pipes is to create a device file of type p in the file system, then unrelated processes pass through this device files for communication. In addition, regardless of whether it is an anonymous pipe or a named pipe, the data written by a process is cached in the kernel. When another process reads data, it is naturally obtained from the kernel. At the same time, the communication data follows the first-in-first-out principle, and lseek is not supported. Such file positioning operations.

Message Queue : It overcomes the problem that the data in the pipeline communication is an unformatted byte stream. The message queue is actually stored in the [message linked list] of the kernel. The message body of the message queue is a data type that can be customized by the user. When sending data , will be divided into independent message bodies one by one. Of course, when receiving data, it must be consistent with the data type of the message body sent by the sender, so as to ensure that the read data is correct. The speed of message queue communication is not the most timely. After all, each data writing and reading needs to go through the copy process between user mode and kernel mode .

Shared memory : It can solve the overhead caused by the data copy process between user mode and kernel mode in message queue communication. It directly allocates a shared space, and each process can directly access it , which is as fast and convenient as accessing its own space. There is no need to fall into the kernel state or system calls, which greatly improves the communication speed and enjoys the name of the fastest inter-process communication method . However, convenient and efficient shared memory communication brings new problems: multiple processes competing for the same resource will cause data confusion .

Therefore, a semaphore is needed to maintain shared resources to ensure that only one process can access shared resources at any time. This method is mutually exclusive access. The semaphore can not only realize the mutual exclusion of access, but also realize the synchronization between processes . The semaphore is actually a counter, which indicates the number of resources, and its value can be controlled by two atomic operations, namely P operation and V operation .

Signal : Similar to the semaphore name but has nothing to do with it. The signal is an asynchronous communication mechanism . The signal can directly interact between the application process and the kernel. The kernel can also use the signal to notify the user space process of which system events have occurred. The signal event The sources mainly include hardware sources (such as keyboard Ctrl + C) and software sources (such as the kill command). Once a signal occurs, the process has three ways to respond to the signal: 1: execute the default operation, 2: catch the signal, 3: ignore the signal . There are two signals that cannot be captured and ignored by the application process, namely SIGKILL and SIGSTOP. At this time, we can end or stop a process at any time for convenience.

Socket communication : used for inter-process communication between different hosts or local hosts . According to the different types of Socket creation, it can be divided into three common communication methods, one is the communication method based on the TCP protocol, the other is the communication method based on the UDP protocol, and the other is the local inter-process communication method.

Threads under the same process share the resources of the process. As long as they are shared variables, inter-thread communication can be achieved, such as global variables. Therefore, the focus on inter-thread communication is not the communication method, but the multi-thread competition for shared resources. Problem, semaphore can also achieve mutual exclusion and synchronization between threads:

Mutual exclusion ensures that only one thread accesses shared resources at any time;
Synchronization, which ensures that thread A should execute before thread B