Linux design process within the message bus

Article Directory

• In the Windows platform processes the message bus
• If there is no message bus, what would be the problem
• Infinite loop contains relations
• High coupling, low cohesion
• Message Bus
• Structure chart
• principle
• The relationship between producers and bus
• Bus relationship with consumers
• Linux design process within the message bus
• Using inter process to achieve real-time signal-process message bus
• Reference Documents
• Overall process
• The main thread registration bus message processing function
• Producer thread generates and sends a message to the bus
• Receives and processes messages
• Receiving the message bus
• Bus message handler
• Consumers bus message handling virtual function
• Consumers message handling real function
• The core principle
• Send message sigqueue
• Message Processing sigaction
• Problems
• Messages which thread is executed, unpredictable
• Can not call non-reentrant functions
• Can not call an asynchronous signal unsafe function
• Use threads within a process to achieve inter-signal message bus
• Overall process
• The main thread registration bus message processing function
• Registration bus message handler
• Create a special bus message processing threads
• Special bus message processing thread waits for messages
• Producer thread generates and sends a message to the bus
• Receives and processes messages
• Bus message processing thread receives a dedicated message
• Bus message handler
• Consumers bus message handling virtual function
• Consumers message handling real function
• The core principle
• sigprocmask blocking all asynchronous messages
• Send message pthread_sigqueue
• Create a dedicated thread sigwaitinfo receive messages using the message bus
• summary

In the Windows platform processes the message bus

  Development under Windows platform C / C ++ program, the same process in different modules, if you want to send a message, you can use PostMessage, SendMessage and other functions. Wherein: PostMessage asynchronous message transmission interface; the SendMessage synchronization message transmission interface.

  For example:
  If the window B sends message A to give window. Then: A window just need to have the window handle of the window B, then call PostMessage to send messages to the handle of the window B; B only need to increase the processing window to the message in your own message handler.
  If window B to give the window A transmitted message. So: Window B need only have window A window handle, then call PostMessage to send messages to handle window A; A window can only increase the processing of the message in your own message handler.
  Therefore, regardless of window A message is sent to a window or window B B A message sent to the window, or A, send messages between B, A, B, if the other party does not need to focus on process messages, but only need to focus on what needs to deal with each other messages.

If there is no message bus, what would be the problem

  Using the example above, if there is no bus messages:
  If the window B sends message A to give window. A window then call the message processing function of the window B. A window requires window B contains the file header. Window A window B need to focus on how to handle the messages. That is: Window A high degree of coupling with the B window.
  As shown below:

  If window B to give the window A transmitted message. Window B then call the message processing function window A. Window B A need to include the document window. Window A window B need to focus on how to handle the message. That is: A window B with high degree of coupling window.
  As shown below:

Infinite loop contains relations

  If the window message transmission window B to give A, while window B also sends a message to the window A. A window then need to include the document window B, but also requires the window B contains the header window A. So there will be A contains B, while B also contains an infinite loop A containment relationship. At compile time error. Of course, even if there has been an endless loop relationship, we can also avoid some tips compile-time error. This is no longer conjugated later.
  As shown below:

High coupling, low cohesion

  If there is no message bus, the message sender must be concerned about how the message recipient to process messages, resulting in strong coupling relationship between the sender and the receiver module module. This design does not meet the high cohesion and low coupling principles of software design.

Message Bus

  The message bus just to solve the above problem.

Structure chart

principle

  In the design of the message bus, a total of three roles, a definition, a message handling process. Three roles are: producers, consumers, bus. A definition is: the message definition. Processing a message are: message handler. Producers are only responsible for the production of news. Consumers are only responsible for processing messages. Bus driver is only responsible for receiving and messages. Between producers and consumers share the message definition. Instance of a subclass of the bus contains all consumers. Each Consumers only need to implement a message handler for their concerns, to complete the message processing.

The relationship between producers and bus

  Production producer responsibility message and sends the message to the bus, the bus is responsible for receiving messages.

Bus relationship with consumers

  Bus is responsible for receiving messages and drive consumers to perform message processing function. Consumers bus inherited from the bus. Informers contains all consumers bus. Each message need only realize their attention to the message processing function to complete message handling.

  Such coupling between producers and consumers is present only message definition. Achieve high cohesion, low coupling between them. At the same time when the two need to send messages, but also lifted the death cycle, including relationship between the two.

  Because the message sender does not need to focus on process messages. Therefore, the message bus programming to bring the good outside:

• To avoid the endless loop between the sender and the receiver contains relationship.
• Module complies with high cohesion, low coupling design principles, the module is easy to maintain.

Linux design process within the message bus

  Implement the message bus within Windows platform process, you can: PostMessage, SendMessage function to achieve. Unfortunately, Linux system itself does not directly provide for message exchange between the different modules within the PostMessage (SendMessage) function is functionally similar function for the process. However, we can use a Linux system-related library functions within their own similar package set with PostMessage (SendMessage) function function processes the message bus.
  Here we will describe in detail two options to achieve progress in the Linux message bus. The first scenario: using inter-process real-time information to achieve. The second option: to implement the use of inter-thread signal. At the same time, we will compare the merits of the two programs.

Using inter process to achieve real-time signal-process message bus

Reference Documents

"Linux-UNIX System Programming Manual" real-time signal Chapter 22.8

Overall process

  The main bus message flow is as follows:

• The main thread registration bus message processing function
  • Registration bus message handler
  • Bus waiting for news
• Producer thread generates and sends a message to the bus
• Receives and processes messages
  • Receiving the message bus
  • Bus message handler
  • Consumers bus message handling virtual function
  • Consumers message handling real function

The main thread registration bus message processing function

int main(int argc, char* argv[])
{
	int error = S_OK;

	if (nullptr != g_Service)
	{
	    ...
		message_register_handler(message_handler, g_Service);
        ...
	}

	message_unregister_handler();
	return 0;
}

int message_register_handler(lpfn_message_handler msgHandler, void *userData)
{
	int hr = S_OK;
	struct sigaction act;

	g_userSig.signum = SIGUSER;
	g_userSig.msgHandler = msgHandler;
	g_userSig.userData = userData;

	//register message process function
	act.sa_sigaction = signal_handler;
	act.sa_flags = SA_SIGINFO;
	//wait signal
	hr = sigaction(SIGUSER, &act, NULL);
	return hr;
}

Producer thread generates and sends a message to the bus

int message_post(int message, void *data1, void *data2)
{
	if (!g_userSig.tidp)
		return 0;

	int hr = 0;
	pid_t pid = getpid();
	union sigval val;
	user_signal_data_t *sig = (user_signal_data_t *)malloc(sizeof(user_signal_data_t));
	memset(sig, 0, sizeof(user_signal_data_t));

	sig->checkCode = USER_SIGNAL_CHECK_CODE;
	sig->message = message;
	sig->data1 = data1;
	sig->data2 = data2;

	val.sival_ptr = sig;
	hr = sigqueue(pid, g_userSig.signum, val);
	if (0 != hr)
	{
		//EAGAIN
		printf("message post: hr:%d, errorcode: %d, errorstr: %s\n", hr, errno, strerror(errno));
		hr = E_PTHREAD_SIGQUEUE_FAILED;
	}
	
	return hr;
}

Receives and processes messages

Receiving the message bus

Please refer to the principle: "Linux-UNIX System Programming Handbook" real-time signal processing 22.8.2

//总线等待消息。收到消息时，即会触发总线消息处理函数
	hr = sigaction(SIGUSER, &act, NULL);

Bus message handler

void signal_handler(int signum, siginfo_t * info, void * context)
{
	user_signal_data_t *pdata = nullptr;
	if (g_userSig.signum == signum)
	{
		if (context)
		{
			pdata = (user_signal_data_t *)info->si_ptr;
			if (g_userSig.msgHandler)
			{
				g_userSig.msgHandler(pdata->message, pdata->data1, pdata->data2, g_userSig.userData);
			}
			free(pdata);
			pdata = nullptr;
		}
	}
	else
	{
		printf("unknown signal: %d\n", signum);
	}
}

int message_handler(int message, void *msgData1, void *msgData2, void *userData)
{
	CBaseService *service = (CBaseService *)userData;
	service->on_message(message, msgData1, msgData2);
	return S_OK;
}

Consumers bus message handling virtual function

int CBaseComsumerService::on_message(int message, void *msgData1, void *msgData2)
{
	int hr = E_NOTIMPL;
	return hr;
}

Consumers message handling real function

int CComsumerService::on_message(int message, void *msgData1, void *msgData2)
{
	int hr = S_OK;
	hr = CBaseComsumerService::on_message(message, msgData1, msgData2);
	if (SUCCEEDED(hr))
		return hr;

    //process messages related to myself only 
	switch (message)
	{
	case UM_MESSAGE_A:
		on_msg_a(message, msgData1, msgData2);
		break;
	case UM_MESSAGE_B:
		on_msg_b(message, msgData1, msgData2);
		break;
	default:
		hr = E_NOTIMPL;
		break;
	}

	return hr;
}

The core principle

Send message sigqueue

  In the design of a message bus, a custom parameter sigqueue carries, sends a message to this process. In order to achieve the in-process sending the message.

Message Processing sigaction

  Use sigaction receive messages, and to signal_handler signal processing function. Registration bus message signal_handler call handler: message_handler. message_handler call the consumer virtual function of news distribution process, realize the message.

Problems

  Most of the contents of this section excerpt from: "Linux-UNIX System Programming Manual" Chapter 21.1.2 reentrant functions and asynchronous signals safety function.

Messages which thread is executed, unpredictable

  When using sigqueue send a signal, a signal of the disposal of part of the process level, all threads in the process shared setting for the disposal of each signal. If a thread using the function sigaction () to create a handler for certain types of signal (for example, SIGINT), then when it receives SIGINT, will go to any thread calls the handler. Similarly, the signal will be set if the disposal signal is ignored (ignore), then all threads will be ignored.
  This means that, in this case, the message bus: The message is executed which thread, unpredictable.

Can not call non-reentrant functions

  To explain reentrant functions, why not reentrant function object, you first need to distinguish between single-threaded and multi-threaded program procedures. A typical UNIX program has a thread of execution, throughout the program, the CPU performs logic to surround a single processing instructions. And for multi-threaded programs, the same process but there is logic to perform multiple independent streams, concurrent.
  However, the concept of multiple execution threads using a signal processor function of the program are also relevant. Because the signal processor function may interrupt execution of the program at any point asynchronous, so in the same process in the actual formation of the two (ie, the main program and the signal processor function) independent (though not concurrently) thread of execution.
  If the same process multiple threads can simultaneously call a function safely, the function is reentrant. Here, the "security" means that, regardless of the other thread calls this function to perform state functions can produce the desired results.
  Update function or global variable static data structure may be not reentrant. (Function only use local variables must be re-entrant.) If two calls to the function (for example: were initiated by the two threads of execution) while trying to update the same data type or global variable, then the two will likely interfere with each other and produce incorrect results. For example, suppose a thread is to add a new list item is a linked list data structure while the other thread is also attempting to update the same list. Because the list is updated to add a new item involves rounds of the pointer, another thread interrupted once these steps and modify the same hands, the result will be confusion.
  In the standard C language function library, this possibility is very common. For example, the malloc () and free () maintains a list for the memory blocks freed for reallocation of memory from the heap. If the main program during a call to malloc () the same as a call to malloc () function of the signal processor interrupted, then the list may be destroyed. Thus, malloc () family of functions thereof and the use of other library functions are not reentrant.
  Some libraries reason not reentrant, because they use static memory allocation was to return information. Examples of such functions include crypt (), getpwnam (), gethostbyname () and
getservbyname (). If the signal processor uses these functions, the main program will cover the same information returned by the function (and vice versa) since the last call.
  Function static data structure for the internal accounting is not reentrant. The most obvious example is stdio library members (printf (), scanf (), etc.), they will for the buffer I / O updates the internal data structure. So, if you call the printf function in signal processor (), and again in the main program calls printf () or other stdio function was during the interrupt handler function, then sometimes we will see strange output, and even lead to program crash or data corruption.
  Even if does not use non-reentrant library functions reentrant problem still can not be ignored. If the signal processor and the main function of global data structure will be updated by the custom programmer, then for the main program, the function of such a signal processor is not reentrant.
  If the function is not reentrant, its manual page typically explicitly or implicitly give tips. For those functions which use or return a statically allocated variables, require special attention.

  This means that, in the present embodiment the message bus: as malloc, printf other system functions are not used, because they are not reentrant function.

Can not call an asynchronous signal unsafe function

  Asynchronous signal safe function means that when a function call from the signal processor, which may be implemented to ensure that it is safe. If a function is reentrant, when or if the signal processor function can not be interrupted, saying that the function is an asynchronous signal safe.
  POSIX.1-1990 provides asynchronous signals safety function table. SUSv3 stressed that all functions except for the table in terms of signal is unsafe, but also pointed out that only when the signal processor function to break execution of an unsafe function, and handler function itself calls this function insecurity this function is unsafe. In other words, the signal processor write function has the following two options.

• The signal processor itself to ensure that the function code is reentrant, and only call the security function of the asynchronous signal.
• When the main program or function to unsafe operation of the signal processor function may also be updated global data structure, transfer of the blocking signal.

  The second method problem is that in a complex process, in order to ensure that the signal processor function is interrupted by a call to the main function is not unsafe, which is a bit difficult. For this reason, generally will not call the above rules simplify unsafe function signal processor function.

  This means that, in the present embodiment the message bus: asynchronous signal unsafe function is not invoked.

Use threads within a process to achieve inter-signal message bus

Overall process

  The main bus message flow is as follows:

• The main thread registration bus message processing function
  • Registration bus message handler
  • Create a special bus message processing threads
  • Special bus message processing thread waits for messages
• Producer thread generates and sends a message to the bus
• Receives and processes messages
  • Bus message processing thread receives a dedicated message
  • Bus message handler
  • Consumers bus message handling virtual function
  • Consumers message handling real function

The main thread registration bus message processing function

Registration bus message handler

int main(int argc, char* argv[])
{
	int error = S_OK;

	if (nullptr != g_Service)
	{
	    ...
		message_register_handler(message_handler, g_Service);
        ...
	}

	message_unregister_handler();
	return 0;
}

Create a special bus message processing threads

int message_register_handler(lpfn_message_handler msgHandler, void *userData)
{
	int hr = S_OK;
	g_userSig.signum = SIGUSER;
	g_userSig.msgHandler = msgHandler;
	g_userSig.userData = userData;

	if (0 != (pthread_create(&g_userSig.tidp, nullptr, message_thread_function, (void*)&g_userSig)))
	{
		printf("create message thread error!\n");
		hr = E_PTHREAD_CREATE_FAILED;
	}

	return hr;
}

Special bus message processing thread waits for messages

static void * message_thread_function(void *arg)
{
	sigset_t set;
	int sig;
	siginfo_t info;
	user_signal_t *usersig = (user_signal_t *)arg;

	/* wait SIGUSER. process must block SIGUSER, otherwise, SIGUSER signal may be processed  by other thread*/
	sigemptyset(&set);
	sigaddset(&set, SIGQUIT);//wait SIGQUIT, to exit thread
	sigaddset(&set, SIGUSER);

	while (true)
	{
		sig = sigwaitinfo(&set, &info);
		if (sig < 0)
		{
			if (errno == EINTR)
			{
				perror("message thread: sigwaitinfo ");
				continue;
			}

			printf("message thread: parent error: %s\n", strerror(errno));
			break;
		}

		if (SIGQUIT == sig)
		{
			printf("message thread: receive quit signal\n");
			break;
		}

		if (SIGUSER == sig)
		{
			user_signal_data_t *pdata = nullptr;
			pdata = (user_signal_data_t *)info.si_ptr;
			if (pdata)
			{
				if (USER_SIGNAL_CHECK_CODE == pdata->checkCode)
				{
					if (usersig->msgHandler)
					{
						usersig->msgHandler(pdata->message, pdata->data1, pdata->data2, g_userSig.userData);
					}

					free(pdata);
					pdata = nullptr;
				}
				else
				{
					printf("message thread: receive unknown message data\n");
				}
			}
			else
			{
				printf("message thread: receive data pointer is null\n");
			}
		}
	}

	printf("message thread: exit!\n");
	return nullptr;
}

Producer thread generates and sends a message to the bus

int message_post(int message, void *data1, void *data2)
{
	if (!g_userSig.tidp)
		return 0;

	int hr = 0;
	pid_t pid = getpid();
	union sigval val;
	user_signal_data_t *sig = (user_signal_data_t *)malloc(sizeof(user_signal_data_t));
	memset(sig, 0, sizeof(user_signal_data_t));

	sig->checkCode = USER_SIGNAL_CHECK_CODE;
	sig->message = message;
	sig->data1 = data1;
	sig->data2 = data2;

	val.sival_ptr = sig;

	/* send signal to a specific thread. this can ensure signals are processed correctly. because:
	* 1. only one thread processes all signals, so it makes asynchronous signal processing to become synchronously.
	* 2. system functions like malloc, printf, etc. will not be interrupted by other signals, so they are reentrant in this prcocess.
	*/
	hr = pthread_sigqueue(g_userSig.tidp, g_userSig.signum, val);
	if (0 != hr)
	{
		//EAGAIN
		printf("message post: hr:%d, errorcode: %d, errorstr: %s\n", hr, errno, strerror(errno));
		hr = E_PTHREAD_SIGQUEUE_FAILED;
	}

	return hr;
}

Receives and processes messages

Bus message processing thread receives a dedicated message

Please refer to the principle: "Linux-UNIX System Programming Manual" mode waiting for the signal to synchronize 22.10

	sig = sigwaitinfo(&set, &info);
	...
	usersig->msgHandler(pdata->message, pdata->data1, pdata->data2, g_userSig.userData);

Bus message handler

int message_handler(int message, void *msgData1, void *msgData2, void *userData)
{
	CBaseService *service = (CBaseService *)userData;
	service->on_message(message, msgData1, msgData2);
	return S_OK;
}

Consumers bus message handling virtual function

int CBaseComsumerService::on_message(int message, void *msgData1, void *msgData2)
{
	int hr = E_NOTIMPL;
	return hr;
}

Consumers message handling real function

int CComsumerService::on_message(int message, void *msgData1, void *msgData2)
{
	int hr = S_OK;
	hr = CBaseComsumerService::on_message(message, msgData1, msgData2);
	if (SUCCEEDED(hr))
		return hr;

    //process messages related to myself only 
	switch (message)
	{
	case UM_MESSAGE_A:
		on_msg_a(message, msgData1, msgData2);
		break;
	case UM_MESSAGE_B:
		on_msg_b(message, msgData1, msgData2);
		break;
	default:
		hr = E_NOTIMPL;
		break;
	}

	return hr;
}

The core principle

  Non-reentrant functions, asynchronous signal unsafe function calls are not to be safe in the signal handler. Because of these reasons, when the multi-threaded application must process a signal generated asynchronously, generally it should not be a function of the signal processing mechanism for receiving notification signals arrive. Instead, the recommended approach is as follows.

• All asynchronous signals in all threads are blocked process may receive. The easiest way is, before creating any other thread, the main thread blocking these signals. Each thread will inherit a copy of the main thread signal mask is subsequently created.
• Then create a dedicated thread, call the function sigwaitinfo (), sigtimedwait () or sigwait () to receive the signal received. Upon receiving the signal, a proprietary thread can safely modify shared variables (under the protection mutex), can be called an asynchronous signal is not a function of safety (non-async-signal-safe) of. It can also send a signal to variable conditions, and the use of communication and synchronization mechanisms other threads or processes.

  Use threads within a process to achieve inter-signal message bus, this is the basis for recommendations to achieve.

sigprocmask blocking all asynchronous messages

	sigset_t newmask, oldmask;

	//block SIGUSER for all threads
	sigemptyset(&oldmask);
	sigemptyset(&newmask);
	sigaddset(&newmask, SIGUSER);
	sigprocmask(SIG_BLOCK, &newmask, &oldmask);

Send message pthread_sigqueue

int message_post(int message, void *data1, void *data2)
{
    ...
	hr = pthread_sigqueue(g_userSig.tidp, g_userSig.signum, val);
    ...
}

Create a dedicated thread sigwaitinfo receive messages using the message bus

static void * message_thread_function(void *arg)
{
	while (true)
	{
		sig = sigwaitinfo(&set, &info);
	
	    ...
		usersig->msgHandler(pdata->message, pdata->data1, pdata->data2, g_userSig.userData);
		...
	}

	return nullptr;
}

summary

  Use threads within a process to achieve inter-signal message bus, basically functions in a similar message bus with Windows PostMessage (SendMessage) function feature processes. Code snippets in this article, is mainly used to help understand the principle and process design. Interested friends have their own based on these pieces, to realize their business functions.