"The Nervous System of the Internet"——Analysis of Message Middleware MOM

New Inter-Process Communication: Message-Oriented Middleware

Author: Once Day Date: May 17, 2023

This article is mainly translated from the article "Message-oriented Middleware: Agile Systems for IT Success (g2.com)", for learning and research purposes only !

Reference documents:

• Message-Oriented Middleware (MOM) (Sun Java System Message Queue 3.7 UR1 Technical Overview) (oracle.com)
• Message-oriented Middleware: Agile Systems for IT Success (g2.com)
• Popular Science Time | Introduction to Message Middleware Series - Functions and Protocols - Zhihu (zhihu.com)

1 Overview

Common inter-process communication methods within the system are generally Socket. Although most components use select, poll, epolland other means to poll, and then cooperate with libeventother frameworks to realize the oneloop model, but with less than a thousand lines of code, it is inevitable to encounter Various problems.

Typical questions are as follows:

1. There is a timeout problem with synchronous sockets, which will cause false business processing failures . Severely affect traffic forwarding.
2. Simple non-blocking sockets cannot perform clipping and valley filling, and simple asynchronous processing can also completely improve its logic.
3. Each Clientsum Serverhandles a wide variety of socket error conditions, which are not easy to deal with.
4. Simple Socketprogramming is too poor to locate, because it is impossible to monitor the actual business news, so once a business problem occurs, it is difficult to locate.

Here, a better middleware - message-oriented middleware ( Message-Oriented Middleware, MOM) is introduced.

MOM ( Message Oriented Middleware) is message-oriented middleware that uses messaging providers to coordinate messaging operations. MOM needs to provide API and management tools. Clients use API calls to send messages to destinations managed by the provider. After sending the message, the client continues to perform other work, and the provider keeps the message until the receiver receives an acknowledgment of the message.

The earliest MOM standard was a specification X/Open groupproposed in the 1990s XATMI, which defined the API for inter-process interaction, and the implementation included Enduro/Xmiddleware and software Oracle Tuxedo.

AMQP is another important benchmark that defines communication protocols and formats between participating software elements to ensure interoperability. AMQP provides flexible routing schemes, transaction management, queuing, security, management, clustering, federation and cross-platform support.

The HLA standard was proposed by IEEE for the interoperability of simulation. It specifies a list of services that can be passed C++or Java APIaccessed. This standard defines a publish/subscribe based information exchange. It also includes synchronous data exchange, time-advancing facilities and synchronization points based on logic simulation time.

The Java environment provides a vendor-neutral API, called JMS, for sending and receiving messages using the Java programming language. JMS APIReduces enterprise messaging expertise, enabling Java developers to build a wide range of messaging applications while maintaining some portability between JMS provider implementations.

XMPP is an XML-based communication protocol for message-oriented middleware. The protocol is also used in publish/subscribe systems, VoIP signaling, file transfers, video, gaming, social networking platforms and IoT applications such as smart grids.

It can be seen from the above development history that the evolution of the MOM standard shows a trend from a proprietary standard to an open standard, and the scope of application is continuously expanded to meet the changing needs. This makes the use of MOM middleware more convenient and rich.

2. Basic working principle of message middleware

A messaging middleware improves inter-component communication for complex IT systems. It makes development more effortless in a distributed environment.

The message queue is a crucial component in message-oriented middleware platforms. Message queuing helps MOM-based systems retain messages in the architecture. A message-oriented middleware architecture can send and receive data with the help of queues.

Queues are also necessary for developing asynchronous communication in a message-oriented middleware architecture. The queue stores messages in a particular order using the first in, first out (FIFO) standard. The message sent initially to the queue will be the first to be retrieved from the queue using FIFO.

A queue’s identity, size, queue save threshold, message filtering mechanism, and other properties are all configurable. In most circumstances, each app has its own queue; however, in some cases, the setup allows the apps to share queues. Message-oriented middleware systems can support a range of queues.

The two most popular messaging models in message-oriented middleware architectures are publish/subscribe (also known as pub/sub) and point-to-point (P2P) models. Both of these approaches rely on message exchange via a queue. A typical system employs a hybrid of these two approaches to achieve various messaging goals and objectives.

Messaging middleware improves communication between components of complex IT systems. It makes development in a distributed environment easier.

Message queue is a key component in message-oriented middleware platform. Message queues help mom-based systems retain messages within the architecture. Message-oriented middleware architecture can send and receive data with the help of queues.

Queues are also necessary for developing asynchronous communication in a message-oriented middleware architecture. Queues store messages in a specific order using the first-in-first-out (FIFO) standard. Messages initially sent to the queue will first be retrieved from the queue using FIFO.

The identity, size, queue saving threshold, message filtering mechanism and other attributes of the queue are all configurable. In most cases, each application has its own queue; however, in some cases, settings allow applications to share queues. A message-oriented middleware system can support a series of queues.

In message-oriented middleware architectures, the two most popular messaging models are publish/subscribe (also known as publish/subscribe) and peer-to-peer (P2P) models. Both approaches rely on message exchange via queues. Typical systems use a mix of these two approaches to achieve various messaging goals.

The following is a summary of common messaging middleware roles:

• The sender (Producer) is responsible for sending the message to the message queue middleware (MOM). A sender can be an application, service, or device that generates and publishes messages to a queue or topic. Senders usually don't care who the recipient of the message is, they only care about sending the message to the correct queue or topic.
• Receiver (Consumer) , responsible for obtaining and processing messages from MOM. Receivers can be applications, services, or devices that subscribe to queues or topics to receive messages. After receiving the message, the receiver can execute corresponding processing logic, such as updating the database, triggering an event or calling other services. The receiver usually doesn't care which sender the message came from.
• Queue is a data structure used in MOM to store messages to be processed. The main function of the queue is to buffer messages and ensure asynchronous communication between sender and receiver. Queues usually provide first-in-first-out (FIFO) message processing order to ensure that messages are processed in the order they arrive at the queue.
• The message body (Message) is a data packet encoded according to a fixed format defined by different communication protocols to encapsulate business data and realize message transmission.
• Topic (Topic) is a concept used in MOM to implement the publish/subscribe model. Similar to queues, topics are also used to store pending messages. However, topics allow multiple receivers to subscribe to the same topic to receive messages. This means that when a sender sends a message to a topic, all receivers subscribed to that topic will receive the message. This facilitates one-to-many communication.
• Broker is the core component of the MOM system and is responsible for message delivery, storage and routing. Brokers receive messages from senders, store them in queues or topics, and route messages to receivers as needed. Brokers are also responsible for handling message acknowledgments, transactions, and other MOM features. Brokers can be centralized or distributed.
• Administrator (Administrator) , responsible for configuring, monitoring and managing the MOM system. Administrator responsibilities include creating and deleting queues, setting routing rules, allocating resources, authorizing access rights, and more. Administrators typically use the administrative control tools provided by MOM to perform these tasks. Administrators are also responsible for ensuring the security and stability of the MOM system.

3. Summary of common middleware types

Middleware is an abstraction layer for interaction between different software architectures. It can shield the differences between different architectures and provide a unified abstract interface. The following are common types of middleware:

• Message-oriented middleware (MOM) (Message-oriented middleware) is an architecture that allows application components to receive and forward messages between them. It supports software deployment across multiple platforms and simplifies the process of developing applications across multiple operating systems and network protocols. It has several advantages over other middleware (such as hardcoded logic) and is one of the most widely used forms of middleware.

• Object middleware , also known as an object request broker, enables applications to exchange objects and access services through an object-oriented system. In short, it controls the communication of objects in a distributed computing system.

• Remote procedure call (RPC) middleware (Remote procedure call middleware) , or RPC-based middleware, allows procedures in one application to call procedures in other applications as if they were local calls. This middleware includes a chaining mechanism for locating remote operations and providing them transparently to clients. RPC-based middleware used to deal with process-based systems, but now incorporates object-based components.

• Transaction processing (TP) middleware (Transaction processing middleware) , including systems such as transaction processing monitors. It also includes web application servers. Its goal is to create an environment for developing and deploying various applications.

• Database middleware (Database middleware) , supports direct access to the database. It facilitates direct interaction with databases and includes several database gateway and connection options.

• Embedded middleware , also known as integration middleware, helps companies develop and deliver better integration frameworks through program or hardware interfaces. It is a bridge between embedded applications and real-time operating systems.

• Portals , also known as enterprise portals, are not technically a type of middleware. However, they are classified as middleware because they allow front-end integration and facilitate interactions between client machines and back-end platforms.

• Content-centric middleware (Content-centric middleware) obtains content through provider-consumer abstraction, and is widely used in content-based systems. It is comparable to publish/subscribe middleware, another software often used as a component of web-based applications.

4. Comparison of message-oriented middleware (MOM vs RPC)

There are two types of middleware for message-oriented use:

• Remote Procedure Call (RPC)-based middleware that allows procedures in one application to call procedures in a remote application as if they were local calls. This middleware implements a chaining mechanism that looks up remote procedures and enables callers to use these procedures in a transparent manner. Previously, this type of middleware dealt with procedure-based programs; now, it also includes object-based components.
• Object Request Broker (ORB)-based middleware that enables application objects to be distributed and shared across heterogeneous networks.
• Message-Oriented Middleware (MOM) enables distributed applications to communicate and exchange data by sending and receiving messages.

Both types of models allow a local component to access (influence) another remote component through a network protocol. The difference is that when the RPC middleware calls the remote component, it is a synchronous operation, and it must wait for the calling process to return before proceeding. The MOM middleware performs asynchronous data transmission through an efficient and reliable message passing mechanism.

Here is a comparison of MOM (Message Queue) and RPC (Remote Procedure Call):

Performance:

• MOM : Message queues use asynchronous communication and can provide a buffering mechanism between sender and receiver. This can improve performance, especially in high concurrency situations. However, serialization and deserialization of message passing may introduce additional performance overhead.

• RPC : Remote Procedure Call uses synchronous communication and typically blocks while the caller waits for the result to return. This can result in a performance hit, but RPC is usually faster than MOM because there is no need to serialize and deserialize messages.

In terms of stability:

• MOM : Message queues have high stability because the communication between sender and receiver is decoupled. Even if the receiver fails or is unable to process the request, the sender can continue to work.

• RPC : Remote Procedure Call can be less stable because it relies on direct communication between the caller and the callee. If the callee fails or is unable to process the request, the caller may be affected.

In terms of positionability:

• MOM : Due to the asynchronous nature of message queues, debugging and locating problems can become difficult. To identify problems, it may be necessary to trace and analyze the flow of messages in message queues.

• RPC : Remote Procedure Call makes debugging and locating problems relatively easy because it uses synchronous communication. Requests and responses between callers and callees can be traced and analyzed directly.

Fault Handling Capability:

• MOM : Message queues have better fault handling capabilities because the communication between sender and receiver is decoupled. If the receiver fails, the sender can continue sending messages to the queue, waiting for the receiver to recover.

• RPC : Remote Procedure Call has weak failure handling. If the callee fails, the caller may be affected and additional fault tolerance mechanisms need to be implemented.

Programming complexity:

• MOM : Message queues can lead to increased programming complexity as issues such as serialization, deserialization, and messaging need to be dealt with.

• RPC : A remote procedure call generally has lower programming complexity because it is similar to a local function call.

maintainability

• MOM : The maintainability of the message queue is high, because the communication between the sender and the receiver is decoupled. This allows various parts of the system to be updated and maintained independently.

• RPC : Remote Procedure Call can be less maintainable because it relies on direct communication between the caller and the callee. This can introduce coupling between various parts of the system, making updates and maintenance difficult.

Application example :

• MOM : Message queues are widely used in distributed systems, microservice architectures, and large enterprise applications. For example, Apache Kafka and RabbitMQ are two widely used message queuing systems.

• RPC : Remote procedure calls are often used in distributed systems or scenarios where remote services need to be called. gRPC and Apache Thrift are two widely used RPC frameworks.

5. Features of message-oriented middleware

(1) Asynchronous messaging (Asynchronous messaging) :

Asynchronous messaging allows the sender and receiver to communicate at different times without waiting for each other. Message queuing middleware (MOM) makes this communication possible by storing messages in queues. A sender can send a message to a queue, and a receiver can get a message from the queue at a later time. This approach helps achieve decoupling and scalability.

(2) Assured message delivery :

MOM ensures that messages are delivered reliably to recipients, even in the event of system failure. MOM usually provides mechanisms such as persistence, message acknowledgment (ACK) and message retry to ensure that messages will not be lost.

(3) Routing :

Path selection means that the MOM selects a transmission path for messages in the system based on specific rules or policies. This can include selecting the appropriate path based on priority, load balancing, or specific routing rules. This facilitates efficient messaging and resource utilization.

(4) Message transformation mechanism (Transformation) :

The message conversion mechanism allows the MOM to convert the format of the message between the sender and receiver. This is useful when communicating across platforms or between heterogeneous systems, as it ensures that the message sent by the sender is understood by the receiver. Transformation can include data format, encoding, compression, or protocol conversion.

(5) Transaction support (Transaction support) :

Transaction support allows MOM to process messages in a transactional context. This means that a group of related messages can be processed as an atomic unit, ensuring data consistency and integrity. If the transaction succeeds, all messages will be successfully processed; if the transaction fails, all messages will be rolled back to the state before the transaction started.

(6) Notification services :

The notification service allows MOM to send notifications to subscribers when certain events occur. This can include system failures, full queues, insufficient resources, etc. This facilitates real-time system monitoring and fault diagnosis.

(7) Monitoring (Monitoring) :

MOM usually provides monitoring functions to monitor the health of the system in real time. This might include monitoring queue lengths, message delivery rates, error rates, resource utilization, etc. Monitoring functions can help operators identify potential problems and take corresponding measures.

(8) Management and control :

Administrative controls allow operators to configure and manage the MOM system. This may include adding or removing queues, modifying queue properties, setting routing rules, granting user access, etc. The management control function can ensure the security and stability of the MOM system.

6. Summary of MOM message-oriented middleware

Message-Oriented Middleware ( Message-Oriented MiddlewareMOM) is a software component used to support asynchronous communication and message passing. It provides an infrastructure for passing messages in a distributed system while supporting multiple communication modes such as peer-to-peer (P2P) and publish/subscribe (pub/sub) modes. The following summarizes the advantages, disadvantages, design ideas and precautions of MOM.

advantage:

1. Decoupling : MOM can decouple the sender and receiver, reducing the dependencies between system components and improving the scalability and maintainability of the system.
2. Asynchronous communication : MOM supports asynchronous communication. The sender and receiver do not need to be online at the same time, and can process messages at different times. This helps improve the responsiveness and processing power of the system.
3. Load balancing : MOM can automatically adjust the message transmission speed and processing capacity according to the system load, so as to achieve load balancing.
4. Fault-tolerant processing : MOM provides mechanisms such as message persistence, message retry, and dead letter queues, which can ensure reliable delivery of messages when the system fails.
5. Cross-platform and cross-language : MOM generally supports multiple programming languages ​​and platforms, and can easily realize cross-platform and cross-language communication.

shortcoming:

1. Performance overhead : MOM introduces additional overhead such as message transmission, serialization, and deserialization, which may affect system performance.
2. Complexity : Using MOM requires learning and understanding related concepts, techniques and tools, which may increase the complexity of the system.
3. Availability : MOM usually depends on the external message broker service. If the message broker service fails, the availability of the system may be affected.

Design ideas:

1. Select the appropriate communication mode : According to the needs of the system, select the appropriate communication mode, such as point-to-point (P2P) or publish/subscribe (pub/sub) mode.
2. Guarantee reliable delivery of messages : Use mechanisms such as persistence, message confirmation and retry to ensure reliable delivery of messages in the system.
3. Improve the performance of message processing : Use message batch processing, parallel processing, and asynchronous processing technologies to improve the system's message processing performance.
4. Design a reasonable message format : According to the requirements of the system, design a reasonable message format, such as using JSON, XML or binary format.

Precautions:

1. Ensure the idempotence of messages : When processing messages in the system, it is necessary to ensure the idempotence of messages to prevent data inconsistency caused by repeated messages.
2. Monitor and manage message queues : monitor and manage message queues to discover and solve problems in the system in a timely manner.
3. Optimize the performance and availability of the message broker service : According to the requirements of the system, optimize the performance and availability of the message broker service, such as using technologies such as clustering, partitioning and backup.
   Reasonable message format, such as using JSON, XML or binary format.