Definition:
Message middleware refers to the container that saves messages during message transmission.
Message middleware acts as a middleman when relaying messages from its source to its destination. The main purpose of
message middleware is to provide routing and ensure message delivery. Delivery, if the receiver is unavailable when the message is sent, the message queue will retain the message until it can be successfully delivered. Of course, the message queue also has a time limit for saving the message.
Features:
Asynchronous processing mode: the message sender can send a message without waiting for a response, the message sender sends the message to a channel, the message receiver subscribes or listens to the channel
, and a message is finally forwarded to one or more Message receivers, these receivers do not need to respond synchronously to the message sender, the whole process is asynchronous, and the
application sequence is loosely coupled: the sender and receiver do not need to know each other, only need to confirm the message; the sender and the The receiver does not have to be online at the same time
(the online seller does not need to arrive at the same time in order to ensure the final consistency of the data, that is, the A state to the B state, and the consistent state can be finally reached after a certain period of time. After the payment system is successful,
the order system is notified to change the order status. , you can use message middleware)
The delivery model of message middleware:
Durable subscriptions will store relationships. Durable subscriptions are used in most cases
In the modern website technology stack, MySQL is the most common database management system. We will extract data from multiple different MySQL instances, store it in a central node,
or directly into Hive. There are many mature extraction software based on SQL query on the market, such as the famous open source project Apache Sqoop,
but these tools do not support real-time data extraction. MySQL Binlog is a real-time data stream for data replication between master and slave nodes, and
we can use it for data extraction. With Alibaba's open source Canal project, we can easily extract data from MySQL to any target storage.
Components of Canal
Simply put, Canal will disguise itself as a MySQL slave node (Slave), and obtain Binlog from the master node (Master), parse and store it for downstream consumers to use.
Canal consists of two components: server and client. The server is responsible for connecting to different MySQL instances and maintaining an event message queue for each instance;
Message middleware refers to the container that saves messages during message transmission.
Message middleware acts as a middleman when relaying messages from its source to its destination. The main purpose of
message middleware is to provide routing and ensure message delivery. Delivery, if the receiver is unavailable when the message is sent, the message queue will retain the message until it can be successfully delivered. Of course, the message queue also has a time limit for saving the message.
Features:
Asynchronous processing mode: the message sender can send a message without waiting for a response, the message sender sends the message to a channel, the message receiver subscribes or listens to the channel
, and a message is finally forwarded to one or more Message receivers, these receivers do not need to respond synchronously to the message sender, the whole process is asynchronous, and the
application sequence is loosely coupled: the sender and receiver do not need to know each other, only need to confirm the message; the sender and the The receiver does not have to be online at the same time
(the online seller does not need to arrive at the same time in order to ensure the final consistency of the data, that is, the A state to the B state, and the consistent state can be finally reached after a certain period of time. After the payment system is successful,
the order system is notified to change the order status. , you can use message middleware)
The delivery model of message middleware:
Point-to-point model: 1. There is only one consumer per message 2. The sender and the consumer have no time dependence 3. The receiver confirms that the message is accepted and processed successfully
Durable subscriptions will store relationships. Durable subscriptions are used in most cases
Application scenarios of Internet message middleware:
Referring to the above scenario, we have to say that a middleware can also achieve the above requirements:
Data extraction is the first step in the ETL process. We will extract data from external systems such as RDBMS or log servers to the data warehouse, and perform operations such as cleaning, transformation, and aggregation.In the modern website technology stack, MySQL is the most common database management system. We will extract data from multiple different MySQL instances, store it in a central node,
or directly into Hive. There are many mature extraction software based on SQL query on the market, such as the famous open source project Apache Sqoop,
but these tools do not support real-time data extraction. MySQL Binlog is a real-time data stream for data replication between master and slave nodes, and
we can use it for data extraction. With Alibaba's open source Canal project, we can easily extract data from MySQL to any target storage.
Components of Canal
Simply put, Canal will disguise itself as a MySQL slave node (Slave), and obtain Binlog from the master node (Master), parse and store it for downstream consumers to use.
Canal consists of two components: server and client. The server is responsible for connecting to different MySQL instances and maintaining an event message queue for each instance;
Clients can subscribe to data change events in these queues, process and store them in the data warehouse. Let's see how to quickly build a Canal service.
