Message Queue (MQ), an application communication method of application. Data for application of application access and retrieve queued by writing (messages) to communicate, most applications can be increased via the messaging service middleware asynchronous communication, the ability to expand decoupled
messaging services in two key concepts:
1. message broker (message broker) and destination (Where do you want)
2. when the message sender to send a message, the message will take over the agency, the message agent to ensure message delivery to a specified destination
messaging is not a remote procedure call directly, but through a message queue
A messaging protocol and common
JMS (Java MessageService) refers JMS API is the most successful one asynchronous messaging technology, java based applications to provide a unified message JVM specification message operations, including Create, Send, the receive
the JMS and JDBC similar role as the user interface are in accordance with the respective service and implements the JMS communication, related operations
JMS comprises a provider (JMS messaging middleware implements interface), producer / publisher, consumer / subscriber, message, queue, topic, etc.
from the producer end by an end consumer to achieve the routing message (message-routing), linked by a producer and a consumer to the same queue (p2p) or topic (pub / sub) to achieve routing messages. JMSconsumer selector support Message (message selector) by a message selector, consumer can consume only those filtered by the message selector
message routing mechanism: producer -> queue / topics - > consumer
provides two models message, peer-2- peer (point to point), publish-subscribe (publish and subscribe) model .
When the point to point model, a message will be sent to a queue, the queue of messages can only be consumed by a consumer.
When you publish subscribe model, messages can be consumed more consumers. Publish subscribe model, completely independent producers and consumers, does not need to sense the presence of each other's
main features:
Java platform for the standard message passing API
Queues and topics two message passing model
supports transactions
can define message formats (message header, properties and content)
portions have achieved JMS API, you can assume the role of JMS provider, such as ActiveMQ, Redis, HornetMQ Wait
AMQP
AMQP (advanced the Message Queuing Protocol) Advanced Message Queuing Protocol, is a message broker specification-compliant JMS
AMQP is a protocol is a binary wire-level protocol (link protocol). This is the essence of their difference and JMS, AMQP not defining the API layer, but directly define data formats exchanged on the network
so as to achieve a natural provider of AMQP is cross-platform. You can use the Java AMQP provider, AQMP can use http to draw an analogy, do not care about language, as long as we have to send a message in accordance with the corresponding data format of the request, a different language client can be and server links different languages
AMQP message routing (messagerouting) and JMS there are some differences, increase the Exchange and binding role in the AMQP. producer sends a message to the Exchange, binding decision of the Exchange message should be sent to the queue, and the consumer directly from the message queue in consumption. queue and exchange of consumer bind have decided
producer -> exchange -> binding - > queue -> consumer
news producers and news consumers need to know the other side of the Queue, the message producer through the Route key and any Exchange Queue binding. Route key news consumers get from any Exchange Queue by
offers five models Message: direct exchange, fanout exchange, topic change, headers exchange, system exchange ( the latter four and JMS pub / sub model is not much difference, only It is on the routing mechanism to do more detailed breakdown)
Key Features:
Platform-independent underlying messaging protocol
cross-platform interoperability and language, the underlying protocols
There are five types of switching direct, fanout, topic, headers, system
cache oriented, high performance can
support transactions (cross message queue) to support distributed transactions (XA, the X-/ the OPEN, MS DTC)
AMQP model: publish -> sorting -> Jobs -> consumption -> confirm receipt the MQTT
RabbitMQ AMQP used to achieve
MQTT
for all other application requirements based on the AMQP and JMS for Java enterprise applications, MQTT designed specifically for small devices
can not adapt to complex operations on the AMQP is not high performance computing equipment, they need a simple and interoperable way communicate. This is the basic requirement MQTT, now MQTT is a major component of Things (IOT) ecosystems
main features:
stream-oriented, low memory usage, to transmit short messages between a small silent low bandwidth devices designed
not support long period store and forward, does not support transactions
Second, the composition of the intermediate
base component
Broker: message queue server entity
producer / Publisher: producers message
consumer: consumers message
topic: dissemination of information topics under a subscription model
queue: PTP mode, sending a message to a particular producer specific queue, consumers subscribe to specific queue specified completed message is received
message: a message header and a message body
RabbitMQ
connection: Fi
exchange: switch to receive messages to route the messages in the server queue (there are four types in: direct (default), fanout, topic, headers) different types of Exchange forwards the message strategy has distinguished
binding: binding for the association between the message queue and switches (that is, based on a binding route the key switch and connect message queue routing rules)
Channle: multiplexing a separate TCP connection multiplexing a bidirectional data connection flow path, regardless of publish message subscription queue, receives the message operation are completed through these channels, reducing overhead created and destroyed TCP
Three, AMQP message routing in Exchange
AMQP routing process messages and Java developers are familiar with JMS there are some differences, AMQP and Binding Exchange increased role, the producers send messages to the Exchange
depending on the type of message distribution when Exchange distribution strategies differ, there are four types: Direct, fanout, Topic, headers
Direct:
message routing keys (routing key) and if the Binding bingding key agreement, the switch will send a message to a corresponding queue , and the queue name routing keys exact match
fanout:
fanout switch does not process routing keys, simply bind the queue up to the switch, each switch type sent to the fanout messages are assigned to all queues bound up,
All queues for each message sent to the exchanger will be forwarded to the switch binding
Topic:
matching attribute assignment message routing keys, two wildcard: the symbol "#" and the symbol "*." # Matches zero or more words to match a word *
Fourth, the advantage of messaging middleware
1. Decoupling: messaging between systems and the system does not directly call relations, expansion of the system do not change the interface. Reduce dependency between projects
2. asynchronous communication: a logical coordinate system of N simultaneous requests to complete the service. By calling a parallel business to improve system response time
3. Overload protection: access to the system is the system peak value of the business can not function properly, and then slowly pre-existing queue buffer processing service pressure, flow clipping
Drawback
1. Reducing availability: The more dependent on external systems introduced, the more easy to hang, you had A system call interface is the three BCD system just fine, and now join a mq, mq hung up in case the whole the system will collapse
complexity 2. increase the system: double-spending problem, the problem message is lost, the order of questions messaging
3. consistency: set of business processes ABC system in which a system failure data inconsistencies
Five, mq message is lost
producers lost:
1. Transaction way: the producer before sending a message, open a transaction channel.txSelect, then send the message. The RabbitMQ message is not successfully received, the transaction for rollback channel.txRollback, then re-transmitted, and vice versa commit the transaction channel.txCommit (producer throughput and performance will be much reduced)
2.Confirm mechanism (send a receipt acknowledgment): Confirm mode is set in the producers, it is time to write each message is assigned a unique ID, and then will return an ACK after receipt of RabbitMQ, told the news producer OK. RabbitMQ if not dealt with this message, then there is a callback interface Nack, this time the producer can resend
transaction mechanism are synchronized, after submitting a transaction is blocked there.
Confirm mechanism is asynchronous, the next can be sent after sending a message a message and that message RabbitMQ After receiving asynchronously callback you an interface to inform you that this news has been received
rabbitmq data loss:
preventing rabbitmq own lost data, to be open rabbitmq persistence, after the message is written to be persisted to disk, even if it is rabbitmq himself hung up the data previously stored will automatically read after recovery, general data will not be lost
rare is, rabbitmq not persistent, and that they hung up, could result in a small amount of data will be lost, but the probability is small
persistence + confirm the mechanism with producer, only after the message was to disk, will inform the producer ack persistence, so even in the persistence to prior to the disk, RabbitMQ hung up, lost data, the producer can not receive ACK, but also their possible retransmission
(deliveryMode to 2)
Consumer data loss :
consumption of time, just to consume, not treatment, such as the results of the process of hanging the restart, RabbitMQ think you are a consumer, this data is lost
ack mechanism provided with RabbitMQ, if the consumer has not received end ack in response to the consumer assigned to the other end of
six, assurance idempotent (repeated consumption)
duplication consumption issues such as the consumer side is not consumed mq ack hung up on the consumer side is repeatedly transmitted after the current starts repeated consumption.
Solution:
1. Producers send messages, each adding a globally unique id, then the time of consumption, save the id to redis inside. Redis go inside to check when the consumer about what there is there, no longer consume
2 database operations can set a unique key to prevent duplicate data insertion
3. Write redis that there is no problem, every set, natural idempotency
Seven, the order of consumption
as producers had two messages: M1, M2, to ensure that the order of these two messages
in a simple way is to send M1, M2 to the same Server, the producers sent after the success of waiting M1 send M2, according to the first to be reached first principle of consumption, M1 to M2 will first be consumed, thus ensuring the order of the messages
time-consuming if the sending time-consuming than the transmission M2 M1, then M2 will remain to be the first consumer, it can not in order to ensure that the message is determined to succeed recurrence M1 M2, retry mechanism
consistent message sequence when the same order in short time message is stored and kept sending, the message is stored and maintained consumption of
eight, the message consistency
to order Creating an example: generate orders, process orders such as order processing is successful, the reasons or due to network downtime MQ, the message was not sent party status unchanged
1. select transactional messaging middleware, to ensure that the messages generated by local transactions the final success of the consistency of the message sent successfully
pre-transaction message delivery mechanism is actually similar to the two-stage submission, the only difference is that the consumer side such as consumer Unsuccessful will not trigger a rollback because the message queues persistent data through MQ retry mechanism until successful consumer
2. local news
in the implementation of business operations, recorded a message data to the DB, and message data and business records recording data must be completed within the same transaction
after the completion of the recording of the message data, we can later by a timed task in rotation to the DB status message to be sent, then the message will be delivered to the MQ
Nine, high availability