Overview
Scenes
- Service decoupling
- Peak-cutting
- Asynchronous buffering: eventual consistency / flexible transactions
MQ application thinking
- Production reliability delivery
- Consumer idempotent: messages can only be consumed once
- High availability, low latency, reliability
- Message stacking capacity
- Scalability
Industry mainstream MQ
- ActiveMQ: suitable for traditional needs, poor concurrency
- RabbitMQ: Poor scalability
- RocketMQ: Strong scalability
- Kafka: Strong scalability, strong concurrency, and poor reliability
Technical selection
- Performance, advantages and disadvantages, business scenarios
- Cluster architecture mode, distributed, scalable, high availability, maintainability
- Comprehensive cost, cluster size, personnel cost
- Future direction, planning, thinking
ActiveMQ
Introduction
- JMS (Java Message Service): Java Message Service, which defines the interface specification for accessing message middleware in Java
- The middleware that implements JMS is called "JMS Provider"
- MOM (Message Oriented Middleware): ActiveMQ, RocketMQ, RabbitMQ, Kafka
the term
- JMS (Java Message Service): Java message service interface
- Provider (MessageProvider): the producer of the message
- Consumer (MessageConsumer): the consumer of the message
- PTP (Point to Point): the point-to-point message model, which is also a very classic model
- Pub / Sub (Publish / Subscribe): the publish / subscribe message model
- Queue: The queue target, which is the message queue that we often say, is generally physically stored.
- Topic: Topic goal
- ConnectionFactory: connection factory, JMS uses it to create connections
- Connection: JMS client to JMS Provider connection
- Destination: the destination of the message
- Session: session, a thread that sends or receives messages (here Session can be compared to Mybatis Session)
Message definition format
- StreamMessage: the data stream of the original value
- MapMessage: a set of name / value pairs
- TextMessage: a string object
- BytesMessage: a data stream of uninterpreted bytes
- ObjectMessage: a serialized Java object
Message delivery mode
- Peer-to-peer: Producers deliver messages to the queue, and only one consumer can listen to this message
- Publish and subscribe: The producer posts a message to the queue, and all consumers listening to the queue can listen to the message
ActiveMQ indicators
- Service performance: suitable for traditional industry needs, insufficient support for high concurrency and big data business scenarios
- Data storage: The default is kahadb storage (index-file), which can be google leveldb (memory database) or MySql / Oracle (relational database)
- Cluster architecture: can build an active and standby cluster model with zookeeper
Cluster architecture pattern
- Master-Slave: master-slave mode, dual hot standby
- ActiveMQ currently recommends high-reliability and fault-tolerant solutions
- The green node is the main node, and the gray node is the backup node. Both nodes are running.
- The role of zookeeper: when the master node is down, it switches to the backup gray node in time, and the master and slave roles are exchanged to achieve high availability
- Defects: Unable to achieve distributed topics and queues. MQ cluster pressure is too large when the amount of messages is huge
- Network: Network communication method (Network of brokers)
- Really solved the problems of distributed message storage and failover, broker switching
- A broker will treat all subscriptions equally
- Two or more sets of (Master-Slave) cluster models are required
- Defects: waste of resources and complex deployment
1 <broker brokerName="receiver" persistent="false" useJmx="false"> 2 <transportConnectors> 3 <transportConnector uri="tcp://localhost:62002"/> 4 </transportConnectors> 5 <networkConnectors> 6 <networkConnector 7 uri="static:( tcp://localhost:61616,tcp://remotehost:61616)"/> 8 </networkConnectors> 9 </broker>
RocketMQ
Introduction
- High concurrency, high reliability, massive data scenarios
- The underlying communication framework uses Netty NIO
- Use NameServer instead of Zookeeper
- Support cluster, load balancing, horizontal expansion
- Smart copy, sequential write, random read
- Message failure retry mechanism, message can be queried
the term
- Producer: Message producer, responsible for generating messages, generally the business system is responsible for generating messages
- Consumer: Message consumer, responsible for consuming messages, generally the background system is responsible for asynchronous consumption
- Push Consumer: A type of Consumer that needs to register and listen to Consumer objects
- Pull Consumer: A type of Consumer that needs to actively request Broker to pull messages
- Producer Group: Producer set, generally used to send a type of message
- Consumer Group: consumer group, generally used to receive a type of message for consumption
- Broker: MQ message service (transit role, used for message storage and production and consumption forwarding)
Cluster architecture
- Single point mode
- Master-slave mode
- Dual master mode
- Dual master dual slave mode
RabbitMQ
Overview
- Open source message broker software (also known as message-oriented middleware) that implements Advanced Message Queuing Protocol (AMQP)
Cluster architecture
- Master / standby mode: warren (rabbit den), the master node hangs up, the slave node provides services, similar to ActiveMQ using Zookeeper as master / standby
- Remote mode: Long-distance communication and replication, realizing HyperMetro mode, referred to as Shovel mode, with complicated configuration
- Mirror mode: high availability, data synchronization, simple implementation, three-node, similar to replicate of MongoDB
- Multi-active mode: A dual-center model is adopted. A RabbitMQ cluster is deployed in each of the two data centers, and some queue messages are shared among the centers
- Federation plug-in: a high-performance plug-in for passing messages between brokers without the need to build a cluster, using the AMQP protocol, which can accept discontinuous transmission
- Federation Exchange: Downstream actively pulls subscription messages from Upstream
Kafka
Introduction
- Pull-based mode handles message consumption and pursues high throughput. The initial purpose is for log collection and transmission
- Supports replication, does not support transactions, and has no strict requirements for message duplication, errors, or loss
- Data collection services suitable for Internet services that generate large amounts of data
- Distributed, cross-platform, real-time, scalable
performance
- Write in sequence
- Improve disk utilization
- Consumer consumes data sequentially through offset without deleting already consumed data, thus avoiding random write
- Page Cache (Air Relay)
- Does not use memory explicitly, and restart data is not lost
- High performance, high throughput
- Asynchronous and active Flush in the background
- Multiple asynchronous levels of schedulers, which combine consecutive small blocks into large physical files
- Read-ahead strategy, IO scheduling
Cache Page
- A major disk storage strategy implemented at the operating system level
- Cache data from disk to memory, reduce IO operations (four copies)
- Highly concurrent Internet project: MySQL-> Sub-database sub-table-> MongoDB-> Redis-> Local cache
ZeroCopy
- Not associated with the application (one copy)
- After copying from the disk to the memory read buffer, directly send the data to the network card interface for consumer use
- The application does not make a copy
- If there are 10 consumers, the traditional file reading and writing needs 40 IO operations, and ZeroCopy only needs 1 + 10 times
Cluster
