Foreword
Compiled some RocketMQ related flowchart / diagram, do some notes, we will study together.
What is RocketMQ
- Queuing model is a messaging middleware, high performance, high reliability, high real-time, distributed nature.
- Producer, Consumer, queues can be distributed.
- Producer turn sends some messages queue, called Topic set of queues, if done Consumer broadcast consumer, the consumer is a consumer of this example all queues corresponding Topic, if done cluster consumption, average consumption of the plurality of instances of this topic Consumer corresponding set of queues .
- To ensure strict message ordering
- Provide rich information pull model
- Efficient subscriber level scalability
- Real-time news subscribe mechanism
- One hundred million messages accumulation capacity
- Less dependent on
RocketMQ core component diagram
RocketMQ messaging middleware is open source, which mainly consists of four parts NameServer, Producer, Broker, Consumer.
NameServer
Topic NameServer responsible for managing and routing information, similar to Dubbo function of zookeeper.
Producer
News producer, responsible for producing the news, the general in charge of the business system generated messages.
Broker
The message transit role, responsible for storing messages, forward messages.
Consumer
Consumer news, responsible for news consumption is generally responsible for back office systems asynchronous consumption.
RokcetMQ physical deployment diagram
NameServer
NameServer node status is a little, no information can be synchronized between cluster deployment, nodes.
Broker
Broker into Master and Slave, Master may correspond to a plurality of Slave, a Slave but corresponds to only one Master, Master and Slave corresponding relationship specified by the same BrokerName, different BrokerId defined, BrokerId 0 indicates Master, the non-0 He expressed Slave. Master can also deploy multiple. Each Broker Name Server build all nodes in the cluster long connection, the timing information to all registered Topic Name Server.
Producer
Producer Name Server and wherein a node cluster (random selection) to establish long connection route taken regularly Topic Name Server information, and provide long Topic Service Master established connection, and the timer sends a heartbeat to the Master. Producer completely stateless, can be clustered deployment.
Consumer
Consumer Name Server cluster with one of the nodes (randomly selected) long connection established, from the routing information taken periodically Topic Name Server, and to provide services Topic Master, Slave long established connections, and the timing of sending a heartbeat to the Master, Slave. Consumer can either subscribe to the Master, you can subscribe to news from the Slave, subscription rules by the Broker configuration decisions.
RocketMQ logical deployment structure
Producer Group
Is used to indicate a message transmission application, the plurality comprising a Group Producer Producer instances, may be a multiple machine, multiple processes may be a machine, or a plurality of process objects Producer. A Producer Group may send multiple messages Topic, Producer Group effects include:
- Identifies a class Producer
- Producer plurality of instances may send the query message application by the operation and maintenance tool
- When sending a message distributed transaction, if the middle of Producer unplanned downtime, Broker will take the initiative callback any machine in the Producer Group to confirm the state of affairs.
Consumer Group
Message is used to indicate a consumer application, comprising a plurality of Consumer instances the Consumer Group, may be multiple machines, may be a plurality of processes, or a plurality of objects Consumer process. Consumer Group under a plurality of Consumer consumption in a manner shared equally message, if a broadcast mode is set, in each instance the total amount of consumption are Consumer Group data.
NameServer Routing Registry, delete mechanism
- Broker to send a heartbeat packet every 30 seconds NameServer, the heartbeat packet contains routing information to topic
- BrokerLiveTable update information, in particular recorded heartbeat lastUpdateTime Broker heartbeat packet received after NarneServer
- NarneServer scanning every 10s brokerLiveTable, the table was last detected heartbeat packets received time, compare the current time and the last time exceeds 120s, that the broker is not available to remove all the information associated with the routing table of the broker
- News producer pulling the routing information about the topics that message producer can not detect and delete the new Broker server immediately.
FIG RocketMQ message domain model
Topic
- Message Topic message indicating that the first stage of type, such as an electricity supplier system can be divided into: news transaction, logistics and other news. A message must have a Topic.
- The most fine-grained subscription unit, a Group Topic can subscribe to multiple messages.
Tag
Tag represents the second stage of the message type, such as transaction messages can be divided into: create a message, the transaction complete messages. Level 2 message classification RocketMQ provide convenient and flexible control.
Group
Group, a group can subscribe to multiple Topic.
Message Queue
Physical management unit message. There can be multiple next Topic Queue, Queue introduction of such messages can be stored in a distributed cluster, with the horizontal scalability.
In RocketMQ, all are persistent message queues, data structures of infinite length, the so-called infinite length refers to each memory cell in the queue is a fixed length, wherein the memory access unit uses to access Offset, offset type as java long , 64, in theory, will not overflow in 100 years, it is considered to be of infinite length.
Message Queue can also be considered an infinite length of the array, Offset is a subscript.
Message sequence diagram
Consumption order consistent with the order of messages to be sent the message, the RocketMQ, the primary partial sequence, i.e., a type of message to satisfy the sequential, single-threaded Producer order must be sent, and sent to the same queue, so that it can follow the Consumer Producer order to send and receive messages to the consumer.
RocketMQ message store design schematics
CommitLog
Message storage files, all messages are stored in the theme of the message CommitLog file. File storage logical view shown in FIG. Commitlog
ConsumeQueue
After the news consumer queue, the message reaches CommitLog file, the message will be forwarded to the consumer asynchronous queue for the message consumer spending. ConsumeQueue storage format as follows:
- ConsumeQueue single file contains the default entries 300 000, the length of a single file is 30w × 20 bytes, a single file can be seen to ConsumeQueue ConsumeQueue array entry, which is the index of the logical offset ConsumeQueue, message storage consumption schedule i.e. offset a logical offset.
- ConsumeQueue Commitlog file is the index file, which mechanism is constructed when the message arrives Commitlog file, generate a message forwarded by a dedicated task threads, thereby constructing the index file and the message queue file consumption mentioned below.
IndexFile
Message index file, correspondence between the main storage of messages Key and Offset.
Message queues are RocketMQ specifically for consumer news Subscribe to build an index file, increase the speed according to the subject and message queue to retrieve messages, in addition RocketMQ introduced Hash indexing mechanism to index the message, HashMap design consists of two basic points: Hash Hash groove list structure conflict. RocketMQ index file layout shown in FIG.
lndexFile contains a total lndexHeader, Hash groove, Hash entries
State Service Affairs
Transaction state store each message.
Timing message service
Each level corresponds to a delay of a message queue consumption, delay queue stores messages pulling progress.
RMQ file storage model layer
RocketMQ business layer processor
Broker end of the message service logic of reads and writes the inlet, which contains the business logic layer mainly related to process operation (operation to distinguish a specific service type based on the analysis of RemotingCommand requestCode, to thereby carry out various business processes), For example the front of the check and validation step, the object configured MessageExtBrokerInner, decode deserialized, configured Response returned objects.
RocketMQ data storage layer component
- This layer is RocketMQ storage core classes -DefaultMessageStore, access to the data file for which the entrance RocketMQ message, message log data stored in the file read and write CommitLog accomplished by class "putMessage ()" and "getMessage ()" method operation (specifically, operation of read and write access is still dependent on the method in the next layer of the object model provides CommitLog);
- In addition, the component initialization time, will start a lot of storage-related services background threads, including AllocateMappedFileService (MappedFile pre-allocated service thread), ReputMessageService (replay stored message service thread), HAService (Broker master-slave synchronization service availability thread), StoreStatsService (message storage service thread count), IndexService (index file server threads) and so on.
RocketMQ logical object storage layer
- This layer contains three model classes IndexFile RocketMQ data file is stored directly related, ConsumerQueue and CommitLog.
- IndexFile provide index data file access services, ConsumerQueue provide access to services for the logical message queue, CommitLog log data file was stored messages provide access to services.
- The three model classes are constituted RocketMQ overall configuration of the storage layer.
Memory mapped file package layer
- JDK NIO RocketMQ mainly in two ways FileChannel MappedByteBuffer and to read and write data files.
- Wherein, using memory-mapped MappedByteBuffer this manner to complete disk file read and write large files, it will be packaged into MappedFile class in the class RocketMQ.
- Here, a single file for each type of read and write operations provided by the service class MappedFile (wherein, MappedFile class provides a sequential write / random read, the brush disc memory data, memory scrubbing and other documents related services).
Disk storage layer
Mainly it refers to deploy disk servers RocketMQ used. Here, we consider the different disk types (e.g., normal or SSD HDD) disk characteristics and performance parameters (e.g. IOPS, throughput and access delay and other indicators) Effect on the sequential write / random read brought.
RocketMQ message brush disc
In RocketMQ message brush disc brush disc can be divided into synchronous and asynchronous two kinds of the brush disc.
Synchronization brush plate
- When writing a successful return to the state, the message has been written to disk.
- Specific process is that after the message is written to the memory of PAGECACHE, immediately notify brush disk brush thread disk, and then wait for the disk to complete the brush, waiting threads wake brush plate thread execution is completed, a return message write success state.
- Generally used only for financial scene.
Asynchronous brush plate
Message flow in the system of FIG.
1.Producer send messages, a message from the socket into the java heap.
2.Producer send a message, the message transferred PAGACACHE, java physical memory from the heap.
3.Producer message transmitted by the asynchronous thread brush plate, the message from the brush PAGECACHE disk.
4.Consumer Pull message (normal consumption), directly from PageCache message (data in physical memory) into the socket, Consumer reach, without java heap. This scenario most consumption, physical memory line 96G according 1K operator message, the message can be 100 million in physical memory cache.
5.Consumer pull messages (abnormal consumption), messages directly from PAGECACHE (data in virtual memory) into the socket.
6.Consumer pull messages (abnormal consumption), due Socket access virtual memory, resulting in a page fault, this time will produce disk IO, Disk Load message to PAGECACHE, and then sent out directly from the socket.
7. the 5 match.
8. same 6 match.
Reference and thanks
- Ten minutes Getting RocketMQ
- Distributed messaging Series: Detailed RocketMQ the introduction and evolution, architecture design, key features and application scenarios
- RocketMQ message store
- "RocketMQ principle Introduction"
Personal Public Number
Welcome to the concern, we will study together, discuss together.