RocketMQ --- Principles

1. Terminology

• Producer

  • Message producers are responsible for generating messages, and generally business systems are responsible for generating messages.

• Consumer

  • Message consumers are responsible for consuming messages, usually the background system is responsible for asynchronous consumption.

• Push Consumer

  • A type of Consumer. The application usually registers a Listener interface with the Consumer object. Once a message is received, the Consumer object immediately calls back the method of the Listener interface.

• Pull Consumer

  • A type of Consumer, the application usually actively calls the Consumer's message pull method to pull messages from the Broker, and the initiative is controlled by the application.

• Producer Group

  • A collection name of a type of Producer. This type of Producer usually sends a type of message, and the sending logic is consistent.

• Consumer Group

  • The collection name of a class of Consumers, which usually consume a class of messages, and the consumption logic is consistent.

• Broker

  • Message transfer role, responsible for storing and forwarding messages, generally also called Server. Called Provider in the JMS specification.

• broadcast consumption

  • A message is consumed by multiple consumers. Even if these consumers belong to the same consumer group, the message will be consumed once by each consumer in the consumer group. The concept of consumer group in broadcast consumption can be considered meaningless in terms of message division.
  • In the CORBA Notification specification, all consumption methods belong to broadcast consumption.
  • In the JMS specification, equivalent to the JMS publish/subscribe model

• cluster consumption

  • Consumer instances in a Consumer Group share consumption messages equally. For example, if a Topic has 9 messages, and a Consumer Group has 3 instances (maybe 3 processes, or 3 machines), then each instance only consumes 3 of the messages.

  • In the CORBA Notification specification, there is no such consumption method.

  • In the JMS specification, the JMS point-to-point model is similar, but RocketMQ's cluster consumption function is much equal to the PTP model.

  • Because consumers in a single RocketMQ Consumer Group are similar to PTP, but a Topic/Queue can be consumed by multiple Consumer Groups.

• sequential message

  • The order of consuming messages must be consistent with the order of sending messages. In RocketMQ, it mainly refers to the local order, that is, in order to satisfy the order, a class of messages must be sent sequentially by the Producer in a single thread and sent to the same queue, so that the Consumer can Consume messages in the order sent by Producer.

• normal sequence message

  • A type of sequential message, which can guarantee complete sequential messages under normal circumstances, but once a communication exception occurs and Broker restarts, due to changes in the total number of queues, the queues positioned after hash modulo will change, resulting in short-term message sequence inconsistencies.
  • If the business can tolerate short-term out-of-order messages under cluster abnormal conditions (such as a Broker downtime or restart), it is more appropriate to use the normal order method.

• strict order messages

  • A kind of sequential message, which can guarantee the sequence regardless of normal and abnormal conditions, but sacrifices the distributed failover feature, that is, as long as one machine in the Broker cluster is unavailable, the entire cluster will be unavailable, and service availability will be greatly reduced.
  • If the server is deployed in synchronous dual-write mode, this defect can be avoided by automatically switching the standby server to the primary server, but the service will still be unavailable for a few minutes. (Relying on synchronous double writing, automatic switching between active and standby, the automatic switching function has not yet been realized)
  • At present, the only known application is the database binlog synchronization that strongly relies on strict order messages. Most other applications can tolerate short-term out-of-order messages. It is recommended to use ordinary order messages.

• Message Queue

  • In RocketMQ, all message queues are persistent data structures with unlimited length. The so-called unlimited length means that each storage unit in the queue is of fixed length, and the storage unit in it is accessed using Offset, which is a java long type. , 64 bits, theoretically it will not overflow within 100 years, so it is considered to be infinite in length. In addition, only the data of the last few days is saved in the queue, and the previous data will be deleted according to the expiration time.
  • It can also be considered that Message Queue is an array of unlimited length, and offset is the subscript.

2. What problems do message middleware need to solve?

2.1、Publish/Subscribe

Publishing and subscribing is the most basic function of message middleware, and it is also relative to traditional RPC communication.

4.2、Message Priority

The priority described in the specification means that in a message queue, each message has a different priority, which is generally described by an integer. The message with a higher priority is delivered first. If the message is completely in a memory queue, then the message is delivered You can sort them by priority, so that the ones with higher priority will be delivered first.

Since all RocketMQ messages are persistent, if they are sorted by priority, the overhead will be very high, so RocketMQ does not specifically support message priority, but similar functions can be achieved in a flexible way, that is, a separate configuration with a high priority Queue, and a normal priority queue, just send different priorities to different queues.

For the priority problem, it can be summarized into 2 categories

• As long as the purpose of priority is achieved, it is not strictly a priority, and the priority is usually divided into high, medium, low, or a few more levels. Each priority can be represented by a different topic. When sending a message, specify a different topic to represent the priority. This method can solve most of the priority problems, but compromises the accuracy of the business priority.
• Strict priority, the priority is represented by an integer, such as 0 ~ 65535, this kind of priority problem is generally not suitable to use different topics to solve. If MQ is to solve this problem, it will have a very large impact on the performance of MQ. One thing to make sure here is whether this kind of strict priority is really needed in business. If the priority is compressed into a few, how much impact will it have on the business?

2.3、Message Order

Message order means that when a class of messages are consumed, they can be consumed in the order in which they are sent. For example: an order generates 3 messages, which are order creation, order payment, and order completion. When consuming, it only makes sense to consume in this order. But at the same time, orders can be consumed in parallel.

RocketMQ can strictly guarantee the order of messages.

2.4、Message Filter

• Broker-side message filtering

  • In the Broker, filtering according to the requirements of the Consumer has the advantage of reducing the network transmission of useless messages for the Consumer. The disadvantage is that it increases the burden on Broker, and the implementation is relatively complicated.
  • Taobao Notify supports a variety of filtering methods, including filtering directly by message type and flexible syntax expression filtering, which can almost meet the most demanding filtering needs.
  • Taobao RocketMQ supports filtering by simple Message Tag, and also supports filtering by Message Header and body.
  • Flexible syntax expression filtering is also supported in the CORBA Notification specification.

• Consumer-side message filtering

  • This filtering method can be fully customized by the application, but the disadvantage is that many useless messages need to be transmitted to the Consumer side.

2.5、Message Persistence

Several persistence methods commonly used by message middleware:

• (1). Persist to the database, such as Mysql.
• (2). Persist to KV storage, such as levelDB, Berkeley DB and other KV storage systems.
• (3). File record persistence, such as Kafka, RocketMQ
• (4). Make a persistent image of memory data, such as beanstalkd, VisiNotify

(1), (2), and (3) all three persistence methods have the ability to expand the memory queue Buffer, (4) is just a mirror image of the memory, and the function is that when the Broker hangs up and restarts, the previous memory can still be restored. The data is restored.

The JMS and CORBA Notification specifications do not specify how to persist, but the performance of the persistence part directly determines the performance of the entire message middleware.

RocketMQ refers to Kafka's persistence method and makes full use of the memory cache of the Linux file system to improve performance.

2.6、Message Reliablity

Several situations that affect message reliability:

• (1). Broker shutdown normally
• (2). Broker Abnormal Crash
• (3). OS Crash
• (4). The machine is powered off, but the power supply can be restored immediately.
• (5). The machine cannot be turned on (maybe the CPU, motherboard, memory and other key equipment are damaged)
• (6). The disk device is damaged.

The four situations (1), (2), (3), and (4) all belong to the situation that the hardware resources can be recovered immediately. RocketMQ can guarantee that the message will not be lost in these four situations, or a small amount of data will be lost (depending on the method of flashing the disk is synchronous or asynchronous).

(5) and (6) are single-point failures and cannot be recovered. Once they occur, all messages on this single point will be lost. In both cases, RocketMQ can ensure that 99% of messages are not lost through asynchronous replication, but there are still very few messages that may be lost. Single points can be completely avoided through synchronous double writing technology, which will inevitably affect performance, and is suitable for occasions that require extremely high message reliability, such as money-related applications.

RocketMQ has supported synchronous double writing since version 3.0.

2.7、Low Latency Messaging

In the case of no message accumulation, after the message reaches the Broker, it can reach the Consumer immediately.

RocketMQ uses the long polling Pull method to ensure that the message is very real-time, and the real-time performance of the message is not lower than that of Push.

2.8、At least Once

means that each message must be delivered once

RocketMQ Consumer first pulls the message to the local, and then returns ack to the server after the consumption is completed. If there is no consumption, it will not ack the message, so RocketMQ can well support this feature.

2.9、Exactly Only Once

(1). In the stage of sending messages, it is not allowed to send repeated messages.
(2). In the stage of consuming messages, it is not allowed to consume duplicate messages.

Only when the above two conditions are met, can the message be considered "Exactly Only Once", and to realize the above two points, in a distributed system environment, it is inevitable to generate huge overhead. Therefore, in order to pursue high performance, RocketMQ does not guarantee this feature, and requires deduplication in business, that is to say, idempotence must be achieved for consuming messages.

Although RocketMQ cannot strictly guarantee non-repetition, under normal circumstances, repeated sending and consumption rarely occur. Only when the network is abnormal, and the consumer starts and stops and other abnormal situations, messages will be repeated.

The essential reason for this problem is that there is uncertainty in the network call, that is, the third state of neither success nor failure, so the problem of message repetition occurs.

2.10. What should I do if the Broker's Buffer is full?

Broker's Buffer usually refers to the memory buffer size of a queue in Broker. This kind of Buffer is usually limited in size. What if the Buffer is full?

The following is how it is handled in the CORBA Notification specification:

• (1). RejectNewEvents

  • Reject new messages and return RejectNewEvents error code to Producer.

• (2). Discard existing messages according to specific strategies

RocketMQ does not have the concept of memory buffer. RocketMQ's queues are all persistent disks, and the data is cleared regularly.

Regarding the solution to this problem, RocketMQ is very different from other MQs. The memory buffer of RocketMQ is abstracted into a queue of infinite length, no matter how much data comes in, it can hold it. Delete expired data. For example, if Broker only saves messages for 3 days, then although the length of this Buffer is unlimited, the data older than 3 days will be deleted from the end of the queue.

2.11. Backtracking consumption

Retrospective consumption refers to the message that the Consumer has successfully consumed. Due to business needs, it needs to be re-consumed. To support this function, after the Broker delivers the successful message to the Consumer, the message still needs to be retained. And re-consumption is generally based on the time dimension. For example, due to a consumer system failure, it is necessary to re-consume the data 1 hour ago after recovery. Then the Broker needs to provide a mechanism to roll back the consumption progress according to the time dimension.

RocketMQ supports backward consumption according to time. The time dimension is accurate to milliseconds, and it can be traced back forward or backward.

2.12. Message accumulation

The main function of message middleware is asynchronous decoupling. Another important function is to block the front-end data flood and ensure the stability of the back-end system. This requires the message middleware to have a certain ability to accumulate messages. The message accumulation points are as follows: Condition:

• Messages are accumulated in the memory Buffer. Once the memory Buffer is exceeded, messages can be discarded according to a certain discarding policy, as described in the CORBA Notification specification. It is suitable for businesses that can tolerate discarded messages. In this case, the ability to accumulate messages mainly depends on the size of the memory buffer, and after messages are accumulated, the performance degradation will not be too large, because the amount of data in the memory has limited impact on the access capabilities provided to the outside world.

• Messages are accumulated in persistent storage systems, such as DB, KV storage, and file records. When the message cannot be hit in the memory cache, it is inevitable to access the disk, which will generate a large number of read IO, and the throughput of the read IO directly determines the access capability after the message is accumulated.

There are four main points to evaluate message accumulation capabilities:

• How many messages and how many bytes can be accumulated? That is, the accumulation capacity of messages.
• After messages are accumulated, will the throughput of sending messages be affected by the accumulation?
• After messages are piled up, will normal consumers be affected?
• After the messages have accumulated, what is the throughput when accessing the messages accumulated on the disk?

2.13. Distributed transactions

Several known distributed transaction specifications, such as XA, JTA, etc. Among them, the XA specification is widely supported by major database vendors, such as Oracle and Mysql. Among them, the best XA TM implementations, such as Oracle Tuxedo, are widely used in finance, telecommunications and other fields.

Distributed transactions involve a two-phase commit problem. In terms of data storage, KV storage must be supported, because the commit rollback of the second phase needs to modify the message state, which must involve the action of searching for the Message according to the Key. In the second stage, RocketMQ bypasses the problem of finding the message based on the Key. When sending the Prepared message in the first stage, it gets the Offset of the message. In the second stage, it uses the Offset to access the message and modify the state. The Offset is the address of the data.

RocketMQ’s way of implementing transactions does not use KV storage, but uses Offset. There is a significant defect, that is, changing data through Offset will cause too many dirty pages in the system, which requires special attention.

2.14. Timing message

Timed message means that after the message is sent to the Broker, it cannot be consumed by the Consumer immediately. It can only be consumed after a specific time or after a specific time.

If you want to support arbitrary time precision, you must sort messages at the Broker level. If persistence is involved, then message sorting will inevitably generate huge performance overhead.

RocketMQ supports timing messages, but does not support arbitrary time precision, and supports specific levels, such as timing 5s, 10s, 1m, etc.

2.15. Message retry

After the consumer fails to consume the message, it needs to provide a retry mechanism to consume the message again. The failure of Consumer to consume messages can usually be considered as the following situations

• Due to the reasons of the message itself, such as deserialization failure, the message data itself cannot be processed (such as phone charge recharge, the mobile phone number of the current message is canceled and cannot be recharged), etc.

  • This kind of error usually needs to skip this message, and then consume other messages, and even if the failed message is retried immediately, 99% of it will not be successful, so it is best to provide a timing retry mechanism, that is, after 10 seconds Try again.

• Because the dependent downstream application services are unavailable, such as the db connection is unavailable, the external system network is unreachable, etc.

  • When encountering this kind of error, even if the current failed message is skipped, an error will also be reported when consuming other messages. In this case, it is recommended to apply sleep for 30s before consuming the next message, which can reduce the pressure on Broker to retry the message.

3. RocketMQ Overview

3.1. What is RocketMQ?

• It is a queue model message middleware with high performance, high reliability, high real-time and distributed characteristics.
• Producer, Consumer, and queues can all be distributed.
• The Producer sends messages to some queues in turn. The queue set is called a Topic. If a Consumer performs broadcast consumption, one consumer instance consumes all the queues corresponding to this Topic. If it performs cluster consumption, multiple Consumer instances consume the queue set corresponding to this topic on average. .
• Can guarantee strict message order
• Provide rich message pull mode
• Efficient subscriber horizontal expansion capability
• Real-time message subscription mechanism
• Billion-level message accumulation capability
• less dependent

3.2. RocketMQ physical deployment structure

RocketMQ Network Deployment Features

• Name Server is an almost stateless node that can be deployed in a cluster without any information synchronization between nodes.
• Broker deployment is relatively complicated. Broker is divided into Master and Slave. One Master can correspond to multiple Slaves, but one Slave can only correspond to one Master. The corresponding relationship between Master and Slave is defined by specifying the same BrokerName and different BrokerId. BrokerId is 0 It means Master, and non-zero means Slave. Master can also deploy multiple. Each Broker establishes long connections with all nodes in the Name Server cluster, and regularly registers Topic information to all Name Servers.
• Producer establishes a long connection with one of the nodes (randomly selected) in the Name Server cluster, periodically fetches Topic routing information from the Name Server, establishes a long connection to the Master that provides Topic services, and sends heartbeats to the Master regularly. Producer is completely stateless and can be deployed in clusters.
• The Consumer establishes a long connection with one of the nodes in the Name Server cluster (selected at random), periodically fetches Topic routing information from the Name Server, establishes a long connection to the Master and Slave that provide Topic services, and sends heartbeats to the Master and Slave regularly. Consumers can subscribe to messages from the Master or from the Slave, and the subscription rules are determined by the Broker configuration.

3.3. RocketMQ logical deployment structure

Producer Group

• It is used to represent a message sending application. A Producer Group contains multiple Producer instances, which can be multiple machines, multiple processes of one machine, or multiple Producer objects of one process. A Producer Group can send multiple Topic messages, and the functions of the Producer Group are as follows:
  • Identify a class of Producer
  • You can query through the operation and maintenance tool that there are multiple Producer instances under this message sending application
  • When sending a distributed transaction message, if the Producer unexpectedly crashes midway, the Broker will actively call back any machine in the Producer Group to confirm the transaction status.

Consumer Group

• It is used to represent a consumer message application. A Consumer Group contains multiple Consumer instances, which can be multiple machines, multiple processes, or multiple Consumer objects of a process. Multiple Consumers under a Consumer Group consume messages in an amortized manner. If broadcast mode is set, each instance under this Consumer Group consumes the full amount of data.

4. RocketMQ Storage Features

4.1. Principle of zero copy

The process of consumer consuming messages uses zero copy, which includes the following two methods

• Use mmap + write method

  • Advantages: Even if called frequently, using small block file transfer, the efficiency is very high
  • Disadvantages: Cannot make good use of the DMA method, consumes more CPU than sendfile, complicated memory security control, and needs to avoid JVM Crash problems.

• Use the sendfile method

  • Advantages: DMA mode can be used, less CPU is consumed, large file transfer efficiency is high, and there is no new problem of memory security.
  • Disadvantage: The efficiency of small block files is lower than that of mmap, and it can only be transferred in BIO mode, and NIO cannot be used.

RocketMQ chose the first method, mmap+write, because there is a need for small data transmission, and the effect will be better than sendfile.

For a more detailed introduction to Zero Copy, please refer to the following article http://www.linuxjournal.com/article/6345

4.2. File system

RocketMQ chooses the Linux Ext4 file system for the following reasons:

The Ext4 file system usually takes less than 50ms to delete a 1G file, while the Ext3 file system takes about 1s. When deleting a file, the disk IO pressure is extremely high, which will cause the IO write timeout.

The following tuning measures need to be done at the file system level

The IO scheduling algorithm of the file system needs to be adjusted to deadline, because the deadline algorithm can merge read requests into a sequential jump mode in the case of random read, thereby improving the read IO throughput.

4.3. Data storage structure

5. Key features of RocketMQ

5.1. A single machine supports more than 10,000 persistent queues

• All data is stored in a single Commit Log, which is completely sequentially written and read randomly.
• The queue displayed to the end user actually only stores the position information of the message in the Commit Log, and is serially flushed.

The benefits of doing this are as follows:

• The queue is lightweight, and the amount of data in a single queue is very small.
• The access to the disk is serialized to avoid disk contention, and the IOWAIT will not increase due to the increase of the queue.

Each scheme has disadvantages, and its disadvantages are as follows:

• Although writing is completely sequential, reading has become completely random.
• To read a message, the Consume Queue will be read first, and then the Commit Log will be read, which increases overhead.
• To ensure that the Commit Log is completely consistent with the Consume Queue increases the complexity of programming.

How to overcome the above shortcomings:

• For random reading, make the read hit the PAGECACHE as much as possible to reduce IO read operations, so the larger the memory, the better. If there are too many messages accumulated in the system, will the system performance drop sharply due to random reading when accessing the disk to read data? The answer is no.
  • When accessing PAGECACHE, even if only 1k messages are accessed, the system will pre-read more data in advance, and the next time it is read, it may hit the memory.
  • For random access to the Commit Log disk data, the system IO scheduling algorithm is set to NOOP mode, which will change the complete random read into the sequential jump mode to a certain extent, and the sequential jump mode will have more than 5 times higher performance than the complete random read. 4K messages can still achieve a read performance of more than 8K times per second under the condition of complete random access.
• Since the Consume Queue stores very little data and reads sequentially, under the effect of PAGECACHE read-ahead, the read performance of the Consume Queue is almost the same as that of the memory, even in the case of accumulation. So it can be considered that the Consume Queue will not hinder the read performance at all.
• Commit Log stores all meta information, including message body, similar to Mysql and Oracle redolog, so as long as there is Commit Log, Consume Queue can recover even if the data is lost.

5.2. Disk brushing strategy

All RocketMQ messages are persistent. First write to the system PAGECACHE, and then flash the disk, which can ensure that both the memory and the disk have a copy of the data. When accessing, it is directly read from the memory.

5.2.1. Asynchronous brushing

When there is a RAID card, the SAS 15000 RPM disk test writes files sequentially, and the speed can reach about 300M per second, while the online network cards are generally Gigabit network cards, and the disk writing speed is significantly faster than the data network entrance speed, so is it possible to do this? After writing the memory, it returns to the user, and the background thread flushes the disk?

• Since the speed of the disk is greater than the speed of the network card, the progress of flashing the disk must be able to keep up with the speed of writing messages.
• In case the system pressure is too high at this time, messages may pile up, in addition to writing IO, there is also reading IO. In case of disk reading lagging behind, will it cause system memory overflow? The answer is no, the reasons are as follows:
  • When writing a message to PAGECACHE, if there is insufficient memory, try to discard the clean PAGE to free up memory for new messages. The strategy is the LRU method.
  • If there are not enough clean pages, writing to PAGECACHE will be blocked at this time, and the system will try to flush some data, about 32 pages each time, to find more clean pages.

5.2.2. Synchronous brushing

The only difference between synchronous flashing and asynchronous flashing is that the asynchronous flashing returns directly after writing the PAGECACHE, while the synchronous flashing needs to wait for the flashing to complete before returning. The synchronous flashing process is as follows:

• After writing to the PAGECACHE, the thread waits and informs the disk-wishing thread to do a disk-wishing.
• After the disk is wiped, the front-end waiting thread is woken up, which may be a batch of threads.
• The front end waits for the thread to return success to the user.

5.3. Message query

5.3.1. Query messages according to Message Id

MsgId is 16 bytes in total, including message storage host address and message Commit Log offset. Parse the address of Broker and the offset address of Commit Log from MsgId, and then parse it into a complete message according to the location of the message buffer in the storage format.

5.3.2. Query messages according to Message Key

• Get the specific slot position according to the hashcode%slotNum of the queried key (slotNum is the maximum number of slots contained in an index file, for example, slotNum=5000000 as shown in the figure).
• Find the last item in the index item list according to slotValue (the value corresponding to the slot position) (in reverse order, slotValue always points to the latest index item).
• Traverse the list of index items and return the result set within the query time range (by default, a maximum of 32 records can be returned at one time)
• Hash conflict; when looking for the slot position of the key, it is equivalent to executing the hash function twice, one key hash, one key hash value modulo, so there are two conflicts here; first, the key hash value is different But the modulus is the same. At this time, the query will compare the hash value of the key once (each index item saves the hash value of the key), and filter out the items with unequal hash values. The second is that the hash values ​​are equal but the keys are not equal. For performance considerations, conflict detection is handled by the client (the original value of the key is stored in the message file to avoid parsing the data file), and the client compares
  once Whether the key of the message body is the same.
• Storage; in order to save space, the time stored in the index item is the time difference (storage time - start time, the start time is stored in the index file header), the entire index file is of fixed length, and the structure is also fixed.

5.4. Server message filtering

RocketMQ's message filtering method is different from other message middleware. It is to filter when subscribing. Let's first look at the storage structure of Consume Queue.

• To compare Message Tags on the Broker side, traverse the Consume Queue first. If the stored Message Tag does not match the subscribed Message Tag, skip it and continue to compare the next one. If it matches, it will be transmitted to the Consumer. Note: Message Tag is in the form of a string, and the corresponding hashcode is stored in the Consume Queue, and the hashcode is also compared when comparing.
• After the Consumer receives the filtered message, it also needs to perform the operation on the Broker side, but the real Message Tag string is compared instead of the Hashcode.

Why is filtering doing this?

• Message Tag stores Hashcode, which is to save space in the Consume Queue in a fixed-length manner.
• Commit Log data will not be accessed during the filtering process, which can ensure efficient filtering even in the case of accumulation.
• Even if there is a Hash conflict, it can be corrected on the consumer side to ensure nothing goes wrong.

5.5, long polling Pull

The Consumers of RocketMQ all pull messages from the Broker to consume, but in order to receive messages in real time, RocketMQ uses the long polling method to ensure that the message real-time performance is consistent with the Push method. This long polling method is similar to the message sending and receiving mechanism of Web QQ.

Please refer to the following information to learn more: http://www.ibm.com/developerworks/cn/web/wa-lo-comet/

5.6. Sequential messages

5.6.1. Sequential message principle

5.6.2. Sequential Message Defects

• Sending sequential messages cannot take advantage of the cluster FailOver feature
• The parallelism of consuming sequential messages depends on the number of queues
• Queue hotspot problem, individual queues have too many messages due to uneven hashing, and the consumption speed cannot keep up, resulting in message accumulation problems
• Encountered a message of message failure, which cannot be skipped, and the current queue consumption is suspended

5.7, transaction message

5.8. Send message load balancing

As shown in the figure, 5 queues can be deployed on one machine or on 5 different machines respectively. Messages are sent by polling the queues, and each queue receives an average amount of messages. By adding machines, the queue capacity can be scaled horizontally. In addition, you can choose which queue to send to in a custom way.

5.9, subscription message load balancing

As shown in the figure, if there are 5 queues and 2 consumers, then the first consumer consumes 3 queues, and the second consumer consumes 2 queues. In this way, the goal of average consumption can be achieved, and consumers can be expanded horizontally to improve consumption capabilities. However, the number of consumers must be less than or equal to the number of queues. If the number of consumers exceeds the number of queues, the redundant consumers will not be able to consume messages.

number of queues	Number of Consumers	Rebalance result
5	2	C1: 3；C2: 2
6	3	C1: 2；C2: 2；C3: 2
10	20	C1~C10: 1；C11~C20: 0
20	6	C1: 4；C2: 4；c3~C6: 3

5.10, single queue parallel consumption

Single-queue parallel consumption adopts the sliding window method for parallel consumption. As shown in the figure, messages 3 to 7 are within a sliding window interval, and multiple threads can consume in parallel, but the Offset submitted each time is the minimum Offset, for example, 3

5.11. A single JVM process can also use the large memory of the machine

• (1). Producer sends a message, and the message enters the java heap from the socket.

• (2). Producer sends a message, and the message is transferred from the java heap to PAGACACHE, physical memory.

• (3). Producer sends a message, which is flushed by an asynchronous thread, and the message is flushed from PAGECACHE to disk.

• (4). The consumer pulls the message (normal consumption), and the message is directly transferred from the PAGECACHE (data in the physical memory) to the socket and reaches the consumer without going through the java heap. This kind of consumption scenario is the most, online 96G physical memory, calculated according to 1K messages, can cache 100 million messages in physical memory.

• (5). Consumer pulls messages (abnormal consumption), and the messages are directly transferred from PAGECACHE (data in virtual memory) to socket.

• (6). The Consumer pulls the message (abnormal consumption). Because the Socket accesses the virtual memory, a page fault interrupt occurs. At this time, disk IO will be generated, from the disk Load message to PAGECACHE, and then sent directly from the socket.

• (7). Same as 5.

• (8). Same as 6.

5.12. Solution to message accumulation problem

	Stacking performance index
message stacking capacity	depends on disk size
The degree to which the throughput of sending messages is affected	If there is no SLAVE, it will be affected to a certain extent; if there is SLAVE, it will not be affected
Will normal consumers be affected?	If there is no SLAVE, it will be affected to a certain extent; if there is SLAVE, it will not be affected
What is the throughput when accessing messages accumulated on disk	It is related to the concurrency of access, and the slowest will drop to about 5000.

In the case of Slave, once the Master discovers that the Consumer accesses the data accumulated on the disk, it will issue a redirection instruction to the Consumer, so that the Consumer will pull the data from the Slave, so that the normal sending of messages and the normal consumption of Consumers will not be affected by the message. Accumulation is affected because the system divides the accumulation scene and the non-accumulation scene into two different nodes for processing.

Another question arises here, whether the write performance of Slave will degrade, and the answer is no. Because Slave’s message writing only pursues throughput, not real-time, as long as the overall throughput is high, and Slave pulls a batch of data from Master every time, such as 1M, this batch sequential writing method even In the case of accumulation, the impact on the overall throughput is relatively small, but the write
RT will be longer.

6. RocketMQ message filtering

6.1. Simple message filtering

/**
 * 订阅指定topic下tags分别等于TagA或TagC或TagD
*/
 consumer.subscribe("TopicTest1", "TagA || TagC || TagD");

As shown in the above code, simple message filtering filters messages by specifying multiple Tags, and the filtering action is performed on the server.

6.2. Advanced message filtering

• The machine where the Broker is located will start multiple FilterServer filtering processes
• After the Consumer starts, it will upload a filtered Java class to the FilterServer
• Consumer pulls messages from FilterServer, and FilterServer forwards the request to Broker. After FilterServer receives messages from Broker, it filters according to the Java filter program uploaded by Consumer, and returns to Consumer after filtering.

Summarize:

• Use CPU resources in exchange for network card traffic resources
• FilterServer and Broker are deployed on the same machine, and the data communicates through the local loopback without using the network card
• A Broker deploys multiple FilterServers to make full use of CPU resources, because it is difficult for a single JVM to fully utilize the high-configuration physical machine Cpu resources
• Because the filtering code is written in the Java language, the application can perform almost any form of server-side message filtering, such as filtering by Message Header, or even by Message Body.
• Using the Java language as a filtering expression is a double-edged sword, which facilitates the filtering operation of the application, but brings security risks on the server side. The application is required to ensure the security of the filtering code, such as avoiding operations such as applying for large memory and creating threads as much as possible in the filtering program. Avoid resource leaks on the Broker server.