RocketMQ cluster construction theory

architecture diagram

Data replication and flashing strategy

copy strategy

The replication strategy is the data synchronization method between the Broker's Master and Slave. Divided into synchronous replication and asynchronous replication:

• Synchronous replication: After the message is written to the master, the master will wait for the slave to successfully synchronize data before returning a successful ACK to the producer

• Asynchronous replication: After the message is written to the master, the master immediately returns a successful ACK to the producer, without waiting for the slave to successfully synchronize data

The asynchronous replication strategy will reduce the write delay of the system, the RT will be smaller, and the throughput of the system will be improved

Brush strategy

The disk flushing strategy refers to the method of placing messages in the broker, that is, the way in which the messages are persisted to the disk after they are sent to the memory of the broker. Divided into synchronous brushing and asynchronous brushing.

• Synchronous disk flushing: the message is considered successful when the message is persisted to the broker's disk
• Asynchronous flushing: When the message is written to the broker's memory, it means that the message is written successfully, and there is no need to wait for the message to be persisted to the disk.

1) The asynchronous flushing strategy will reduce the write delay of the system, and the RT will be smaller, which improves the throughput of the system. 2) The message is written to the Broker's memory
, usually to the PageCache.
A successful ACK is returned immediately after writing to PageCache. However, it will not perform the disk placement operation immediately, but will automatically perform disk placement when the PageCache reaches a certain amount.

Broker cluster mode

According to the relationship between each node in the Broker cluster, the Broker cluster can be divided into the following categories:

Single Master

There is only one broker (it cannot be called a cluster in essence). This method can only be used during testing, and cannot be used in a production environment because there is a single point of problem.

Multi-Master

The broker cluster only consists of multiple masters, and there are no slaves. Each Queue of the same Topic will be evenly distributed on each master node.

• Advantages: simple configuration, single Master downtime or restart maintenance has no impact on the application, when the disk is configured as RAID10, even if the machine downtime is unrecoverable, because the RAID10 disk is very reliable, the message will not be lost (a small amount of asynchronous brushing is lost message, synchronously brushing the disk without losing one piece), the performance is the highest;
• Disadvantage: When a single machine is down, unconsumed messages on this machine cannot be subscribed (unconsumable) until the machine is restored, and the real-time performance of messages will be affected.
  The premise of the above advantages is that these Masters are configured with RAID disk arrays. If there is no configuration, once a Master goes down, a large number of messages will be lost.

Multi-Master and multi-Slave mode - asynchronous replication

Multi-Master and multi-Slave mode - synchronous double write