Handwritten message queue (based on RabbitMQ)

1. What is a message queue?

Does the mention of message queues awaken memories deep in your mind? Looking back at this previous article:"Java Multithreading Series IV (Singleton Mode + Blocking Queue + Timer + Thread Pool)", among which When we introduced blocking queues, we said that the biggest use of blocking queues is to implementproducer consumer model。

We know that the producer-consumer model has two very eye-catching features:

1. Decoupling.
2. Cut peaks and fill valleys.

(1) Decoupling
Before the producer-consumer model was introduced, two servers usually interacted directly. This interaction mode made the coupling between servers very big. After the introduction of the producer-consumer model, there is no longer direct communication between the two servers, but the blocking queue is used for business processing, which achieves a decoupling effect.

(2) Peak shaving and valley filling
Before the introduction of the producer-consumer model, the same two servers communicated directly. If at a point in time, server A suddenly sends a When the group request peaks, server B will also feel the peak at this moment. In this case, it is likely to cause server failure. If a blocking queue is used at this time, A will send the received request to the queue. Although there are many requests in the queue, server B still reads the requests at the original rhythm.

In fact, it is precisely because the producer-consumer model has many of the above benefits that in actual back-end development, especially in distributed systems, it is a very common requirement to use the producer-consumer model across hosts. Therefore, the blocking queue is usually separated, given richer functions, and encapsulated into an independent server program. This program is calledmessage queue。

2. Requirements sorting

1. Core concepts of the producer-consumer model

1. Producer: is responsible for sending messages to the message queue.

2. Consumer: Receives and processes messages from the message queue. .

3. Broker: It is responsible for receiving messages sent by publishers, storing these messages in the queue, and then delivering these messages to subscribers.

4. Publish: The process by which a producer delivers a message to an intermediary.

5. Subscribe: The process by which consumers register with the middleman. Only after the consumer is registered, when a message is published to the message queue, the message will be sent to the corresponding subscriber.

Based on the above concepts, we can roughly draw a conceptual diagram of the producer-consumer model: (PS: Each module below represents a server)

One producer, one consumer:

N producers, N consumers:

2. Broker design outline

Our current purpose is to implement a message queue, of which Broker is the core part and is mainly responsible for messagestorageandForward, the core concepts involved are as follows:

1. VirtualHost: Similar to MySQL’s “database”, it is a logical collection. In actual development, a BrokerServer may manage multiple sets of business line data at the same time. In this case, different VirtualHosts can be used to differentiate.

2. Exchange: The producer first sends the message to the Broker's Exchange, and then forwards the message to different Queues according to different rules.

3. Queue (Queue): The part actually used to store messages. Each consumer decides which Queue to read messages from (according to the subscribed queue). ⼼One Exchange can be bound to multiple Queues (messages can be forwarded to multiple Queues), and one Queue can also be bound to multiple Exchanges
   (Messages in one Queue can come from multiple Exchange).

4. Binding: The association between Exchange and Queue. Exchange and Queue can be understood as a "many-to-many" relationship. These two concepts can be linked using an association table.

5. Message: Specifically, requests and responses between servers. A message can be regarded as a string (binary data), which is customized by the programmer.

The above concepts are reflected in Broker as shown in the figure:

Supplementary Note 1: Data Storage

The data corresponding to the above concepts needs to be stored in both memory and hard disk, with memory as the main method and hard disk as the supplement:

• Memory storage: For MQ, it is a very critical indicator to efficiently process data forwarding. Therefore, using memory to organize the above data can achieve higher efficiency.
• Hard disk storage: Mainly to prevent the data in the memory from being lost when the process/host restarts.

Supplementary Note: 2: Switch type and forwarding rules

As we mentioned above, when a producer sends a message, it will first send the message to the Broker's switch, and then the switch will forward it to the corresponding queue according to different rules. MQ supports four types of switches, which are: Direct (direct switch), Fanout (fanout switch), Topic (topic switch), and Header (head switch). Among them, the Header method is more complex and rare. The first three are mainly implemented in the current project. They are introduced in detail below:

Before explaining:

• The following bindingKeys are keywords specified when creating queue and switch bindings.
• The following routingKey (routing key) is the key specified by the producer when sending the message.

(1)Direct (direct switch)

1. When the producer sends a message, it will specify the name of a "target queue" (the routingKey at this time is the name of the queue, and the bindingKey is invalid)
2. After the switch receives the message, it checks whether there is a queue with the queue name routingKey in the binding corresponding to the current switch.
3. If there is, forward it (put the message into the corresponding "target queue")
4. If not, the message is discarded directly.

(2)Fanout (fan-out switch)

1. The producer does not need to specify routingKey and directly sends messages to the designated switch.
2. After receiving the message, the switch directly forwards the message to all queues bound to the current switch. (The bindingKey and routingKey at this time are not valid for fan-out switches.)

(3)Topic(topic switch)

1. When the producer sends a message, specify a routingKey
2. After the switch receives the message, it checks whether there is a bindingKey in the binding corresponding to the current switch and matches the routingKey through certain rules.
3. If so, forward the message to the corresponding binding queue.
4. If not, the message is discarded.

PS: All the above concepts come from the AMQP protocol: an application layer standard Advanced Message Queuing Protocol that provides unified messaging services. It is an open standard for application layer protocols and is designed for message-oriented middleware.

3. Broker core API

Based on the above concepts and functions, Broker needs to implement the following core APIs:

1. Create queue (queueDeclare)
2. Destroy the queue (queueDelete)
3. Create an exchange (exchangeDeclare)
4. Destroy the switch (exchangeDelete)
5. Create binding (queueBind)
6. Unbind (queueUnbind)
7. Publish a message (basicPublish)
8. Subscribe to messages (basicConsume)
9. Acknowledgment message (basicAck)

Additional instructions:

1. As can be seen from the above, create is not used during the creation operation, but declare is used. This is semantically explained. The effect of creation here is to create it if it does not exist, and do nothing if it exists.
2. The above does not create an API for "consuming messages". This is because the current working mode we use is push, and the Broker will actively push messages to subscribed consumers. Of course, there is also a pull working mode, which requires consumers to actively call the Broker's API to obtain messages. The current project does not involve this mode.
3. There are two response methods in MQ. One is automatic response. In this method, the response is completed after the Broker pushes the message to the subscribed consumer. Another response method is manual response. The effect of the above confirmation message (basicAck) is that it allows consumers toexplicitTell Broker that I have finished processing this message, improving the reliability of the entire system.

4. Client design outline

Producers and consumers are both client programs, and the broker acts as a server and communicates through the network. It is set here to use TCP + customized application layer protocol to realize the interaction between producer/consumer and BrokerServer. Here, a set of APIs need to be provided to the client for the client's business code to call, so as to communicate through the network. method to remotely call methods on the brokerserver.

Client core API:

1. Create Connection
2. Close Connection
3. Create Channel
4. Close Channel
5. Create queue (queueDeclare)
6. Destroy the queue (queueDelete)
7. Create an exchange (exchangeDeclare)
8. Destroy the switch (exchangeDelete)
9. Create binding (queueBind)
10. Unbind (queueUnbind)
11. Publish a message (basicPublish)
12. Subscribe to messages (basicConsume)
13. Acknowledgment message (basicAck)

Additional instructions:

1. Compared with the Broker server API, the client program also provides the following 4 APIs: Create Connection, Close Connection, Create Channel, and Close Channel.
2. A Connection object here represents a TCP connection.
3. Channel is a logical link within Connection. Multiple Channels reuse the same TCP connection. A Connection can contain multiple Channels. Each Channel is responsible for different modules in the client, and the data transmitted therein are independent of each other.
4. This setting is mainly to better reuse TCP connections, achieve the effect of long connections, and avoid frequently creating and closing TCP connections.
5. The API provided by the above client is only called by the business code, and the actual method execution is handed over to the BrokerServer. This process is called a Remote Procedure Call (RPC), a computer communication protocol that allows a program to call a procedure or function in another address space (usually a different computer) without the programmer having to explicitly write it. Network code. By using RPC, an application can call a process on a remote server as if it were a local process.

5. Summary

Finally, a brief summary of the work we need to do roughly, and the details involved, we will add later:

1. Implement three parts: producer, BrokerServer and consumer
2. For producers and consumers, it mainly writes the network communication part of the client and server.
3. Focus on implementing BrokerServer, including basic internal concepts and core APIs.
4. Persistent storage of data.

3. Specific implementation

Attached is the link:

1. Detailed design and implementation mind map of message queue
2. Gitee complete code address