Based on the source code, simulate and implement RabbitMQ - from requirement analysis to implementation of core classes (1)

Table of contents

1. Requirements analysis

1.1. Understanding of Message Queue

1.2. Core concepts of message queue

1.3. Broker Server internal key concepts

1.4. Broker Server core API (key implementation)

1.5. Switch type

Direct direct switch

Fanout fanout switch

Topic topic switch

1.6, Endurance

1.7. Network communication

communication process

Remote call design ideas

1.8. Module design drawing

2. Implement core classes

2.1. Properties and binding relationships of switches and queues

2.2. Message

---

1. Requirements analysis

---

1.1. Understanding of Message Queue

The message queue is to extract the data structure such as the blocking queue into an independent program for deployment, and realize the producer-consumer model of "between processes and processes/between services and services" (in distributed systems, it is a A cluster composed of group servers).

The benefits of the producer-consumer model are as follows:

1. Decoupling: In a distributed system, server A sends a request to server B, and B returns a response to A, so the coupling between A and B is relatively large (once B hangs up, A cannot receive the response normally); introduction After the message queue, A sends the request to the message queue, and B obtains the request from the message queue, which reduces the degree of coupling (even if B hangs up, A does not need to worry about it and continues to send requests to the message queue, and the queue responds).
2. Peak shaving and valley filling: Assume that A is the entrance server, and A then calls B to complete some specific services. If A and B communicate directly, suddenly one day A receives a peak of user requests, and B will also follow suit. After introducing the message queue, A sends the request to the queue. At this time, B can take the request from the queue according to its own original rhythm, and will not receive too much concurrency all at once.

1.2. Core concepts of message queue

Producer: A client application that publishes messages.

Consumer: A client application that subscribes to messages and is used to process messages generated by producers.

Broker: In order for consumers to get information from producers, they need to go through a middleman, which is used to cut peaks and fill valleys.

Publish: The process in which the producer delivers messages to the intermediary.

Subscription: Similar to subscribing to a newspaper, the premise for consumers to get the corresponding news from the intermediary is to subscribe to the news first.

Consume: The operation of the consumer to obtain data from the middleman.

1.3. Broker Server internal key concepts

1. Virtual Host: Similar to the database in MySQL, it is a "logical" data collection.

In actual development, a Broker Server can also have many different types of data, and may be used to manage multiple sets of business line data at the same time. Virtual Host can be used for logical distinction~

2. Exchange: In fact, the producer first sends the message to an exchange on the Broker Server, and then the exchange forwards the message to the corresponding queue.

The switch is similar to the front desk lady of the company. One day when you come for an interview, you tell the front desk lady and she will take you to the corresponding floor (queue).

3. Queue: In a large message queue, there can be many specific small queues. They are used to store message entities, and subsequent consumers also fetch data from the corresponding queues.

4. Binding: Establish an association between the switch and the queue.

You can think of switches and queues as "many-to-many" relationships in the database.

5. Message: The request sent by server A to B (forwarded through MQ) is a message, and the response returned by server B to A (forwarded through MQ) is also a message.

A message can be regarded as a string (binary data). The specific data contained in the message is customized by the programmer.

Ps: These concepts need to be stored both in memory (convenient and fast) and in the hard disk (persistence).

1.4. Broker Server core API (key implementation)

1. Create queue (queueDeclare)

The term "Create" is not used here, but "Declare" is used because it is modeled on RabbitMQ. Declare means creating if it does not exist, and doing nothing if it exists.

2. Destroy the queue (queueDelete)

3. Create the switch (exchangeDeclare)

4. Destroy the switch (exchangeDelete)

5. Create binding (queueBind)

6. Unbind (queueUnbind)

7. Publish news (basicPublish)

8. Subscribe to messages (basicConsume)

9. Confirm message (basicAck)

Similar to TCP's confirmation response mechanism, the confirmation message API allows consumers to explicitly tell the broker server that I have completed processing this message, which improves the reliability of the entire system.

In addition to providing the above 9 methods, the client also needs to provide 4 methods:

1.Create Connection

2.Close Connection

3. Create Channel

4.Close Channel

 A Connection object represents a TCP connection, which can contain multiple Channels. The data transmitted on each Channel is independent of each other.

Now that we have a Connection, why do we need a Channel?

In TCP, the cost of establishing/disconnecting a connection is still quite high. Therefore, in many cases, we do not want to frequently establish and disconnect TCP connections, so the introduction of Channel is much lighter than TCP. Between Connection and Channel The relationship is similar to that of "network cable".

1.5. Switch type

When the switch forwards messages, it provides several different switch types (ExchangeType) according to the forwarding rules to describe the different forwarding rules here.

RabbitMQ mainly implements four switch types (also defined by the AMQP protocol)

1. Direct direct switch
2. Fanout fanout switch
3. Topic topic switch
4. Header message header switch

Ps: The first three switch types are mainly implemented here, because the fourth switch has complex rules and few application scenarios, and the first three are enough.

Direct direct switch

There are two key concepts:

bindingKey: Specify a word (similar to a password) when binding the queue to the switch.

routingKey: When the producer sends a message, he also specifies a word.

When routingKey and BindingKey match the secret key, the message can be forwarded to the corresponding queue.

When the producer sends a message, it will specify a "name of the target queue" (routingKey). After receiving it, the switch will check whether there is a matching queue (BindingKey) in the bound queue, and if so, forward it ( Put the message into the corresponding queue), if not, the message is discarded directly.

Fanout fanout switch

The received message will be broadcast to each queue bound to it, and finally handed over to the corresponding consumer. It is worth noting that exchange is responsible for message routing, not storage. If routing fails, the message will be lost.

Topic topic switch

There are two key concepts:

bindingKey: Specify a word (similar to a password) when binding the queue to the switch.

routingKey: When the producer sends a message, he also specifies a word.

When routingKey and BindingKey match the secret key, the message can be forwarded to the corresponding queue.

Ps: TopicExchange is very similar to DirectExchange. The difference is that routingKey must be a list of multiple words and  separated by  .

1.6, Endurance

The data corresponding to the above concepts (switch, queue, binding, message...) need to be stored and managed. At this time, both the memory and the hard disk will store one copy each, with the memory as the main one and the hard disk as the auxiliary.

The reason for storing in memory: For MQ, it is very important to be able to forward and process data efficiently, so organizing data in memory is much faster than on the hard disk.

The reason for storing on the hard disk: to prevent the data in the memory from being lost when the process restarts/the host restarts.

Ps: Hard disk storage, this persistence is relative to memory. For a hard disk, the lifespan of stored messages is generally several years to decades (when it is not powered on all the time).

1.7. Network communication

communication process

Producers and consumers are both client programs, and the broker server acts as a server. They communicate through the network.

For example, the following process:

1. The client sends a request: There needs to be a method queueDeclare in the producer's code to create a queue. What this method needs to do internally is to send a request to the server, telling the server that we want to create a queue and what the queue looks like... ...
2. The server processes the request and returns a response: After receiving the request, the broker server executes the queueDeclare method on the server side, actually writes some data to the memory/hard disk, actually creates the queue, and then reports the success/failure of the creation. The result is packaged into a response and returned to the client.
3. The client receives the response: When the response comes back, the client's queueDeclare will get the response and see that the queue is created successfully. At this time, the queueDeclare is completed.

Remote call design ideas

Here, the client called a local method. As a result, behind this method, a series of messages were sent to the server, and the server completed a series of work. From the caller's perspective, he only saw that this function was completed, and Don't know the implementation details behind it.

Although a local method is called, it actually seems to be calling a method on another remote server~

This can be thought of as writing a client server program, a design idea in the communication process - remote call (RPC)

for example

There used to be an American guy who worked in a big factory, but he really wanted to fish, so he found a Chinese to help him (for a fee). For programmers of the same level, the salary working in the United States is the same as in China. 3-4 times.

1.8. Module design drawing

2. Implement core classes

---

Ps：Spring boot、MyBatis

2.1. Properties and binding relationships of switches and queues

We can use an enumeration class to represent three types of switches.

public enum ExchangeType {

    DIRECT(0),
    FANOUT(1),
    TOPIC(2);

    private final int type;

    private ExchangeType(int type) {
        this.type = type;
    }

    public int getType() {
        return type;
    }

}

The switch properties are as follows

public class Exchange {

    //模仿 rabbitmq 使用 name 作为唯一身份标识
    private String name;
    //交换价类型，DIRECT,FANOUT,TOPIC
    private ExchangeType type = ExchangeType.DIRECT;
    //当前交换机是否要持久化存储，true 标识持久化.
    private boolean durable = false;
    //TODO: 若当前交换机没人使用了，就会自动删除
    private boolean autoDelete = false;
    //TODO: 表示创建交换机时可以指定一些额外的选项
    private Map<String, Object> arguments = new HashMap<>();

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public ExchangeType getType() {
        return type;
    }

    public void setType(ExchangeType type) {
        this.type = type;
    }

    public boolean isDurable() {
        return durable;
    }

    public void setDurable(boolean durable) {
        this.durable = durable;
    }

    public boolean isAutoDelete() {
        return autoDelete;
    }

    public void setAutoDelete(boolean autoDelete) {
        this.autoDelete = autoDelete;
    }

    public Map<String, Object> getArguments() {
        return arguments;
    }

    public void setArguments(Map<String, Object> arguments) {
        this.arguments = arguments;
    }
}

The queue properties are as follows

public class MSGQueue {
    //表示队列的身份标识
    private String name;
    //是否持久化，true 表示支持
    private boolean durable;
    //TODO: 这个属性为 true，表示队列只能被一个消费者使用, false表示大家都能用
    private boolean exclusive = false;
    //TODO: 自动删除，为 true 表示没有人使用以后自动删除
    private boolean autoDelete = false;
    //TODO: 扩展参数，自定义选项
    private Map<String, Object> arguments = new HashMap<>();

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public boolean isDurable() {
        return durable;
    }

    public void setDurable(boolean durable) {
        this.durable = durable;
    }

    public boolean isExclusive() {
        return exclusive;
    }

    public void setExclusive(boolean exclusive) {
        this.exclusive = exclusive;
    }

    public boolean isAutoDelete() {
        return autoDelete;
    }

    public void setAutoDelete(boolean autoDelete) {
        this.autoDelete = autoDelete;
    }

    public Map<String, Object> getArguments() {
        return arguments;
    }

    public void setArguments(Map<String, Object> arguments) {
        this.arguments = arguments;
    }
}

The binding relationship between the switch and the queue is as follows

public class Binding {
    //交换机身份标识
    private String exchangeName;
    //队列身份标识
    private String queueName;
    private String bindingKey;

    //绑定没必要设置持久化，因为没有意义，他存在的意义前提是 交换机和队列存在（持久化）

    public String getExchangeName() {
        return exchangeName;
    }

    public void setExchangeName(String exchangeName) {
        this.exchangeName = exchangeName;
    }

    public String getQueueName() {
        return queueName;
    }

    public void setQueueName(String queueName) {
        this.queueName = queueName;
    }

    public String getBindingKey() {
        return bindingKey;
    }

    public void setBindingKey(String bindingKey) {
        this.bindingKey = bindingKey;
    }
}

2.2. Message

Message messages are mainly divided into:

1.  Attribute part BasicProperties (network): This is a custom class that carries the core properties of Message, such as the unique identity of the message, routingKey, whether persistence is required...
2.  Text part byte[]: carries the specific message content.
3. Auxiliary attributes: Use offsetBeg and offsetEnd to quickly find the specific location of a message in the file (a file will store many messages). The rule agreed here is "close first and open later" (in bytes); isValid identifies the message to be Delete, if the deletion adopts logical deletion (not really deleted, but marked as invalid data), it is agreed that 0x1 is valid data and 0x0 represents invalid data.

Ps: The first two pieces of information need to be transmitted over the network and written to files, so they need to be serialized and deserialized through Serializable (just inherit the interface directly, no specific implementation is required)~~ 

The third item does not need to be serialized and saved to the file. This attribute is mainly for the Message object in the memory to quickly find the Message object in the file. Therefore, using trasient modification can prevent serialization. 

public class Message implements Serializable {
    //这两个属性是 Message 最核心的属性
    private BasicProperties basicProperties = new BasicProperties();
    private byte[] body;

    //以下是辅助类型的属性
    private transient long offsetBeg = 0; //消息数据的开头距离文件开头的位置偏移量(字节)
    private transient long offsetEnd = 0; //消息数据的结尾距离文件开头的位置偏移量(字节)
    //表示文件中的消息是否是有效消息(对文件中的消息，如果删除，使用逻辑删除)
    // 0x1 表示有效， 0x0 表示无效
    private byte isValid = 0x1;

    //创建一个工厂方法，让工厂方法创建 Message 对象
    //此方法中创建的 Message 对象，会自动生成唯一的 MessageId
    //万一 routingKey 和 basicProperties 里的 routingKey 冲突，以外面为主
    public static Message createMessageWithId(String routingKey, BasicProperties basicProperties, byte[] body) {
        Message message = new Message();
        if(basicProperties != null) {
            message.setBasicProperties(basicProperties);
        }
        //生成的 MessageId 以 M- 作为前缀
        message.setMessageId("M-" + UUID.randomUUID());
        message.setRoutingKey(routingKey);
        message.body = body;
        // offsetBeg, offsetEnd, isValid ，是消息持久化的时候才会用到，消息写入文件之前在进行设定
        return message;
    }

    public String getMessageId() {
        return basicProperties.getMessageId();
    }

    public void setMessageId(String messageId) {
        basicProperties.setMessageId(messageId);
    }

    public String getRoutingKey() {
        return basicProperties.getRoutingKey();
    }

    public void setRoutingKey(String routingKey) {
        basicProperties.setRoutingKey(routingKey);
    }

    public int getDeliverMode() {
        return basicProperties.getDeliverMode();
    }

    public void setDeliverMode(int mode) {
        basicProperties.setDeliverMode(mode);
    }

    public BasicProperties getBasicProperties() {
        return basicProperties;
    }

    public void setBasicProperties(BasicProperties basicProperties) {
        this.basicProperties = basicProperties;
    }

    public byte[] getBody() {
        return body;
    }

    public void setBody(byte[] body) {
        this.body = body;
    }

    public long getOffsetBeg() {
        return offsetBeg;
    }

    public void setOffsetBeg(long offsetBeg) {
        this.offsetBeg = offsetBeg;
    }

    public long getOffsetEnd() {
        return offsetEnd;
    }

    public void setOffsetEnd(long offsetEnd) {
        this.offsetEnd = offsetEnd;
    }

    public byte getIsValid() {
        return isValid;
    }

    public void setIsValid(byte isValid) {
        this.isValid = isValid;
    }
}

The core properties in Message are as follows

public class BasicProperties implements Serializable {

    //消息的唯一标识(使用 UUID 保证唯一性)
    private String messageId;
    //和 bindingKey 做匹配
    //如果当前交换机类型是 DIRECT ，此时 routingKey 就表示要转发的队列名
    //如果当前交换机类型是 FANOUT ，此时 routingKey 无意义(不使用)
    //如果当前交换机类型是 TOPIC ，此时 routingKey 就是要和 bindingKey 做匹配，匹配才进行转发
    private String routingKey;
    //标识消息是否要持久化，1 表示不持久化， 2 表示持久化(rabbitmq 是这么搞得)
    private int deliverMode = 1;

    //RabbitMQ 还有其他属性，这里就不考虑了


    public String getMessageId() {
        return messageId;
    }

    public void setMessageId(String messageId) {
        this.messageId = messageId;
    }

    public String getRoutingKey() {
        return routingKey;
    }

    public void setRoutingKey(String routingKey) {
        this.routingKey = routingKey;
    }

    public int getDeliverMode() {
        return deliverMode;
    }

    public void setDeliverMode(int deliverMode) {
        this.deliverMode = deliverMode;
    }
}