Table of contents
1.1.2. What is the virtual host used for?
1.1.3. How to express the affiliation between the switch and the virtual host?
2. Implement the VirtualHost class
2.4. Virtual host construction method
2.5. Switch related operations
2.6. Binding related operations
2.7. Message related operations
2.8. Supplementary thread safety issues
1. Virtual host design
1.1. Demand analysis
1.1.1. Core API
The concept of a virtual host is similar to MySQL's database, which logically isolates switches, queues, bindings, messages... so it not only needs to manage data, but also needs to provide some core APIs for upper-layer code to call.
Core API (which is to connect the previously written data management in the memory and the data management on the hard disk):
- Create switch exchangeDeclare
- delete switchexchagneDelete
- Create queuequeueDeclare
- Delete queuequeueDelete
- Create binding queueBind
- Remove binding queueUnbind
- Send messagebasicPublish
- Subscribe to messagesbasicCosume
- Acknowledgment message basicAck
1.1.2. What is the virtual host used for?
The purpose of virtual hosts is to ensure isolation and the content between different virtual hosts should not be affected.
For example, an exchange is created in virtual host 1, called "testExchange", and an exchange is also created in virtual host 2, called "testExchange". Although they have the same name, they are exchanges in different spaces.
From the caller's perspective, although the two exchanges were named testExchange when they were created, they are automatically processed within the virtual host and prefixed with "virtualHost1testExchange" and "virtualHost2testExchange".
In this way, different switches can be distinguished, and different queues can be further distinguished (one switch corresponds to multiple queues). Furthermore, bindings are isolated (bindings are related to switches and queues), and further messages are Strongly related to queues, messages are naturally distinguished.
1.1.3. How to express the affiliation between the switch and the virtual host?
Solution 1: Refer to the design of the database, "one-to-many" solution, such as adding an attribute to the switch table, virtual host id/name...
Solution 2 (the strategy adopted by RabbitMQ): Re-agreement, the name of the switch = the name of the virtual host + the name of the switch
Option 3: A more elegant way is to assign a different set of databases and files to each virtual host, but it is a bit troublesome.
Here we follow the strategy adopted by RabbitMQ~~
2. Implement the VirtualHost class
2.1. Properties
private String virtualHostName;
private MemoryDataCenter memoryDataCenter = new MemoryDataCenter();
private DiskDataCenter diskDataCenter = new DiskDataCenter();
private Router router;
The Router class describes whether routingKey and bindingKey are legal, as well as the matching rules between messages and queues.
2.2. Lock object
Used to handle thread safety issues in switch operations, queue operations, binding operations, and message operations
For example: locking operations for multi-threaded operations "add switch" and "add while deleting" (multi-threaded deletion at the same time is actually nothing, because it is deleted by SQL delete, even if deleted multiple times, there are no side effects)
//交换机锁对象
private final Object exchangeLocker = new Object(); //final 修饰是为了防止程序员修改引用，导致不是一个锁对象(其实可有可无，自己注意就 ok)
//队列锁对象
private final Object queueLocker = new Object();
2.3. Public examples
This allows the upper-layer code to obtain the virtual host name, memory data processing center, and hard disk data processing center respectively when calling.
public String getVirtualHostName() {
return virtualHostName;
}
public MemoryDataCenter getMemoryDataCenter() {
return memoryDataCenter;
}
public DiskDataCenter getDiskDataCenter() {
2.4. Virtual host construction method
It mainly performs initialization operations on the host name, hard disk, and memory data processing center.
public VirtualHost(String name) {
this.virtualHostName = name;
//对于 MemoryDataCenter 来说，不需要进行初始化工作，只需要 new 出来即可
//但是，对于 DiskDataCenter 来说，需要进行初始化操作，建库建表和初始数据的设定
diskDataCenter.init();
//另外还需要针对硬盘初始化的数据，恢复到内存中
try {
memoryDataCenter.recovery(diskDataCenter);
} catch (IOException | MqException | ClassNotFoundException e) {
e.printStackTrace();
System.out.println("[VirtualHost] 恢复内存数据失败！");
}
}
2.5. Switch related operations
Create a switch: Get the corresponding switch from the memory. If it does not exist, create it. If it exists, return it directly.
Delete the switch: First check whether the switch exists. If it does not exist, an exception will be thrown. If it exists, delete it.
Ps: The return value is of boolean type, true indicates success, false indicates failure
/**
* 创建交换机
* @param exchangeName
* @param exchangeType
* @param durable
* @param autoDelete
* @param arguments
* @return
*/
public boolean exchangeDeclare(String exchangeName, ExchangeType exchangeType, boolean durable, boolean autoDelete,
Map<String, Object> arguments) {
//1.把交换机加上虚拟主机的名字作为前缀(虚拟主机的隔离效果)
exchangeName = virtualHostName + exchangeName;
try {
synchronized (exchangeLocker) {
//2.通过内存查询交换机是否存在
Exchange exchange = memoryDataCenter.getExchange(exchangeName);
if(exchange != null) {
//已经存在就直接返回
System.out.println("[VirtualHost] 交换机已经存在！exchangeName=" + exchangeName);
return true;
}
//3.不存在就创建一个
exchange = new Exchange();
exchange.setName(exchangeName);
exchange.setType(exchangeType);
exchange.setDurable(durable);
exchange.setAutoDelete(autoDelete);
exchange.setArguments(arguments);
//4.内存中是一定要创建的，硬盘中是否存储，就要传来的参数 durable 是否为 true 了(持久化)
if(durable) {
diskDataCenter.insertExchange(exchange);
}
memoryDataCenter.insertExchange(exchange);
System.out.println("[VirtualHost] 交换机创建成功！exchangeName=" + exchangeName);
//上述逻辑中，先写硬盘，后写内存，就是因为硬盘更容易写失败，如果硬盘写失败了，内存就不用写了
//但要是先写内存，一旦内存写成功了，硬盘写失败了，还需要把内存中的数据给删掉，比较麻烦
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 交换机创建失败！exchangeName=" + exchangeName);
e.printStackTrace();
return false;
}
}
/**
* 删除交换机
* @param exchangeName
* @return
*/
public boolean exchangeDelete(String exchangeName) {
//1.虚拟主机前缀
exchangeName = virtualHostName + exchangeName;
try {
synchronized (exchangeLocker) {
//2.检查当前内存中是否存在该交换机
Exchange exchange = memoryDataCenter.getExchange(exchangeName);
if(exchange == null) {
throw new MqException("[VirtualHost] 该交换机不存在无法删除！exchangeName=" + exchangeName);
}
//3.如果持久化到硬盘上，就把硬盘的先删除了
if(exchange.isDurable()) {
diskDataCenter.deleteExchange(exchangeName);
}
//4.存在就删除
memoryDataCenter.deleteExchange(exchangeName);
System.out.println("[VirtualHost] 交换机删除成功！exchangeName=" + exchangeName);
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 交换机删除失败！exchangeName=" + exchangeName);
e.printStackTrace();
return false;
}
}
2.5. Queue related operations
The operating logic here is similar to that of a switch~~
/**
* 创建队列
* @param queueName
* @param durable
* @param exclusive
* @param autoDelete
* @param arguments
* @return
*/
public boolean queueDeclare(String queueName, boolean durable, boolean exclusive, boolean autoDelete,
Map<String, Object> arguments) {
//1.虚拟主机前缀
queueName = virtualHostName + queueName;
try {
synchronized (queueLocker) {
//2.先检查内存是否存在此队列
MSGQueue queue = memoryDataCenter.getQueue(queueName);
if(queue != null) {
System.out.println("[VirtualHost] 当前队列已存在！queueName=" + queueName);
return true;
}
//3.不存在则创建并赋值
queue = new MSGQueue();
queue.setName(queueName);
queue.setExclusive(exclusive);
queue.setDurable(durable);
queue.setAutoDelete(autoDelete);
queue.setArguments(arguments);
//4.如果设置了持久化，就先在硬盘上保存一份
if(durable) {
diskDataCenter.insertQueue(queue);
}
//5.写内存
memoryDataCenter.insertQueue(queue);
System.out.println("[VirtualHost] 队列创建成功！queueName=" + queueName);
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 队列创建失败！queueName=" + queueName);
e.printStackTrace();
return false;
}
}
/**
* 队列删除
* @param queueName
* @return
*/
public boolean queueDelete(String queueName) {
//1.虚拟主机前缀
queueName = virtualHostName + queueName;
try {
synchronized (queueLocker) {
//2.判断队列是否存在，若不存在则抛出异常
MSGQueue queue = memoryDataCenter.getQueue(queueName);
if(queue == null) {
throw new MqException("[VirtualHost] 队列不存在，删除失败！queueName=" + queueName);
}
//3.存在则判断是否持久化，若持久化到硬盘上，则先删硬盘
if(queue.isDurable()) {
diskDataCenter.deleteQueue(queueName);
}
//4.删内存
memoryDataCenter.deleteQueue(queueName);
System.out.println("[VirtualHost] 队列删除成功！queueName=" + queueName);
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 队列删除失败！queueName=" + queueName);
e.printStackTrace();
return false;
}
}
2.6. Binding related operations
Create a binding: first determine whether such a binding exists in the memory. If it exists, the creation will fail. If it does not exist, you need to check whether the switch and queue exist at the same time. If they exist, the creation can be successful.
Delete binding: First determine whether this binding exists in the memory. No exception will be thrown. If it exists, there is no need to determine whether the switch and queue exist (which complicates the steps), and there is no need to determine whether the binding exists on the hard disk. Just delete it directly from the memory and hard disk, because even if there is no data in the hard disk, there will be no side effects (the essence is that the bottom layer of MyBatis calls the delete statement to delete).
/**
* 创建队列交换机绑定关系
* @param queueName
* @param exchangeName
* @param bindingKey
* @return
*/
public boolean queueBind(String queueName, String exchangeName, String bindingKey) {
//1.虚拟主机前缀
queueName = virtualHostName + queueName;
exchangeName = virtualHostName + exchangeName;
try {
synchronized (exchangeLocker) {
synchronized (queueLocker) {
//2.检测内存中是否存在绑定
Binding existsBinding = memoryDataCenter.getBinding(exchangeName, queueName);
if(existsBinding != null) {
throw new MqException("[VirtualHost] 绑定已存在，创建失败！exchangeName=" + exchangeName +
"queueName=" + queueName);
}
//3.不存在，先检验 bindingKey 是否合法
if(!router.checkBindingKey(bindingKey)) {
throw new MqException("[VirtualHost] bindingKey 非法！bingdingKey=" + bindingKey);
}
//4.检验队列和交换机是否存在，不存在抛出异常
Exchange existsExchange = memoryDataCenter.getExchange(exchangeName);
if(existsExchange == null) {
throw new MqException("[VirtualHost] 交换机不存在！exchangeName=" + exchangeName);
}
MSGQueue existsQueue = memoryDataCenter.getQueue(queueName);
if(existsQueue == null) {
throw new MqException("[VirtualHost] 队列不存在！queueName=" + queueName);
}
//5.创建绑定
Binding binding = new Binding();
binding.setBindingKey(bindingKey);
binding.setQueueName(queueName);
binding.setExchangeName(exchangeName);
//6.硬盘中创建绑定的前提是队列和交换机都是持久化的，否则绑定在硬盘中无意义
if(existsQueue.isDurable() && existsExchange.isDurable()) {
diskDataCenter.insertBinding(binding);
}
//7.写内存
memoryDataCenter.insertBinding(binding);
System.out.println("[VirtualHost] 绑定创建成功！exchangeName=" + exchangeName +
"queueName=" + queueName);
}
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 绑定创建失败！exchangeName=" + exchangeName +
"queueName=" + queueName);
e.printStackTrace();
return false;
}
}
/**
* 删除绑定关系
* @param queueName
* @param exchangeName
* @return
*/
public boolean queueUnBind(String queueName, String exchangeName) {
//1.虚拟主机前缀
queueName = virtualHostName + queueName;
exchangeName = virtualHostName + exchangeName;
try {
//这里的加锁顺序一定要和上面的两次加锁保持一致(防止死锁)
synchronized (exchangeLocker) {
synchronized (queueLocker) {
//2.检查绑定是否存在，不存在则抛出异常
Binding binding = memoryDataCenter.getBinding(exchangeName, queueName);
if(binding == null) {
throw new MqException("[VirtualHost] 绑定不存在, 删除失败！binding=" + binding);
}
//3.直接删除硬盘数据，不用判断是否存在，因为即使不存在绑定，直接删也不会有什么副作用，因为本质就是调用 sql 删除
diskDataCenter.deleteBinding(binding);
//4.删内存
memoryDataCenter.deleteBinding(binding);
System.out.println("[VirtualHost] 绑定删除成功！queueName=" + queueName +
", exchangeName=" + exchangeName);
}
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 绑定删除失败！queueName=" + queueName +
", exchangeName=" + exchangeName);
e.printStackTrace();
return false;
}
}
2.7. Message related operations
Publishing a message: The essence is to send a message to a designated switch/queue. First, determine whether the routingKey is legal, then obtain the switch, and then perform different processing according to the type of different switches (different switches send messages to the queue in different ways), and finally Call the send message method
Delivering a message: Sending a message essentially means writing the message to the memory/hard disk. Whether the message needs to be written to the hard disk depends on whether the message supports persistence (deliverMode of 1 means no persistence, and deliverMode of 2 means persistence). , and finally a logic needs to be added to notify the consumer that the message can be consumed (I won’t add it here, we will talk about it in the next chapter).
Ps: For the direct switch here, it is agreed to use routingKey as the queue name, and send the message directly to the specified queue , so that the binding relationship can be directly ignored (this is the most common scenario in actual development, so it is set like this here)
/**
* 发送消息到指定的 交换机/队列 中
* @param exchangeName
* @return
*/
public boolean basicPublish(String exchangeName, String routingKey, BasicProperties basicProperties, byte[] body) {
//1.虚拟主机前缀
exchangeName = virtualHostName + exchangeName;
try {
//2.判定 routingKey 是否合法
if(!router.checkRoutingKey(routingKey)) {
throw new MqException("[VirtualHost] routingKey 非法！routingKey=" + routingKey);
}
//3.获取交换机
Exchange exchange = memoryDataCenter.getExchange(exchangeName);
if(exchange == null) {
throw new MqException("[VirtualHost] 交换机不存在，消息发送失败！exchangeName=" + exchangeName);
}
//4.判定交换机类型
if(exchange.getType() == ExchangeType.DIRECT) {
//直接交换机转发消息，以 routingKey 作为队列名字
//直接把消息写入指定队列，也就可以无视绑定关系(这是实际开发环境中最常用的，因此这里这样设定)
String queueName = virtualHostName + routingKey;
//5.构造消息对象
Message message = Message.createMessageWithId(routingKey, basicProperties, body);
//6.查找该队列名对应的对象
MSGQueue queue = memoryDataCenter.getQueue(queueName);
if(queue == null) {
throw new MqException("[VirtualHost] 队列不存在！queueName=" + queueName);
}
//7.发送消息
sendMessage(queue, message);
} else {
// 按照 fanout 和 topic 的方式转发
//5.找到该交换机的所有绑定，并遍历绑定对象
ConcurrentHashMap<String, Binding> bindingsMap = memoryDataCenter.getBindings(exchangeName);
for(Map.Entry<String, Binding> entry : bindingsMap.entrySet()) {
// 1) 获取到绑定对象，判定对应的队列是否存在
Binding binding = entry.getValue();
MSGQueue queue = memoryDataCenter.getQueue(binding.getQueueName());
if(queue == null) {
//此处不抛异常了，因为又多个这样的队列
//希望不要因为一个队列匹配失败，影响到其他队列
System.out.println("[VirtualHost] basicPublish 发送消息时，发现队列不存在！queueName=" + binding.getQueueName());
}
// 2) 构造消息对象
Message message = Message.createMessageWithId(routingKey, basicProperties, body);
// 3) 判定这个消息是否能转发给该队列
// 如果是 fanout。所有绑定的队列都要转发
// 如果是 topic，还需要判定下，bindingKey 和 routingKey 是不是匹配
if(!router.route(exchange.getType(), binding, message)) {
continue;
}
sendMessage(queue, message);
}
}
return true;
} catch (Exception e) {
System.out.println("[VirtualHost] 消息发送失败！exchangeName=" + exchangeName +
", routingKey=" + routingKey);
e.printStackTrace();
return false;
}
}
/**
* 发送消息
* 实际上就是把消息写道 硬盘 和 内存 上
* @param queue
* @param message
*/
private void sendMessage(MSGQueue queue, Message message) throws IOException, MqException {
//判断当前消息是否要进行持久化
//deliverMode 为 1 表示不持久化，deliverMode 为 2 表示要持久化
int deliverMode = message.getDeliverMode();
if(deliverMode == 1) {
diskDataCenter.sendMessage(queue, message);
}
//写内存
memoryDataCenter.sendMessage(queue, message);
//TODO 此处还需要补充一个逻辑，通知消费者可以消费消息了
}
2.8. Supplementary thread safety issues
The lock granularity is too large. Do you need to adjust it?
A lot of locks are added here, and the lock granularity is still quite large, but adjustments can be made to refine the lock granularity, but the impact will not be big, because switches are created here, bindings are created, queues are created, switches are deleted... these They are all low-frequency operations!
Since it is a low-frequency operation, the probability of encountering a situation where both threads are operating to create a queue is very low. In addition, synchronized itself is also biased towards the lock state, and the lock is only really locked when there is actual competition. Therefore There is no need to adjust the lock granularity.
It is still necessary to add a lock here to deal with some rare extreme situations.
Since the VirtualHost layer of code is locked, do the operations in the MemoryDataCenter inside do not need to be locked? Does it make sense to lock it before?
We don't know which class the method of this class is called for. Currently, VirtualHost itself guarantees thread safety. At this time, VirtualHost internally calls MemoryDataCenter, and the problem of not locking it is not big, but if it is another class, If you call MemoryDataCenter from multiple threads, it will be unpredictable~~
