Table of contents
1) Implement ConnectionFactory
1. Client code implementation
1.1. Demand analysis
RabbitMQ client settings: A client can have multiple modules, and each module can establish a "logical connection" (channel) with the broker server. The channels of these modules do not affect each other. At the same time, these channels reuse the same TCP connection, eliminating the overhead of frequently establishing/destroying TCP connections (three handshakes, four waves...).
Here, we also implement the message queue client according to this logic, which mainly involves the following three core classes:
- ConnectionFactory: Connection factory. This class holds the address of the server. Its main function is to create Connection objects.
- Connection: Represents a TCP connection, holds a Socket object, is used to write requests/read responses, and manages multiple Channel objects.
- Channel: Represents a logical connection and needs to provide a series of methods to correspond to the core API provided by the server (internally, these methods provided by the client write a specific request and then wait for the server to respond).
1.2. Specific implementation
1) Implement ConnectionFactory
Mainly used to create Connection objects.
public class ConnectionFactory {
//broker server 的 ip 地址
private String host;
//broker server 的端口号
private int port;
// //访问 broker server 的哪个虚拟主机
// //这里暂时先不涉及
// private String virtualHostName;
// private String username;
// private String password;
public Connection newConnection() throws IOException {
Connection connection = new Connection(host, port);
return connection;
}
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public int getPort() {
return port;
}
public void setPort(int port) {
this.port = port;
}
}
2) Implement Connection
The properties are as follows
private Socket socket;
//一个 socket 连接需要管理多个 channel
private ConcurrentHashMap<String, Channel> channelMap = new ConcurrentHashMap<>();
private InputStream inputStream;
private OutputStream outputStream;
// DataXXX 主要用来 读取/写入 特定格式数据(例如 readInt())
private DataInputStream dataInputStream;
private DataOutputStream dataOutputStream;
//用来处理 0xc 的回调，这里开销可能会很大，不希望把 Connection 阻塞住，因此使用 线程池 来处理
private ExecutorService callbackPool;
The structure is as follows
Not only do you need to initialize the properties, you also need to create a scanning thread. This thread is responsible for continuously reading the response data from the socket, and then handing the response data to the corresponding channel for processing.
public Connection(String host, int port) throws IOException {
socket = new Socket(host, port);
inputStream = socket.getInputStream();
outputStream = socket.getOutputStream();
dataInputStream = new DataInputStream(inputStream);
dataOutputStream = new DataOutputStream(outputStream);
callbackPool = Executors.newFixedThreadPool(4);
//创建一个扫描线程，由这个线程负责不停的从 socket 中读取响应数据，把这个响应数据再交给对应的 channel 负责处理
Thread t = new Thread(() -> {
try {
while(!socket.isClosed()) {
Response response = readResponse();
dispatchResponse(response);
}
} catch (SocketException e) {
//连接正常断开的，此时这个异常可以忽略
System.out.println("[Connection] 连接正常断开！");
} catch(IOException | ClassNotFoundException | MqException e) {
System.out.println("[Connection] 连接异常断开！");
e.printStackTrace();
}
});
t.start();
}
Release Connection related resources
public void close() {
try {
callbackPool.shutdown();
channelMap.clear();
inputStream.close();
outputStream.close();
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Use this method to distinguish whether the current response is a response to a control request or a message pushed by the server.
If it is a message pushed by the server, the response indicates 0xc, which is a callback and is processed through the thread pool;
If it is just a normal response, put the result into the channel's hash table (then the channel will wake up all threads blocked waiting for the response to get the data from the map).
public void dispatchResponse(Response response) throws IOException, ClassNotFoundException, MqException {
if(response.getType() == 0xc) {
//服务器推送过来的消息数据
SubScribeReturns subScribeReturns = (SubScribeReturns) BinaryTool.fromBytes(response.getPayload());
//根据 channelId 找到对应的 channel 对象
Channel channel = channelMap.get(subScribeReturns.getChannelId());
if(channel == null) {
throw new MqException("[Connection] 该消息对应的 channel 再客户端中不存在！channelId=" + channel.getChannelId());
}
//执行该 channel 对象内部的回调(这里的开销未知，有可能很大，同时不希望把这里阻塞住，所以使用线程池来执行)
callbackPool.submit(() -> {
try {
channel.getConsumer().handlerDelivery(subScribeReturns.getConsumerTag(), subScribeReturns.getBasicProperties(),
subScribeReturns.getBody());
} catch(MqException | IOException e) {
e.printStackTrace();
}
});
} else {
//当前响应是针对刚才的控制请求的响应
BasicReturns basicReturns = (BasicReturns) BinaryTool.fromBytes(response.getPayload());
//把这个结果放到 channel 的 哈希表中
Channel channel = channelMap.get(basicReturns.getChannelId());
if(channel == null) {
throw new MqException("[Connection] 该消息对应的 channel 在客户端中不存在！channelId=" + channel.getChannelId());
}
channel.putReturns(basicReturns);
}
}
Send requests and read responses
/**
* 发送请求
* @param request
* @throws IOException
*/
public void writeRequest(Request request) throws IOException {
dataOutputStream.writeInt(request.getType());
dataOutputStream.writeInt(request.getLength());
dataOutputStream.write(request.getPayload());
dataOutputStream.flush();
System.out.println("[Connection] 发送请求！type=" + request.getType() + ", length=" + request.getLength());
}
/**
* 读取响应
*/
public Response readResponse() throws IOException {
Response response = new Response();
response.setType(dataInputStream.readInt());
response.setLength(dataInputStream.readInt());
byte[] payload = new byte[response.getLength()];
int n = dataInputStream.read(payload);
if(n != response.getLength()) {
throw new IOException("读取的响应格式不完整! n=" + n + ", responseLen=" + response.getLength());
}
response.setPayload(payload);
System.out.println("[Connection] 收到响应！type=" + response.getType() + ", length=" + response.getLength());
return response;
}
Provide methods to create Channel in Connection
public Channel createChannel() throws IOException {
String channelId = "C-" + UUID.randomUUID().toString();
Channel channel = new Channel(channelId, this);
//放到 Connection 管理的 channel 的 Map 集合中
channelMap.put(channelId, channel);
//同时也需要把 “创建channel” 这个消息告诉服务器
boolean ok = channel.createChannel();
if(!ok) {
//如果创建失败，就说明这次创建 channel 操作不顺利
//把刚才加入 hash 表的键值对再删了
channelMap.remove(channelId);
return null;
}
return channel;
}
Ps: UUIDs are used many times in the code. Here we use prefixes to distinguish them as before.
3) Implement Channel
The properties and construction are as follows
private String channelId;
// 当前这个 channel 是属于哪一个连接
private Connection connection;
//用来存储后续客户端收到的服务器响应，已经辨别是哪个响应(要对的上号) key 是 rid
private ConcurrentHashMap<String, BasicReturns> basicReturnsMap = new ConcurrentHashMap<>();
//如果当前 Channel 订阅了某个队列，就需要记录对应的回调是什么，当该队列消息返回回来的时候，调用回调
//此处约定一个 Channel 只能有一个回调
private Consumer consumer;
public Channel(String channelId, Connection connection) {
this.channelId = channelId;
this.connection = connection;
}
public String getChannelId() {
return channelId;
}
public void setChannelId(String channelId) {
this.channelId = channelId;
}
public Connection getConnection() {
return connection;
}
public void setConnection(Connection connection) {
this.connection = connection;
}
public ConcurrentHashMap<String, BasicReturns> getBasicReturnsMap() {
return basicReturnsMap;
}
public void setBasicReturnsMap(ConcurrentHashMap<String, BasicReturns> basicReturnsMap) {
this.basicReturnsMap = basicReturnsMap;
}
public Consumer getConsumer() {
return consumer;
}
public void setConsumer(Consumer consumer) {
this.consumer = consumer;
Implement 0x1 to create channel
The main thing is to construct the request, then send the request to the BrokerServer server, and block waiting for the server response.
/**
* 0x1
* 和服务器进行交互，告诉服务器，此处客户端已经创建了新的 channel 了
* @return
*/
public boolean createChannel() throws IOException {
//构造 payload
BasicArguments arguments = new BasicArguments();
arguments.setChannelId(channelId);
arguments.setRid(generateRid());
byte[] payload = BinaryTool.toBytes(arguments);
//发送请求
Request request = new Request();
request.setType(0x1);
request.setLength(payload.length);
request.setPayload(payload);
connection.writeRequest(request);
//等待服务器响应
BasicReturns basicReturns = waitResult(arguments.getRid());
return basicReturns.isOk();
}
/**
* 生成 rid
* @return
*/
public String generateRid() {
return "R-" + UUID.randomUUID().toString();
}
/**
* 阻塞等待服务器响应
* @param rid
* @return
*/
private BasicReturns waitResult(String rid) {
BasicReturns basicReturns = null;
while((basicReturns = basicReturnsMap.get(rid)) == null) {
//查询结果为空，就说明咱们去菜鸟驿站要取的包裹还没到
//此时就需要阻塞等待
synchronized (this) {
try {
wait();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
basicReturnsMap.remove(rid);
return basicReturns;
}
/**
* 由 Connection 中的方法调用，区分为普通响应之后触发
* 将响应放回到 channel 管理的 map 中，并唤醒所有线程
* @param basicReturns
*/
public void putReturns(BasicReturns basicReturns) {
basicReturnsMap.put(basicReturns.getRid(), basicReturns);
synchronized (this) {
//当前也不知道有多少线程再等待上述的这个响应
//因此就把所有等待的线程唤醒
notifyAll();
}
}
Ps: Other request operations are almost the same as 0x1. I will not show them one by one here. I will mainly talk about 0xa.
0xa consumers subscribe to queue messages. Here, the callback must be set to the attribute first, so that Connection can handle the callback through this attribute.
It is worth noting that we agree that channelId is consumerTag
public boolean basicConsume(String queueName, boolean autoAck, Consumer consumer) throws IOException, MqException {
//先设置回调
if(this.consumer != null) {
throw new MqException("该 channel 已经设置过消费消息回调了，不能重复！");
}
this.consumer = consumer;
BasicConsumeArguments basicConsumeArguments = new BasicConsumeArguments();
basicConsumeArguments.setRid(generateRid());
basicConsumeArguments.setChannelId(channelId);
basicConsumeArguments.setConsumerTag(channelId); // 注意：此处的 consumerTag 使用 channelId 来表示
basicConsumeArguments.setQueueName(queueName);
basicConsumeArguments.setAutoAck(autoAck);
byte[] payload = BinaryTool.toBytes(basicConsumeArguments);
Request request = new Request();
request.setType(0xa);
request.setLength(payload.length);
request.setPayload(payload);
connection.writeRequest(request);
BasicReturns basicReturns = waitResult(basicConsumeArguments.getRid());
return basicReturns.isOk();
}
2. Supplement: About the callback execution process
The process is as follows:
- The client calls the basicConsume method, creates a consumer to subscribe to the queue message, and brings the Consumer (the specific actions the consumer will do after getting the message);
- Channel saves the Consumer sent by the client (waiting to receive the 0xc response to actually execute the callback), sends the subscription queue message request (0xa), and waits for the response
- After receiving the request, BrokerServer parsed out 0xa and created a Consumer (the purpose is to wait for the server to get the message to be consumed, and then package the message data into a 0xc response. After the client receives the response, it executes "Consumer" The specific actions to be done after getting the message" This is the setting of RabbitMQ), and then BrokerServer calls VirtualHost.
- After VirtualHost creates a new consumer subscription queue, if it finds another message in the queue, it will consume it immediately.
- Specifically, it calls the newly created callback and then returns a 0xc response to the client. After the client receives the response, it executes "the specific actions that the consumer will do after getting the message."
3. Write Demo
3.1. Example
At this point, a producer-consumer model across hosts/servers has been basically implemented (functionally capable of meeting the use of message queues in daily development), but it also has strong scalability. You can continue to refer to RabbitMQ. If If you have any ideas, or if you encounter problems that you don’t know, you can send me a private message~
Next, I will write a demo to simulate the producer-consumer model across hosts/servers (for convenience, I will demonstrate it on this machine).
First create the BrokerServer in the startup class of the spring boot project, bind the port number, and then start
@SpringBootApplication
public class RabbitmqProjectApplication {
public static ConfigurableApplicationContext context;
public static void main(String[] args) throws IOException {
context = SpringApplication.run(RabbitmqProjectApplication.class, args);
BrokerServer brokerServer = new BrokerServer(9090);
brokerServer.start();
}
}
Write consumer
public class DemoConsumer {
public static void main(String[] args) throws IOException, MqException, InterruptedException {
//建立连接
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setPort(9090);
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
//创建交换机和队列(这里和生产者创建交换机和队列不冲突，谁先启动，就按照谁的创建，即使已经存在交换机和队列，再创建也不会有什么副作用)
channel.exchangeDeclare("demoExchange", ExchangeType.DIRECT, true, false, null);
channel.queueDeclare("demoQueue", true, false, false, null);
//消费者消费消息
channel.basicConsume("demoQueue", true, new Consumer() {
@Override
public void handlerDelivery(String consumerTag, BasicProperties basicProperties, byte[] body) throws MqException, IOException {
System.out.println("开销消费");
System.out.println("consumerTag=" + consumerTag);
System.out.println("body=" + new String(body));
System.out.println("消费完毕");
}
});
//由于消费者不知道生产者要生产多少，就在这里通过循环模拟一直等待
while(true) {
Thread.sleep(500);
}
}
}
Write producer
public class DemoProducer {
public static void main(String[] args) throws IOException, InterruptedException {
//建立连接
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setPort(9090);
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
//创建交换机和队列(这里和消费者创建交换机和队列不冲突，谁先启动，就按照谁的创建，即使已经存在交换机和队列，再创建也不会有什么副作用)
channel.exchangeDeclare("demoExchange", ExchangeType.DIRECT, true, false, null);
channel.queueDeclare("demoQueue", true, false, false, null);
//生产消息
byte[] body1 = "Im cyk1 !".getBytes();
channel.basicPublish("demoExchange", "demoQueue", null, body1);
Thread.sleep(500);
//关闭连接
channel.close();
connection.close();
}
}
3.2. Example demonstration
Start the spring boot project (start BrokerServer)
Run the consumer (either the consumer or the producer can be run sequentially)
Run the producer
