Since the bottom layer of Dubbo uses Socket for communication, I am not very clear about the theory of communication, so I also learn the knowledge of communication by the way.
communication theory
The exchange of information between a computer and the outside world is called communication. The basic communication methods are parallel communication and serial communication.
1. A communication method in which each bit of a set of information (usually bytes) is transmitted simultaneously is called parallel communication. Parallel communication relies on the parallel I/O interface to realize. Parallel communication is fast, but the number of transmission lines is large, and it is only suitable for short-distance (a few meters) communication.
2. The communication method in which each bit data of a group of information is sequentially transmitted bit by bit is called serial communication. Serial communication can be achieved through a serial interface. The serial communication speed is slow, but there are few transmission lines, which is suitable for long-distance communication.
Serial communication is divided into the following three types according to the direction of information transmission:
1) Simplex
Data can only be transferred in one direction
2) Half duplex
Information can be transmitted in both directions, but not both at the same time
3) Full duplex
Can transmit in both directions and can transmit in both directions at the same time
Socket
Socket is an application interface, and TCP/IP is a network transmission protocol. Although the interface is the same, different protocols have different service properties. When creating a Socket connection, you can specify the transport layer protocol used. Socket can support different transport layer protocols (TCP or UDP). When using the TCP protocol to connect, the Socket connection is a TCP connection. Soket is not necessarily related to TCP/IP. When the Socket programming interface was designed, it was hoped that it could also adapt to other network protocols. Therefore, the appearance of socket is only to make it more convenient to use the TCP/IP protocol stack.
Quoted from: http://hi.baidu.com/lewutian/blog/item/b28e27fd446d641d09244d08.html
The last communication theory is actually to say that Socket (TCP) communication is a full-duplex method
Analysis of the principle of Dubbo remote synchronization call
From the Dubbo open source documentation, I learned that a calling process is as follows
http://code.alibabatech.com/wiki/display/dubbo/User+Guide#UserGuide-APIReference
In addition, it is stated in the document: Dubbo's default protocol uses a single long connection and NIO asynchronous communication, which is suitable for small data volume and large concurrent service calls, and the number of service consumer machines is much larger than the number of service provider machines.
Dubbo default protocol, using tbremoting interaction based on mina1.1.7+hessian3.2.1.
• Number of connections: single connection
•Connection method: long connection
•Transport protocol: TCP
• Transmission method: NIO asynchronous transmission
• Serialization: Hessian binary serialization
• Scope of application: The incoming and outgoing parameter data packets are small (recommended less than 100K), the number of consumers is more than the number of providers, and a single consumer cannot fill the provider. Try not to use the dubbo protocol to transmit large files or large strings .
• Applicable scenarios: regular remote service method calls
In general, a typical synchronous remote call would look like this:
1. The client thread calls the remote interface and sends a request to the server. At the same time, the current thread should be in a "suspended" state, that is, the thread cannot be executed backwards. It must get the result given to itself by the server before executing backwards.
2. After the server receives the client request, it processes the request and sends the result to the client
3. The client receives the result, and then the current thread continues to execute
Dubbo uses Socket (using the apache mina framework as the underlying call) to establish a long connection, send and receive data, and the bottom layer uses the IoSession of the apache mina framework to send messages.
Looking at the Dubbo documentation and source code, it can be seen that the bottom layer of Dubbo uses Socket to send messages for data transmission. Combined with the mina framework, the IoSession.write() method is used. After this method is called, the entire remote call (from sending the request to receiving the result) is called. It is asynchronous, that is, for the current thread, when the request is sent, the thread can be executed later. As for the result of the server, it is sent to the client in the form of a message after the server has completed the processing. So here are 2 questions:
• How does the current thread "pause" and execute backwards after the result comes back?
• As mentioned earlier, Socket communication is a full-duplex method. If there are multiple threads performing remote method calls at the same time, there will be many messages sent by both parties on the socket connection established between the client and server. It may also be messy. After the server processes the results, it sends the result message to the client. The client receives a lot of messages. How do you know which message result was called by which thread?
________________________________________
Analysis of the source code, the basic principles are as follows:
1. A client thread calls the remote interface to generate a unique ID (such as a random string, UUID, etc.), Dubbo uses AtomicLong to accumulate numbers from 0
2. Encapsulate the packaged method invocation information (such as the interface name, method name, parameter value list, etc.) and the callback object callback of the processing result together to form an object object
3. Put(ID, object) into the global ConcurrentHashMap that stores the call information
4. Encapsulate the ID and the packaged method call information into an object connRequest, and use IoSession.write(connRequest) to send it asynchronously
5. The current thread then uses the get() method of callback to try to obtain the result returned by the remote. Inside get(), use synchronized to obtain the lock of the callback object callback, and then first check whether the result has been obtained, if not, then call the callback The wait() method releases the lock on the callback, leaving the current thread in a waiting state.
6. After the server receives the request and processes it, it sends the result (this result contains the previous ID, that is, the return) to the client. The thread on the client's socket connection that monitors the message receives the message, analyzes the result, and retrieves the message. To the ID, then get(ID) from the previous ConcurrentHashMap, so as to find the callback, and set the result of the method call to the callback object.
7. The listening thread then uses synchronized to obtain the lock of the callback object callback (because wait() was called before, the thread has released the lock of the callback), and then notifyAll(), wake up the thread in the waiting state to continue execution (callback get () method continues to execute to get the result of the call), at this point, the whole process ends.
There is also a need to draw a large picture to describe, which will be added later.
It should be noted that the callback object here is a new one for each call and cannot be shared, otherwise there will be problems; in addition, the ID must be at least unique within a Socket connection.
________________________________________
Now, the first two questions have been answered,
• How does the current thread "pause" and execute backwards after the result comes back?
Answer: First generate an object obj, store it in a global map with put(ID, obj), then use synchronized to acquire the obj lock, then call obj.wait() to make the current thread wait, and then another message listener thread waits until After the server-side result comes, map.get(ID) to find obj, then use synchronized to acquire the obj lock, and then call obj.notifyAll() to wake up the thread in the waiting state.
• As mentioned earlier, Socket communication is a full-duplex method. If there are multiple threads performing remote method calls at the same time, there will be many messages sent by both parties on the socket connection established between the client and server. It may also be messy. After the server processes the results, it sends the result message to the client. The client receives a lot of messages. How do you know which message result was called by which thread?
Answer: Use an ID to make it unique, then pass it to the server, and then the server returns it back, so that you know which thread the result belongs to.
This is not the first time I have seen this practice. In the last company for 10 years, it is also a remote interface call, but the message middleware rabbitmq is used. The principle of synchronous calling is similar to this. For details, see: rabbitmq learning-9- RpcClient sends messages and the principle of synchronously receiving messages
Key code:
| com.taobao.remoting.impl.DefaultClient.java // Call the remote interface synchronously public Object invokeWithSync(Object appRequest, RequestControl control) throws RemotingException, InterruptedException { byte protocol = getProtocol(control); if (!TRConstants.isValidProtocol(protocol)) { throw new RemotingException("Invalid serialization protocol [" + protocol + "] on invokeWithSync."); } ResponseFuture future = invokeWithFuture(appRequest, control); return future.get () ; // Let the current thread wait when getting the result, ResponseFuture is actually the callback mentioned earlier } public ResponseFuture invokeWithFuture(Object appRequest, RequestControl control) { byte protocol = getProtocol(control); long timeout = getTimeout(control); ConnectionRequest request = new ConnectionRequest(appRequest); request.setSerializeProtocol(protocol); Callback2FutureAdapter adapter = new Callback2FutureAdapter(request); connection.sendRequestWithCallback(request, adapter, timeout); return adapter; } |
|
Callback2FutureAdapter implements ResponseFuture public Object get() throws RemotingException, InterruptedException { synchronized (this) { // 旋锁 while (!isDone) { // 是否有结果了 wait(); //没结果是释放锁，让当前线程处于等待状态 } } if (errorCode == TRConstants.RESULT_TIMEOUT) { throw new TimeoutException("Wait response timeout, request[" + connectionRequest.getAppRequest() + "]."); } else if (errorCode > 0) { throw new RemotingException(errorMsg); } else { return appResp; } } 客户端收到服务端结果后，回调时相关方法，即设置isDone = true并notifyAll() public void handleResponse(Object _appResponse) { appResp = _appResponse; //将远程调用结果设置到callback中来 setDone(); } public void onRemotingException(int _errorType, String _errorMsg) { errorCode = _errorType; errorMsg = _errorMsg; setDone(); } private void setDone() { isDone = true; synchronized (this) { //获取锁，因为前面wait()已经释放了callback的锁了 notifyAll(); // 唤醒处于等待的线程 } } |
| com.taobao.remoting.impl.DefaultConnection.java
// 用来存放请求和回调的MAP private final ConcurrentHashMap<Long, Object[]> requestResidents;
//发送消息出去 void sendRequestWithCallback(ConnectionRequest connRequest, ResponseCallback callback, long timeoutMs) { long requestId = connRequest.getId(); long waitBegin = System.currentTimeMillis(); long waitEnd = waitBegin + timeoutMs; Object[] queue = new Object[4]; int idx = 0; queue[idx++] = waitEnd; queue[idx++] = waitBegin; //用于记录日志 queue[idx++] = connRequest; //用于记录日志 queue[idx++] = callback; requestResidents.put(requestId, queue); // 记录响应队列 write(connRequest);
// 埋点记录等待响应的Map的大小 StatLog.addStat("TBRemoting-ResponseQueues", "size", requestResidents.size(), 1L); } public void write(final Object connectionMsg) { //mina里的IoSession.write()发送消息 WriteFuture writeFuture = ioSession.write(connectionMsg); // 注册FutureListener，当请求发送失败后，能够立即做出响应 writeFuture.addListener(new MsgWrittenListener(this, connectionMsg)); }
/** * 在得到响应后，删除对应的请求队列，并执行回调 * 调用者：MINA线程 */ public void putResponse(final ConnectionResponse connResp) { final long requestId = connResp.getRequestId(); Object[] queue = requestResidents.remove(requestId); if (null == queue) { Object appResp = connResp.getAppResponse(); String appRespClazz = (null == appResp) ? "null" : appResp.getClass().getName(); StringBuilder sb = new StringBuilder(); sb.append("Not found response receiver for requestId=[").append(requestId).append("],"); sb.append("from [").append(connResp.getHost()).append("],"); sb.append("response type [").append(appRespClazz).append("]."); LOGGER.warn(sb.toString()); return; } int idx = 0; idx++; long waitBegin = (Long) queue[idx++]; ConnectionRequest connRequest = (ConnectionRequest) queue[idx++]; ResponseCallback callback = (ResponseCallback) queue[idx++]; // ** 把回调任务交给业务提供的线程池执行 ** Executor callbackExecutor = callback.getExecutor(); callbackExecutor.execute(newCallbackExecutorTask(connResp, callback));
long duration = System.currentTimeMillis() - waitBegin; // 实际读响应时间 logIfResponseError(connResp, duration, connRequest.getAppRequest()); } |
| CallbackExecutorTask static private class CallbackExecutorTask implements Runnable { final ConnectionResponse resp; final ResponseCallback callback; final Thread createThread;
CallbackExecutorTask(ConnectionResponse _resp, ResponseCallback _cb) { resp = _resp; callback = _cb; createThread = Thread.currentThread(); }
public void run() { // 预防这种情况：业务提供的Executor，让调用者线程来执行任务 if (createThread == Thread.currentThread() && callback.getExecutor() != DIYExecutor.getInstance()) { StringBuilder sb = new StringBuilder(); sb.append("The network callback task [" + resp.getRequestId() + "] cancelled, cause:"); sb.append("Can not callback task on the network io thhread."); LOGGER.warn(sb.toString()); return; }
if (TRConstants.RESULT_SUCCESS == resp.getResult()) { callback.handleResponse(resp.getAppResponse()); //设置调用结果 } else { callback.onRemotingException(resp.getResult(), resp .getErrorMsg()); //处理调用异常 } } } |
另外：
1， 服务端在处理客户端的消息，然后再处理时，使用了线程池来并行处理，不用一个一个消息的处理
同样，客户端接收到服务端的消息，也是使用线程池来处理消息，再回调