In Open Source China, I saw an entry-level RPC framework implementation project using Spring + Netty + Protostuff + ZooKeeper. The first time I saw the introduction, I felt that this is a simplified version of dubbo, although only this function of the rpc is implemented, it is worth learning.
Directory :
1 Introduction
2. Usage and implementation ideas
3. Partial source code analysis
<!----Separation line---->
1 Introduction
rpc process diagram:
spring: mainly used for dependency injection, after serializing the object, use its proxy to call the target interface
netty: Simplifies the development of nio, and implements the development of communication protocols by adding codecs in the data flow process. Here, nio is used, and the serialization framework is integrated in the form of codecs.
protostuff: serialization framework, there are many types of products, mainly because the serialization that comes with jdk has been criticized
zk: used to maintain a list of services, mainly to use its strong data consistency to realize dynamic online and offline of services, to achieve service exposure, registration, discovery, etc.
Integration is a common implementation of rpc at present. rpc can be implemented based on application layer protocols or transport layer protocols, each with its own advantages and disadvantages. Implementing rpc based on tcp is more efficient, there is not as much redundant information as http requests, but for many problems such as handshake connection, disconnection and reconnection, heartbeat detection, etc., you need to develop yourself to increase the difficulty of development, and the result of http request is here for a long time. Development has taken all kinds of issues into consideration.
2. Usage and implementation ideas
The usage and implementation ideas are similar to dubbo.
How to use :
Service provider:
(a) Write the service provider interface, implement the service provider interface, configure the service provider into sprimg, and specify a port for publishing the service. The author added support for annotations.
(b) Encapsulate the zk client instance into the bean, configure it into spring, and specify the zk service ip address for service registration.
(c) Load the spring configuration file, and the service provider runs.
Service consumer:
(a) Configure the service provider interface into spring. When the user calls him, he can call the service provider remotely
(b) As above, encapsulate the zk client instance and configure it in spring, and specify the ip address of the zk service. The consumer discovers the service provider from the zk service (that is, obtains the To see if the service provider is online, run under that port)
(c) Create a proxy interface for remote calls
Implementation ideas :
The core is to customize an rpc protocol and use netty to send messages on the consumer side (the message contains the methods, classes, parameters, etc. that you want to call), and after the provider receives the local call of the message, a call result return message is generated (the message contains call status, error, result, etc.).
On the consumer side, the dynamic proxy of jdk is used to generate a proxy object of the provider interface, the target method of the proxy object is called (triggering the above process) and the result of the remote call is returned.
3. Partial source code analysis
(a) Encapsulate the zk client for service registration :
First, manually establish a permanent node/registry, and registered services will establish temporary nodes under this node. The core is the createNode method in ServiceRegistry.
//Constant, zk client timeout time, prefix and suffix of temporary nodes, etc.
public interface Constant {
int ZK_SESSION_TIMEOUT = 5000;
String ZK_REGISTRY_PATH = "/registry";
String ZK_DATA_PATH = ZK_REGISTRY_PATH + "/data";
}
//Registry center implementation (injected into spring)
public class ServiceRegistry {
private static final Logger LOGGER = LoggerFactory.getLogger(ServiceRegistry.class);
private CountDownLatch latch = new CountDownLatch(1);
private String registryAddress;
public ServiceRegistry(String registryAddress) {
this.registryAddress = registryAddress;
}
public void register(String data) {
if (data != null) {
ZooKeeper zk = connectServer();
if (zk != null) {
createNode(zk, data);
}
}
}
private ZooKeeper connectServer() {
ZooKeeper zk = null;
try {
zk = new ZooKeeper(registryAddress, Constant.ZK_SESSION_TIMEOUT, new Watcher() {
@Override
public void process(WatchedEvent event) {
if (event.getState() == Event.KeeperState.SyncConnected) {
latch.countDown();
}
}
});
latch.await();
} catch (IOException | InterruptedException e) {
LOGGER.error("", e);
}
return zk;
}
private void createNode(ZooKeeper zk, String data) {
try {
byte[] bytes = data.getBytes();
String path = zk.create(Constant.ZK_DATA_PATH, bytes, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
LOGGER.debug("create zookeeper node ({} => {})", path, data);
} catch (KeeperException | InterruptedException e) {
LOGGER.error("", e);
}
}
}
(b) Encapsulate the zk client for service discovery : the core is the watchNode method used to read the temporary node under zk, and the reading proves that the service is online and available.
public class ServiceDiscovery {
private static final Logger LOGGER = LoggerFactory.getLogger(ServiceDiscovery.class);
private CountDownLatch latch = new CountDownLatch(1);
private volatile List<String> dataList = new ArrayList<>();
private String registryAddress;
public ServiceDiscovery(String registryAddress) {
this.registryAddress = registryAddress;
ZooKeeper zk = connectServer();
if (zk != null) {
watchNode(zk);
}
}
public String discover() {
String data = null;
int size = dataList.size();
if (size > 0) {
if (size == 1) {
data = dataList.get(0);
LOGGER.debug("using only data: {}", data);
} else {
data = dataList.get(ThreadLocalRandom.current().nextInt(size));
LOGGER.debug("using random data: {}", data);
}
}
return data;
}
private ZooKeeper connectServer() {
ZooKeeper zk = null;
try {
zk = new ZooKeeper(registryAddress, Constant.ZK_SESSION_TIMEOUT, new Watcher() {
@Override
public void process(WatchedEvent event) {
if (event.getState() == Event.KeeperState.SyncConnected) {
latch.countDown();
}
}
});
latch.await();
} catch (IOException | InterruptedException e) {
LOGGER.error("", e);
}
return zk;
}
private void watchNode(final ZooKeeper zk) {
try {
List<String> nodeList = zk.getChildren(Constant.ZK_REGISTRY_PATH, new Watcher() {
@Override
public void process(WatchedEvent event) {
if (event.getType() == Event.EventType.NodeChildrenChanged) {
watchNode(zk);
}
}
});
List<String> dataList = new ArrayList<>();
for (String node : nodeList) {
byte[] bytes = zk.getData(Constant.ZK_REGISTRY_PATH + "/" + node, false, null);
dataList.add(new String(bytes));
}
LOGGER.debug("node data: {}", dataList);
this.dataList = dataList;
} catch (KeeperException | InterruptedException e) {
LOGGER.error("", e);
}
}
}
(c) Develop netty :
Service registration and discovery are mainly used to dynamically manage services and perform service governance.
The positive rpc function still relies on netty to achieve. The author's idea is to use netty to encapsulate an rpc protocol based on tcp (rpcRequest is used to transmit serialized objects including the interface class name, method name, parameters and other information that you want to call remotely, rpcRespnse is used to return the call status, call result, etc.).
Then customize the encoder and decoder of the rpc protocol for message communication, as follows: first obtain the registered service address from zk, use netty on the consumer side to send rpcRequest to the service provider, and the service provider side uses the customized rpc The decoder parses the rpcRequest to obtain a method that the consumer wants to call, then uses reflection to call it, and then constructs the result into an rpcResponse and sends it back to the consumer. The consumer again uses netty to parse the rpcResponse to obtain the call result.
The related technologies used in the whole process include: custom rpcRequest class, rpcResponse class, two decoders, zk client reads temporary nodes, and SimpleChannelInboundHandler is used between decoding and encoding to process rpc requests (the real use of reflection invoke() to call is in In this class), serialization process (serialization is mainly performed on the consumer side, and reflection calls are performed on the provider side, at this time, the efficient serialization framework is used)
rpcRequest/rpcResponse：
public class RpcRequest {
private String requestId;
private String className;
private String methodName;
private Class<?>[] parameterTypes;
private Object[] parameters;
// getter/setter...
}
public class RpcResponse {
private String requestId;
private Throwable error;
private Object result;
// getter/setter...
}
Codec for rpcRequest/rpcResponse:
public class RpcDecoder extends ByteToMessageDecoder {
private Class<?> genericClass;
public RpcDecoder(Class<?> genericClass) {
this.genericClass = genericClass;
}
@Override
public void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
if (in.readableBytes() < 4) {
return;
}
in.markReaderIndex();
int dataLength = in.readInt();
if (dataLength < 0) {
ctx.close();
}
if (in.readableBytes() < dataLength) {
in.resetReaderIndex();
return;
}
byte[] data = new byte[dataLength];
in.readBytes(data);
Object obj = SerializationUtil.deserialize(data, genericClass);
out.add(obj);
}
}
public class RpcEncoder extends MessageToByteEncoder {
private Class<?> genericClass;
public RpcEncoder(Class<?> genericClass) {
this.genericClass = genericClass;
}
@Override
public void encode(ChannelHandlerContext ctx, Object in, ByteBuf out) throws Exception {
if (genericClass.isInstance(in)) {
byte[] data = SerializationUtil.serialize(in);
out.writeInt(data.length);
out.writeBytes(data);
}
}
}
Serializer class:
The serialization framework is integrated. Of course, you can use the native jdk first. If you want to use other ones such as marshling and protobuf, you can directly modify the implementation in the tool class.
public class SerializationUtil {
private static Map<Class<?>, Schema<?>> cachedSchema = new ConcurrentHashMap<>();
private static Objenesis objenesis = new ObjenesisStd(true);
private SerializationUtil() {
}
@SuppressWarnings("unchecked")
private static <T> Schema<T> getSchema(Class<T> cls) {
Schema<T> schema = (Schema<T>) cachedSchema.get(cls);
if (schema == null) {
schema = RuntimeSchema.createFrom(cls);
if (schema != null) {
cachedSchema.put(cls, schema);
}
}
return schema;
}
@SuppressWarnings("unchecked")
public static <T> byte[] serialize(T obj) {
Class<T> cls = (Class<T>) obj.getClass();
LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
try {
Schema<T> schema = getSchema(cls);
return ProtostuffIOUtil.toByteArray(obj, schema, buffer);
} catch (Exception e) {
throw new IllegalStateException(e.getMessage(), e);
} finally {
buffer.clear();
}
}
public static <T> T deserialize(byte[] data, Class<T> cls) {
try {
T message = (T) objenesis.newInstance(cls);
Schema<T> schema = getSchema(cls);
ProtostuffIOUtil.mergeFrom(data, message, schema);
return message;
} catch (Exception e) {
throw new IllegalStateException(e.getMessage(), e);
}
}
}
The handler for processing rpc requests on the provider side:
You can directly inherit Netty's SimpleChannelInboundHandler.
public class RpcHandler extends SimpleChannelInboundHandler<RpcRequest> {
private static final Logger LOGGER = LoggerFactory.getLogger(RpcHandler.class);
private final Map<String, Object> handlerMap;
public RpcHandler(Map<String, Object> handlerMap) {
this.handlerMap = handlerMap;
}
@Override
public void channelRead0(final ChannelHandlerContext ctx, RpcRequest request) throws Exception {
RpcResponse response = new RpcResponse();
response.setRequestId(request.getRequestId());
try {
Object result = handle(request);
response.setResult(result);
} catch (Throwable t) {
response.setError(t);
}
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
}
private Object handle(RpcRequest request) throws Throwable {
String className = request.getClassName();
Object serviceBean = handlerMap.get(className);
Class<?> serviceClass = serviceBean.getClass();
String methodName = request.getMethodName();
Class<?>[] parameterTypes = request.getParameterTypes();
Object[] parameters = request.getParameters();
/*Method method = serviceClass.getMethod(methodName, parameterTypes);
method.setAccessible(true);
return method.invoke(serviceBean, parameters);*/
FastClass serviceFastClass = FastClass.create(serviceClass);
FastMethod serviceFastMethod = serviceFastClass.getMethod(methodName, parameterTypes);
return serviceFastMethod.invoke(serviceBean, parameters);
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
LOGGER.error("server caught exception", cause);
ctx.close();
}
}
public class RpcClient extends SimpleChannelInboundHandler<RpcResponse> {
private static final Logger LOGGER = LoggerFactory.getLogger(RpcClient.class);
private String host;
private int port;
private RpcResponse response;
private final Object obj = new Object();
public RpcClient(String host, int port) {
this.host = host;
this.port = port;
}
@Override
public void channelRead0(ChannelHandlerContext ctx, RpcResponse response) throws Exception {
this.response = response;
synchronized (obj) {
obj.notifyAll(); // Receive the response, wake up the thread
}
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
LOGGER.error("client caught exception", cause);
ctx.close();
}
public RpcResponse send(RpcRequest request) throws Exception {
EventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(group).channel(NioSocketChannel.class)
.handler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel channel) throws Exception {
channel.pipeline()
.addLast(new RpcEncoder(RpcRequest.class)) // Encode the RPC request (for sending the request)
.addLast(new RpcDecoder(RpcResponse.class)) // Decode the RPC response (in order to process the response)
.addLast(RpcClient.this); // Send RPC request using RpcClient
}
})
.option(ChannelOption.SO_KEEPALIVE, true);
ChannelFuture future = bootstrap.connect(host, port).sync();
future.channel().writeAndFlush(request).sync();
synchronized (obj) {
obj.wait(); // no response, make thread wait
}
if (response != null) {
future.channel().closeFuture().sync();
}
return response;
} finally {
group.shutdownGracefully();
}
}
}
(d) Agency development :
The consumer wants to call the provider and needs an object to call its method. The consumer uses a proxy object because it is a remote call. The main purpose is to generate a proxy object according to the interface to be called. The core lies in the use of the above netty in the InvocationHandler method of the proxy object. Send rpc request, get rpc want and then generate call result and return. So when you use the proxy object to call the target method, you get the result of the remote call. This creates the effect of a remote method being called locally
public class RpcProxy {
private String serverAddress;
private ServiceDiscovery serviceDiscovery;
public RpcProxy(String serverAddress) {
this.serverAddress = serverAddress;
}
public RpcProxy(ServiceDiscovery serviceDiscovery) {
this.serviceDiscovery = serviceDiscovery;
}
@SuppressWarnings("unchecked")
public <T> T create(Class<?> interfaceClass) {
return (T) Proxy.newProxyInstance(
interfaceClass.getClassLoader(),
new Class<?>[]{interfaceClass},
new InvocationHandler() {
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
RpcRequest request = new RpcRequest(); // Create and initialize RPC request
request.setRequestId(UUID.randomUUID().toString());
request.setClassName(method.getDeclaringClass().getName());
request.setMethodName(method.getName());
request.setParameterTypes(method.getParameterTypes());
request.setParameters(args);
if (serviceDiscovery != null) {
serverAddress = serviceDiscovery.discover(); // discover service
}
String[] array = serverAddress.split(":");
String host = array[0];
int port = Integer.parseInt(array[1]);
RpcClient client = new RpcClient(host, port); // Initialize RPC client
RpcResponse response = client.send(request); // Send RPC request through RPC client and get RPC response
if (response.isError()) {
throw response.getError();
} else {
return response.getResult();
}
}
}
);
}
}