Previous studies Hadoop source code, the RPC module source code under hadoop-common module read through it, and spent three months Hadoop RPC code that copied it, learn a lot. I think the most effective way to learn programming is to copy the code, I think this process of helpful friends are learning programming, so I want to make the tutorial, here is the beginning of a weekend of writing, in the form of follow-up tutorials and progress will be released in public No, interested friends welcome to end the concern .
1. Origins
The story was talking about 19 years from the first half, when I was planning to look into Hadoop source. Now we've all heard Hadoop, it is a framework for distributed storage and computing. As a distributed system, communication between nodes, interactive essential. Hadoop implements its own RPC (Remote Procedure Call, Remote Procedure Call) module to meet this demand. With curiosity, I read the source code for the entire Hadoop RPC module, found that after reading this module design is very good, no coupling with other modules, can separate out as a separate frame. To be able to learn the relevant knowledge of programming, but also can see how the top-level Apache open source project code written, so I put Hadoop RPC module made tutorial.
Although this project is to implement an RPC function, but I think we should not focus too much attention RPC itself, but should focus on learning the client involved in the development of RPC, core services side development, network programming, multithreaded, concurrent programming, design patterns, etc. knowledge, especially for just learning Java is no actual contact line projects friend, master this knowledge, write other items will be more ideas.
2. named
In order to allow unfamiliar with Hadoop friend can follow this tutorial, we will Hadoop RPC gives new meaning business.
Suppose we have such a scenario, the company developed a new database, the underlying database may be Mysql, it may be MongoDB, the company may even be self-development of database technology, no matter what, can be used as the database server. As a user, using the underlying database technology is not what relationship, but about how to use. Database need to provide a user-friendly API calls, so you need to have a client.
Implement technology to connect the client request and server response is RPC. I give this project has a name called Manis, then the name of the database is ManisDb. We also work to encourage people to compare their project a name, has a name, it's like their own children, we will be more a sense of responsibility to do it well.
3. Advantages
Advantages are relative. Hadoop RPC compared to the average real items, it is the result of online testing, Hadoop clusters on the largest reach million units, a single RPC module can be used independent of actual combat. At the same time, we also accumulated experience in the development of top open source projects, large architectural design, small design patterns, code specifications. RPC client-side development, server development, network programming in three areas are involved, and are the main focus.
Hadoop RPC compared to the tutorial source code analysis class, the advantage of relatively strong real meaning. We will follow the source Hadoop RPC Manis from 0-1 to complete a wild knocking over our projects, reduction of 95%. Under explain why not 100%, on the one hand to focus, I will not too important, not very discard the core technology. On the other hand in order to meet the new definition of business, I would do some improvements, instead of copying Hadoop RPC. For example: Hadoop RPC version only supports 2.6 to Protobuf serialization protocol, but in order to reflect the coupling between the low and high expansion modules, Manis support multiple serialization protocol.
Before writing a tutorial, it took me about three months to control the source Hadoop RPC knocked out Manis. After learned Manis, you are fully capable of reading source Hadoop RPC, it would be the interview of bonus items now. Some readers may find this approach more LOW, but I believe Mr. Ma said, "copy code for your sensitivity," write code like learning calligraphy, learning how not to copy good work in people. Really want to see things into their own, the final solid way is to find their own path again. As part Manis is discarded, I will explain in the tutorial, when necessary, intercept Hadoop source code analysis together.
4. Manis Architecture FIG.
FIG 1-1 is a schematic diagram of Manis, substantially RPC general architecture of FIG.
Figure 1-1 Manis Chart
5. Manis timing chart core components
FIG timing diagram of core components 1-2 Manis
6. Manis core component concept
The core component in conjunction with FIG. 1-2 to Manis concepts involved will be described.
ManisDb : not illustrated, it represents the database is responsible for starting the server (ie ipc.Server class), need to explain here Our focus is on RPC development, database here only give an example, it does not involve a real database development.
ManisClient : available to the general user client classes, for ManisDb for CRUD, Protobuf protocol it uses to communicate with ManisDb.
Manager : providing the client class to the administrator, for ManisDb manage it uses Serializable protocol (Java native serialization) communicates with ManisDb
ProtoBufRpcEngine : Protobuf support RPC protocol engine, which defines two classes and inner classes --Invoker ProtoBufRpcInvoker class. Invoker class encapsulates the request for invoking the client, and protocol using Protobuf serialization. A method for performing class ProtoBufRpcInvoker client requests call (transfer server).
SerializableRpcEngine : Serializable support RPC protocol engine, which defines two classes and inner classes --Invoker SerializableRpcInvoker class. Invoker class encapsulates the request for invoking the client, and the Serializable serialization protocol. A method for performing class SerializableRpcInvoker client requests call (transfer server).
Client : establish socket connection with the server, the client receives the call and send a call request to the server, waiting for the server to return and returns the results.
ipc.Server：该类定义在ipc包下面，通过Reactor模式接收并处理客户端请求，最终调用ProtoBufRpcInvoker或SerializableRpcInvoker的方法获得结果，并返回给客户端。
公众号「渡码」，分享更多高质量内容
