Some time ago we discussed a cluster-based comprehensive data platform solution from SDP, consisting of multiple databases, including: JDBC, Cassandra and MongoDB. Among them, Cassandra and MongoDB are distributed databases that can be called by any deployment node in the cluster. The JDBC database is non-distributed and cannot be deployed on multiple nodes. Assuming that we provide the data processing functions of each database in the form of microservices, any invocation of JDBC database microservices from other cluster nodes requires data serialization. Although Cassandra and MongoDB are distributed, they are limited to communicating freely on the different nodes where they have been deployed. Now it seems that the interaction between different data microservices still needs to be realized through serialization. That is to say, the data needs to be serialized and marshalled before it can be sent, and after receiving, the data can be restored by deserialization and unmarshalling. I spent some time researching akka-http: the so-called system integration tool I initially selected at that time, it integrates the system through data exchange in json format. JSON is a standard data format, so data exchange between heterogeneous systems can be realized. I have been considering that if the SDP data platform microservices are integrated in the form of data exchange through akka-http, the internal interaction efficiency of this integrated system will be very low, because 1. json is a character data. , occupies a large space, and the transmission efficiency is naturally low. 2. Subject to the http1.0 interactive mode, it is convenient for data downlink but uplink data is limited to request instructions. This mode has limited interaction capabilities at the system level, or is very unnatural and inconvenient. It was also frustrating for a while. In fact, the threshold for using akka-http is very high, even though akka-http has provided many types of help http operations, but just understanding the content of the http protocol and httprequest, httpresponse details, construction, and usage took me a few Weeks of energy, and has only just reached a shallow level of understanding, if you can really mobilize nature in practical applications, you need to add more effort.
Google gRPC is a brand-new RPC framework, which has been an integration tool used within Google before open source. gRPC supports data exchange in protobuf format via http/2. Protobuf is protocol buffer, which is a new set of serialization-protocol invented by google. It is binary-encoded. Compared with java-object, XML, Json, etc., it has an advantage in space, so the data transmission efficiency is high. . Since gRPC supports the http/2 protocol, it can realize duplex-communication of two-way communication, which solves many limitations of the independent request/response interaction mode in software programming. This is a bright spot relative to aka-http in system integration programming. Data in protobuf format can be easily converted into data in json format, which supports open protocol data exchange to external systems. This is also the main reason why I decided to choose gRPC as a large-scale system microservice integration development tool. More importantly: client/server interaction with protobuf and gRPC does not involve any http objects including httprequest, httpresponse, it is easy to use, and it has successful experience in large companies such as google, so it will be more reliable to use.
gRPC supports the following four interaction protocols:
1. Unary: an independent pair of client-request/server-response, which is our commonly used http interaction mode
2. Server-Streaming: After the client sends a request, it receives a string of multiple responses from the server
3. Client-Streaming: The client sends a series of multiple requests to the server and then receives a response from the server
4. Bidirectional-Streaming: The client first sends a request to start the connection, and then the client/server can continue to exchange information on this connection.
I feel that the fourth way is most suitable for interaction at the program flow level. In other words, it can represent a natural program flow, although it still requires the client to actively initiate an interactive connection. Since the generated source code does not involve any http protocol related types and operations, it is easier to use.
In the scala programming world, we can use scalaPB to implement the use of gRPC and protobuf.
The usage process of google gRPC is as follows:
1. Create a .proto file to define data types and services in IDL language (Interface Definition Language).
2. Compile the .proto file to generate relevant java data types and abstract service frameworks
3. In java programming, you can directly call the data type generated by compilation and operate on the data
4. Inherit and implement the generated service class
scalaPB is a scala version of the protobuf compiler. After compiling the .proto file, the data types and abstract service classes of the scala language are generated, so that we can use protobuf and gRPC to implement integrated programming of microservices in the scala environment.
In the following blog posts, I will introduce the use of scalaPB for protobuf data conversion, gRPC microservice implementation, gRPC streaming operations, and conversion methods between gRPC streams and json.