Foreword : Speaking of JSON, you may be familiar with it. It is the most widely used serialization format. It is simple and convenient to use, and has high readability. But in more and more application scenarios, JSON's verbose shortcomings make it not an optimal choice.
1. Introduction to common serialization formats
Currently, JAVA commonly used serialization includes protobuf, json, xml, Serializable, hessian, and kryo. Their advantages and disadvantages are as follows:
-
JSON: Not much to say, it has a wide range of uses. The serialization method has also derived better transcoding tools such as Ali's fastjson, Meituan's MSON, and Google's GSON.
Advantages: easy to use.
Disadvantages: The data is verbose, and the transcoding performance is average. -
XML: Transcoding method long ago, not much used now.
Advantages: not found yet.
Disadvantages: The data is verbose, and the transcoding performance is average. -
Serialzable: Serialization that comes with JDK.
Advantages: easy to use.
Disadvantage: low transcoding performance. -
hessian: A remote communication library based on binary-RPC, using binary to transmit data.
Advantage: small data length.
Cons: Poor performance.
Having said so much, it's all about low performance, a group of mentally retarded MMPs? of course not! kryo is a fast and efficient serialization framework, but it is not our protagonist today, because it can only be used in java, and there is a great gap between communication with front-end non-java languages. Our protagonist today is protobuf? emmm, that's right, but it's not all, let me tell you about protobuf first.
- protobuf: An open source project developed by Google, with good performance, high efficiency, and supports multiple languages, such as: java, C++, python, etc.
Advantages: high transcoding performance, support for multiple languages.
Disadvantages: There are few Chinese documents, and the use is relatively complicated.
2. Detailed explanation of protobuf
Before using protobuf, you need to install the protobuf compiler and runtime environment.
Since protobuf is cross-platform and cross-language, you need to download and install the corresponding version of the compiler and runtime dependencies.
2.1 Introduction to proto syntax
| .proto Type | illustrate | C++ Type | Java Type | Python Type[2] | Go Type |
|---|---|---|---|---|---|
| double | double | double | float | float64 | |
| float | float | float | float | float32 | |
| int32 | Use variable length encoding. Inefficient when encoding negative numbers - if your field is going to use negative numbers, use sint32 instead. | int32 | int | int | int |
| int64 | Use variable length encoding. Inefficient when encoding negative numbers - if your field uses negative values, use sint64 instead. | int64 | long | int/long[3] | int64 |
| uint32 | Use variable length encoding | uint32 | int[1] | int/long[3] | uint32 |
| uint64 | Use variable length encoding | uint64 | long[1] | int/long[3] | uint64 |
The detailed grammar is not introduced here because there are too many chapters. For details, click on another blog post:
<!-- protobuf-google-->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.5.1</version>
</dependency>
2.2.1 Download the compiler compilation file
Download address: https://github.com/google/protobuf/releases
Select the version corresponding to the system, and unzip it after downloading.
Java code can be generated through the defined .proto file, and the protocol buffer compiler protoc needs to be run based on the .proto file. If you don't have the compiler installed, download the installation package and follow the README to install it.
Invoke the protocol compiler as follows:
protoc --proto_path=IMPORT_PATH --cpp_out=DST_DIR --java_out=DST_DIR --python_out=DST_DIR --go_out=DST_DIR --ruby_out=DST_DIR --javanano_out=DST_DIR --objc_out=DST_DIR --csharp_out=DST_DIR path/to/file.proto
IMPORT_PATHDeclare a specific directory for parsing import where a .proto file is located. If this value is omitted, the current directory is used. It can be called multiple times if there are multiple directories--proto_path, and they will be accessed and imported sequentially. Simplified form of-I=IMPORT_PATHyes .--proto_path- Of course one or more output paths can also be provided:
--java_outGenerate Java code in the target directory DST_DIR, see more in the Java code generation reference .
As a convenience extension, if DST_DIR ends with .zip or .jar, the compiler will write the output to a ZIP format file or a JAR-compliant .jar file. Note that the output will be overwritten if it already exists, the compiler is not yet smart enough to append the file.
- You must propose one or more .proto files as input, multiple .proto files can be specified only once. Although file paths are relative to the current directory, each file must be located under its IMPORT_PATH so that each file can determine its canonical name.
2.2.3 protobuf usage tutorial
// Take user as an example and encode it into byte[] UserOuterClass.User.Builder userBuild = UserOuterClass.User.newBuilder(); userBuild.setUserId(user.getUserId()); userBuild.setUserName(user.getUserName()); userBuild.setPhoneNum(user.getPhoneNum()); userBuild.setCreateTime(user.getCreateTime()); userBuild.setOpenId(user.getOpenId()); userBuild.setIntroduct(user.getIntroduct()); userBuild.setSex(user.isSex()); userBuild.setUserImg(user.getUserImg()); userBuild .toByteArray()；//得到byte[] // Take user as an example to decode UserOuterClass.User.Builder userBuild = UserOuterClass.User.newBuilder(); User user= user.build(); user=User.parseFrom(data.getValue().getBytes());
3. Problems existing in protobuf in actual operation
Protobuf is mainly used to communicate with the front-end codec, then how to store the binary received in the background into the database, or how to map the data obtained from the database to protobean.
Since the java files generated by protoc are different from the java files we usually write, but in fact they all have getset methods, children's shoes that are not afraid of trouble can be directly converted through the getset methods of the values of the two classes, which is quite efficient, but it does have some operations. trouble. Here we provide a more convenient tool class.
/**
* This method converts the javabean object into the bean corresponding to protobuf
*
* @param javaBean
* @param protoBuilder
*/
@SuppressWarnings({ "unchecked", "rawtypes" })
public static Object javaBeanToProtoBean(Object javaBean, Object protoBuilder) {
try {
Method mm = protoBuilder.getClass().getMethod("getDescriptorForType");
Descriptors.Descriptor descriptor = (Descriptor) mm.invoke(protoBuilder);
Field[] fields = javaBean.getClass().getDeclaredFields();
for (Field item : fields) {
try{
String fName = item.getName();
item.setAccessible(true);
Object jObject = item.get(javaBean);
if(null == jObject){
break;
}
FieldDescriptor fd = descriptor.findFieldByName(fName);
if(null != fd){
if(fd.isRepeated()){
boolean isDefined = false;
Method[] mmm = protoBuilder.getClass().getMethods();
for(Method mItem : mmm){
try{
String mName = mItem.getName();
String mName1 = "add" + StringUtil.firstToUpper(fName);
if(mName1.equals(mName) && mItem.getParameterTypes().length == 1){
Class[] ccList = mItem.getParameterTypes();
Class cc = ccList[0];
Method me = cc.getMethod("newBuilder");
Object oBuilder = me.invoke(null);//Get custom object builder
List<Object> dList = (List<Object>) jObject; //The data is a List collection
List<Object> pBeanList = new ArrayList<Object>();
for(Object oItem : dList){
Object pBean = javaBeanToProtoBean(oItem,oBuilder);
pBeanList.add(pBean);
}
Method mee = protoBuilder.getClass().getMethod("addAll"+StringUtil.firstToUpper(fName),Iterable.class);
mee.invoke(protoBuilder, pBeanList);
isDefined = true;
}
}catch(Exception e){
}
}
if(!isDefined){
try{
Method me = protoBuilder.getClass().getMethod("addAll"+StringUtil.firstToUpper(fName),Iterable.class);
me.invoke(protoBuilder, jObject);
}catch(Exception e){
logger .info("this repeated field is a user-defined field");
e.printStackTrace ();
}
}
}else{
boolean isDefined1 = false;
try{
// Custom objects continue to need to be parsed and processed through the builder, and callbacks and this block take up a lot of computing time. to be optimized
Method bM = protoBuilder.getClass().getMethod("getFieldBuilder", FieldDescriptor.class);
Object subBuilder = bM.invoke(protoBuilder, fd);
Object pBean = javaBeanToProtoBean(jObject,subBuilder);
Method me = protoBuilder.getClass().getMethod("setField", FieldDescriptor.class, Object.class);
me.invoke(protoBuilder, fd, pBean);
isDefined1 = true;
}catch(Exception e){
// logger .info("this required field is not a user-defined field");
}
if(!isDefined1){
Method me = protoBuilder.getClass().getMethod("setField", FieldDescriptor.class, Object.class);
me.invoke(protoBuilder, fd, jObject);
}
}
}
}catch(Exception e){
logger .error("javaBeanToProtoBean method item reflect error, item name:"+item.getName());
}
}
Method buildM = protoBuilder.getClass().getMethod("build");
Object rObject = buildM.invoke(protoBuilder);
/* Method byteM = rObject.getClass().getMethod("toByteArray");
Object byteObject = byteM.invoke(rObject);
byte[] pbByte = (byte[]) byteObject;
String pbStr = new String(Base64.getEncoder().encode(pbByte), "UTF-8");*/
return rObject;
} catch (Exception e) {
e.printStackTrace ();
logger.error("convert javabean to protobuf bean error,e:", e);
return null;
}
}
The above method can generally say that the protobean sent from the front end is converted into the ordinary javabean we need, but it is much slower than getset in performance. It is no problem to use ordinary projects, and it can reach tens of thousands of times per second, but it is not good for performance. The required children's shoes can pay attention to the line of code I annotated.
try{
// Custom objects continue to need to be parsed and processed through the builder, and callbacks and this block take up a lot of computing time. to be optimized
Method bM = protoBuilder.getClass().getMethod("getFieldBuilder", FieldDescriptor.class);
Object subBuilder = bM.invoke(protoBuilder, fd);
Object pBean = javaBeanToProtoBean(jObject,subBuilder);
Method me = protoBuilder.getClass().getMethod("setField", FieldDescriptor.class, Object.class);
me.invoke(protoBuilder, fd, pBean);
isDefined1 = true;
}catch(Exception e){
// logger .info("this required field is not a user-defined field");
}
Since there are other beans included in the conversion, the catch code block is often entered during ordinary operations, so it wastes a long time (as we all know, catch is a waste of time), but remove this code to transfer the inclusion relationship There is a problem with the bean, and the blogger has not solved this problem for the time being. I leave it to you. If you can solve it, you can leave a message below. If you can't solve it but still want something simple and convenient, you can follow my next blog post, protostuff.