0. Preface

Protobuf is a binary serialized data format developed by Google that can be used to transfer structured data between different computer systems . There are many other data exchange formats such as JSON, XML, etc., so what benefits does Protobuf have over them? Mainly reflected in the following points:

Faster serialization and deserialization : Protobuf is a binary format and therefore can serialize and deserialize messages faster than other data interchange formats such as JSON and XML. This makes the use of Protobuf significantly improve the performance and efficiency of applications in scenarios such as high concurrency and large data processing.
Smaller message size : Since Protobuf is a compact binary format, it can generate smaller message size, which is very beneficial for network transmission and storage. This saves bandwidth and storage space, while also improving application performance.
Compatibility and Version Control : Compatibility and version control are easier with Protobuf. With easy backward and forward compatibility through modification and version control of message definitions, this is very helpful in maintaining code stability and reliability over long development cycles.
Simplified code : Using Protobuf can make the code more concise and easier to maintain. Since Protobuf can automatically generate code for structures and other data structures, there is no need to manually write complex data parsing and serialization code, which can make the code clearer and easier to read.

This article will explain how to use Protobuf in Go, including installation , how to define Protobuf messages , generate Go code , serialize and deserialize messages , use Protobuf in the network, and version control and compatibility .

1. Install

Installing Protocol Buffers (protobuf) can be divided into two parts:

Install the protoc compiler
Install Go's protobuf library

1. Install protoc compiler

The protoc compiler can compile .proto files into codes in multiple programming languages, such as C++, Java, Python, Go, etc., and is the core component of protobuf.

The latest compiler releases can be found on Protocol Buffers' GitHub: https://github.com/protocolbuffers/protobuf/releases

Find the appropriate system version download on the download page and then add the bin directory to the environment variable .

The following uses Windows x64 as an example.

Download: Click here to download the package , and extract the downloaded compressed package to a certain folder.
**Environment variable:** Open the place where the environment variable is set, and add the bin folder in the compressed package to the PATHenvironment variable.
**Test:** Open cmd, enter protoc --version, if the version is output, the installation is successful.

2. Install Go's protobuf library

Next, you need to install Go's protobuf library. This library can use the protoc compiler to generate GO code, and can be installed with the following command:

go get -u github.com/golang/protobuf/protoc-gen-go

Note that protoc-gen-go will be automatically installed in $GOPATH/binthe directory , and this directory needs to be added to the environment variable .

2. Define the Protobuf message type

2.1 Write .proto package file

The message type is defined in .protothe package, we create one here person.proto 文件, and write the following classic example writing method:

syntax = "proto3";

option go_package = "/person";

package example;

message Person {
    string name = 1;
    int32 age = 2;
    repeated string hobbies = 3;
}

syntax = "proto3": Used to specify .protothe version of the file, here is the Protocol Buffers 3 version.
package example：指定消息类型所在的包名，这里包名是 example`。
option go_package = "/person";Used to specify the package name (package name) and import path (import path) of the generated Go code.
message Person { ... }: Defines a message type Personnamed .
string name = 1: Defines a string-type field namenamed with a tag number of 1.
int32 age = 2: Define an integer field agenamed , whose tag number is 2.
repeated string hobbies = 3: Defines a string-array-type field hobbiesnamed with a tag number of 3. repeatedThe keyword indicates that the field is an array type.

This .protofile defines a message type Personnamed , which contains three fields name, , and . and are common single-value type fields, which are string array type fields. In this file, each field has a unique tag number, which is used to identify the position and type of this field in the binary encoding.agehobbiesnameagehobbies

2.2 Use protoc to generate GO code

In the directory of this file, run protoc --go_out=. *.protothe command to generate GO code.

After running, we can see that there is an additional person folder in this directory, which contains Go files person.pb.go. This file internally defines a structure Person and related methods:

type Person struct {
   state         protoimpl.MessageState
   sizeCache     protoimpl.SizeCache
   unknownFields protoimpl.UnknownFields

   Name    string   `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
   Age     int32    `protobuf:"varint,2,opt,name=age,proto3" json:"age,omitempty"`
   Hobbies []string `protobuf:"bytes,3,rep,name=hobbies,proto3" json:"hobbies,omitempty"`
}

In addition to the structure, there are many methods. These methods provide the infrastructure for encoding, decoding and manipulating Protocol Buffers messages. There are the following main methods.

func (*Person) Reset(): Reset Personmessage to default.
func (*Person) String() string: Returns a string containing the textual representation of Personthe message .
func (*Person) ProtoMessage(): Make Personthe struct implement proto.Messagethe interface, which is required when serializing and deserializing Protobuf messages.
func (*Person) Descriptor() ([]byte, []int): Returns descriptor information about Personthe message type.
func (*Person) GetName() string: PersonReturns Namethe value of the field in the message.
func (*Person) GetAge() int32: PersonReturns Agethe value of the field in the message.

Write a simple test!

func main() {
	p:=&person.Person{Name: "yzy",Age: 23,Hobbies: []string{"music","sport"}}
	fmt.Println("string",p.String())
	fmt.Println("the data:",p.Name,p.Age,p.Hobbies)

	fmt.Println("-----------")
	fmt.Println("reset the person")
	fmt.Println("-----------")
	
	p.Reset()
	fmt.Println("string",p.String())
	fmt.Println("the data:",p.Name,p.Age,p.Hobbies)
}

After running, you can see that get, string, and reset are all running normally.

2.3 Serializing and deserializing messages

The serialization and deserialization functions are in github.com/golang/protobuf/protothe package. We have just obtained this package through go get, so it can be used directly. The following is an example of serialization and deserialization, and compares the sequence before and after serialization Whether the transformed data is consistent.

func TestPersonSerialization(t *testing.T) {
	// 创建一个 Person 消息实例并设置其字段
	p:=&Person{Name: "yzy",Age: 23,Hobbies: []string{"music","sport"}}

	// 将消息序列化为二进制格式
	data, err := proto.Marshal(p)
	if err != nil {
		t.Fatal("marshaling error: ", err)
	}

	// 反序列化消息
	p2 := &Person{}
	err = proto.Unmarshal(data, p2)
	if err != nil {
		t.Fatal("unmarshaling error: ", err)
	}

	// 比较原始消息和反序列化后的消息
	if p.String()!=p2.String() {
		t.Fatalf("original message %v != unmarshaled message %v", p, p2)
	}
}

2.4 Field Type

The field type section refers to Go Protobuf brief tutorial .

2.4.1 Scalar type (Scalar)

proto type	type of go	Remark	proto type	type of go	Remark
double	float64		float	float32
int32	int32		int64	int64
uint32	uint32		uint64	uint64
sint32	int32	suitable for negative numbers	sint64	int64	suitable for negative numbers
fixed32	uint32	Fixed-length encoding, suitable for values greater than 2^28	fixed64	uint64	Fixed-length encoding, suitable for values greater than 2^56
sfixed32	int32	fixed length code	sfixed64	int64	fixed length code
bool	bool		string	string	UTF8 encoding, the length does not exceed 2^32
bytes	[]byte	Arbitrary byte sequence, the length does not exceed 2^32

If the scalar type is not assigned, it will not be serialized, and will be assigned a default value when parsing.

strings: empty string
bytes: empty sequence
bools：false
Value type: 0

2.4.2 Enumerations¶

Enumerated types are suitable for providing a predefined set of values, one of which is chosen. For example we define gender as an enum type.

message Student {
  string name = 1;
  enum Gender {
    FEMALE = 0;
    MALE = 1;
  }
  Gender gender = 2;
  repeated int32 scores = 3;
}

The identifier of the first option of an enumeration type must be 0, which is also the default value of the enumeration type.
Alias (Alias), allows to give the same identifier to different enumeration values, called alias, need to open allow_aliasthe option.

message EnumAllowAlias {
  enum Status {
    option allow_alias = true;
    UNKOWN = 0;
    STARTED = 1;
    RUNNING = 1;
  }
}

2.4.3 Using other message types

Resultis another message type used in SearchReponse as a message field type.

message SearchResponse {
  repeated Result results = 1; 
}

message Result {
  string url = 1;
  string title = 2;
  repeated string snippets = 3;
}

Nested writes are also supported:

message SearchResponse {
  message Result {
    string url = 1;
    string title = 2;
    repeated string snippets = 3;
  }
  repeated Result results = 1;
}

If defined in other files, other message types can be imported for use:

import "myproject/other_protos.proto";

2.4.4 Any type (Any)

Any can mean that any built-in type is not defined in .proto.

import "google/protobuf/any.proto";

message ErrorStatus {
  string message = 1;
  repeated google.protobuf.Any details = 2;
}

2.4.5 oneof

message SampleMessage {
  oneof test_oneof {
    string name = 4;
    SubMessage sub_message = 9;
  }
}

2.4.6 map

message MapRequest {
  map<string, int32> points = 1;
}

3. Summary

So far, it is possible to generate a .go file through the same .proto file on the server and client side, and then use the serialized binary format for transmission through this .go file, and deserialize the data. It is more efficient than using the traditional json format.

Probuf can also be used to define the RPC interface, which will be described in future articles.

Introduction to Protobuf and how to use Protobuf in GO language?