rpc and grpc learning records

1、RPC

RPC is the abbreviation of Remote Procedure Call, which is called remote procedure call in Chinese.

The goal of the RPC framework is to make remote service calls simpler and more transparent . The RPC framework is responsible for shielding the underlying transmission methods (TCP or UDP or HTTP), serialization methods (XML/Json/binary) and communication details. The service caller can call the remote service provider just like calling the local interface, and does not need to care about the details of the underlying communication and the calling process, making it easier to develop applications including network distributed programs.

To put it bluntly, it can be understood as follows: now there are two servers A and B. An application deployed on server A wants to call a method provided by another application deployed on server B. Since it is not in the same memory space, it cannot be called directly, and the effect of the call needs to be achieved through the network.

Now, we encapsulate the logic of calling B in a local method of A service, and then only need to use this method locally to achieve the effect of calling B. For the user, the details are blocked. You only need to know the result returned by calling this method, without paying attention to the underlying logic.

So, from the perspective of the encapsulated method, what do we need to know before calling B?

Certainly some conventions. For example:
1. The semantics of calling can also be understood as interface specification. (such as RESTful)
2. Network transmission protocol. (such as HTTP)
3. Data serialization and deserialization specifications (such as JSON).

With these agreements, I know how to send you data, what kind of data to send, and what kind of data you return to me.

As can be seen from the above figure, RPC is a client-server (Client/Server) mode .

What is the difference between HTTP and RPC?

First of all, the question itself is not rigorous. HTTP is just a communication protocol that works on the seventh layer of OSI. And RPC is a complete remote calling scheme. It includes: interface specification, transmission protocol, data serialization and deserialization specification.

Looking at it this way, the relationship between RPC and HTTP can only be an inclusion relationship. Why is it possible? Because the RPC transport protocol can be based on TCP or UDP instead of HTTP.

So this question should be changed to: What is the difference between an HTTP-based remote call solution (such as: HTTP+RESTful+JSON) and a direct RPC remote call solution.

The mainstream RPC frameworks in the industry are generally divided into three categories:

• Support multi-language RPC frameworks, such as Google's gRPC and Apache (Facebook) Thrift;
• RPC frameworks that only support specific languages, such as Sina Weibo's Motan;
• For distributed service frameworks that support service-oriented features such as service governance, the underlying core is still an RPC framework, such as Ali's
  Dubbo. "Understanding the Dubbo Framework in Seconds (Principle)"

---

2、gRPC

gRPC is a type of RPC. It is free and open source, produced by Google, mainly for mobile application development and designed based on the HTTP/2 protocol standard, and supports most popular programming languages.

A typical feature of gRPC is to use protobuf (full name protocol buffers) as its interface definition language (Interface Definition Language, abbreviated as IDL), and the underlying message exchange format also uses protobuf.

gRPC requires the client to store a stub (stub: provide the same methods and functions as the server), and the stub is automatically generated by the gRPC framework. With the stub, by directly calling the method in the stub, the developer only needs to care about the specific business logic, and does not need to care about the implementation principles of network communication.

The stub of the client is automatically generated by the gRPC framework after compiling the .protoc file. You can use the plug-in to quickly generate the stub required by gRPC from the .proto file.

Reference:

"What is gRPC? What is RPC? " : In plain language, first explain rpc, and then explain the relationship between rpc and gprc, the explanation is very straightforward and clear.

"Getting to Know gRPC" : A good explanation of stubs

"gRPC Detailed Explanation" : small demo and parameter description

"Technical Practice: Teach You to Build gRPC Services with Python"

"Using Python to Realize gRPC Communication"

"Introduction to gRPC"

Multithreaded python
"Concurrent and asynchronous programming in Python"

"Python threading to achieve multi-threading basics"

3. grpc code

3.1. Install the libraries required by python:

install grpc

pip install grpcio

Install gRPC tools

pip install grpcio-tools 

Note: gRPC tools includes the protobuf compiler protoc, and the compilation plug-in grpc_python_out (which will be used for later compilation). And the protobuf library has been automatically installed in it, so there is no need pip install protobufto install .

---

3.2, grpc programming steps

Python implementation of grpc requires the following steps:

1. Write the .proto file, that is, define the interface and data type specification;
2. Generate 2 stub files by compiling the .proto file;
3. Write the server code. Realize and start the interface defined in the first step. The definition of these interfaces is in the stub file;
4. Write the client code. The client uses the stub file to call the function of the server. Although the function called by the client is implemented by the server, it is called like a local function.

---

3.3、Demo1

The directory structure is as follows:

Specifically, you can use the following command to view:

apt-get install tree

tree ./

3.3.1. Write a .proto file to define interfaces and data types

For the specific proto common grammar, please refer to: "gRPC proto grammar"

// helloworld.proto：定义接口和数据类型规范

// 限定该文件使用的是proto3的语法
syntax = "proto3";

// 可以为proto文件指定包名，防止消息命名冲突
package helloworld;

// 基础Demo
service Greeter {
    
    
    //一个服务中可以定义多个接口，也就是多个函数功能。注意return加了个s
    //   方法名      方法参数                 返回值
    rpc SayHello (HelloRequest) returns (HelloResponse) {
    
    } 
}

// 请求的参数
message HelloRequest {
    
    
    string name = 1; //数字1是参数的位置顺序，并不是对参数赋值
}

// 返回的对象
message HelloResponse {
    
    
    string message = 1;
}

3.3.2. Compile the .proto file to generate a stub file

Use the compilation tool to convert the .proto file into a .py file, and run the code below directly in the current file directory.

cd test0_zct

python3 -m grpc_tools.protoc -I./proto --python_out=. --grpc_python_out=. proto/helloworld.proto

Parameter Description:

1. -m specifies that the .py file (that is, the stub file) is automatically generated by the protoc tool
2. -I specifies the directory where .proto is located
3. –python_out specifies the output path of the generated .py file
4. hello.proto input .proto file

After compiling, two stub files (stub) will be generated, as shown in the figure below, helloworld_pb2.py and helloworld_pb2_grpc.py. It can be roughly understood as: the data structure is defined in _pb2, and the related methods are defined in _pb2_grpc.

3.3.3. Write server-side code

vim hello_server.py

The code content is as follows:

# -*- coding: utf-8 -*-
# hello_server.py
'''
服务器端代码
'''

import grpc
from concurrent import futures
import helloworld_pb2
import helloworld_pb2_grpc

class Hello(helloworld_pb2_grpc.GreeterServicer):
    '''
    实现在helloworld.proto中'service'定义的SayHello方法。说明：

    1、定义一个类，要继承父类GreeterServicer。
    这里为何是GreeterServicer？因为helloworld_pb2_grpc.py是通过helloworld.proto编译后自动生成的，
    而在helloworld.proto文件中定义的'service'名为'Greeter'，编译后会自动生成'名+Servcier'，
    继承'GreeterServicer'就相当于是helloworld.proto的service Greeter。

    2、初始化

    3、具体实现SayHello方法。注意该方法的参数和返回值，要和helloworld.proto中'service'定义的SayHello方法相对应。
    此方法虽存在服务器端，但是通过rpc，客户端可以实现远程调用。
    具体实现细节是客户端可以调用本地的存根文件里的方法。存根文件来源于编译.proto文件后，grpc自动生成的。
    '''
    def __init__(self):
        pass

    def SayHello(self, request, context):
        '''
        这里具体实现之前定义的SayHello()
        :param request: request是在.proto文件中定义的HelloRequest消息类型
        :param context: context是保留字段，这里不用管
        :return: .proto文件中定义的HelloResponse类型
        '''
        # message = 'Hello {msg}'.format(msg=request.name)
        # return helloworld_pb2.HelloResponse(message)
        return helloworld_pb2.HelloResponse(message='Hello {msg}'.format(msg=request.name))

def serve():
    '''
    模拟服务启动
    :return:
    '''
    # 这里通过thread pool来并发处理server的任务
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    # 将对应的任务处理函数添加到rpc server中，第一个参数为上面定义的类名
    helloworld_pb2_grpc.add_GreeterServicer_to_server(Hello(), server)
    # 添加端口
    server.add_insecure_port('[::]:50054')
    # 开始监听
    server.start()
    print('gRPC 服务端已开启，端口为50054...')
    # 堵塞监测本地50054端口，等待接收数据
    server.wait_for_termination()
    # try:
    #     while True:
    #         time.sleep(_ONE_DAY_IN_SECONDS)
    # except KeyboardInterrupt:
    #     server.stop(0)

if __name__ == '__main__':
    serve()

3.3.4. Write client code:

vim hello_client.py

The code content is as follows:

# -*- coding: utf-8 -*-
# hello_client.py
'''
客户端代码
'''

import grpc
import helloworld_pb2, helloworld_pb2_grpc

def run():
    # 本次不使用SSL，所以channel是不安全的
    channel = grpc.insecure_channel('localhost:50054')
    # 客户端实例
    stub = helloworld_pb2_grpc.GreeterStub(channel)
    # 调用服务端方法，这里其实是通过调用client本地的存根文件中方法实现的
    response = stub.SayHello(helloworld_pb2.HelloRequest(name='World'))
    print("Greeter client received: " + response.message)

if __name__ == '__main__':
    run()

3.3.5. Test

A terminal running hello_server.py

python3 hello_server.py

Another terminal runs hello_client.py

python3 hello_client.py

The result of the operation is as follows

3.4、Demo2

3.4.1. Write .proto files to define interfaces and data types

// helloworld.proto：定义接口和数据类型


// 限定该文件使用的是proto3的语法
syntax = "proto3";

// 可以为proto文件指定包名，防止消息命名冲突
package helloworld;

// 进阶Demo
service Greeter {
    
    
    // 获取部门员工信息接口
    //    方法名            方法参数                       返回值
    rpc GetDeptUser (GetDeptUserRequest) returns (GetDeptUserResponse) {
    
    }
}

// 请求的参数
// 数字1,2,3是参数的位置顺序，并不是对参数赋值
message GetDeptUserRequest {
    
    
    uint32 dept_id = 1; // 部门
    string dept_name = 2; // 部门名称
    repeated uint32 uid_list = 3; // 用户id列表
}

// 返回的对象
message GetDeptUserResponse {
    
    
    repeated BasicUser user_list = 1; // 用户列表
    map<uint32, BasicUser> user_map = 2; // 用户哈希表，即字典
}
// 用户基本信息
message BasicUser {
    
    
    uint32 id = 1;
    string name = 2;
}

3.4.2. Compile the .proto file to generate a stub file

cd test1_zct

python3 -m grpc_tools.protoc -I./proto --python_out=. --grpc_python_out=. proto/helloworld.proto

3.4.3. Write server-side code

# -*- coding: utf-8 -*-

import grpc
import random
from concurrent import futures
import helloworld_pb2
import helloworld_pb2_grpc


# 实现.protow文件中定义的方法
class Greeter(helloworld_pb2_grpc.GreeterServicer):
    def __init__(self):
        pass
        
    def GetDeptUser(self, request, context):
        '''
        这里具体实现之前定义的GetDeptUser()
        :param request: request是在.proto文件中定义的GetInfoRequest消息类型
        :param context: context是保留字段，这里不用管
        :return: .proto文件中定义的GetDeptUserResponse类型
        '''
        # 字段使用点号获取
        dept_id = request.dept_id
        dept_name = request.dept_name
        uid_list = request.uid_list
        if dept_id <= 0 or dept_name == '' or len(uid_list) <= 0:
            return helloworld_pb2.GetDeptUserResponse()
        print('dept_id is {0}, dept_name is {1}'.format(dept_id, dept_name))
        user_list = []
        user_map = {
    
    }
        for id_ in uid_list:
            uid = id_ + random.randint(0, 1000)
            letters = 'qwertyuiopasdfghjklzxcvbnm'
            name = "".join(random.sample(letters, 10))
            
            user = helloworld_pb2.BasicUser()
            user.id = uid
            user.name = name
            user_list.append(user) # 与正常的添加操作差不多
            user_map[uid] = user
        return helloworld_pb2.GetDeptUserResponse(user_list=user_list, user_map=user_map)


def serve():
    '''
    模拟服务启动
    :return:
    '''
    # 这里通过thread pool来并发处理server的任务
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    # 将对应的任务处理函数添加到rpc server中，第一个参数为上面定义的类名
    helloworld_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
    # 添加端口
    server.add_insecure_port('[::]:50054')
    # 开始监听
    server.start()
    print('gRPC 服务端已开启，端口为50054...')
    # 堵塞监测本地50054端口，等待接收数据
    server.wait_for_termination()


if __name__ == '__main__':
    serve()

3.4.4. Write client code

# -*- coding: utf-8 -*-

import grpc

import helloworld_pb2, helloworld_pb2_grpc


def run():
    # 本次不使用SSL，所以channel是不安全的
    channel = grpc.insecure_channel('localhost:50054')
    # 客户端实例
    stub = helloworld_pb2_grpc.GreeterStub(channel)
    # 调用服务端方法，这里其实是通过client本地的存根文件实现的
    response = stub.GetDeptUser(helloworld_pb2.etDeptUserRequeGst(dept_id=1, dept_name='dd', uid_list=[1, 2, 3]))
    print(response.user_list)
    print(response.user_map)


if __name__ == '__main__':
    run()

3.4.5. Test