Apache Thrift charm

WhyApacheThrift

Because of the recent need to be integrated in the project come in a machine-learning algorithms written in Python, but my rear end of the main use of the SpringCloud technology stack. So faced with the choice of communication between heterogeneous implementations of the language. Because the business logic is this

The main part of this is to achieve 2-3 in response to the request, a lot of ways to achieve, as long as there is even the ability to package a WebServer py provide services or choose to use message middleware, but the middle of the message communication models are most one-way, that publication subscriptions, but also to achieve the above business needs

Project implementation began in a fact like this, the choice simple and crude directly use socket programming, py with a socket to write server, java client with socket implementation, heterogeneous communication between the two sides, as in the following code two, running locally, then the speed of communication between the parties can be, but when I will make him into a mirror when the docket packaged and released to the line of communication between the two sides even need 9s

A request needs nine seconds, which is certainly not acceptable

        InetAddress localhost = InetAddress.getByName("192.168.88.1");
        Socket socket = new Socket(localhost.getHostName(), 9999);
        OutputStream outputStream = socket.getOutputStream();
        InputStream inputStream = socket.getInputStream();
        // 向py发送消息
        PrintStream out = new PrintStream(outputStream);
        // 发送内容
        out.print("hi python");
        // 获取消息
        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
        String targetName = bufferedReader.readLine();
        System.err.println("获取返回的消息   " + targetName);

import socket
import  time
# 建立一个服务端
server = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.bind(('localhost',9999)) #绑定要监听的端口
server.listen(5) #开始监听 表示可以使用五个链接排队
while True:# conn就是客户端链接过来而在服务端为期生成的一个链接实例
    conn,addr = server.accept() #等待链接,多个链接的时候就会出现问题,其实返回了两个值
    print(conn,addr)
    while True:
        try:
            data = conn.recv(1024)  #接收数据
            print('recive:',data.decode()) #打印接收到的数据
            conn.send(data.upper()) #然后再发送数据
            time.sleep(3)
        except ConnectionResetError as e:
            print('关闭了正在占线的链接！')
            break
    conn.close()

In fact, most of the problems we now encounter predecessors have encountered, like on the demand for, in fact, achieve the industry there could be many, Apache top-level project Thrift can be the perfect solution to this problem, and the speed of communication, the effect and stability are ideal for super

Thrift Profile

Apache Thrift is facebook donated to Apache, and now it is also Apace top-level project, Thrift meaning of thrift, thrift essentially is an interface description language and binary communications protocol, originally designed is to achieve service call service across languages, RPC is an absolutely excellent framework

Currently, Thrift supported languages ​​are many, such as: C #, C ++ (based on POSIX compatible systems), Cappuccino, Cocoa, Delphi, Erlang, Go, Haskell, Java, Node.js, OCaml, Perl, PHP, Python, Ruby and Smalltalk .

installation

Download the official website: http://thrift.apache.org/download

Download compiler

Why use this compiler? Because we're really at the beginning, using the code are automatically generated using the Thrift out for us, plainly, is that we just need a good .thrift files based on business logic defined by this compilation its compiles the configuration file, and use to tell the compiler what code language generation for us to ok, Thrift framework automatically would codec logical communication between the parties generate for our code, also includes package to the socket, as a whole is one-stop service

Because of this use of windows, so the end of the download .exe compiler

Adding Environment Variables

When the code is complete after a while the generated code in the idea, so this step is inevitable, otherwise it will say can not find this command

The first pit: Rename about this compiler, remove the previous version of the small number go to add environment variables

The second pit: If you enter a thrift version number is displayed in the console, but the idea is not identified, restart the computer

Thrift system architecture

Why look Thrift system architecture it? Although Thrift will help us to automatically generate some template code, but still need to manually encode the client and server code, understand its architecture system, look at the server and if you write the client code can also be read to understand, structure of the object in your code in the following system architecture are all traceable

As shown above, Thrift is a typical CS architecture we can thrift called IDL (Interface DescriptionLanguage) language, and the figure above server and client can use a different language development, those two different server-side language development and What use to communicate between the client be implemented by Thrift

• YourCode: is our own business code
• The client and server objects to communicate: FooServiceClient
• Foo write () / read (): This is the auto-generated our Thrift code logic bottom package transmission of data through the socket
• TProtocol: 协议层, 在这一层中规定了数据传输使用的哪种协议
• TTransport: 传输层: 在这一层中规定了数传输的格式,比如需不需要进行压缩
• Underlying IO : 数据在网络中的IO交互

Thrift的传输协议

这种传输协议实际上就是规定了数据在网络上采用什么样的格式进行传输

• TBinaryProtocol : 以二进制格式进行传输
• TCompactProtocol: 对二进制数据进一步压缩的格式
• TJsonProtocol: json格式
• TSimpleJsonProtocol: 针对Json的只写协议
• TDebugProtocol: 简单易懂的文本格式, 常用于去 调试代码使用

Thrift的数据传输方式

• TSocket : 阻塞式的Socket 效率最低
• TFrameTransport: 在非阻塞应用服务中常用. 以Frame为单位进行传输
• TFileTransport: 以文本格式进行传输
• TMemoryTransport: 使用内存进行IO, 在java中的实现是 ByteArrayOutPutStream

Thrift支持的服务模型

这种服务类型说的就是服务端示例的类型, 有如下几种

• TSimpleServer: 简单的单线程服务模型
• TThreadPoolServer: 虽然是表中的阻塞式IO, 但是采用多线程模型处理
• TNonblockingServer: 多线程服务模型, 使用的是非阻塞IO常和TFramedTransport数据传输方式搭配使用
• THsHaServer: 引用线程池去处理, 采用的是半同步,半异步的模式, 针对不同类型的消息, 进行不同的处理, 比如对IO类型的消息异步处理, 对Handler的RPC远程过程调用进行同步处理

Thrift的数据类型

thrift不支持无符号数据类型

简单的数据类型:

名称	简介
byte	有符号字节
i16	16位有符号整数
i32	32位有符号整数
i64	64位有符号整数
double	64位浮点数
string	字符串类型

thrift的容器类型: (支持泛型)

• list: 表示一系列T类型的数据组成的有序列表, 元素可以重复
• set: 一系列T类型的数据组合成的无序集合, 元素不重复
• map: 一个字典结构, key为K类型, value为V类型,相当于java中的hashmap

结构体: 这个struct类似C语言中的结构体, 初衷也是将不同的数据聚合在一起,方便传输,经过编译器编译完成后其实就是java中的类

struct Student{
    1:string name;
    2:i32 age;
}

枚举类型

enum Gender{
    MALE,
    FEMALE
}

异常类型: thrift 支持异常类型表示服务端和客户端之间的通信所可能抛出来的异常, 并且我们可以在service中的方法上throws 异常, 用描述异常出现的时间,异常的类型

exception RequestException{
    1:i32 coed
    2: string reason
}

服务类型: 服务端和客户端通信使用到的接口 , 就好比java中的接口, 它是一系列方法的集合, thrift 会将service转换成客户端和服务端的框架的代码 , 定义形式如下

service MyService{
    string ask(1:string name,2:i32 age)
}

类型定义: 可以像下面这样,使用类似C语言的语法为变量取别名, 转换成我们习惯的命名格式

typedef i32 int
typedef i64 long

常量const类型: thrift 同样支持常量的定义, 使用const关键字:

const string NAME="XXX"

命名空间类型: 关键字是 namespace , thrift的命名空间相当于java中的package, 实际使用上thrift也会将生成的代码放在这下面指定的包中

格式: namespace 语言  路径
实例: namespace java com.changwu.thrift.Demo

文件包含: 同样向C/C++那样,支持文件之间相互包含的操作. 在java中这个动作就是Import

include "global.thrift"

注释: thrift 中的注释有一下几种

// XXXX
#  XXX
/*XXX*/

Optional and mandatory option , keywords are required and optional, represent the corresponding field is optional or mandatory

struct Student{
    1:required string name;
    2:optional string age;
}

Real

Thrift write file

We use the above defined thrift file, we expect to generate the target language code

namespace java com.changwu.thrift
namespace py py.thrift

// 去别名字
typedef string String
typedef bool boolean
// 我们通过 .thrift文件 描述对象(struct), 方法(service), 类型, 异常等信息
struct Message {
    1: optional String msg,
}

exception MyExcetion{
    1:optional String message,
    2:optional String callStack,
    3:optional String date
}

service PersonService{
     Message getResultFromPy(1:required String message) throws(1:MyExcetion e),
}

Code Generation

After executing the command code generation will be more gen-java directory under the root path, there is also generated code

命令  thrift --gen 语言 .thrift文件路径
java实例: thrift --gen java src/thrift/data.thrift
py  实例: thrift --gen py src/thrift/data.thrift

After you perform these two commands we'll get this result

Dependence on import run

Dependent jar package introduced runtime

<!-- https://mvnrepository.com/artifact/org.apache.thrift/libthrift -->
<dependency>
    <groupId>org.apache.thrift</groupId>
    <artifactId>libthrift</artifactId>
    <version>0.13.0</version>
</dependency>

coding

The use of java implementation of the client, Thrift client code generation in the service class type, that is to say above us in the MessageService,

By understanding the architecture of Thrift above we can know, and need to select TProtocol TTransport build client and server have to be consistent and, therefore following sample code will clear the

        // 帮点端口号和超时时间
        TTransport tTransport = new TFramedTransport(new TSocket("localhost",9999),600);
        TProtocol protocol = new TCompactProtocol(tTransport);
        MessageService.Client client = new MessageService.Client(protocol);
        try{
            // 打开socket 
            tTransport.open();
            // 发送
            Message p = client.getResultFromPy("张三123");
            System.out.println("结果: "+p.getMsg());
        }catch (Exception  e){
            e.printStackTrace();
        }finally {
            tTransport.close();
        }

py end Server implementation

Py in the first end, implement the interface defined previously, when receiving a specific message callback business logic here is defined

class MessageServiceImpl:

    def getResultFromPy(self, msg):
        print("获取到msg = " + msg)
        message = ttypes.Message()
        message.msg = 'hello java'
        return messag

Write server-side

Write server and client code actually follow the previously mentioned Thrift architecture specification, and secondly, the need to use Idea installed at thrift library associated with the py

Download http://www.apache.org/dyn/closer.cgi?path=/thrift/0.13.0/thrift-0.13.0.tar.gz

In the local decompression open, and then find inside py directory, enter lib find py py install support for the

Excuting an order:

python setup.py install

from thrift.transport import TTransport
from thrift.protocol import TCompactProtocol
from thrift.server import TServer
from py.thrift import MessageService
from MessageServiceImpl import MessageServiceImpl


try:
    # handler就是我们对 ThriftService的实现
    personServiceHandler = MessageServiceImpl()
    # Foo read()/write() 被Thrift生成在 MesageServive中
    processor = MessageService.Processor(personServiceHandler)
    # TSocket TProtocol 和 Transport 从上面下载的类库导入进来
    # 这三者的作用在上面的架构图上也能提现出来
    serverSocket = TSocket.TServerSocket(host="127.0.0.1",port=9999)
    transportFactory = TTransport.TFramedTransportFactory()
    protocolFactory = TCompactProtocol.TCompactProtocolFactory()
    
    # py提供了四种Server , TThreadPoolServer TServer THeaderProtocolFactory
    server = TServer.TThreadPoolServer(processor,serverSocket,transportFactory,protocolFactory)
    print(">>>>>>>>>>>>>>>>>>服务端启动>>>>>>>>>>>>>>>>>>>>")
    server.serve()

except Thrift.TException as ex:
    print("%s" % ex.message)

ok, so far finished writing the code, but try to run it, to experience the charm of a heterogeneous communication