Examples of RPC communication Hadoop

2019-06-05

Keywords: Hadoop, Hadoop RPC instance, Hadoop RPC calls, Hadoop Remote Procedure Call

---

 

This article briefly introduce the concept of RPC and RPC gives a development instance of Hadoop.

 

RPC concept

 

RPC is a request for a communication service, but need not be concerned about the details of the underlying network communication protocol from a remote computer over a network. Briefly, RPC is a communication protocol.

 

RPC communication process is constructed based on TCP or UDP. It is because of RPC does not need to spend energy to deal with the underlying network traffic details, you can fully concentrate on the business logic of communication features, it has been widely distributed in the field.

 

RPC is usually architectural model C / S model. A typical composition has the following structure RPC portion

1, a communication module

The communication module is used to transmit data packets on the network, it is generally not process any packet content. RPC communications synchronous and asynchronous communication modes.

2, Stub program

It can be simply understood as a proxy. It recorded above all support remote call interface, service and client each hold an identical Stub program. Our RPC process, its essence is to call interface on this program Stub their hands of it.

3, the scheduler

Scheduler, to run on the server. Receiving a request from the communication module, and selecting a corresponding program based on the request Stub identifier to perform.

 

RPC communication process

 

The following describes a typical communication procedure RPC

 

Step one: The client calls the appropriate interface on the Stub program their own hands;

Step two: Stub application client-side invocation request information into the packet as required by network communication module and a communication module to transmit to the remote server;

Step three: a communication module in the remote server will forward this message to the dispatcher package after receiving the message packet;

Step Four: The scheduler will the message packet identifier, then the message is forwarded to the relevant service program Stub end side;

Step Five: Stub Service opened end of the message packet will be formed in the form of the adjustment process, and a function to call the corresponding methods;

Step Six: the called function executes in accordance with the obtained parameters, and the final result is returned to the program Stub;

Step 7: Stub program will return a result of the server-side packed into packets, transmitted to the client program via a network communication module stepwise.

 

The following is a frame of the above-described step of FIG.

 

 

Hadoop RPC examples

 

Although Hadoop is a Java language development, but it did not directly follow the RPC framework of Java RPC framework. But overall, the differences are not large, the development process is relatively simple.

 

Here a small example of a simple Hadoop RPC program.

 

step one

All custom Hadoop RPC interfaces need to inherit org.apache.hadoop.ipc.VersionedProtocol interface. Here we define two numbers together an implementation of the interface.

import org.apache.hadoop.ipc.VersionedProtocol;

public interface IRPCDemo extends VersionedProtocol {

    int myAdd(int a, int b);

}

 

Step Two

Next, we want to create an RPC server. We need to implement the interface defined in step one.

import java.io.IOException;

import org.apache.hadoop.ipc.ProtocolSignature;

public class RPCDemoImpl implements IRPCDemo {

    @Override
    public long getProtocolVersion(String protocol, long clientVersion) throws IOException {
        return 0;
    }

    @Override
    public ProtocolSignature getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash)
            throws IOException {
        try {
            return ProtocolSignature.getProtocolSignature(protocol, clientVersion);
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }
        return null;
    }

    @Override
    public int myAdd(int a, int b) {
        if(a < 0 || b < 0)
            return 0;
        
        return a + b;
    }

}

Here to explain the above two methods.

 

About getProtocolVersion () method, which returns a version number. RPC communication is very strict for the version number, the version number of the communication must not be different. Need to be taken seriously return value of this method in a production environment, but in our sample program where it does not matter.

 

About the middle of that getProtocolSignature () method, according to written enough.

 

Step Three

Then to achieve our server.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.ipc.RPC.Server;

public class RPCDemoServer {

    public static void main(String[] args) {
        Configuration conf = new Configuration();
        try {
            Server server = new RPC.Builder(conf)
                    .setBindAddress("usmaster")
                    .setNumHandlers(2)
                    .setPort(17173)
                    .setInstance(new RPCDemoImpl())
                    .setProtocol(IRPCDemo.class)
                    .build();
            server.start();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}

In fact, nothing special, according to write just fine. Be sure to put your own server and server interface class interface class to replace about it. That part of the above code marked red.

 

Step Four

Then it is to achieve our client.

import java.io.IOException;
import java.net.InetSocketAddress;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;


public class RPCDemoClient {

    public static void main(String[] args) {
        Configuration conf = new Configuration();
        try {
            IRPCDemo lmt = RPC.getProxy(IRPCDemo.class,    0,
                    new InetSocketAddress("usmaster", 17173),
                    conf);
            int ret = lmt.myAdd(3, 7);
            System.out.println(ret);
            ret = lmt.myAdd(-3, 1);
            System.out.println(ret);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

}

 

 

At this point, our version of Hadoop RPC example is complete. If you want to test it, to ensure the correct after-related Hadoop environment can be run directly Client and Server side to see the end result.

 

---