例如在client put数据时，会调用htable的flushCommits，再调HConnectionImplementationr的processBatch，再调processBatchCallback
中，在这里异步调用线程，并使用future取得结果，最终执行的是call方法。
// step 2: make the requests
Map<HRegionLocation, Future<MultiResponse>> futures =
new HashMap<HRegionLocation, Future<MultiResponse>>(
actionsByServer.size());
for (Entry<HRegionLocation, MultiAction<R>> e: actionsByServer.entrySet()) {
futures.put(e.getKey(), pool.submit(createCallable(e.getKey(), e.getValue(), tableName)));
}
// step 3: collect the failures and successes and prepare for retry
for (Entry<HRegionLocation, Future<MultiResponse>> responsePerServer
: futures.entrySet()) {
HRegionLocation loc = responsePerServer.getKey();
private <R> Callable<MultiResponse> createCallable(final HRegionLocation loc,
final MultiAction<R> multi, final byte [] tableName) {
// TODO: This does not belong in here!!! St.Ack HConnections should
// not be dealing in Callables; Callables have HConnections, not other
// way around.
final HConnection connection = this;
return new Callable<MultiResponse>() {
public MultiResponse call() throws IOException {
ServerCallable<MultiResponse> callable =
new ServerCallable<MultiResponse>(connection, tableName, null) {
public MultiResponse call() throws IOException {
return server.multi(multi);
}
@Override
public void connect(boolean reload) throws IOException {
server = connection.getHRegionConnection(loc.getHostname(), loc.getPort());
}
};
return callable.withoutRetries();
}
};
}
其中的异步调用的线程池
int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.MAX_VALUE);
if (maxThreads == 0) {
maxThreads = 1; // is there a better default?
}
long keepAliveTime = conf.getLong("hbase.htable.threads.keepalivetime", 60);
// Using the "direct handoff" approach, new threads will only be created
// if it is necessary and will grow unbounded. This could be bad but in HCM
// we only create as many Runnables as there are region servers. It means
// it also scales when new region servers are added.
this.pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueue<Runnable>(),
new DaemonThreadFactory());
((ThreadPoolExecutor)this.pool).allowCoreThreadTimeOut(true);
this.finishSetup();
HMaster or HRegionServer初始化创建HBaseServer调用HBaseRPC.getServer->
this.rpcServer = HBaseRPC.getServer(this,
new Class<?>[]{HMasterInterface.class, HMasterRegionInterface.class},
initialIsa.getHostName(), // BindAddress is IP we got for this server.
initialIsa.getPort(),
numHandlers,
0, // we dont use high priority handlers in master
conf.getBoolean("hbase.rpc.verbose", false), conf,
0); // this is a DNC w/o high priority handlers
HBase 0.94 HRegionServer默认启动10个handler线程用于处理rpc请求（hbase.regionserver.handler.count），
this.rpcServer = HBaseRPC.getServer(this,
new Class<?>[]{HRegionInterface.class, HBaseRPCErrorHandler.class,
OnlineRegions.class},
initialIsa.getHostName(), // BindAddress is IP we got for this server.
initialIsa.getPort(),
conf.getInt("hbase.regionserver.handler.count", 10),
conf.getInt("hbase.regionserver.metahandler.count", 10),
conf.getBoolean("hbase.rpc.verbose", false),
conf, QOS_THRESHOLD);
// Set our address.
this.isa = this.rpcServer.getListenerAddress();
this.rpcServer.setErrorHandler(this);
this.rpcServer.setQosFunction(new QosFunction());
HBaseServer的Listerner的reader线程接到RPC请求后，会丢到queue中，
其中的reader线程数是由一个线程池决定
this.readThreads = conf.getInt(
"ipc.server.read.threadpool.size",
10);
public Listener() throws IOException {
address = new InetSocketAddress(bindAddress, port);
// Create a new server socket and set to non blocking mode
acceptChannel = ServerSocketChannel.open();
acceptChannel.configureBlocking(false);
// Bind the server socket to the local host and port
bind(acceptChannel.socket(), address, backlogLength);
port = acceptChannel.socket().getLocalPort(); //Could be an ephemeral port
// create a selector;
selector= Selector.open();
readers = new Reader[readThreads];
readPool = Executors.newFixedThreadPool(readThreads,
new ThreadFactoryBuilder().setNameFormat(
"IPC Reader %d on port " + port).setDaemon(true).build());
for (int i = 0; i < readThreads; ++i) {
Reader reader = new Reader();
readers[i] = reader;
readPool.execute(reader);
}
// Register accepts on the server socket with the selector.
acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
this.setName("IPC Server listener on " + port);
this.setDaemon(true);
}
1. HBaseServer创建后有几个重要的角色.
1.1 Listener deamon线程,负责接收HMaster,HRegionServer,HBase Client的http请求.
1.2 Responder demon线程,负责将处理完的请求,发送回调用者.
1.3 Connection listener接收到的每个Socket请求都会创建一个Connection 实例.
1.4 Call 每一个客户端的发送的请求由Connection读取到有效数据后都会生成一个Call实例
1.5 LinkedBlockingQueue callQueue 每个由新生成的call都会放入到callQueue这个队列里.
1.6 Handler 从callQueue中取出call,并对call进行反射的调用,生成的结果value,交由responder处理.
1.7 LinkedList Connection.responseQueue ,用来存放已经由handler处理过的属于同一个Connection的call.
HBaseServer->listener = new Listener();->Reader.run->doRunLoop->doRead->Connection.readAndProcess->processData中将rpc请求生成call并加入到queue中。
Call call = new Call(id, param, this, responder, callSize);
callQueueSize.add(callSize);
HBaseServer的下面方法决定了接收RPC请求的queue大小maxQueueLength ，默认是100
String oldMaxQueueSize = this.conf.get("ipc.server.max.queue.size");
if (oldMaxQueueSize == null) {
this.maxQueueLength =
this.conf.getInt("ipc.server.max.callqueue.length",
handlerCount * DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER);
} else {
LOG.warn("ipc.server.max.queue.size was renamed " +
"ipc.server.max.callqueue.length, " +
"please update your configuration");
this.maxQueueLength = Integer.getInteger(oldMaxQueueSize);
}
this.maxQueueSize =
this.conf.getInt("ipc.server.max.callqueue.size",
DEFAULT_MAX_CALLQUEUE_SIZE);
this.readThreads = conf.getInt(
"ipc.server.read.threadpool.size",
10);
this.callQueue = new LinkedBlockingQueue<Call>(maxQueueLength);
当regionserver启动时调用startServiceThreads内会调用HBaseServer的下面方法启动10个hander线程。
public synchronized void startThreads() {
responder.start();
listener.start();
handlers = new Handler[handlerCount];
for (int i = 0; i < handlerCount; i++) {
handlers[i] = new Handler(callQueue, i);
handlers[i].start();
}
if (priorityHandlerCount > 0) {
priorityHandlers = new Handler[priorityHandlerCount];
for (int i = 0 ; i < priorityHandlerCount; i++) {
priorityHandlers[i] = new Handler(priorityCallQueue, i);
priorityHandlers[i].start();
}
}
}
handler线程最终会调用到run的call方法，再此方法中会射出相应的方法并最终调用，也就是会调用相应的regionserver的方法。
Method method =
protocol.getMethod(call.getMethodName(),
call.getParameterClasses());
method.setAccessible(true);
//Verify protocol version.
//Bypass the version check for VersionedProtocol
if (!method.getDeclaringClass().equals(VersionedProtocol.class)) {
long clientVersion = call.getProtocolVersion();
ProtocolSignature serverInfo = ((VersionedProtocol) instance)
.getProtocolSignature(protocol.getCanonicalName(), call
.getProtocolVersion(), call.getClientMethodsHash());
long serverVersion = serverInfo.getVersion();
if (serverVersion != clientVersion) {
LOG.warn("Version mismatch: client version=" + clientVersion
+ ", server version=" + serverVersion);
throw new RPC.VersionMismatch(protocol.getName(), clientVersion,
serverVersion);
}
}
Object impl = null;
if (protocol.isAssignableFrom(this.implementation)) {
impl = this.instance;
}
else {
throw new HBaseRPC.UnknownProtocolException(protocol);
}
long startTime = System.currentTimeMillis();
Object[] params = call.getParameters();
Object value = method.invoke(impl, params);