【运行环境】
java SE 1.8.0_101，hadoop-2.7.3，HBase-1.1.2，Zookeeper-3.4.6，python2.7，happybase1.1.0
【作业报错信息】
IOError: IOError(message='org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 69992 actions: IOException: 69992 times, \n\tat org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:234)\n\tat org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:214)\n\tat org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1623)\n\tat org.apache.hadoop.hbase.client.HTable.batch(HTable.java:909)\n\tat org.apache.hadoop.hbase.client.HTable.batch(HTable.java:923)\n\tat org.apache.hadoop.hbase.client.HTable.get(HTable.java:884)\n\tat org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.getRowsWithColumnsTs(ThriftServerRunner.java:1108)\n\tat org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.getRowsWithColumns(ThriftServerRunner.java:1064)\n\tat sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat com.sun.proxy.$Proxy11.getRowsWithColumns(Unknown Source)\n\tat org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$getRowsWithColumns.getResult(Hbase.java:4266)\n\tat org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$getRowsWithColumns.getResult(Hbase.java:4250)\n\tat org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\n\tat org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\n\tat org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n')Thu Apr 19 06:28:20 2018 <Greenlet at 0x7f9710c08e10: import_user_profile(<happybase.table.Table name='all_users:city_sales'>, ['P_2CAD1AC8878528C06AAB6EF859AB6DD8', 'P_2CAD1B1A, ['basic:pc_label', 'pref:shopping)> failed with IOError
【日志信息】
查看{HBASE_LOG_DIR}/hbase 目录下的日志文件 hbase-root-thrift-YOUR_HOSTNAME-master-1.log，找到作业报错时间附近的日志消息
2018-04-19 16:07:16,284 WARN [thrift-worker-0] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
2018-04-19 16:07:21,456 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:07:21,903 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:07:23,779 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 135571 actions to finish
2018-04-19 16:07:26,322 WARN [hconnection-0x26338bf1-shared--pool2-t75] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
2018-04-19 16:07:31,463 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:07:31,913 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:07:33,780 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 118358 actions to finish
2018-04-19 16:07:34,380 INFO [thrift-worker-0] client.AsyncProcess: #1, waiting for 135571 actions to finish
2018-04-19 16:07:41,465 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:07:41,922 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:07:43,792 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 118358 actions to finish
2018-04-19 16:07:44,386 INFO [thrift-worker-0] client.AsyncProcess: #1, waiting for 135571 actions to finish
2018-04-19 16:07:51,466 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:07:51,928 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:07:53,800 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 118358 actions to finish
2018-04-19 16:07:54,416 INFO [thrift-worker-0] client.AsyncProcess: #1, waiting for 135571 actions to finish
2018-04-19 16:08:01,470 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:08:01,934 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:08:03,803 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 52007 actions to finish
2018-04-19 16:08:04,419 INFO [thrift-worker-0] client.AsyncProcess: #1, waiting for 135571 actions to finish
2018-04-19 16:08:11,479 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:08:11,940 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:08:13,803 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 52007 actions to finish
2018-04-19 16:08:14,428 INFO [thrift-worker-0] client.AsyncProcess: #1, waiting for 135571 actions to finish
2018-04-19 16:08:21,481 INFO [thrift-worker-3] client.AsyncProcess: #1, waiting for 83858 actions to finish
2018-04-19 16:08:21,943 INFO [thrift-worker-2] client.AsyncProcess: #1, waiting for 65870 actions to finish
2018-04-19 16:08:23,809 INFO [thrift-worker-1] client.AsyncProcess: #1, waiting for 52007 actions to finish
【日志解读】
thrift-worker就是thriftServer的线程。
client.AsyncProcess: #1, waiting for 65870 actions to finish 是说此客户端在等待进程号65870的作业完成。
再结合正在运行的大查询作业，发现有一个客户端程序正在 scan 一张Hbase的表。
可以推测出，出错原因就是thriftServer线程数不足。
【解决】增加thriftServer线程数量
# 在两个hmaster都做ThriftServer启动参数配置：最小线程数量300个，最大线程数500个
cd {your_hbase_home}/bin && ./hbase-daemon.sh start thrift --threadpool -m 300 -w 500
ssh {hmaster_ip} "cd {your_hbase_home}/bin/ && ./hbase-daemon.sh start thrift --threadpool -m 300 -w 500"
# 检查刚才重启的thrift进程，查看设置是否生效
[root@tony-cluster-master-001 bin]# ps aux | grep -v grep | grep thrift
root 27239 0.0 0.0 113132 1556 pts/3 S 11:58 0:00 bash {your_hbase_home}/bin/hbase-daemon.sh --config {your_hbase_home}/bin/../conf foreground_start thrift --threadpool -m 300 -w 500
root 27253 104 0.7 10324400 244132 pts/3 Sl 11:58 0:04 /usr/java/oracle-jdk/bin/java -Dproc_thrift -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Dhdp.version=2.6.1.0-129 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/mnt/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/mnt/log/hbase/gc.log-201804201158 -Dhbase.log.dir=/mnt/log/hbase -Dhbase.log.file=hbase-root-thrift-kmr-a20125dd-gn-a05044c6-master-1-001.ksc.com.log -Dhbase.home.dir={your_hbase_home}/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.6.1.0-129/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-129/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.thrift.ThriftServer --threadpool -m 300 -w 500 start
# 启动参数生效！
重新运行python客户端的作业，成功完成。