一个hive客户端登录问题
20191226
公司开始部署Spark版本的程序，之前使用的是MapReduce框架。我主要工作是配合部署工程师调试程序。
首先是部署Spark版程序的运行环境。我把spark streaming的包放在服务器上并添加bin目录到环境变量，然后hive相关的任务脚本都执行失败了。
这些脚本是把SQL嵌入到shell脚本的，cron调度运行了近一年。忽然执行失败，要么集群出现问题，要么就是任务失败前的变更导致。
使用其它主机提交相同任务可以运行，证明hadoop集群没有问题。那么就是最近的变更有问题。后来发现出问题的主机连hive客户端都不能登录。以下是登录时的报错：
Connecting to jdbc:hive2://10.236.160.74:2181,10.236.160.154:2181,10.236.160.54:2181,10.236.160.61:2181,10.216.160.216:2181,10.236.160.37:2181,10.236.160.180:2181/icaasiainfo;principal=hive/hivecluster@BCHKDC;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2?mapreduce.job.queuename=root.bdoc.renter_1.renter_49.dev_73
19/12/24 08:26:02 INFO Utils: Supplied authorities: 10.236.160.74:2181,10.236.160.154:2181,10.236.160.54:2181,10.236.160.61:2181,10.216.160.216:2181,10.236.160.37:2181,10.236.160.180:2181
19/12/24 08:26:02 INFO CuratorFrameworkImpl: Starting
19/12/24 08:26:02 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-bc1.3.6--1, built on 09/13/2017 15:18 GMT
19/12/24 08:26:02 INFO ZooKeeper: Client environment:host.name=host-10-236-157-70.openstacklocal
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.version=1.7.0_67
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.7.0_67/jre
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.class.path=/cmss/bch/bc1.3.6/spark//conf/:/cmss/bch/bc1.3.6/spark//lib/spark-assembly-1.6.2-bc1.3.6-hadoop2.6.0-bc1.3.6.jar:/cmss/bch/bc1.3.6/spark/lib/datanucleus-core-3.2.10.jar:/cmss/bch/bc1.3.6/spark/lib/datanucleus-rdbms-3.2.9.jar:/cmss/bch/bc1.3.6/spark/lib/datanucleus-api-jdo-3.2.6.jar:/cmss/bch/bc1.3.6/hadoop/etc/hadoop/
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
19/12/24 08:26:02 INFO ZooKeeper: Client environment:java.compiler=<NA>
19/12/24 08:26:02 INFO ZooKeeper: Client environment:os.name=Linux
19/12/24 08:26:02 INFO ZooKeeper: Client environment:os.arch=amd64
19/12/24 08:26:02 INFO ZooKeeper: Client environment:os.version=3.10.0-514.el7.x86_64
19/12/24 08:26:02 INFO ZooKeeper: Client environment:user.name=aiica
19/12/24 08:26:02 INFO ZooKeeper: Client environment:user.home=/home/louhl
19/12/24 08:26:02 INFO ZooKeeper: Client environment:user.dir=/home/louhl
19/12/24 08:26:02 INFO ZooKeeper: Initiating client connection, connectString=10.236.160.74:2181,10.236.160.154:2181,10.236.160.54:2181,10.236.160.61:2181,10.216.160.216:2181,10.236.160.37:2181,10.236.160.180:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2e7c470b
19/12/24 08:26:02 INFO ClientCnxn: Opening socket connection to server 10.236.160.61/10.236.160.61:2181. Will not attempt to authenticate using SASL (unknown error)
19/12/24 08:26:02 INFO ClientCnxn: Socket connection established to 10.236.160.61/10.236.160.61:2181, initiating session
19/12/24 08:26:02 INFO ClientCnxn: Session establishment complete on server 10.236.160.61/10.236.160.61:2181, sessionid = 0x16ec5efe277d92b, negotiated timeout = 60000
19/12/24 08:26:02 INFO ConnectionStateManager: State change: CONNECTED
19/12/24 08:26:02 WARN ConnectionStateManager: There are no ConnectionStateListeners registered.
19/12/24 08:26:03 INFO ZooKeeperHiveClientHelper: Selected HiveServer2 instance with uri: ZW201-A03U03-HDP-NOD011:10000
19/12/24 08:26:03 INFO ZooKeeper: Session: 0x16ec5efe277d92b closed
19/12/24 08:26:03 INFO ClientCnxn: EventThread shut down
19/12/24 08:26:03 INFO Utils: Resolved authority: ZW201-A03U03-HDP-NOD011:10000
19/12/24 08:26:04 INFO HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://ZW201-A03U03-HDP-NOD011:10000/icaasiainfo;principal=hive/hivecluster@BCHKDC;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2?mapreduce.job.queuename=root.bdoc.renter_1.renter_49.dev_73
19/12/24 08:26:04 ERROR HiveConnection: Error opening session
org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:mapreduce.job.queuename=root.bdoc.renter_1.renter_49.dev_73, use:database=icaasiainfo})
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:156)
at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:143)
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:583)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:192)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
at org.apache.hive.beeline.Commands.connect(Commands.java:1149)
at org.apache.hive.beeline.Commands.connect(Commands.java:1070)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
Error: Could not establish connection to jdbc:hive2://ZW201-A03U03-HDP-NOD011:10000/icaasiainfo;principal=hive/hivecluster@BCHKDC;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2?mapreduce.job.queuename=root.bdoc.renter_1.renter_49.dev_73: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{set:hiveconf:mapreduce.job.queuename=root.bdoc.renter_1.renter_49.dev_73, use:database=icaasiainfo}) (state=08S01,code=0)
Beeline version 1.6.2-bc1.3.6 by Apache Hive
0: jdbc:hive2://10.236.160.74:2181,10.236.160 (closed)>
ERROR HiveConnection: Error opening session的前一行显示beeline正在连接zookeeper主机的10000端口：Will try to open client transport with JDBC Uri: jdbc:hive2://ZW201-A03U03-HDP-NOD011:10000。实际上zk监听的是2181端口。
再看INFO ZooKeeper: Client environment:java.class.path=/cmss/bch/bc1.3.6/spark//conf/，意思应该是beeline加载的apark路径下的java.class.path。这处有大问题，beeline作为hive客户端应该是在hive/bin路径下，不该在spark的目录下。
查看spark环境变量，ll spark/bin|grep beeline，果然在spark/bin下也有beeline客户端。原来是配置环境变量的问题：我将spark加在原来PATH前面，说以hive登录时在spark/bin下找到了beeline。将spark/bin加在PATH后面问题解决。
