使用Hive-JDBC遇到的一些问题解决

其他 2018-07-23 09:30:07 阅读次数: 0

1.java.lang.NoClassDefFoundError: org/apache/hive/service/cli/thrift/TCLIService$Iface

使用JDBC驱动连接到hive的时候，出现了NoClassDefFoundError的错误，提示找不到Iface接口这个类。

Hive版本1.1

java.lang.NoClassDefFoundError: org/apache/hive/service/cli/thrift/TCLIService$Iface
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:208)
	at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:116)
	at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:107)
	at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:191)
	at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:161)
	at com.bonc.vbap.data.query.jdbc.ConnectionManager.getConnection(ConnectionManager.java:63)
	at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext$SchemaHelper.getConnection(JdbcSchemaContext.java:308)
	at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext$SchemaHelper.<init>(JdbcSchemaContext.java:112)
	at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext.getSchemaHelper(JdbcSchemaContext.java:68)
	at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext.getSchemaNames(JdbcSchemaContext.java:43)
	at com.bonc.vbap.data.service.impl.SchemaServiceImpl.getSchemaNames(SchemaServiceImpl.java:34)
	at com.bonc.vbap.data.schema.jdbc.TestJdbcColumn.testHiveDataQuery(TestJdbcColumn.java:252)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.ClassNotFoundException: org.apache.hive.service.cli.thrift.TCLIService$Iface
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 44 more

根据错误定位到..HiveDriver类105行的位置

 
public Connection connect(String url, Properties info) throws SQLException {

return acceptsURL(url) ? new HiveConnection(url, info) : null;

}

发现在HiveConnection中有调用TCLIService.Iface接口


...

TCLIService.Iface client = new TCLIService.Client(new TBinaryProtocol(transport));

TOpenSessionResp openResp = client.OpenSession(new TOpenSessionReq());

if (openResp != null) {

client.CloseSession(new TCloseSessionReq(openResp.getSessionHandle()));

}

...
这里抛了找不到Iface的异常，但是点进去TCLIService类中，确是存在Iface接口，所以说找不到不是因为不存在，而是出现重复的类。
 

@SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})

@Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")

public class TCLIService {


public interface Iface {


public TOpenSessionResp OpenSession(TOpenSessionReq req) throws org.apache.thrift.TException;


public TCloseSessionResp CloseSession(TCloseSessionReq req) throws org.apache.thrift.TException;

.....

所以去看下引入的依赖包，发现有hive-service和hive-service-rpc两个包中有相同的源码，所以在这里会抛异常，我就先把这其中一个依赖从hive-jdbc中去除掉，但是发现还是抛异常，我这里用的hive-jdbc1.1版本，发现hive-service是2.1版本，所以立马把两个都去除掉，重新引入一个hive-service进来就好拉

<dependency>

<groupId>org.apache.hive</groupId>

<artifactId>hive-jdbc</artifactId>

<exclusions>

<exclusion>

<groupId>org.apache.hive</groupId>

<artifactId>hive-service-rpc</artifactId>

</exclusion>

<exclusion>

<groupId>org.apache.hive</groupId>

<artifactId>hive-service</artifactId>

</exclusion>

</exclusions>

<version>1.1.0</version>

</dependency>

<dependency>

<groupId>org.apache.hive</groupId>

<artifactId>hive-service</artifactId>

<version>1.1.0</version>

</dependency>


 


2.上述的问题解决之后，又出现了一个新的问题：java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

异常详情：


java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:393)

at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:187)

at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

at java.sql.DriverManager.getConnection(DriverManager.java:664)

at java.sql.DriverManager.getConnection(DriverManager.java:208)

at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:116)

at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:107)

at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:191)

at com.bonc.vbap.data.query.jdbc.spring.datasource.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:161)

at com.bonc.vbap.data.query.jdbc.ConnectionManager.getConnection(ConnectionManager.java:63)

at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext$SchemaHelper.getConnection(JdbcSchemaContext.java:308)

at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext$SchemaHelper.<init>(JdbcSchemaContext.java:112)

at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext.getSchemaHelper(JdbcSchemaContext.java:68)

at com.bonc.vbap.data.schema.jdbc.JdbcSchemaContext.getSchemaNames(JdbcSchemaContext.java:43)

at com.bonc.vbap.data.service.impl.SchemaServiceImpl.getSchemaNames(SchemaServiceImpl.java:34)

at com.bonc.vbap.data.schema.jdbc.TestJdbcColumn.testHiveDataQuery(TestJdbcColumn.java:252)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:52)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)

at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)

at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)

at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)

at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)

at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)

at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 47 more

这是因为要缺少一个hadoop-common包，引入即可：


<dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-common</artifactId>

<version>2.6.5</version>

</dependency>

3. 在使用DatabaseMetaData获取元数据信息时，抛出异常：

"Method not supported"

这是因为HiveDatabaseMetaData在实现DatabaseMetaData接口时，没有做相应方法的实现，这里我使用到的是一个getUserName接口：

 
...

public String getUserName() throws SQLException {

throw new SQLException("Method not supported");

}

...

可以看到在HiveDatabaseMetaData中没有对getUserName支持，所以会抛异常，getUserName方法是获取到当前HiveConnection连接的用户名(Schema名称)，但是这里不支持，那就得另外想办法，这里可以调用HiveConnection中的一个getConnParams方法得到连接参数，再调用getSchema方法获取到Schema名称。代码：

...

HiveDatabaseMetaData hiveMetaData = (HiveDatabaseMetaData) metadata;

HiveConnection hiveConn = (HiveConnection) hiveMetaData.getConnection();

String hiveSchema = hiveConn.getSchema();

...

猜你喜欢

转载自blog.csdn.net/qq_34246546/article/details/81075452
今日推荐
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
周排行
浏览器对同一域名进行请求的最大并发连接数
React Hook之自定义Hook
【转】MyBatis缓存机制
-Java-泛型
自动化测试常用脚本-发送邮件
LeetCode#859: Buddy Strings
java、Python处理字符串
第二篇の博客
Hadoop伪分布式环境安装
SQL Server进阶（十一）临时表、表变量
每日归档
更多
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022