spark thriftserver

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/bigdataf/article/details/78933505

1.hive mr 太慢了，基于次 适配 spark-sql的功能 ，于是乎，想通过spar启动thriftserver来解决

问题1：

17/12/29 16:28:59 WARN thrift.ThriftCLIService: Error opening session:
java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.schshim.FairSchedulerShim
        at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:149)
        at org.apache.hadoop.hive.shims.ShimLoader.getSchedulerShims(ShimLoader.java:133)
        at org.apache.hadoop.hive.shims.Hadoop23Shims.refreshDefaultQueue(Hadoop23Shims.java:296)
        at org.apache.hive.service.cli.session.HiveSessionImpl.<init>(HiveSessionImpl.java:109)
        at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:251)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
        at org.apache.hive.service.cli.CLIService.openSession(CLIService.java:194)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:354)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:246)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.schshim.FairSchedulerShim
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:146)
        ... 17 more

原因编译的时候没有
加org.apache.hive.shims hive-shims-scheduler 1.2.1
可以借用hive的包

cp  hive/lib/hive-shims*  .spark/jars/

问题2：

Exception in thread "HiveServer2-Handler-Pool: Thread-154" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService$L
istener
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:146)
        at org.apache.hadoop.hive.shims.ShimLoader.getSchedulerShims(ShimLoader.java:133)
        at org.apache.hadoop.hive.shims.Hadoop23Shims.refreshDefaultQueue(Hadoop23Shims.java:296)
        at org.apache.hive.service.cli.session.HiveSessionImpl.<init>(HiveSessionImpl.java:109)
        at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:251)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
        at org.apache.hive.service.cli.CLIService.openSession(CLIService.java:194)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:354)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:246)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
        at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService$Listener
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 20 more

解决方法 hadoop 包应用有问题
可以直接cp到jars中来

cp ./hadoop_home/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.4.jar .／jars/
 1096  cp ./hadoop_home/share/hadoop/yarn/sources/hadoop-yarn-server-resourcemanager-2.6.4-sources.jar  .／jars／