SparkSql connects to hive, but the Hive database or Hive table cannot be found

20/09/27 18:32:57 WARN ObjectStore: Failed to get database spark, returning NoSuchObjectException
Exception in thread “main” org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database ‘spark’ not found;

Project scene:

Spark integrates Hive, and idea uses spark locally to read data in Hive

Problem Description:

Check that the database spark exists on the hive client, but connect to Hive with spark cannot find the database spark
exception is as follows
20/09/27 18:32:57 WARN ObjectStore: Failed to get database spark, returning NoSuchObjectException
Exception in thread “main” org .apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database'spark' not found;

hive client view database

hive> show databases;
OK
default
spark
Time taken: 0.11 seconds, Fetched: 3 row(s)

Cause Analysis:

Spark did not find Hive metadata


solution:

Plus metadata configuration, hive.metastore.uris

Integrate Hive
hdfs-site.xml in scala and add configuration

    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://node003:9083</value>
    </property>


Spark integrates Hive spark in java to add configuration property config("hive.metastore.uris", "thrift://node003:9083")

SparkSession  spark = SparkSession.builder().appName("随便写").config("hive.metastore.uris","thrift://node003:9083").enableHiveSupport().getOrCreate();

Guess you like

Origin blog.csdn.net/weixin_43614067/article/details/108833075