I. Overview
There are two states of SparkSQL metadata:
1, in_memory, when the metadata is used up, it will be lost
2. hive, which is saved through hive, that is to say, wherever the metadata of hive exists, its metadata also exists.
In other words, SparkSQL's data warehouse is built on top of Hive . When we use SparkSQL to build a data warehouse, we must rely on Hive.
2. Spark-SQL script
If the user runs the bin/spark-sql command directly. This will cause our metadata to have two states:
1. In-memory status:
If the hive-site.xml file is not placed in the SPARK-HOME/conf directory, the status of the metadata is in-memory
2. Hive status:
If we put the hive-site.xml file in the SPARK-HOME/conf directory, then by default
The state of spark-sql's metadata is hive.