1 Brief description
Spark Sql supports reading and writing data stored in Hive. When operating the data in Hive, you can create HiveContext instead of SqlContext. HiveContext inherits from SqlContext, but adds the function of looking up tables in the Hive metadata database, and writing sql functions in HiveQl syntax. In addition to the sql method, the hql method is also provided. So use hive syntax to compile sql.
Using HiveContext, you can perform most of the functions of Hive, including creating tables, importing data into tables, and querying data in tables with Sql statements. The query result is a Row array.
2 Save data
Spark Sql also allows data to be saved to the Hive table. Call the saveAsTable command of the DataFrame to save the data in the DataSet to the Hive table. Unlike registerTempTable, saveAsTable will materialize the data in the DataFrame to the Hive table, and Metadata for tables is also created in the Hive metabase.
By default, SaveAsTable will create a Hive Manager Table, that is, the location of the data is controlled by the information in the metabase. When ManagerTable is deleted, the data in the table is also deleted.
RegisterTempTable just registers a temporary table. As long as the Application is stopped or restarted, the table will be gone. The saveAsTable stone creates a physical table that will always exist regardless of whether the Application is restarted or stopped.
Welcome to pay attention, more benefits
