Spark SQL structured data processing, it may be stored in a two-dimensional table, similar to the table in the database as stored data
Spark1.x
val sqlContext = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
// The RDD Schema and related information together, 1, RDD and case class 2, RDD and StructType
// case class Person converted into data in the RDD case class corresponding to the attribute type, and then provided to the case class of
Val eet: eet [Person] = ....
// Convert the RDD into DataFrame
df = rdd.toDF Val
// to operate df (1, directly on the operator DSL.2 DataFrame, write SQL)
// df will be registered as a temporary table
df.registerTempTable("t_person")
// execute SQL
val result :DataFrame = sqlContext.sql("select * from t_person");
result.show()
Spark2.x
val spark = SparkSession.builder().appName("a").master("local[*]").getOrCreate()
// Create DF
val df = spark.createDataFrame(RDD[Row], schema)
// DSL and SQL
df.createTempView("v_user")
// execute SQL
val result:DataFrame = spark.sql("select * from t_user")
// Perform action
result.show()
//