Spark-data loading and data landing (consolidation)

Regarding the difference between creating temporary views  

## 1. 从范围上说分为带Global和非Global的,带Global代表是当前SparkApplication可用的,而非Global的表示只在当前的SparkSession中可以使用
## 2. 从创建的角度上比较,createTempView,创建临时视图,如果该视图存在就报错。createOrReplaceTempView创建临时视图,如果视图存在就会覆盖之。
createOrReplaceTempView
createTempView 

createOrReplaceGlobalTempView
createGlobalTempView

data loading

       There are generally two ways to load data:

  1. spark.read.format("xx format").load("path")
  2. spark.read.xx format("path") 
//标准的加载方式:
spark.read.format("数据格式").load(path)
//简写:
spark.read.json("file:///d:/1.json")默认加载parquet 
 //读取JDBC操作1
    val df = spark.read.format("jdbc")
        .option("url","jdbc:mysql://master:3306/spark-sql") //路径
        .option("dbtable","user") //指定哪张数据表
        .option("user","root") //指定用户名
        .option("password","p@ssw0rd") //指定密码
        .load() //读取
    //读取JDBC操作2
    val pro = new Properties()
    pro.put("user","root") //指定用户名
    pro.put("password","p@ssw0rd") //指定密码
    //jdbc("路径","表名","Properties对象")
    val df1 = spark.read.jdbc("jdbc:mysql://master:3306/spark-sql","user",pro)

data landing

        There are generally two ways of data landing:

1. spark.wirte.format("file format").mode("storage mode").save("path")

2. spark.wirte.format("path")

标准格式:

df.write.format("text").save("file:///C:\\real_win10\\1.txt")

简写

df.write.text("file:///C:\\real_win10\\1.txt")

Guess you like

Origin blog.csdn.net/dafsq/article/details/129623328