先启动pyspark第一篇博客有
from pyspark import SparkContext,SparkConf
from pyspark.sql import SparkSession
spark = SparkSession.builder.config(conf=SparkConf()).getOrCreate()
spark.read.text(“people.txt”)#读取文件创建dataframe,加上路径一样
spark.read.json(“people.json”)
spark.read.parquet(“people.parquet”)
data.show()#看看简介
data.write.txt(“people.txt”)#json文件就是换成json，写入dataframe
data.write.format(“text”).save(“people.txt”)#写入保存
下面从把一个文件创建DataFrame并加载到另一个文件中去
p = spark.read.format(“json”).
…load(“file:///usr/local/spark/example/src/main/resources/people.json”)
p.select(“name”,“age”).write.format(“json”).
…save(“file:///usr/local/spark/mycode/sparksql/newpeople.json”)
#注意这里newpeole.json是个目录不是文件
df=spark.read.json(“peole.json”)
df.printSchema()
df.select(df[“name”])
df.filter(df[“age”]>0).show()
df.groupBy(“age”).count().show()
df.sort(df[“age”].desc()).show()
df.sort(df[“age”].desc(),df[“name”].asc()).show()#之前那个有问题的二次排序代码在这里轻松实现
