Spark Core:RDD编程Action

reduce(f: (T, T) => T): T

:通过func函数聚集RDD中的所有元素

scala> var rdd=sc.parallelize(1 to 10,2).collect
rdd: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> rdd.reduce(_+_)
res1: Int = 55

scala> sc.makeRDD(Array(("A",0),("A",2),("B",1),("B",2),("C",1)))
res3: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[3] at makeRDD at <console>:25

scala> res3.reduce((x,y)=>(x._1+y._1,x._2+y._2))
res4: (String, Int) = (BBCAA,6)

collect(): Array[T]

:在驱动程序中，以数组的形式返回数据集的所有元素

scala> var rdd=sc.parallelize(1 to 10,2).collect
rdd: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

count(): Long

:返回RDD中的元素个数

scala> var rdd=sc.makeRDD(1 to 10)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[5] at makeRDD at <console>:24

scala> rdd.count
res9: Long = 10

first(): T

:返回RDD中的第一个元素

scala> var rdd=sc.makeRDD(1 to 10)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[6] at makeRDD at <console>:24

scala> rdd.first
res10: Int = 1

take(num: Int): Array[T]

:返回RDD中的前n个元素

scala> var rdd=sc.makeRDD(1 to 10)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[7] at makeRDD at <console>:24

scala> rdd.take(3)
res11: Array[Int] = Array(1, 2, 3)

takeOrdered(num: Int)(implicit ord: Ordering[T])

:返回前几个的排序

scala> var rdd=sc.makeRDD(Array(1,3,6,2,4,5))
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[10] at makeRDD at <console>:24

scala> rdd.takeOrdered(3)
res14: Array[Int] = Array(1, 2, 3)

aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T) => U, combOp: (U, U) => U): U

:aggregate函数将每个分区里面的元素通过seqOp和初始值进行聚合，然后用combine函数将每个分区的结果和初始值(zeroValue)进行combine操作。这个函数最终返回的类型不需要和RDD中元素类型一致。

scala> var rdd=sc.makeRDD(1 to 10,2)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[12] at makeRDD at <console>:24

scala> rdd.aggregate(1)(
     | (x:Int,y:Int)=>x+y,
     | (a:Int,b:Int)=>a+b
     | )
res16: Int = 58

scala> rdd.aggregate(1)(
     | (x:Int,y:Int)=>x*y,
     | (a:Int,b:Int)=>a*b
     | )
res17: Int = 3628800

fold(zeroValue: T)(op: (T, T) => T): T

:折叠操作，aggregate的简化操作，seqop和combop一样。

scala> var rdd=sc.makeRDD(1 to 10,2)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[12] at makeRDD at <console>:24

scala> rdd.aggregate(1)(
     | (x:Int,y:Int)=>x+y,
     | (a:Int,b:Int)=>a+b
     | )
res16: Int = 58

scala> rdd.fold(1)(_+_)
res18: Int = 58

saveAsTextFile(path: String): Unit

:将RDD以文本文件的方式保存到本地或者HDFS中

scala> var rdd=sc.makeRDD(1 to 10,2)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[12] at makeRDD at <console>:24

scala> rdd.saveAsTextFile("hdfs://hadoop01:9000/rdd")

[root@hadoop01 ~]# hadoop fs -cat /rdd/*
1
2
3
4
5
6
7
8
9
10

saveAsObjectFile(path: String): Unit

:将RDD中的元素以序列化后对象形式保存到本地或者HDFS中。

scala> rdd.saveAsObjectFile("hdfs://hadoop01:9000/rdd")

countByKey(): Map[K, Long]

:针对(K,V)类型的RDD，返回一个(K,Int)的map，表示每一个key对应的元素个数。

scala> var rdd=sc.makeRDD(List((1,3),(1,2),(1,4),(2,3),(3,6),(3,8)),3)
rdd: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[17] at makeRDD at <console>:24

scala> rdd.countByKey()
res22: scala.collection.Map[Int,Long] = Map(3 -> 2, 1 -> 3, 2 -> 1)

foreach(f: T => Unit): Unit

:在数据集的每一个元素上，运行函数func进行更新。

scala> var rdd=sc.makeRDD(1 to 10 ,2)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[20] at makeRDD at <console>:24

scala> rdd.collect().foreach(println)
1
2
3
4
5
6
7
8
9
10