Spark RDD Action operation

reduce

def reduce (f: (T, T) => T): T
aggregation by all elements in RDD func function, this function must be parallel and may be used interchangeably

      
      
       
       scala> val rdd1 = sc.makeRDD(1 to 10,2)
      
      
      
      
       
       rdd1: org.apache.spark.rdd.RDD [Int] = ParallelCollectionRDD [85] to makeRDD to <console>: 24
      
      
      
      
      
      
       
       scala> rdd1.reduce(_+_)
      
      
      
      
       
       res50: Int = 55
      
      
      
      
      
      
       
       scala> val rdd2 = sc.makeRDD(Array(("a",1),("a",3),("c",3),("d",5)))
      
      
      
      
       
       rdd2: org.apache.spark.rdd.RDD [(String, Int)] = ParallelCollectionRDD [86] to makeRDD to <console>: 24
      
      
      
      
      
      
       
       scala> rdd2.reduce((x,y)=>(x._1 + y._1,x._2 + y._2))
      
      
      
      
       
       res51: (String, Int) = (adca,12)

collect

def collect (): Array [T ]
in the driver returns the data set all the elements of an array

count

def count(): Long
返回RDD中的元素个数

first

def first(): T 返回RDD中的第一个元素

take

def take(num: Int): Array[T] 返回RDD中的前n个元素

takeOrdered

def takeOrdered(num: Int)(implicit ord: Ordering[T]) 返回前几个的排序

takeSample

def takeSample( withReplacement: Boolean, num: Int, seed: Long = Utils.random.nextLong): Array[T] 抽样但是返回一个scala集合。

aggregate

def aggregateU: ClassTag(seqOp: (U, T) => U, combOp: (U, U) => U): U aggregate函数将每个分区里面的元素通过seqOp和初始值进行聚合，然后用combine函数将每个分区的结果和初始值(zeroValue)进行combine操作。这个函数最终返回的类型不需要和RDD中元素类型一致。

fold

def fold(zeroValue: T)(op: (T, T) => T): T 折叠操作，aggregate的简化操作，seqop和combop一样。

saveAsTextFile

def saveAsTextFile(path: String): Unit 将RDD以文本文件的方式保存到本地或者HDFS中

saveAsObjectFile

def saveAsObjectFile(path: String): Unit 将RDD中的元素以序列化后对象形式保存到本地或者HDFS中。

countByKey

def countByKey (): Map [K, Long] for (K, V) type RDD, a return (K, Int) of the map, represents the number of elements corresponding to each key.

foreach

def foreach (f: T => Unit): Unit on each element of the data set, function func update operation.

Original: Big Box Spark RDD Action operation