Accumulator (two)

Accumulator

Application scenario : The driver side defines a shared variable and accumulates the data to the variable. If you directly use iterative operators such as foreach or map, the accumulated variable cannot be returned to the driver side, because the accumulation process takes place on the Executor side. Generally used in counting scenarios, variables are often declared on the Driver side.

Features : Variables are on the Driver side, and the accumulation process is on the Executor side. In the accumulation process, the Executor side cannot read its value. If you want to read its value, you
can only read it on the Driver side.
Use :
1. Create an instance of the Accumulator
2. Register an accumulator through sc.register()
3. Add data
through the real name of the accumulator . Add 4. Pass the accumulator instance name. value to and from the accumulator value

import org.apache.spark.util.{
    
    DoubleAccumulator, LongAccumulator}
import org.apache.spark.{
    
    SparkConf, SparkContext}

object AccumlatorV2Demo {
    
    
  def main(args: Array[String]): Unit = {
    
    
    val conf = new SparkConf().setAppName(this.getClass.getName).setMaster("local[2]")
    val sc = new SparkContext(conf)
    val nums1 = sc.parallelize(List(1,2,3,4,5,6,7,8,9),2)
    val nums2 = sc.parallelize(List(1.2,2.4,3.4,4.0,5.0,6.0,7.0,8.0,9.0),2)
   //注册一个long累加器，并初始化累加器
    def longAcc(name:String) : LongAccumulator={
    
    
      val acc = new LongAccumulator
      sc.register(acc,name)
      acc
    }
    //注册一个double累加器，并初始化累加器
    def doubleAcc(name:String) : DoubleAccumulator={
    
    
      val acc = new DoubleAccumulator
      sc.register(acc,name)
      acc
    }
    val acc1: LongAccumulator = longAcc("LongAccumulator")
    nums1.foreach(x=>acc1.add(x))
    val acc2: DoubleAccumulator = doubleAcc(" DoubleAccumulator ")
    nums2.foreach(x=>acc2.add(x))
    println(acc1.value)
    println(acc2.value)

    sc.stop()
  }
}