var counter = 0
val data = Seq(1,2,3)
var rdd = sc.parallelize (data)
rdd.foreach (x => counter + = x)
println(counter)
Why is this? All function calls to RDD, although the code appears to be running in the Driver program, but the actual calculation is not local. Even when running in stand-alone local mode, it runs on a separate executor process. The execution of each job will go through the process of serialization, network transmission, deserialization and execution.
During the execution process, only the counter on the computing node will increase automatically, but the counter of the Driver program will not change. After the execution is completed, the result is returned to the Driver program (foreach in the example has no return value), and the counter in the Driver is still the original Driver, with a value of 0.