Spark 零件

SparkEnv

SparkEnv 是spark的执行环境对象，存在driver 或 executor 进程中。

BlockManager

Driver Application 和 Executor 都会创建 BlockManager .

Manager running on every node (driver and executors) which provides interfaces for putting and retrieving blocks both locally and remotely into various stores (memory, disk, and off-heap).

BlockManagerMaster

BlockManagerMaster 是在SparkEnv中创建

如果当前应用程序是Driver，则创建BlockManagerMasterEndpoint，并且注册到RpcEnv中；
如果当前应用程序是Executor，则从RpcEnv中找到BlockManagerMasterEndpoint的引用。

// SparkEnv.scala

val blockManagerMaster = new BlockManagerMaster(registerOrLookupEndpoint(
      BlockManagerMaster.DRIVER_ENDPOINT_NAME,
      new BlockManagerMasterEndpoint(rpcEnv, isLocal, conf, listenerBus)),
      conf, isDriver)



def registerOrLookupEndpoint(
        name: String, endpointCreator: => RpcEndpoint):
      RpcEndpointRef = {
      if (isDriver) {
        logInfo("Registering " + name)
        rpcEnv.setupEndpoint(name, endpointCreator)
      } else {
        RpcUtils.makeDriverRef(name, conf, rpcEnv)
      }
    }