Spark core之BlockManager

BlockManager源码分析

  • key-value形式存储的数据块,用block ID表示
  • 充当在Spark集群中的每个节点上运行的本地缓存(driver and executors)

SparkEnv实例化一个BlockManager对象blockManager

    // NB: blockManager is not valid until initialize() is called later.
    val blockManager = new BlockManager(
    					 executorId, 
    					 rpcEnv,
    					 blockManagerMaster,
      					 serializerManager, 
      					 conf, 
      					 memoryManager, 
      					 mapOutputTracker, 
      					 shuffleManager,
     					 blockTransferService, 
     					 securityManager, 
     					 numUsableCores)

调用BlockManager的主构造器(主构造器会执行类定义中的所有语句)

 // 定义变量blockManagerId,初始值为null
 var blockManagerId: BlockManagerId = _  
 ......
 blockManagerId = 
     if (idFromMaster != null) 	          
           idFromMaster 
     else id   

猜你喜欢

转载自blog.csdn.net/qq_29573903/article/details/83176573