Spark - 源码解析-LocalCluster模式启动分析

Spark常用的几种模式

Local

StandAlone

Yarn -

Mesos

Local模式又分为几种启动方式

```
local - local单核
```

LOCAL_N_REGEX(threads) - local[N]/local[*] 设置N个线程或启动与cpu core个数相同的线程

LOCAL_N_FAILURES_REGEX(threads, maxFailures) - 设置N线程数与最大失败次数

LOCAL_CLUSTER_REGEX(numWorkers, coresPerWorker, memoryPerWorker) - local伪分布式

简单介绍下local-cluster伪分布式模式的启动流程

说明，源码基于spark-3.1.2

通过match case判断传入master的启动类型

/**
 * numWorkers:worker的个数
 * coresPerWorker:核心数
 * memoryPerWorker:worker的内存
 * 在local启动Master和Worker
  */
/**
 * local-cluster伪分布式，类似standalone模式
 * local-cluster的master和worker都运行在local
 */
case LOCAL_CLUSTER_REGEX(numWorkers, coresPerWorker, memoryPerWorker) =>
  checkResourcesPerTask(coresPerWorker.toInt)
  // Check to make sure memory requested <= memoryPerWorker. Otherwise Spark will just hang.
  val memoryPerWorkerInt = memoryPerWorker.toInt
  if (sc.executorMemory > memoryPerWorkerInt) {
    throw new SparkException(
      "Asked to launch cluster with %d MiB RAM / worker but requested %d MiB/worker".format(
        memoryPerWorkerInt, sc.executorMemory))
  }

根据指定的worker，core，内存创建new LocalSparkCluster() ，并调用LocalSparkCluster的start()方法

val scheduler = new TaskSchedulerImpl(sc) // 创建TaskScheduler实例
val localCluster = new LocalSparkCluster(
  numWorkers.toInt, coresPerWorker.toInt, memoryPerWorkerInt, sc.conf)
val masterUrls = localCluster.start()

localCluster.start()方法主要启动Master和Worker

/* Start the Master */
val (rpcEnv, webUiPort, _) = Master.startRpcEnvAndEndpoint(localHostname, 0, 0, _conf)
masterWebUIPort = webUiPort
masterRpcEnvs += rpcEnv
val masterUrl = "spark://" + Utils.localHostNameForURI() + ":" + rpcEnv.address.port
val masters = Array(masterUrl)

/* Start the Workers */
for (workerNum <- 1 to numWorkers) {
  val workerEnv = Worker.startRpcEnvAndEndpoint(localHostname, 0, 0, coresPerWorker,
    memoryPerWorker, masters, null, Some(workerNum), _conf,
    conf.get(config.Worker.SPARK_WORKER_RESOURCE_FILE))
  workerRpcEnvs += workerEnv
}

通过Master和Worker的startRpcEnvAndEndpoint方法启动Rpc通讯

/**
 * 启动master，并返回一个tuple(RpcEnv对象,webUIPort,)
 * 为Master创建RpcEnv，并将Master注册到RpcEnv中(setupEndpoint)
 * Wroker有自己的RpcEnv
 * @return
 */
def startRpcEnvAndEndpoint(
    host: String,
    port: Int,
    webUiPort: Int,
    conf: SparkConf): (RpcEnv, Int, Option[Int]) = {
  val securityMgr = new SecurityManager(conf)
  val rpcEnv = RpcEnv.create(SYSTEM_NAME, host, port, conf, securityMgr)
  // setupEndpoint,实现Master注册RpcEnv
  val masterEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME,
    new Master(rpcEnv, rpcEnv.address, webUiPort, securityMgr, conf))
  val portsResponse = masterEndpoint.askSync[BoundPortsResponse](BoundPortsRequest)
  (rpcEnv, portsResponse.webUIPort, portsResponse.restPort)
}

/**
 * Worker启动RpcEnv
 * @return
 */
def startRpcEnvAndEndpoint(
    host: String,
    port: Int,
    webUiPort: Int,
    cores: Int,
    memory: Int,
    masterUrls: Array[String],
    workDir: String,
    workerNumber: Option[Int] = None,
    conf: SparkConf = new SparkConf,
    resourceFileOpt: Option[String] = None): RpcEnv = {

  // The LocalSparkCluster runs multiple local sparkWorkerX RPC Environments
  val systemName = SYSTEM_NAME + workerNumber.map(_.toString).getOrElse("")
  val securityMgr = new SecurityManager(conf)
  val rpcEnv = RpcEnv.create(systemName, host, port, conf, securityMgr)
  val masterAddresses = masterUrls.map(RpcAddress.fromSparkURL)
  rpcEnv.setupEndpoint(ENDPOINT_NAME, new Worker(rpcEnv, webUiPort, cores, memory,
    masterAddresses, ENDPOINT_NAME, workDir, conf, securityMgr, resourceFileOpt))
  rpcEnv
}

初始化TaskSchedule和Application

/**
 * StandaloneSchedulerBackend实现Spark的独立群集管理器
 * 主要实现了Rpc和ApplicationDescription以及StandaloneAppClient的构建
 */
val backend = new StandaloneSchedulerBackend(scheduler, sc, masterUrls)
scheduler.initialize(backend) // 初始化scheduler，返回调度池
backend.shutdownCallback = (backend: StandaloneSchedulerBackend) => {
  localCluster.stop()
}
(backend, scheduler) // 最终返回一个(SchedulerBackend, TaskScheduler)类型Tuple

欢迎大佬指正

Spark - 源码解析-LocalCluster模式启动分析

猜你喜欢