Spark kernel analysis (2) Principle analysis of Spark's three deployment modes

Spark supports three cluster managers (Cluster Manager), namely:

1 Standalone.: Standalone mode, Spark's native simple cluster manager, comes with a complete service, can be deployed to a cluster separately, without relying on any other resource management system, using Standalone can easily build a cluster;

2 Apache Mesos.: A powerful distributed resource management framework, which allows a variety of different frameworks to be deployed on it, including yarn;

3 Hadoop YARN.: A unified resource management mechanism, which can run multiple computing frameworks, such as map reduce, storm, etc., according to the location of the driver in the cluster, divided into yarn client and yarncluster.

In fact, in addition to the above-mentioned general cluster managers, Spark also provides some simple cluster deployment modes that are convenient for users to test and learn. Since most of the cluster managers used in the actual factory environment are Hadoop YARN, so we 关注的重点是 Hadoop YARN 模式下的 Spark 集群部署.

The running mode of Spark depends on the value of the MASTER environment variable passed to SparkContext. Some modes also require auxiliary program interfaces to cooperate. The currently supported Master strings and URLs include: When the

user submits a task to Spark for processing, the following two The parameters together determine how Spark runs:

（1）master MASTER_URL ：决定了 Spark 任务提交给哪种集群处理。
（2）deploy-mode DEPLOY_MODE：决定了 Driver 的运行方式，可选值为 Client 或者 Cluster。

1. Standalone mode operation mechanism

Standalone cluster has four important components, namely:

(1) Driver: It is a process. The Spark application we write runs on the Driver and is executed by the Driver process;

(2) Master: It is a process, which is mainly responsible for resource scheduling and allocation, and performs cluster monitoring and other responsibilities;

(3) Worker: It is a process, a Worker runs on a server in the cluster, and is mainly responsible for two responsibilities. One is to store one or some partitions of the RDD with its own memory; the other is to start other processes and threads (Executor), parallel processing and calculation of partitions on RDD.

(4) Executor: It is a process. Multiple Executors can be run on a Worker. Executor executes parallel calculations on RDD partitions by starting multiple threads (tasks), that is, executes the RDD definitions such as map, flatMap, reduce Wait for operator operations.

1.1 Standalone Client mode

In the Standalone Client mode, the Driver runs on the local machine where the task is submitted. After the Driver is started, the application is registered with the Master. The Master finds internal resources according to the resource requirements of the submit script and can start all the Workers of at least one Executor, and then between these Workers Assign the Executor, the Executor on the Worker will register with the Driver reversely after it is started. After all the Executors are registered, the Driver starts to execute the main function, and then when the execution reaches the Action operator, it starts to divide the stages, and each stage generates the corresponding taskSet. Distribute tasks to each Executor for execution.

1.2 Standalone Cluster mode

In the Standalone Cluster mode, after the task is submitted, the Master will find a Worker to start the Driver process. After the Driver is started, the application will be registered with the Master. The Master finds internal resources according to the resource requirements of the submit script and can start all the Workers of at least one Executor. The Executor is allocated among the Workers. After the Executor on the Worker is started, it will register with the Driver in the reverse direction. After all the Executors are registered, the Driver starts to execute the main function, and then when the execution reaches the Action operator, it starts to divide the stages, and each stage generates the corresponding taskSet, and then distribute the task to each Executor for execution.

Note :Standalone 的两种模式下（client/Cluster），Master 在接到 Driver 注册Spark 应用程序的请求后，会获取其所管理的剩余资源能够启动一个 Executor 的所有 Worker，然后在这些 Worker 之间分发 Executor，此时的分发只考虑 Worker 上的资源是否足够使用，直到当前应用程序所需的所有 Executor 都分配完毕，Executor反向注册完毕后，Driver 开始执行 main 程序。

2. YARN mode operation mechanism

2.1 YARN Client mode

In the YARN Client mode, the Driver runs on the local machine where the task is submitted. After the Driver is started, it will communicate with the ResourceManager to apply to start the ApplicationMaster, and then the ResourceManager will allocate the container and start the ApplicationMaster on the appropriate NodeManager.此时的ApplicationMaster 的功能相当于一个 ExecutorLaucher，只负责向 ResourceManager申请 Executor 内存。

ResourceManager will be allocated after receiving the resource application from ApplicationMaster container, and then ApplicationMaster will start the Executor process on the NodeManager specified in the resource allocation. After the Executor process is started, it will register with Driver in the reverse direction. After all Executor registration is completed, Driver will start to execute the main function, and then execute to Action When an operator is used, a job is triggered, and the stages are divided according to wide dependencies. Each stage generates a corresponding taskSet, and then distributes the tasks to each Executor for execution.

2.2 YARN Cluster mode

In the YARN Cluster mode, after the task is submitted, it will communicate with the ResourceManager to apply to start the ApplicationMaster, and then the ResourceManager will allocate a container and start the ApplicationMaster on the appropriate NodeManager. At this time, the ApplicationMaster is the Driver.

After the Driver is started, it applies to the ResourceManager for the Executor memory. After the ResourceManager receives the resource request from the ApplicationMaster, it will allocate the container, and then start the Executor process on the appropriate NodeManager. After the Executor process is started, it will register with the Driver in the reverse direction. After the Executor is fully registered, the Driver will start to execute The main function, after the execution of the Action operator, triggers a job, and starts to divide the stages according to wide dependencies, and each stage generates a corresponding taskSet, and then distributes the tasks to each Executor for execution.