Spark cluster scheduling actually quite simple, here is a summary of some of the basic principles of cluster resource scheduling.
spark cluster is how to manage computing resources and memory resource cluster it? In fact, behind this there is a mechanism for resource registration. Simply put, Worker nodes to speak their own resources report to the master.
First, when the start of the Worker node, corresponding to the process worker Worker class implements a class WorkerArguments Worker Process read configuration information from a configuration file related to, or use the default value, including the number of cores assigned to the Worker and memory capacity. If these resources are not configured in the configuration file, then you take the default value, may have been inferDefaultCores methods and inferDefaultMemory method. If the configuration file is, and then it takes a value SPARK_WORKER_CORES SPARK_WORKER_MEMORY provided. If the Worker restart when explicitly specify the parameters -core (-c) or -memory (-m), to develop an explicit value is employed.
Then, you need to register with the Master Worker process, reporting computing resources and memory capacity Worker-owned. When implemented, the method calls tryRegisterAllMaster Worker class, sending a registration message to the Master node RegisterWorker Akka communication mechanism. Worker information including the node number, the host address, port, memory and user-assigned number of cores and the like.
Finally, Master node information transmitted Worker recorded, stored in WorkerInfo object.
These are the resource registration process Worker.
Resource allocation and application
In a clustered environment, the communication between the cluster program Driver Driver program implemented by instantiating CoarseGrainedSchedulerBackend class. Examples of different trunked mode CoarseGrainedSchedulerBackend different subclasses. For Standalone, using a SparkDeploySchedulerBackend class that will start when instantiating a APPClient class, APPClient would think Master node sends an application registration request, a registration request containing the resources required by the application.
Master node after receiving a registration request for an application, the application will wait in the queue, and calls Master.scheduler method, which corresponds to the driver Master process / application dispatcher logic: First Driver process scheduling, in YARN cluster mode, Driver program can run on worker nodes, Master nodes need to dedicate appropriate resources to run Driver cluster process, the approach is to randomly assign to the Driver on idle Worker.
Scheduling a particular application, the method using FIFO scheduling among multiple applications, according to the registration order of the application resource allocation. For between a single application, there SpreadOut and non SpreadOut two scheduling strategies. Before allocating resources, Master need to query the current cluster of memory resources meets the minimum requirements to run the application, and Worker Executor previously allocated for this application too can not participate in resource scheduling, an application on a Worker only one Executor .
SpreadOut strategy, Master polling way to keep every available Worker assigned to a core application, know the distribution of the core to meet the needs of the application.
Non-SpreadOut strategy will put all one-time allocation of all core can be allocated on a Worker to the application.
Worker is assigned the task calls the appropriate function to add addExecutor the Executor, and the call start method launchExecutor the Executor, will change recording method Worker resources are consumed, and to send the corresponding message Worker, notify start Executor. Worker after receiving the message, the record consumption of resources, and start a new Executor process.
The figure is a flow chart of resource scheduling under YARN cluster model, according to this picture, we can also understand the resource scheduling.
Spark submit an application, first by ResourceManager request to start a Client Application, also check whether there are sufficient resources to meet the needs of the Application, if the resource condition is satisfied, ready to start the context of ApplicationMaster, to the ResourceManager, and cycle monitoring Application state.
When the resource queue submission has the resources, ResourceManager will start on a NodeManager ApplicationMaster process, ApplicationMaster starts Driver separate background thread, after the Driver starts, ApplicationMaster Driver will be connected through a local RPC, and begin to apply ResourceManager Container Resources Executor running process (a Executor correspond to a Container), Container return when the ResourceManager resources, start Executor on the corresponding Container.
Driver main thread is initialized SparkContext objects, ready to run the required context, and then on the one hand to maintain the RPC connection ApplicationMaster by ApplicationMaster application resources, on the other hand according to user business logic start scheduling tasks, it will be sent to existing idle next task Executor on.
When the ResourceManager return Container resources to ApplicationMaster, ApplicationMaster you try to start Executor process on the corresponding Container, the Executor process up, register with the Driver, keep the heartbeat and Driver After successful registration, while waiting for Driver distribute tasks, when the task distribution after machining, the task status will be reported to the Driver.
Original: Big Box Road --Spark learning (5)
the Spark cluster resource scheduling