Importing
Job running from a process point of view DAGScheduler Driver is run at the end of its work processes as shown below:
Figure vocabulary concepts involved:
RDD 1. --Resillient a Dataset Flexible Distributed distributed data.
Operation 2. - acting on the various operations and RDD transformation into action.
The Job 3. - jobs JOB comprising a plurality of various operation RDD and acting on the respective RDD.
The Stage 4. - a job into multiple stages.
The Partition 5. The - data partition, the data may be divided into a plurality of RDD in different zones.
The DAG 6. The --Directed Acycle Graph, directed acyclic graph, the reaction dependencies between RDD.
Narrow dependency 7. The - narrow dependent, dependent on the parent sub RDD RDD immobilized data partition.
Wide the Dependency 8. The - width dependence, for all the sub-RDD RDD parent data partition is dependent on both.
The Caching Managenment 9. The - cache management, the intermediate calculation result of RDD buffer management to speed up the overall processing speed.
Driver at the end, run a Job, involving DAGSheduler process is as follows:
1) call applicaiton_jar.jar (application entry function), run the process applications that depend SparkContext, and the need to initialize SparkContext sc, you can create the RDD by sc (sc to call because RDD is created);
2) SparkContext initialization process initializes DAGScheduler, and calls SparkContext.createTaskScheduler (this, master, deployMode) to initialize TaskScheduler, ShedulerBackend;
3) applications are actually transform RDD to perform a function or action, when RDD # action trigger function, in fact, such action inside the function calls sc.submitJob (...) method, in SparkContext # submitJob (.. internal) method creates ResultStage based on action, and find all ShuffleMapStage its dependence. Between the stage in the order of execution, before execution pending the successful completion of a stage to perform the next stage, after all the stage is successful, the job execution be considered complete.
4) stage can actually be seen as TaskSet, it actually represents is an independent Task collection, DAGScheduler will call for TaskSet TaskScheduler to perform job scheduling;
5) TaskScheduler scheduling process is passed to the task serialized Executor by RPC, will be used to run the task on TaskRunner Executor;
6) TaskScheduler If the task fails, TaskScheduler will retry processing; at the same stage after the failure, DAGScheduler will trigger stage retry processing. Note: If this stage fails, the recalculation of the current stage, rather than starting from a stage, this is the cause of division stage of the DAG.
DAGScheduler source code analysis
DAGScheduler features:
1) the top layers of the scheduling, to achieve a stage-oriented (Stage oriented) scheduling. DAGScheduler calculate a DAG described stages for each job, tracking what RDD and output stage to achieve and find the minimum scheduled to run the job. Then, it is TaskSets stages package submitted to the operation of their underlying TaskScheduler implement (TaskScheduler implementation class is the only TaskSchedulerImpl) on the cluster. Completely independent task set contains a set of tasks that can be run immediately according to the data (for example, the first few stages of mapping the output file) cluster already exists, but if this data is not available, it may fail.
2) Spark stages RDD is disconnecting created in FIG Shuffle boundary. Having a "narrow (Narrow)" dependency RDD operation (e.g., map () and filter ()) is connected to a line in the set of tasks in each stage, but having a shuffle operation requires dependency (a plurality of phases writing a set of mapping the output file, read the files in another stage after the barrier). Finally, each stage having only shuffle dependence on other stages, and can be calculated in which a plurality of operations. The actual pipeline of these operations occur in a variety of RDD rdd.compute () function.
3) In addition to dividing stages of DAG, DAGScheduler also determines the preferred location of each task runs according to the current state of the cache, and passes them to a position underlying the TaskScheduler (task scheduler). It also deal with failure due to the shuffle caused by the loss of output file, in which case, you may need to re-submit the old stages. Inside TaskScheduler will handle stage is not caused by the failure of shuffle files are missing, it will cancel the entire stage before each task a few retries.
4) recovery from a failure, the same stage may need to run many times, this is known as "attempts". If the report of a task due TaskScheduler the previous stage of mapping the output file is missing and fails, DAGScheduler will resubmit the missing phase. This is detected by CompletionEvent or ExecutorLost of events with FetchFailed. DAGScheduler will wait a short time to see if the other node fails or task, and then re-submit TaskSets (set of tasks) to calculate the loss of any missing phase tasks. As part of this process, we may also have to clean up old stage objects as before (Completed) stage to create a stage objects. Since the stage of the old "attempts" in the task may still be running, so care must be taken to map any event received in the correct stage object.
When you view this code, there are a few key concepts:
-Jobs: (represented by [ActiveJob]) is a top-level work items submitted to the scheduler. For example, when a user operation such as call count () and the like, jobs submitted by sc.submitJob () method. Each job may need to perform a plurality of stages of intermediate data to build.
-Stages: the Stage is a set of tasks (taskset), an intermediate calculation result of the job, wherein each task compute the same functions in the same partition RDD.
Stage separation boundary shuffle, which will introduce a barrier (we must wait for the completion of the previous phase to obtain output).
There are two types of Stage (Stage): [ResultStage] (for the final stage of the operation performed); [ShuffleMapStage] (the output file into the mapping shuffle).
If these jobs reuse the same RDD, the Stage (stage) is typically shared among multiple jobs.
-Tasks: it is a single unit of work, transmitting each unit of work to a machine.
Tracking -cache: DagScheduler will find out what RDD cache to avoid re-calculate them, the same will remember what shuffle map stage has been generated output file, to avoid duplication shuffle mapping end.
Locations -Preferred: DAGScheduler further preferred according to the position of the underlying RDD or cache location or order processing data, calculates the position of each task in the running phase.
-Cleanup: When a running job depending on their completion, all the data structures will be cleared, in order to prevent a memory leak occurs in long-running applications.
DAGScheduler life cycle
It will be combined with the following code DAGScheduler introduced throughout the life cycle, DAGScheduler life cycle:
1) initialization DAGScheduler
2) The divided Stages RDD DAG
3) Stage scheduling, fault tolerance Stage
4) Examples of destruction
1) initialization of DAGScheduler
2) The RDD DAG division of Stages DAGScheduler
3) DAGScheduler Stage of the scheduling, fault tolerance
4) dispose of example DAGScheduler
reference:
1) " DAG principle of the operating mechanism of the Spark "
2) " the Spark Scheduler module Detailed -DAGScheduler achieve "
3) " the Spark source of commuting DAGScheduler "