1) The memory size of the Worker node must meet the executor-memory size set by the application;
2) The number of cores of the Worker node can meet the number of executor-core set by the application. ;
Worker nodes that meet the conditions will start an Executor process independently for each application. The Executor process is solely responsible for the Task tasks of the application. After the Executor processes of multiple Worker nodes in the cluster receive the Task, they will Start to complete the Task tasks in parallel independently of each other. Each Worker node can start multiple Executor processes to meet the requests of multiple applications. Each Executor process runs independently in a JVM process of the Worker node, and each Task runs in a thread in the Executor.
Once the Executor process of an application is created, it will continue to run, and its resources can be reused by multiple batches of tasks until the Spark program is completed and will be released and exited. This avoids the time overhead caused by repeatedly applying for resources.
The Executor will report the execution status of the task in reverse step by step, as shown in the following figure:
The Executor identifies the task status in four cases:
1. After the Executor receives the task, it has not yet executed the task on the Task, and the status of the Task is RUNNING at this time. ;
2. If an error occurs during the running of the Task, the status of the Task is FAILED, and the Executor will report the cause of the error;
3. If the task is killed by humans or other reasons in the middle, the status of the task is KILLED, and the Executor will report the reason for the kill to the TaskScheduler;
4. If the task calculation is completed, the status of the task is FINISHED, and the Executor will Save the results, feed the results and status information back to the TaskScheduler, and wait for the TaskScheduler to assign the next Task to it;
after the TaskScheduler receives the results and status information sent by the Executor, it will find the TaskSetManager corresponding to the Task, and the TaskScheduler will The completion is notified to the corresponding TaskSetManager. If the tasks maintained by the TaskSetManager are completed, the TaskSetManager will automatically end and close.
If the Stage corresponding to the TaskSetManager is FinalStage, the running result itself will be returned to DAGScheduler. If it corresponds to an intermediate Stage separated by a Shuffle operation, the operation result returned to DAGScheduler is related to the location information of the storage module. These storage location information will schedule the input data of the next Stage, and the next Stage will read the information. Input data for calculation, and repeat task distribution, task execution, and task execution information reporting until all tasks are completed, and the application program is completed.