工作中经常会出现 Spark SQL 执行很慢或者失败的情况，如果要排查问题，就必须要学会看 Spark Web UI。可以参考官网来学习：https://spark.apache.org/docs/3.2.1/web-ui.html#content。关于 Spark Web UI，上面有很多个 tab 页，后面逐一学习。
昨天已经学习了 Jobs，今天学习 Stages。
Stages Tab
The Stages tab displays a summary page that shows the current state of all stages of all jobs in the Spark application.
At the beginning of the page is the summary with the count of all stages by status (active, pending, completed, skipped, and failed)
Stages 选项卡显示一个摘要页面，该页面显示 Spark 应用程序中所有作业的所有阶段的当前状态。 页面开头是按状态（活动、待处理、已完成、已跳过和失败）统计所有阶段的摘要。
In Fair scheduling mode there is a table that displays pools properties
在公平调度模式下，有一个显示池属性的表。
After that are the details of stages per status (active, pending, completed, skipped, failed). In active stages, it’s possible to kill the stage with the kill link. Only in failed stages, failure reason is shown. Task detail can be accessed by clicking on the description.
之后是每个状态的阶段的详细信息（活动、待处理、已完成、已跳过、失败）。 在活动阶段，可以使用终止链接终止阶段。 只有在失败阶段，才会显示失败原因。 单击描述可以访问任务详细信息。
Stage detail
The stage detail page begins with information like total time across all tasks, Locality level summary, Shuffle Read Size / Records and Associated Job IDs.
阶段详细信息页面以所有任务的总时间、位置级别摘要、随机读取大小/记录和相关作业 ID 等信息开头。
There is also a visual representation of the directed acyclic graph (DAG) of this stage, where vertices represent the RDDs or DataFrames and the edges represent an operation to be applied. Nodes are grouped by operation scope in the DAG visualization and labelled with the operation scope name (BatchScan, WholeStageCodegen, Exchange, etc). Notably, Whole Stage Code Generation operations are also annotated with the code generation id. For stages belonging to Spark DataFrame or SQL execution, this allows to cross-reference Stage execution details to the relevant details in the Web-UI SQL Tab page where SQL plan graphs and execution plans are reported.
这个阶段还有一个有向无环图（DAG）的可视化表示，其中顶点表示 RDD 或 DataFrame，边表示要应用的操作。 节点在 DAG 可视化中按操作范围分组，并标有操作范围名称（BatchScan、WholeStageCodegen、Exchange 等）。 值得注意的是，整个阶段代码生成操作也使用代码生成 ID 进行注释。 对于属于 Spark DataFrame 或 SQL 执行的阶段，这允许将阶段执行详细信息与报告 SQL 计划图和执行计划的 Web-UI SQL 选项卡页面中的相关详细信息进行交叉引用。
Summary metrics for all task are represented in a table and in a timeline.
- Tasks deserialization time
- Duration of tasks.
- GC time is the total JVM garbage collection time.
- Result serialization time is the time spent serializing the task result on an executor before sending it back to the driver.
- Getting result time is the time that the driver spends fetching task results from workers.
- Scheduler delay is the time the task waits to be scheduled for execution.
- Peak execution memory is the maximum memory used by the internal data structures created during shuffles, aggregations and joins.
- Shuffle Read Size / Records. Total shuffle bytes read, includes both data read locally and data read from remote executors.
- Shuffle Read Blocked Time is the time that tasks spent blocked waiting for shuffle data to be read from remote machines.
- Shuffle Remote Reads is the total shuffle bytes read from remote executors.
- Shuffle Write Time is the time that tasks spent writing shuffle data.
- Shuffle spill (memory) is the size of the deserialized form of the shuffled data in memory.
- Shuffle spill (disk) is the size of the serialized form of the data on disk.
所有任务的汇总指标都以表格和时间线的形式表示。
- 任务反序列化时间。
- 任务的持续时间。
- GC 时间是 JVM 垃圾收集的总时间。
- 结果序列化时间是在将任务结果发送回 driver 之前在 executor 上序列化任务结果所花费的时间。
- 获取结果时间是 driver 从 worker 那里获取任务结果所花费的时间。
- 调度程序延迟是任务等待被调度执行的时间。
- 峰值执行内存是在混洗、聚合和连接期间创建的内部数据结构使用的最大内存。
- Shuffle Read Size / Records。读取的总 shuffle 字节数，包括本地读取的数据和从远程执行程序读取的数据。
- Shuffle Read Blocked Time 是任务阻塞等待从远程机器读取 shuffle 数据所花费的时间。
- Shuffle Remote Reads 是从远程执行器读取的总 shuffle 字节数。
- Shuffle Write Time 是任务写入 shuffle 数据所花费的时间。
- **Shuffle 溢出（内存）**是内存中随机数据的反序列化形式的大小。
- **Shuffle 溢出（磁盘）**是磁盘上数据的序列化形式的大小。
Aggregated metrics by executor show the same information aggregated by executor.
executor 的聚合指标显示了 executor 聚合的相同信息。
Accumulators are a type of shared variables. It provides a mutable variable that can be updated inside of a variety of transformations. It is possible to create accumulators with and without name, but only named accumulators are displayed.
累加器是一种共享变量。它提供了一个可变变量，可以在各种转换中进行更新。可以创建带名称和不带名称的累加器，但只显示命名的累加器。
Tasks details basically includes the same information as in the summary section but detailed by task. It also includes links to review the logs and the task attempt number if it fails for any reason. If there are named accumulators, here it is possible to see the accumulator value at the end of each task.
任务详细信息基本上包括与摘要部分相同的信息，但按任务详细说明。 它还包括用于查看日志和任务尝试次数的链接（如果它因任何原因失败）。如果有命名的累加器，这里可以看到每个任务结束时的累加器值。
欢迎点击此处关注公众号。