Spark1.0.0 can monitor Spark applications in the following ways:
- WebUI for Spark applications or cluster monitoring for Spark Standalone
- indicators, and then monitor them through a cluster monitoring system that supports indicator collection, such as ganglia
- Auxiliary monitoring tool
1：WebUI
After the Spark application is submitted, the driver and the Executor constantly exchange running information. You can obtain useful Spark application running information through the driver's port 4040 (default port), such as:
- Stage和Task
- RDD size and memory usage
- environment variable information
- executor running information
- ...
If multiple Spark applications are submitted in client mode on the same client, the driver's WebUI port will be bound to serial ports starting from 4040, such as 4040, 4041, 4042....
It should be noted that you can only view the information of the Spark application during the running period of the Spark application after using the WebUI. Once the Spark application is finished running, the information cannot be viewed, because the WebUI port is closed with the completion of the Spark application. If you want to view the running information of the Spark application afterwards, you need to configure the history server to persist the running information of the Spark application. For the history server, see Spark1.0.0 history server configuration (in writing, link later).
2: Indicators
Spark adopts a configurable indicator system based on
Coda Hale Metrics Library
, and can perform summary reports through various indicator collectors, such as JMX, CSV, GraphiteSink, Ganglia, etc. The configuration file of the indicator system is located in conf/metrics.properties (generated by copying conf/metrics.properties.template or self-built), if you want to use a custom configuration file, you also need to configure spark.metrics.conf on the property configuration .
Spark's indicator system is decomposed into corresponding instances for different Spark components, and each instance covers a set of indicators. The instances that Spark currently supports are:
- master
- worker
- applications
- driver
- executor
Spark's indicator system supports multiple collectors, and each instance can use multiple collectors or not. The collectors supported by Spark are defined in org.apache.spark.metrics.sink. The currently supported collectors are:
- ConsoleSink
- CSVSink.
- JmxSink
- MetricsServlet
- GraphiteSink
- GangliaSink Due to copyright issues, the deployment package does not contain this collector by default; if necessary, recompile the source code embedded in the LGPL licensed code. For details, see Monitoring Spark 1.0.0 with ganglia (in writing, link later).
3: Auxiliary monitoring tools
You can use some auxiliary monitoring tools to monitor the Spark application for changes in system performance before and after the Spark application runs and during the running process. These aids are:
- Cluster monitoring systems, such as ganglia, negios, zabbix, etc., these tools can monitor the disk, network, memory utilization and performance bottlenecks of the entire cluster;
- Operating system performance analysis tools, such as dstat, iostat, iotop, these tools can analyze the performance of a single machine in detail;
- JVM性能分析工具，如 jstack、jmap、jstat 、jconsole，这些工具可以对JVM进行详细的性能分析