SparkのデモPI計算では、3つのモード（ローカル、クライアント、クラスター）を使用して、Sparkの実行中のプロセスを分析します。
Sparkタスク実行プロセス（例としてクライアント）
プロセスの概要：このプロセスは単純なプロセスであり、多くの詳細は反映されていません。詳細なプロセスは、スパークオンヤーンの詳細な実行プロセスで確認できます。
1.启动Driver和WebUI
2.为ApplicationMaser准备Container以及设置运行环境并启动AM
3.上传配置信息以及Jar包
4.提交到Yarn上面进行执行
5.监控任务的运行状态
6.提交missing task和Result Stage
7.运行Task得到运行结果
8.停止WebUI、停止Executor、停止SparkContext、删除临时目录
注：Yarn-ClusterモードのPIの値は、コンソールに出力されません。
詳細なログ情報は次のとおりです。
1.Running Spark version 2.4.5 SparkContext----->开始准备启动
2.INFO Utils: Successfully started service 'sparkDriver' on port 36991.----->启动Driver
3.Start server connector----->开始准备连接
4.Started SparkUI----->启动SparkwebUI
5.Added JAR file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar----->上传Jar包到Spark
6.Connecting to ResourceManager----->连接到Resourcemanager
7.Setting up container launch context for our AM-----> 为ApplicationMaster准备container
8.Setting up the launch environment for our AM container----->为ApplicationMaster设置container的运行环境
9.Preparing resources for our AM container----->为ApplicationMaster 准备资源
10.Uploading resource file:/tmp/xx/__spark_conf__14378.zip -> hdfs://xxx/__spark_conf__.zip----->上传Spark 的配置文件
11.Submitting application application_1604816619741_0001 to ResourceManager ----->提交任务到ResourceManager
12.Application report for application_1604816619741_0001 (state: ACCEPTED) ----->监控任务的运行状态
13.Application report for application_1604816619741_0001 (state: RUNNING)
14.Application application_1604816619741_0001 has started running.----->资源分配合适开始运行Spark任务
15.NettyBlockTransferService: Server created on bigdata01:44993----->创建Netty连接
16.INFO BlockManagerMaster: Registered BlockManager ----->注册Blockmanager
17.Registered executor NettyRpcEndpointRef----->注册executor NettyRpcEndpointRef
18.INFO SparkContext: Starting job----->启动任务
19.INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions-----> partition信息
20.INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 ----->提交missing的task fro ResultStage
21.INFO YarnScheduler: Adding task set 0.0 with 2 tasks
22.INFO TaskSetManager: Starting task 0.0 in stage 0.0 ----->启动Task
23.INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:44924
24.INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) ----->TaskS完成
25.INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 5.051 s
26.INFO DAGScheduler: Job 0 finished:----->job完成
27.Pi is roughly 3.1423357116785584 ----->得到PI的计算结果
28.INFO SparkUI: Stopped Spark web UI at http://bigdata01:4040 ----->停止SparkWebUI
29.INFO YarnClientSchedulerBackend: Shutting down all executors----->停止所有的executor
30.INFO YarnClientSchedulerBackend: Stopped
ヤーン-クライアントモード
注文を送信
[hadoop@bigdata01 ~]$ spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode client \
> /home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar \
> 2
ログ情報
20/11/08 14:57:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/08 14:57:02 INFO SparkContext: Running Spark version 2.4.5
20/11/08 14:57:02 INFO SparkContext: Submitted application: Spark Pi
20/11/08 14:57:02 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 14:57:02 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 14:57:02 INFO SecurityManager: Changing view acls groups to:
20/11/08 14:57:02 INFO SecurityManager: Changing modify acls groups to:
20/11/08 14:57:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 14:57:03 INFO Utils: Successfully started service 'sparkDriver' on port 36991.
20/11/08 14:57:03 INFO SparkEnv: Registering MapOutputTracker
20/11/08 14:57:03 INFO SparkEnv: Registering BlockManagerMaster
20/11/08 14:57:03 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/11/08 14:57:03 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/11/08 14:57:03 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-7fafcfe2-eec5-4008-911a-d065cac2cdc8
20/11/08 14:57:03 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
20/11/08 14:57:03 INFO SparkEnv: Registering OutputCommitCoordinator
20/11/08 14:57:03 INFO log: Logging initialized @2901ms
20/11/08 14:57:03 INFO Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
20/11/08 14:57:03 INFO Server: Started @3043ms
20/11/08 14:57:03 INFO AbstractConnector: Started ServerConnector@bc57b40{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 14:57:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a62b2a4{
/jobs,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a021cb9{
/jobs/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@51768776{
/jobs/job,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b93aad{
/jobs/job/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4a9419d7{
/stages,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2f3c6ac4{
/stages/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2e8ab815{
/stages/stage,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@41394595{
/stages/stage/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3a0807b7{
/stages/pool,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21a5fd96{
/stages/pool/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5769e7ae{
/storage,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5c77053b{
/storage/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@26b894bd{
/storage/rdd,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@287f94b1{
/storage/rdd/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@30b34287{
/environment,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5489c777{
/environment/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3676ac27{
/executors,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@62f87c44{
/executors/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@48f5bde6{
/executors/threadDump,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@525d79f0{
/executors/threadDump/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5149f008{
/static,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@324c64cd{
/,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@13579834{
/api,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2555fff0{
/jobs/job/kill,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@70d2e40b{
/stages/stage/kill,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://bigdata01:4040
20/11/08 14:57:03 INFO SparkContext: Added JAR file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar at spark://bigdata01:36991/jars/spark-examples_2.12-2.4.5.jar with timestamp 1604818623822
20/11/08 14:57:04 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/08 14:57:04 INFO Client: Requesting a new application from cluster with 1 NodeManagers
20/11/08 14:57:04 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
20/11/08 14:57:04 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
20/11/08 14:57:04 INFO Client: Setting up container launch context for our AM
20/11/08 14:57:04 INFO Client: Setting up the launch environment for our AM container
20/11/08 14:57:04 INFO Client: Preparing resources for our AM container
20/11/08 14:57:05 INFO Client: Source and destination file systems are the same. Not copying hdfs://bigdata01:9000/spark/jars/sparkjars.zip
20/11/08 14:57:05 INFO Client: Uploading resource file:/tmp/spark-e49788a4-fa66-4ba5-926f-7daf3977b84b/__spark_conf__1437485475950579958.zip -> hdfs://bigdata01:9000/user/hadoop/.sparkStaging/application_1604816619741_0001/__spark_conf__.zip
20/11/08 14:57:05 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 14:57:05 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 14:57:05 INFO SecurityManager: Changing view acls groups to:
20/11/08 14:57:05 INFO SecurityManager: Changing modify acls groups to:
20/11/08 14:57:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 14:57:07 INFO Client: Submitting application application_1604816619741_0001 to ResourceManager
20/11/08 14:57:07 INFO YarnClientImpl: Submitted application application_1604816619741_0001
20/11/08 14:57:07 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1604816619741_0001 and attemptId None
20/11/08 14:57:09 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:09 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1604818627705
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0001/
user: hadoop
20/11/08 14:57:10 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:11 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:12 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:13 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:14 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:15 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:16 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:17 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:18 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:19 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:19 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> bigdata01, PROXY_URI_BASES -> http://bigdata01:18088/proxy/application_1604816619741_0001), /proxy/application_1604816619741_0001
20/11/08 14:57:19 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
20/11/08 14:57:20 INFO Client: Application report for application_1604816619741_0001 (state: RUNNING)
20/11/08 14:57:20 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.52.50
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1604818627705
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0001/
user: hadoop
20/11/08 14:57:20 INFO YarnClientSchedulerBackend: Application application_1604816619741_0001 has started running.
20/11/08 14:57:20 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44993.
20/11/08 14:57:20 INFO NettyBlockTransferService: Server created on bigdata01:44993
20/11/08 14:57:20 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/11/08 14:57:20 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:44993 with 413.9 MB RAM, BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
20/11/08 14:57:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7aaad0{
/metrics/json,null,AVAILABLE,@Spark}
20/11/08 14:57:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.52.50:58832) with ID 1
20/11/08 14:57:27 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:44924 with 413.9 MB RAM, BlockManagerId(1, bigdata01, 44924, None)
20/11/08 14:57:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.52.50:58836) with ID 2
20/11/08 14:57:29 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
20/11/08 14:57:30 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:42724 with 413.9 MB RAM, BlockManagerId(2, bigdata01, 42724, None)
20/11/08 14:57:30 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
20/11/08 14:57:30 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
20/11/08 14:57:30 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
20/11/08 14:57:30 INFO DAGScheduler: Parents of final stage: List()
20/11/08 14:57:30 INFO DAGScheduler: Missing parents: List()
20/11/08 14:57:30 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
20/11/08 14:57:30 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.2 KB, free 413.9 MB)
20/11/08 14:57:31 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1826.0 B, free 413.9 MB)
20/11/08 14:57:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:44993 (size: 1826.0 B, free: 413.9 MB)
20/11/08 14:57:31 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163
20/11/08 14:57:31 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
20/11/08 14:57:31 INFO YarnScheduler: Adding task set 0.0 with 2 tasks
20/11/08 14:57:31 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, bigdata01, executor 1, partition 0, PROCESS_LOCAL, 7217 bytes)
20/11/08 14:57:31 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, bigdata01, executor 2, partition 1, PROCESS_LOCAL, 7219 bytes)
20/11/08 14:57:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:44924 (size: 1826.0 B, free: 413.9 MB)
20/11/08 14:57:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:42724 (size: 1826.0 B, free: 413.9 MB)
20/11/08 14:57:35 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4401 ms on bigdata01 (executor 1) (1/2)
20/11/08 14:57:35 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 4481 ms on bigdata01 (executor 2) (2/2)
20/11/08 14:57:35 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 5.051 s
20/11/08 14:57:35 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/11/08 14:57:35 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 5.165122 s
Pi is roughly 3.1423357116785584
20/11/08 14:57:35 INFO AbstractConnector: Stopped Spark@bc57b40{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 14:57:35 INFO SparkUI: Stopped Spark web UI at http://bigdata01:4040
20/11/08 14:57:35 INFO YarnClientSchedulerBackend: Interrupting monitor thread
20/11/08 14:57:35 INFO YarnClientSchedulerBackend: Shutting down all executors
20/11/08 14:57:35 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
20/11/08 14:57:35 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
20/11/08 14:57:35 INFO YarnClientSchedulerBackend: Stopped
20/11/08 14:57:35 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/11/08 14:57:35 INFO MemoryStore: MemoryStore cleared
20/11/08 14:57:35 INFO BlockManager: BlockManager stopped
20/11/08 14:57:35 INFO BlockManagerMaster: BlockManagerMaster stopped
20/11/08 14:57:35 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/11/08 14:57:36 INFO SparkContext: Successfully stopped SparkContext
20/11/08 14:57:36 INFO ShutdownHookManager: Shutdown hook called
20/11/08 14:57:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-dd5062ee-711c-4455-b53e-031d6b75da58
20/11/08 14:57:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-e49788a4-fa66-4ba5-926f-7daf3977b84b
糸クラスターモード
コマンドを送信します。
[hadoop@bigdata01 native]$ spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode cluster \
> /home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar \
> 2
ログ情報：
20/11/08 15:19:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/08 15:19:42 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/08 15:19:42 INFO Client: Requesting a new application from cluster with 1 NodeManagers
20/11/08 15:19:42 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
20/11/08 15:19:42 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
20/11/08 15:19:42 INFO Client: Setting up container launch context for our AM
20/11/08 15:19:42 INFO Client: Setting up the launch environment for our AM container
20/11/08 15:19:42 INFO Client: Preparing resources for our AM container
20/11/08 15:19:43 INFO Client: Source and destination file systems are the same. Not copying hdfs://bigdata01:9000/spark/jars/sparkjars.zip
20/11/08 15:19:43 INFO Client: Uploading resource file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar -> hdfs://bigdata01:9000/user/hadoop/.sparkStaging/application_1604816619741_0002/spark-examples_2.12-2.4.5.jar
20/11/08 15:19:43 INFO Client: Uploading resource file:/tmp/spark-a93a1160-b473-498b-b3df-4912a955ff54/__spark_conf__1880777542668876131.zip -> hdfs://bigdata01:9000/user/hadoop/.sparkStaging/application_1604816619741_0002/__spark_conf__.zip
20/11/08 15:19:43 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 15:19:43 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 15:19:43 INFO SecurityManager: Changing view acls groups to:
20/11/08 15:19:43 INFO SecurityManager: Changing modify acls groups to:
20/11/08 15:19:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 15:19:45 INFO Client: Submitting application application_1604816619741_0002 to ResourceManager
20/11/08 15:19:45 INFO YarnClientImpl: Submitted application application_1604816619741_0002
20/11/08 15:19:46 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:46 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1604819985284
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0002/
user: hadoop
20/11/08 15:19:47 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:48 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:49 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:50 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:51 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:52 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:52 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: bigdata01
ApplicationMaster RPC port: 37147
queue: root.hadoop
start time: 1604819985284
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0002/
user: hadoop
20/11/08 15:19:53 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:54 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:55 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:56 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:57 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:58 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:59 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:00 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:01 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:02 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:03 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:04 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:05 INFO Client: Application report for application_1604816619741_0002 (state: FINISHED)
20/11/08 15:20:05 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: bigdata01
ApplicationMaster RPC port: 37147
queue: root.hadoop
start time: 1604819985284
final status: SUCCEEDED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0002/
user: hadoop
20/11/08 15:20:05 INFO ShutdownHookManager: Shutdown hook called
20/11/08 15:20:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-658608fd-a353-4804-9a37-ce267d732efa
20/11/08 15:20:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-a93a1160-b473-498b-b3df-4912a955ff54
ローカルモード
コマンドを送信します。
[hadoop@bigdata01 native]$ spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master local[2] \
> /home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar \
> 2
実行ログ：
20/11/08 15:23:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/08 15:23:25 INFO SparkContext: Running Spark version 2.4.5
20/11/08 15:23:25 INFO SparkContext: Submitted application: Spark Pi
20/11/08 15:23:25 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 15:23:25 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 15:23:25 INFO SecurityManager: Changing view acls groups to:
20/11/08 15:23:25 INFO SecurityManager: Changing modify acls groups to:
20/11/08 15:23:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 15:23:26 INFO Utils: Successfully started service 'sparkDriver' on port 42466.
20/11/08 15:23:26 INFO SparkEnv: Registering MapOutputTracker
20/11/08 15:23:26 INFO SparkEnv: Registering BlockManagerMaster
20/11/08 15:23:26 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/11/08 15:23:26 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/11/08 15:23:26 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-e01c3a97-0f5c-4019-8d78-47d5a600b923
20/11/08 15:23:26 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
20/11/08 15:23:26 INFO SparkEnv: Registering OutputCommitCoordinator
20/11/08 15:23:26 INFO log: Logging initialized @2476ms
20/11/08 15:23:26 INFO Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
20/11/08 15:23:26 INFO Server: Started @2635ms
20/11/08 15:23:26 INFO AbstractConnector: Started ServerConnector@64f857e7{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 15:23:26 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@733c423e{
/jobs,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@27eb3298{
/jobs/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@200a26bc{
/jobs/job,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1b5bc39d{
/jobs/job/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@655a5d9c{
/stages,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1494b84d{
/stages/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@34abdee4{
/stages/stage,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@77cf3f8b{
/stages/stage/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1df98368{
/stages/pool,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21ca139c{
/stages/pool/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@226f885f{
/storage,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2cd2c8fe{
/storage/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d61eccf{
/storage/rdd,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@cc6460c{
/storage/rdd/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@52350abb{
/environment,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@681aad3b{
/environment/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1a6f2363{
/executors,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2427e004{
/executors/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5ebd56e9{
/executors/threadDump,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@63f34b70{
/executors/threadDump/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@641856{
/static,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@23c388c2{
/,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@486be205{
/api,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5ca17ab0{
/jobs/job/kill,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a62b2a4{
/stages/stage/kill,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://bigdata01:4040
20/11/08 15:23:26 INFO SparkContext: Added JAR file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar at spark://bigdata01:42466/jars/spark-examples_2.12-2.4.5.jar with timestamp 1604820206605
20/11/08 15:23:26 INFO Executor: Starting executor ID driver on host localhost
20/11/08 15:23:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39948.
20/11/08 15:23:26 INFO NettyBlockTransferService: Server created on bigdata01:39948
20/11/08 15:23:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/11/08 15:23:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:39948 with 413.9 MB RAM, BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@779de014{
/metrics/json,null,AVAILABLE,@Spark}
20/11/08 15:23:27 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
20/11/08 15:23:27 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
20/11/08 15:23:27 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
20/11/08 15:23:27 INFO DAGScheduler: Parents of final stage: List()
20/11/08 15:23:27 INFO DAGScheduler: Missing parents: List()
20/11/08 15:23:27 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
20/11/08 15:23:27 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.2 KB, free 413.9 MB)
20/11/08 15:23:28 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1826.0 B, free 413.9 MB)
20/11/08 15:23:28 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:39948 (size: 1826.0 B, free: 413.9 MB)
20/11/08 15:23:28 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163
20/11/08 15:23:28 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
20/11/08 15:23:28 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
20/11/08 15:23:28 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7206 bytes)
20/11/08 15:23:28 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7208 bytes)
20/11/08 15:23:28 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
20/11/08 15:23:28 INFO Executor: Fetching spark://bigdata01:42466/jars/spark-examples_2.12-2.4.5.jar with timestamp 1604820206605
20/11/08 15:23:28 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
20/11/08 15:23:28 INFO TransportClientFactory: Successfully created connection to bigdata01/192.168.52.50:42466 after 65 ms (0 ms spent in bootstraps)
20/11/08 15:23:28 INFO Utils: Fetching spark://bigdata01:42466/jars/spark-examples_2.12-2.4.5.jar to /tmp/spark-d2b27239-b4e7-4af8-b9b3-07b9bfd60cb3/userFiles-195a5986-155d-4e68-a31b-15fa7ae4e5ec/fetchFileTemp6323009668907142489.tmp
20/11/08 15:23:28 INFO Executor: Adding file:/tmp/spark-d2b27239-b4e7-4af8-b9b3-07b9bfd60cb3/userFiles-195a5986-155d-4e68-a31b-15fa7ae4e5ec/spark-examples_2.12-2.4.5.jar to class loader
20/11/08 15:23:29 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 676 bytes result sent to driver
20/11/08 15:23:29 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 676 bytes result sent to driver
20/11/08 15:23:29 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 915 ms on localhost (executor driver) (1/2)
20/11/08 15:23:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 971 ms on localhost (executor driver) (2/2)
20/11/08 15:23:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/11/08 15:23:29 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 1.645 s
20/11/08 15:23:29 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.756773 s
Pi is roughly 3.144195720978605
20/11/08 15:23:29 INFO AbstractConnector: Stopped Spark@64f857e7{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 15:23:29 INFO SparkUI: Stopped Spark web UI at http://bigdata01:4040
20/11/08 15:23:29 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/11/08 15:23:29 INFO MemoryStore: MemoryStore cleared
20/11/08 15:23:29 INFO BlockManager: BlockManager stopped
20/11/08 15:23:29 INFO BlockManagerMaster: BlockManagerMaster stopped
20/11/08 15:23:29 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/11/08 15:23:29 INFO SparkContext: Successfully stopped SparkContext
20/11/08 15:23:29 INFO ShutdownHookManager: Shutdown hook called
20/11/08 15:23:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-c675b673-6b0d-44b1-8748-1114de6f96e3
20/11/08 15:23:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-d2b27239-b4e7-4af8-b9b3-07b9bfd60cb3