A través del cálculo de Demo PI de Spark, se utilizan tres modos (local, cliente, clúster) para analizar el proceso de ejecución de Spark
Proceso de ejecución de tareas de Spark (cliente como ejemplo)
Descripción general del proceso: este proceso es solo un proceso simple, muchos detalles no se reflejan, el proceso detallado se puede ver en el proceso de ejecución detallado de Spark on Yarn
1.启动Driver和WebUI
2.为ApplicationMaser准备Container以及设置运行环境并启动AM
3.上传配置信息以及Jar包
4.提交到Yarn上面进行执行
5.监控任务的运行状态
6.提交missing task和Result Stage
7.运行Task得到运行结果
8.停止WebUI、停止Executor、停止SparkContext、删除临时目录
Nota: El valor de PI en el modo Yarn-Cluster no está impreso en la consola
La información de registro detallada es la siguiente:
1.Running Spark version 2.4.5 SparkContext----->开始准备启动
2.INFO Utils: Successfully started service 'sparkDriver' on port 36991.----->启动Driver
3.Start server connector----->开始准备连接
4.Started SparkUI----->启动SparkwebUI
5.Added JAR file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar----->上传Jar包到Spark
6.Connecting to ResourceManager----->连接到Resourcemanager
7.Setting up container launch context for our AM-----> 为ApplicationMaster准备container
8.Setting up the launch environment for our AM container----->为ApplicationMaster设置container的运行环境
9.Preparing resources for our AM container----->为ApplicationMaster 准备资源
10.Uploading resource file:/tmp/xx/__spark_conf__14378.zip -> hdfs://xxx/__spark_conf__.zip----->上传Spark 的配置文件
11.Submitting application application_1604816619741_0001 to ResourceManager ----->提交任务到ResourceManager
12.Application report for application_1604816619741_0001 (state: ACCEPTED) ----->监控任务的运行状态
13.Application report for application_1604816619741_0001 (state: RUNNING)
14.Application application_1604816619741_0001 has started running.----->资源分配合适开始运行Spark任务
15.NettyBlockTransferService: Server created on bigdata01:44993----->创建Netty连接
16.INFO BlockManagerMaster: Registered BlockManager ----->注册Blockmanager
17.Registered executor NettyRpcEndpointRef----->注册executor NettyRpcEndpointRef
18.INFO SparkContext: Starting job----->启动任务
19.INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions-----> partition信息
20.INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 ----->提交missing的task fro ResultStage
21.INFO YarnScheduler: Adding task set 0.0 with 2 tasks
22.INFO TaskSetManager: Starting task 0.0 in stage 0.0 ----->启动Task
23.INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:44924
24.INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) ----->TaskS完成
25.INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 5.051 s
26.INFO DAGScheduler: Job 0 finished:----->job完成
27.Pi is roughly 3.1423357116785584 ----->得到PI的计算结果
28.INFO SparkUI: Stopped Spark web UI at http://bigdata01:4040 ----->停止SparkWebUI
29.INFO YarnClientSchedulerBackend: Shutting down all executors----->停止所有的executor
30.INFO YarnClientSchedulerBackend: Stopped
Modo Yarn-Client
Orden de envio
[hadoop@bigdata01 ~]$ spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode client \
> /home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar \
> 2
Información de registro
20/11/08 14:57:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/08 14:57:02 INFO SparkContext: Running Spark version 2.4.5
20/11/08 14:57:02 INFO SparkContext: Submitted application: Spark Pi
20/11/08 14:57:02 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 14:57:02 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 14:57:02 INFO SecurityManager: Changing view acls groups to:
20/11/08 14:57:02 INFO SecurityManager: Changing modify acls groups to:
20/11/08 14:57:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 14:57:03 INFO Utils: Successfully started service 'sparkDriver' on port 36991.
20/11/08 14:57:03 INFO SparkEnv: Registering MapOutputTracker
20/11/08 14:57:03 INFO SparkEnv: Registering BlockManagerMaster
20/11/08 14:57:03 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/11/08 14:57:03 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/11/08 14:57:03 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-7fafcfe2-eec5-4008-911a-d065cac2cdc8
20/11/08 14:57:03 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
20/11/08 14:57:03 INFO SparkEnv: Registering OutputCommitCoordinator
20/11/08 14:57:03 INFO log: Logging initialized @2901ms
20/11/08 14:57:03 INFO Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
20/11/08 14:57:03 INFO Server: Started @3043ms
20/11/08 14:57:03 INFO AbstractConnector: Started ServerConnector@bc57b40{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 14:57:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a62b2a4{
/jobs,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a021cb9{
/jobs/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@51768776{
/jobs/job,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b93aad{
/jobs/job/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4a9419d7{
/stages,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2f3c6ac4{
/stages/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2e8ab815{
/stages/stage,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@41394595{
/stages/stage/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3a0807b7{
/stages/pool,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21a5fd96{
/stages/pool/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5769e7ae{
/storage,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5c77053b{
/storage/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@26b894bd{
/storage/rdd,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@287f94b1{
/storage/rdd/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@30b34287{
/environment,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5489c777{
/environment/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3676ac27{
/executors,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@62f87c44{
/executors/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@48f5bde6{
/executors/threadDump,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@525d79f0{
/executors/threadDump/json,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5149f008{
/static,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@324c64cd{
/,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@13579834{
/api,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2555fff0{
/jobs/job/kill,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@70d2e40b{
/stages/stage/kill,null,AVAILABLE,@Spark}
20/11/08 14:57:03 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://bigdata01:4040
20/11/08 14:57:03 INFO SparkContext: Added JAR file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar at spark://bigdata01:36991/jars/spark-examples_2.12-2.4.5.jar with timestamp 1604818623822
20/11/08 14:57:04 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/08 14:57:04 INFO Client: Requesting a new application from cluster with 1 NodeManagers
20/11/08 14:57:04 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
20/11/08 14:57:04 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
20/11/08 14:57:04 INFO Client: Setting up container launch context for our AM
20/11/08 14:57:04 INFO Client: Setting up the launch environment for our AM container
20/11/08 14:57:04 INFO Client: Preparing resources for our AM container
20/11/08 14:57:05 INFO Client: Source and destination file systems are the same. Not copying hdfs://bigdata01:9000/spark/jars/sparkjars.zip
20/11/08 14:57:05 INFO Client: Uploading resource file:/tmp/spark-e49788a4-fa66-4ba5-926f-7daf3977b84b/__spark_conf__1437485475950579958.zip -> hdfs://bigdata01:9000/user/hadoop/.sparkStaging/application_1604816619741_0001/__spark_conf__.zip
20/11/08 14:57:05 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 14:57:05 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 14:57:05 INFO SecurityManager: Changing view acls groups to:
20/11/08 14:57:05 INFO SecurityManager: Changing modify acls groups to:
20/11/08 14:57:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 14:57:07 INFO Client: Submitting application application_1604816619741_0001 to ResourceManager
20/11/08 14:57:07 INFO YarnClientImpl: Submitted application application_1604816619741_0001
20/11/08 14:57:07 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1604816619741_0001 and attemptId None
20/11/08 14:57:09 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:09 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1604818627705
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0001/
user: hadoop
20/11/08 14:57:10 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:11 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:12 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:13 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:14 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:15 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:16 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:17 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:18 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:19 INFO Client: Application report for application_1604816619741_0001 (state: ACCEPTED)
20/11/08 14:57:19 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> bigdata01, PROXY_URI_BASES -> http://bigdata01:18088/proxy/application_1604816619741_0001), /proxy/application_1604816619741_0001
20/11/08 14:57:19 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
20/11/08 14:57:20 INFO Client: Application report for application_1604816619741_0001 (state: RUNNING)
20/11/08 14:57:20 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.52.50
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1604818627705
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0001/
user: hadoop
20/11/08 14:57:20 INFO YarnClientSchedulerBackend: Application application_1604816619741_0001 has started running.
20/11/08 14:57:20 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44993.
20/11/08 14:57:20 INFO NettyBlockTransferService: Server created on bigdata01:44993
20/11/08 14:57:20 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/11/08 14:57:20 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:44993 with 413.9 MB RAM, BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, bigdata01, 44993, None)
20/11/08 14:57:20 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
20/11/08 14:57:20 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7aaad0{
/metrics/json,null,AVAILABLE,@Spark}
20/11/08 14:57:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.52.50:58832) with ID 1
20/11/08 14:57:27 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:44924 with 413.9 MB RAM, BlockManagerId(1, bigdata01, 44924, None)
20/11/08 14:57:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.52.50:58836) with ID 2
20/11/08 14:57:29 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
20/11/08 14:57:30 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:42724 with 413.9 MB RAM, BlockManagerId(2, bigdata01, 42724, None)
20/11/08 14:57:30 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
20/11/08 14:57:30 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
20/11/08 14:57:30 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
20/11/08 14:57:30 INFO DAGScheduler: Parents of final stage: List()
20/11/08 14:57:30 INFO DAGScheduler: Missing parents: List()
20/11/08 14:57:30 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
20/11/08 14:57:30 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.2 KB, free 413.9 MB)
20/11/08 14:57:31 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1826.0 B, free 413.9 MB)
20/11/08 14:57:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:44993 (size: 1826.0 B, free: 413.9 MB)
20/11/08 14:57:31 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163
20/11/08 14:57:31 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
20/11/08 14:57:31 INFO YarnScheduler: Adding task set 0.0 with 2 tasks
20/11/08 14:57:31 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, bigdata01, executor 1, partition 0, PROCESS_LOCAL, 7217 bytes)
20/11/08 14:57:31 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, bigdata01, executor 2, partition 1, PROCESS_LOCAL, 7219 bytes)
20/11/08 14:57:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:44924 (size: 1826.0 B, free: 413.9 MB)
20/11/08 14:57:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:42724 (size: 1826.0 B, free: 413.9 MB)
20/11/08 14:57:35 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 4401 ms on bigdata01 (executor 1) (1/2)
20/11/08 14:57:35 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 4481 ms on bigdata01 (executor 2) (2/2)
20/11/08 14:57:35 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 5.051 s
20/11/08 14:57:35 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/11/08 14:57:35 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 5.165122 s
Pi is roughly 3.1423357116785584
20/11/08 14:57:35 INFO AbstractConnector: Stopped Spark@bc57b40{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 14:57:35 INFO SparkUI: Stopped Spark web UI at http://bigdata01:4040
20/11/08 14:57:35 INFO YarnClientSchedulerBackend: Interrupting monitor thread
20/11/08 14:57:35 INFO YarnClientSchedulerBackend: Shutting down all executors
20/11/08 14:57:35 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
20/11/08 14:57:35 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
20/11/08 14:57:35 INFO YarnClientSchedulerBackend: Stopped
20/11/08 14:57:35 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/11/08 14:57:35 INFO MemoryStore: MemoryStore cleared
20/11/08 14:57:35 INFO BlockManager: BlockManager stopped
20/11/08 14:57:35 INFO BlockManagerMaster: BlockManagerMaster stopped
20/11/08 14:57:35 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/11/08 14:57:36 INFO SparkContext: Successfully stopped SparkContext
20/11/08 14:57:36 INFO ShutdownHookManager: Shutdown hook called
20/11/08 14:57:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-dd5062ee-711c-4455-b53e-031d6b75da58
20/11/08 14:57:36 INFO ShutdownHookManager: Deleting directory /tmp/spark-e49788a4-fa66-4ba5-926f-7daf3977b84b
Modo de racimo de hilos
Envíe el comando:
[hadoop@bigdata01 native]$ spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode cluster \
> /home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar \
> 2
Información de registro:
20/11/08 15:19:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/08 15:19:42 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/08 15:19:42 INFO Client: Requesting a new application from cluster with 1 NodeManagers
20/11/08 15:19:42 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
20/11/08 15:19:42 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
20/11/08 15:19:42 INFO Client: Setting up container launch context for our AM
20/11/08 15:19:42 INFO Client: Setting up the launch environment for our AM container
20/11/08 15:19:42 INFO Client: Preparing resources for our AM container
20/11/08 15:19:43 INFO Client: Source and destination file systems are the same. Not copying hdfs://bigdata01:9000/spark/jars/sparkjars.zip
20/11/08 15:19:43 INFO Client: Uploading resource file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar -> hdfs://bigdata01:9000/user/hadoop/.sparkStaging/application_1604816619741_0002/spark-examples_2.12-2.4.5.jar
20/11/08 15:19:43 INFO Client: Uploading resource file:/tmp/spark-a93a1160-b473-498b-b3df-4912a955ff54/__spark_conf__1880777542668876131.zip -> hdfs://bigdata01:9000/user/hadoop/.sparkStaging/application_1604816619741_0002/__spark_conf__.zip
20/11/08 15:19:43 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 15:19:43 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 15:19:43 INFO SecurityManager: Changing view acls groups to:
20/11/08 15:19:43 INFO SecurityManager: Changing modify acls groups to:
20/11/08 15:19:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 15:19:45 INFO Client: Submitting application application_1604816619741_0002 to ResourceManager
20/11/08 15:19:45 INFO YarnClientImpl: Submitted application application_1604816619741_0002
20/11/08 15:19:46 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:46 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1604819985284
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0002/
user: hadoop
20/11/08 15:19:47 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:48 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:49 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:50 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:51 INFO Client: Application report for application_1604816619741_0002 (state: ACCEPTED)
20/11/08 15:19:52 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:52 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: bigdata01
ApplicationMaster RPC port: 37147
queue: root.hadoop
start time: 1604819985284
final status: UNDEFINED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0002/
user: hadoop
20/11/08 15:19:53 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:54 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:55 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:56 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:57 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:58 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:19:59 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:00 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:01 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:02 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:03 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:04 INFO Client: Application report for application_1604816619741_0002 (state: RUNNING)
20/11/08 15:20:05 INFO Client: Application report for application_1604816619741_0002 (state: FINISHED)
20/11/08 15:20:05 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: bigdata01
ApplicationMaster RPC port: 37147
queue: root.hadoop
start time: 1604819985284
final status: SUCCEEDED
tracking URL: http://bigdata01:18088/proxy/application_1604816619741_0002/
user: hadoop
20/11/08 15:20:05 INFO ShutdownHookManager: Shutdown hook called
20/11/08 15:20:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-658608fd-a353-4804-9a37-ce267d732efa
20/11/08 15:20:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-a93a1160-b473-498b-b3df-4912a955ff54
Modo local
Envíe el comando:
[hadoop@bigdata01 native]$ spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master local[2] \
> /home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar \
> 2
Ejecutar registro:
20/11/08 15:23:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/11/08 15:23:25 INFO SparkContext: Running Spark version 2.4.5
20/11/08 15:23:25 INFO SparkContext: Submitted application: Spark Pi
20/11/08 15:23:25 INFO SecurityManager: Changing view acls to: hadoop
20/11/08 15:23:25 INFO SecurityManager: Changing modify acls to: hadoop
20/11/08 15:23:25 INFO SecurityManager: Changing view acls groups to:
20/11/08 15:23:25 INFO SecurityManager: Changing modify acls groups to:
20/11/08 15:23:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
20/11/08 15:23:26 INFO Utils: Successfully started service 'sparkDriver' on port 42466.
20/11/08 15:23:26 INFO SparkEnv: Registering MapOutputTracker
20/11/08 15:23:26 INFO SparkEnv: Registering BlockManagerMaster
20/11/08 15:23:26 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/11/08 15:23:26 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/11/08 15:23:26 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-e01c3a97-0f5c-4019-8d78-47d5a600b923
20/11/08 15:23:26 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
20/11/08 15:23:26 INFO SparkEnv: Registering OutputCommitCoordinator
20/11/08 15:23:26 INFO log: Logging initialized @2476ms
20/11/08 15:23:26 INFO Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
20/11/08 15:23:26 INFO Server: Started @2635ms
20/11/08 15:23:26 INFO AbstractConnector: Started ServerConnector@64f857e7{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 15:23:26 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@733c423e{
/jobs,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@27eb3298{
/jobs/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@200a26bc{
/jobs/job,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1b5bc39d{
/jobs/job/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@655a5d9c{
/stages,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1494b84d{
/stages/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@34abdee4{
/stages/stage,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@77cf3f8b{
/stages/stage/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1df98368{
/stages/pool,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21ca139c{
/stages/pool/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@226f885f{
/storage,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2cd2c8fe{
/storage/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7d61eccf{
/storage/rdd,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@cc6460c{
/storage/rdd/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@52350abb{
/environment,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@681aad3b{
/environment/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1a6f2363{
/executors,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2427e004{
/executors/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5ebd56e9{
/executors/threadDump,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@63f34b70{
/executors/threadDump/json,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@641856{
/static,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@23c388c2{
/,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@486be205{
/api,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5ca17ab0{
/jobs/job/kill,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5a62b2a4{
/stages/stage/kill,null,AVAILABLE,@Spark}
20/11/08 15:23:26 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://bigdata01:4040
20/11/08 15:23:26 INFO SparkContext: Added JAR file:/home/hadoop/app/spark/examples/jars/spark-examples_2.12-2.4.5.jar at spark://bigdata01:42466/jars/spark-examples_2.12-2.4.5.jar with timestamp 1604820206605
20/11/08 15:23:26 INFO Executor: Starting executor ID driver on host localhost
20/11/08 15:23:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39948.
20/11/08 15:23:26 INFO NettyBlockTransferService: Server created on bigdata01:39948
20/11/08 15:23:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/11/08 15:23:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO BlockManagerMasterEndpoint: Registering block manager bigdata01:39948 with 413.9 MB RAM, BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, bigdata01, 39948, None)
20/11/08 15:23:26 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@779de014{
/metrics/json,null,AVAILABLE,@Spark}
20/11/08 15:23:27 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
20/11/08 15:23:27 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
20/11/08 15:23:27 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
20/11/08 15:23:27 INFO DAGScheduler: Parents of final stage: List()
20/11/08 15:23:27 INFO DAGScheduler: Missing parents: List()
20/11/08 15:23:27 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
20/11/08 15:23:27 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.2 KB, free 413.9 MB)
20/11/08 15:23:28 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1826.0 B, free 413.9 MB)
20/11/08 15:23:28 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on bigdata01:39948 (size: 1826.0 B, free: 413.9 MB)
20/11/08 15:23:28 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1163
20/11/08 15:23:28 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
20/11/08 15:23:28 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
20/11/08 15:23:28 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7206 bytes)
20/11/08 15:23:28 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7208 bytes)
20/11/08 15:23:28 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
20/11/08 15:23:28 INFO Executor: Fetching spark://bigdata01:42466/jars/spark-examples_2.12-2.4.5.jar with timestamp 1604820206605
20/11/08 15:23:28 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
20/11/08 15:23:28 INFO TransportClientFactory: Successfully created connection to bigdata01/192.168.52.50:42466 after 65 ms (0 ms spent in bootstraps)
20/11/08 15:23:28 INFO Utils: Fetching spark://bigdata01:42466/jars/spark-examples_2.12-2.4.5.jar to /tmp/spark-d2b27239-b4e7-4af8-b9b3-07b9bfd60cb3/userFiles-195a5986-155d-4e68-a31b-15fa7ae4e5ec/fetchFileTemp6323009668907142489.tmp
20/11/08 15:23:28 INFO Executor: Adding file:/tmp/spark-d2b27239-b4e7-4af8-b9b3-07b9bfd60cb3/userFiles-195a5986-155d-4e68-a31b-15fa7ae4e5ec/spark-examples_2.12-2.4.5.jar to class loader
20/11/08 15:23:29 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 676 bytes result sent to driver
20/11/08 15:23:29 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 676 bytes result sent to driver
20/11/08 15:23:29 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 915 ms on localhost (executor driver) (1/2)
20/11/08 15:23:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 971 ms on localhost (executor driver) (2/2)
20/11/08 15:23:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/11/08 15:23:29 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 1.645 s
20/11/08 15:23:29 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.756773 s
Pi is roughly 3.144195720978605
20/11/08 15:23:29 INFO AbstractConnector: Stopped Spark@64f857e7{
HTTP/1.1,[http/1.1]}{
0.0.0.0:4040}
20/11/08 15:23:29 INFO SparkUI: Stopped Spark web UI at http://bigdata01:4040
20/11/08 15:23:29 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/11/08 15:23:29 INFO MemoryStore: MemoryStore cleared
20/11/08 15:23:29 INFO BlockManager: BlockManager stopped
20/11/08 15:23:29 INFO BlockManagerMaster: BlockManagerMaster stopped
20/11/08 15:23:29 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/11/08 15:23:29 INFO SparkContext: Successfully stopped SparkContext
20/11/08 15:23:29 INFO ShutdownHookManager: Shutdown hook called
20/11/08 15:23:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-c675b673-6b0d-44b1-8748-1114de6f96e3
20/11/08 15:23:29 INFO ShutdownHookManager: Deleting directory /tmp/spark-d2b27239-b4e7-4af8-b9b3-07b9bfd60cb3