Spark Tutorial - (3) prepared by spark-sql program read timings report HBase

Others 2019-07-26 05:05:12 views: null

 

plugin zoned red line error:

maven-scala-plugin
maven-shade-plugin

Find Maven repository, did not find a jar package, a package can not unzip jar open, delete bad Maven in a jar, and Reimport success, IDEA is no longer being given:

 

In Maven panel of IDEA set to skip test, to prepare for the package:

Packaging Spark Program: 

 

Packager error:

Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile

wrap: org.apache.commons.exec.ExecuteException

error: scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found.

Re-run Maven using the -X switch to enable full debug logging.

Refer to the relevant blog inferred Maven repository jar package there is a problem, but not the specific error, unable to locate the problematic jar package, set up Maven output more detailed information, the Output level by the Info tune into Debug:

First Clean, after performing Compile:

Error message is returned by filtration as follows:

[FATAL] Non-parseable POM D:\Development\MavenRepository\org\apache\hadoop\hadoop-mapreduce-client-core\2.6.0-cdh5.14.2\hadoop-mapreduce-client-core-2.6.0-cdh5.14.2.pom: end tag name </head> must be the same as start tag <link> from line 21 (position: TEXT seen ...<![endif]-->\r\n</head>... @66:8)  @ line 66, column 8

[FATAL] Non-parseable POM D:\Development\MavenRepository\org\apache\phoenix\phoenix-core\4.14.0-cdh5.14.2\phoenix-core-4.14.0-cdh5.14.2.pom: end tag name </head> must be the same as start tag <link> from line 21 (position: TEXT seen ...<![endif]-->\r\n</head>... @66:8)  @ line 66, column 8

[FATAL] Non-parseable POM D:\Development\MavenRepository\com\lmax\disruptor\3.3.8\disruptor-3.3.8.pom: end tag name </head> must be the same as start tag <link> from line 21 (position: TEXT seen ...<![endif]-->\r\n</head>... @66:8)  @ line 66, column 8

……

到对应目录下查看jar包是否正常，解压打开报错，说明对应jar包确实有问题，删除Maven中坏的jar包，执行Reimport重新导入jar包：

 

虽然IDEA依然有红线报错，但是画红线的jar包已经导入并能解压缩打开，尝试执行Compile，Compile成功，执行Package，Package成功，返回如下信息：

[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing D:\Development\asset\statistics-master-190725\target\statistics-1.0-SNAPSHOT.jar with D:\Development\asset\statistics-master-190725\target\statistics-1.0-SNAPSHOT-shaded.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:49 min
[INFO] Finished at: 2019-07-25T16:41:04+08:00
[INFO] ------------------------------------------------------------------------

上传到服务器：

 

执行打包的Spark程序：

[root@node2 ~]# spark-submit  --master yarn-cluster --driver-memory 4g --num-executors 2 --executor-memory 2g --executor-cores 2  --class statistics.CostAsset   --conf spark.driver.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/*  --conf spark.executor.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/* /home/microservices/statistics-1.0-SNAPSHOT.jar total model

执行后返回如下信息：

[root@node2 ~]# spark-submit  --master yarn-cluster --driver-memory 4g --num-executors 2 --executor-memory 2g --executor-cores 2  --class statistics.CostAsset   --conf spark.driver.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/*  --conf spark.executor.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/* /home/microservices/statistics-1.0-SNAPSHOT.jar total model
19/07/25 16:54:52 INFO client.RMProxy: Connecting to ResourceManager at node1/10.200.101.131:8032
19/07/25 16:54:52 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/07/25 16:54:52 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (40874 MB per container)
19/07/25 16:54:52 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
19/07/25 16:54:52 INFO yarn.Client: Setting up container launch context for our AM
19/07/25 16:54:52 INFO yarn.Client: Setting up the launch environment for our AM container
19/07/25 16:54:52 INFO yarn.Client: Preparing resources for our AM container
19/07/25 16:54:53 INFO yarn.Client: Uploading resource file:/home/microservices/statistics-1.0-SNAPSHOT.jar -> hdfs://node1:8020/user/root/.sparkStaging/application_1563417834812_0018/statistics-1.0-SNAPSHOT.jar
19/07/25 16:54:54 INFO yarn.Client: Uploading resource file:/tmp/spark-2573c8b3-f471-452f-85b1-d3582877290e/__spark_conf__4623511860207833838.zip -> hdfs://node1:8020/user/root/.sparkStaging/application_1563417834812_0018/__spark_conf__4623511860207833838.zip
19/07/25 16:54:54 INFO spark.SecurityManager: Changing view acls to: root
19/07/25 16:54:54 INFO spark.SecurityManager: Changing modify acls to: root
19/07/25 16:54:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
19/07/25 16:54:54 INFO yarn.Client: Submitting application 18 to ResourceManager
19/07/25 16:54:54 INFO impl.YarnClientImpl: Submitted application application_1563417834812_0018
19/07/25 16:54:55 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:54:55 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.users.root
     start time: 1564044894496
     final status: UNDEFINED
     tracking URL: http://node1:8088/proxy/application_1563417834812_0018/
     user: root
19/07/25 16:54:56 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:54:57 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:54:58 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:54:59 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:55:00 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:00 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 10.200.101.133
     ApplicationMaster RPC port: 0
     queue: root.users.root
     start time: 1564044894496
     final status: UNDEFINED
     tracking URL: http://node1:8088/proxy/application_1563417834812_0018/
     user: root
19/07/25 16:55:01 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:02 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:03 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:04 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:05 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:06 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:07 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:08 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:09 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:10 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:55:10 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.users.root
     start time: 1564044894496
     final status: UNDEFINED
     tracking URL: http://node1:8088/proxy/application_1563417834812_0018/
     user: root
19/07/25 16:55:11 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:55:12 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:55:13 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:55:14 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED)
19/07/25 16:55:15 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:15 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 10.200.101.135
     ApplicationMaster RPC port: 0
     queue: root.users.root
     start time: 1564044894496
     final status: UNDEFINED
     tracking URL: http://node1:8088/proxy/application_1563417834812_0018/
     user: root
19/07/25 16:55:16 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:17 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:18 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:19 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:20 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:21 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:22 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:23 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:24 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:25 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:26 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:27 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:28 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:29 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:30 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:31 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:32 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING)
19/07/25 16:55:33 INFO yarn.Client: Application report for application_1563417834812_0018 (state: FINISHED)
19/07/25 16:55:33 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 10.200.101.135
     ApplicationMaster RPC port: 0
     queue: root.users.root
     start time: 1564044894496
     final status: SUCCEEDED
     tracking URL: http://node1:8088/proxy/application_1563417834812_0018/
     user: root
19/07/25 16:55:33 INFO util.ShutdownHookManager: Shutdown hook called
19/07/25 16:55:33 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2573c8b3-f471-452f-85b1-d3582877290e

通过界面分析执行的程序：

 

 

 

 

 

 

 

参考：

https://www.cnblogs.com/nurseryboy/p/6155925.html

https://www.oschina.net/question/1422726_2263380?sort=time

Guess you like

Origin www.cnblogs.com/ratels/p/11246898.html
Recommended
Arc Browser for Windows 1.0 officially GA
A programmer born in the 1990s developed a video porting software and made over 7 million in less than a year. The ending was very punishing!
Ranking
1. Select Sort
Create a thread thread
3 press to play ball that reach 6
Programmation CUDA (4) : gestion de la mémoire
SpringBoot database connection pool Druid error
E Diudiu App redesign summary
4EVERLAND Hosting now supports SNS+IPFS
About HTTPS
[vue3+vite+ts+element-plu+sass] uses bug records in sass
Interpretation of HUAWEI CLOUD GaussDB (for Influx): Best Practice Data Modeling
Daily
More
2024-05-03(8)
2024-05-02(0)
2024-05-01(4)
2024-04-30(36)
2024-04-29(5)
2024-04-28(12)
2024-04-27(29)
2024-04-26(22)
2024-04-25(32)
2024-04-24(30)

Code World

© 2018 codetd.com