IDEA+Maven build scala project [one piece is enough]

1. New project

Enter the project name: SparkDemo_WordCount, click "Finish"

The project is successfully created, as shown below:

"C:\Program Files\Java\jdk1.8.0_45\bin\java" -Dmaven.multiModuleProjectDirectory=C:\Users\Administrator\AppData\Local\Temp\archetype1tmp -Dmaven.home=C:\20170801\software\ideaIU-2017.3\plugins\maven\lib\maven3 -Dclassworlds.conf=C:\20170801\software\ideaIU-2017.3\plugins\maven\lib\maven3\bin\m2.conf -Dfile.encoding=UTF-8 -classpath C:\20170801\software\ideaIU-2017.3\plugins\maven\lib\maven3\boot\plexus-classworlds-2.5.2.jar org.codehaus.classworlds.Launcher -Didea.version=2017.3 -DinteractiveMode=false -DgroupId=com.sjmz.sparkdemo -DartifactId=WordCount -Dversion=1.0-SNAPSHOT -DarchetypeGroupId=org.scala-tools.archetypes -DarchetypeArtifactId=scala-archetype-simple -DarchetypeVersion=1.2 -DarchetypeRepository=http://scala-tools.org/repo-releases org.apache.maven.plugins:maven-archetype-plugin:RELEASE:generate
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO] 
[INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO] 
[INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Batch mode
[INFO] Archetype repository not defined. Using the one from [org.scala-tools.archetypes:scala-archetype-simple:1.3] found in catalog remote
Downloading: https://repo.maven.apache.org/maven2/org/scala-tools/archetypes/scala-archetype-simple/1.2/scala-archetype-simple-1.2.pom
Downloaded: https://repo.maven.apache.org/maven2/org/scala-tools/archetypes/scala-archetype-simple/1.2/scala-archetype-simple-1.2.pom (4 KB at 4.7 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/scala-tools/scala-tools-parent/1.2/scala-tools-parent-1.2.pom
Downloaded: https://repo.maven.apache.org/maven2/org/scala-tools/scala-tools-parent/1.2/scala-tools-parent-1.2.pom (9 KB at 12.5 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/scala-tools/archetypes/scala-archetype-simple/1.2/scala-archetype-simple-1.2.jar
Downloaded: https://repo.maven.apache.org/maven2/org/scala-tools/archetypes/scala-archetype-simple/1.2/scala-archetype-simple-1.2.jar (6 KB at 8.4 KB/sec)
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Old (1.x) Archetype: scala-archetype-simple:1.2
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: basedir, Value: C:\Users\Administrator\AppData\Local\Temp\archetype1tmp
[INFO] Parameter: package, Value: com.sjmz.sparkdemo
[INFO] Parameter: groupId, Value: com.sjmz.sparkdemo
[INFO] Parameter: artifactId, Value: WordCount
[INFO] Parameter: packageName, Value: com.sjmz.sparkdemo
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] project created from Old (1.x) Archetype in dir: C:\Users\Administrator\AppData\Local\Temp\archetype1tmp\WordCount
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:08 min
[INFO] Finished at: 2019-01-05T18:00:45+08:00
[INFO] Final Memory: 15M/197M
[INFO] ------------------------------------------------------------------------
[INFO] Maven execution finished

The following error is reported when compile is executed for the first time:

[Problem analysis] The scala project created by the prototype scala-archetype-simple uses the scala version 2.7.0, which is presumably too old.

[Problem solving] Replace the scala version with 2.11.12, the problem is solved.

"C:\Program Files\Java\jdk1.8.0_45\bin\java" -Dmaven.multiModuleProjectDirectory=C:\20170801\GitProject\SparkDemo_WordCount -Dmaven.home=C:\20170801\software\ideaIU-2017.3\plugins\maven\lib\maven3 -Dclassworlds.conf=C:\20170801\software\ideaIU-2017.3\plugins\maven\lib\maven3\bin\m2.conf -javaagent:C:\20170801\software\ideaIU-2017.3\lib\idea_rt.jar=56281:C:\20170801\software\ideaIU-2017.3\bin -Dfile.encoding=UTF-8 -classpath C:\20170801\software\ideaIU-2017.3\plugins\maven\lib\maven3\boot\plexus-classworlds-2.5.2.jar org.codehaus.classworlds.Launcher -Didea.version=2017.3 compile
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.sjmz.sparkdemo:SparkDemo_WordCount:jar:1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.scala-tools:maven-scala-plugin is missing. @ line 51, column 15
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 69, column 15
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building SparkDemo_WordCount 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: http://scala-tools.org/repo-releases/org/scala-tools/maven-scala-plugin/maven-metadata.xml
[WARNING] Could not transfer metadata org.scala-tools:maven-scala-plugin/maven-metadata.xml from/to scala-tools.org (http://scala-tools.org/repo-releases): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Downloading: http://scala-tools.org/repo-releases/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml
[WARNING] Could not transfer metadata org.apache.maven.plugins:maven-eclipse-plugin/maven-metadata.xml from/to scala-tools.org (http://scala-tools.org/repo-releases): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ SparkDemo_WordCount ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\20170801\GitProject\SparkDemo_WordCount\src\main\resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ SparkDemo_WordCount ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-scala-plugin:2.15.2:compile (default) @ SparkDemo_WordCount ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.scala,]
[INFO] excludes = []
[INFO] C:\20170801\GitProject\SparkDemo_WordCount\src\main\scala:-1: info: compiling
[INFO] Compiling 1 source files to C:\20170801\GitProject\SparkDemo_WordCount\target\classes at 1546707444981
[ERROR] error: error while loading Consumer, class file 'C:\Program Files\Java\jdk1.8.0_45\jre\lib\rt.jar(java/util/function/Consumer.class)' is broken
[INFO] (bad constant pool tag 15 at byte 1074)
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.079 s
[INFO] Finished at: 2019-01-06T00:57:27+08:00
[INFO] Final Memory: 12M/172M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile (default) on project SparkDemo_WordCount: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Process finished with exit code 1

2. Generate the jar package and put it on the Linux server for execution

package com.sjmz.sparkdemo

/**
 * Hello world!
 *
 */
object App {
  def main(args: Array[String]): Unit = {
    println( "Hello World!" )
  }
}

[root@master ~]# spark-submit --class com.sjmz.sparkdemo.App SparkDemo_WordCount_v1.0.jar 
2019-01-06 02:36:35 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Hello World!
2019-01-06 02:36:36 INFO  ShutdownHookManager:54 - Shutdown hook called
2019-01-06 02:36:36 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-b59bdc7f-0f98-4ca9-ba1b-4eabdd5fcce8
[root@master ~]#

IDEA+Maven build scala project [one piece is enough]

Guess you like