IDEA设置默认的maven环境、加入scala环境、spark-scala项目pom设置、打包、提交命令

当前项目选settings...，以后所有其他项目选settings for new projects...

搜索到maven选项卡，设置三处：maven home、settings.xml、local repository

新建maven项目，加入scala环境，file-project structure-copy to project libraries

新建scala文件夹，右键文件夹make dir as--sources root，在其中new一个scala class，注意类型为object，名称需包含包名（com.baidu.sh.xzq.wordcount）

pom设置

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.baidu.sh.xzq</groupId>
    <artifactId>spark</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.3.2</version>
        </dependency>
    </dependencies>
    
    <build>
        <plugins>

            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.1</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

</project>

示例代码

package com.baidu.sh.xzq

import org.apache.spark.{SparkConf, SparkContext}

object wordcount {
  def main(args: Array[String]): Unit = {
    val conf=new SparkConf()
    conf.setAppName("spark-scala")
    val sc=new SparkContext(conf)
    val rdd=sc.textFile("/hadoop/profile")
    rdd.flatMap(_.split(""))
      .map((_,1))
      .reduceByKey(_+_)
      .sortBy(_._2)
      .saveAsTextFile("/hadoop/prof")
    sc.stop()
  }
}

打包，右侧栏选择clean，再执行package

之后可在target文件加下发现打包好的jar包，clean后又可删除这些文件夹

提交命令（class名可在类名上copy reference得到）

spark-submit --class com.baidu.sh.xzq.workcount /share/spark-1.0-SNAPSHOT.jar

IDEA设置默认的maven环境、加入scala环境、spark-scala项目pom设置、打包、提交命令

猜你喜欢