IDEA 中开发第一个Spark 程序

1. 创建一个Maven 项目

2. 添加SCALA依赖库

****注意scala 的版本 相对于spark2.4 ,scala 的版本必须是2.11.x

修改POM.xml 文件

加入 hadoop-client 和spark-core_2.11 的库依赖

 <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.7.3</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.4.0</version>
    </dependency>

在经过一段漫长的等待之后，可以发现所有依赖的JAR包都下载完毕，并添加到依赖的库中

3. HDFS client 的配置

把Hadoop_home下的配置文件拷贝到项目中

$HADOOP_HOME/etc/hadoop/下的文件 core-site.xml 和hdfs-site.xml 拷贝到Maven 项目的的resources 的文件夹下

*** 注意resources 文件夹的路径层级 是在/src/main/resources

本次实验从HDFS读取的测试文件如下

[hadoop@hadoop01 ~]$ hdfs dfs -cat  /user/hadoop/worddir/word.txt
Jason chen welcome TJ
TJ welcome Jason
Bye Bye 
Bye

3. 编写WorldCount 程序

创建一个 Scala 的 object : WorldCount

package com.jason

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
  def main(args: Array[String]): Unit = {
    //初始化conf配置
    val sparkConf = new SparkConf()
      .setAppName("WordCount sample")
     // .setMaster("192.168.1.10:4040")
      .setMaster("local[2]")
      .set("spark.testing.memory", "2147480000");
    val sc = new SparkContext(sparkConf);
    val rdd = sc.textFile("/user/hadoop/worddir/word.txt");
    val tupleRDD = rdd.flatMap(line => {line.split(" ")
        .toList.map(word => (word.trim,1))
    });
    val resultRDD :RDD[(String,Int)] =tupleRDD.reduceByKey((a,b)=> a + b);
    resultRDD.foreach(elm => println(elm._1+"="+elm._2));

    Thread.sleep(10000);
    sc.stop();


  }
}

4. 查看输出:

18/12/16 16:19:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 93 ms
18/12/16 16:19:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 94 ms
Jason=2
TJ=2
Bye=3
chen=1
welcome=2
18/12/16 16:19:28 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 1138 bytes result sent to driver
18/12/16 16:19:28 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 1138 bytes result sent to driver