spark----词频统计(一)

利用Linux系统中安装的spark来统计:

1.选择目录,并创建一个存放文本的目录,将要处理的文本保存在该目录下以供查找操作:

① cd /usr/local ②mkdir mycode ③ cd mycode ④查看当前新目录: ll

⑤新建文本: vim wordcount.txt (文本内容随机copy一段英文)

[root@node01 mycode]# vim  wordcount.txt
  uded among the most successful influencers in Open Source, The Apache Software Foundation's commitment to collaborative development has long served as a model for producing consistently high quality software that advances the future of open development. https://s.apache.org/PIRA

2.为方便查询文本和其它操作,可以在当前操作节点上复制另一个操作节点,作为它的第二个终端操作窗口:

如:打开:node01------>复制node01 ,然后在复制的节点上去查询之前所创建的目录及文本.

 >>>cd /usr/local/mycode/
  >>>ll

3.启动spark: 本机spark安装在(/home/mysoft/spark-1.6),以具体路径为准!

① 跳转路径

 cd /home/mysoft/spark-1.6

②启动命令: (或者 cd bin ----->pyspark (enter) 亦可)

./bin/pyspark

-------出现spark的正常启动信息即启动成功!

 Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /__ / .__/\_,_/_/ /_/\_\   version 1.6.0
        /_/
  
  Using Python version 3.5.0 (default, Jul 12 2018 03:34:21)
  SparkContext available as sc, HiveContext available as sqlContext.
  >>>

4.加载文件:

 >>>textFile = sc.textFile("file:///usr/local/mycode/wordcount.txt")
  >>> textFile.first()

之后会在屏幕显示之前创建的文本!

注:first()是一个“行动”（Action）类型的操作，会启动真正的计算过程，从文件中加载数据到变量textFile中，并取

出第一行文本,另因为Spark采用了惰性机制，在执行转换操作的时候，即使我们输入了错误的语句，pyspark也不

会马上报错，而是等到执行“行动”类型的语句时启动真正的计算，那个时候“转换”操作语句中的错误就会显示出来:

拒绝连接!

5.统计词频:(继续上述代码输入)

 <<<Count = textFile.flatMap(lambda line: line.split(" ")).map(lambda word:       (word,1)).reduceByKey(lambda a, b : a + b)
  <<<Count.collect()

6.打印结果:

 [('development', 1), ('producing', 1), ('among', 1), ('Source,', 1), ('for', 1), ('quality', 1), ('to', 1), ('influencers', 1), ('advances', 1), ('collaborative', 1), ('model', 1), ('in', 1), ('the', 2), ('of', 1), ('has', 1), ('successful', 1), ('Software', 1), ("Foundation's", 1), ('most', 1), ('long', 1), ('that', 1), ('uded', 1), ('as', 1), ('Open', 1), ('The', 1), ('commitment', 1), ('software', 1), ('consistently', 1), ('a', 1), ('development.', 1), ('high', 1), ('future', 1), ('Apache', 1), ('served', 1), ('open', 1), ('https://s.apache.org/PIRA', 1)]

spark----词频统计(一)

猜你喜欢