Spark环境搭建之本地模式(开箱即用无需配置)

安装

1. 下载安装包
   
   

2. 上传并解压安装包

tar -zxvf spark-2.4.7-bin-hadoop2.6.tgz  

3. 修改权限

chown -R root /export/server/spark-2.4.7-bin-hadoop2.6
chgrp -R root /export/server/spark-2.4.7-bin-hadoop2.6

4. 创建软连接

ln -s /export/server/spark-2.4.7-bin-hadoop2.6 /export/server/spark

测试

1. 启动spark交互式窗口

/export/server/spark/bin/spark-shell

2. 测试Spark的WordCount

1.准备文件
vim /root/words.txt
添加以下内容：
hello me you her
hello me you
hello me
hello

2.执行WordCount
val textFile = sc.textFile("file:///root/words.txt")
val counts = textFile.flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _)
counts.collect

结果：