为了教学方便，考虑ALL IN ONE，一台虚拟机构建整个实训环境，因此是伪分布式搭建spark
环境：
hadoop2.6.0-cdh5.15.1
jdk1.8
centos7 64位
1. 安装scala环境
版本是scala-2.12.7，官网下载地址http://www.scala-lang.org/download/
scala-2.12.7.tgz
tar -zxvf scala-2.12.7.tgz
nano /etc/profile
export SCALA_HOME=/home/linbin/software/scala-2.12.7
export PATH=$PATH:$SCALA_HOME/bin
[root@centos7 bin]# scala -version
Scala code runner version 2.12.7 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.
2. 下载spark
wget http://archive.cloudera.com/cdh5/cdh/5/spark1.6.0-cdh5.15.1.tar.gz
3.解压 tar -zxvf spark1.6.0-cdh5.15.1.tar.gz
4.进入spark/conf目录，把 spark-env.sh.template文件重命名为 spark-env.sh，并且在文件末尾加上如下配置
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64
export SCALA_HOME=/home/linbin/software/scala-2.12.7
export SPARK_MASTER_IP=centos7
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/home/linbin/software/hadoop-2.6.0-cdh5.15.1/etc/hadoop
export SPARK_DIST_CLASSPATH=$(/home/linbin/software/hadoop-2.6.0-cdh5.15.1/bin/hadoop classpath)
5. slaves.template文件重命名为slaves，文件的默认内容是localhost，把localhost删除，并添加本机名字centos7
6. 提示log4j类未发现，在其它组件下找到并复制到
sparkXX/lib/slf4j-api-1.7.5.jar
7. worker启动提示如下错误
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
解决办法：
把 /home/linbin/software/hadoop-2.6.0-cdh5.15.1/share/hadoop/mapreduce1/lib/目录下的
jackson-databind-2.2.3.jar
jackson-core-2.2.3.jar
jackson-annotations-2.2.3.jar
复制到/home/linbin/software/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib 目录
8.启动
start-dfs.sh
start-yarn.sh
sparkXX/sbin/start-all.sh //spark 的start-all.sh 和 hdfs 的 start-all.sh 不要搞混了
看到master和Worker 说明spark启动成功
9.启动spark-shell
出现 java.lang.NoClassDefFoundError: parquet/hadoop/ParquetOutputCommitter
解决：
在sqoop目录找到
sqoop2-1.99.5-cdh5.15.1/server/webapps/sqoop/WEB-INF/lib/parquet-hadoop-1.5.0-cdh5.15.1.jar
复制到/home/linbin/software/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib 目录
最后，成功启动
10.通过http监控spark master 和worker
有价值的参考文章：
https://www.jianshu.com/p/a0c38dc46b89 搭建Spark所遇过的坑