软件包
- JDK1.8.0_45
- apache-maven-3.6.0
- Scala2.11.8
- hadoop-2.6.0-cdh5.7.0
注意设置内存*****
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
安装git,源码编译过程中会需要用到git
[hadoop001@iZuf6dz84cmy6o2y05pb8bZ scala-2.11.8]$ sudo yum install -y git
下载spark2.4.0源码
[hadoop001@iZuf6dz84cmy6o2y05pb8bZ software]$ wget https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0.tgz
--2019-01-03 07:05:31-- https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0.tgz
Resolving archive.apache.org... 163.172.17.199
Connecting to archive.apache.org|163.172.17.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16118412 (15M) [application/x-gzip]
Saving to: “spark-2.4.0.tgz”
100%[===============================================================================================================>] 16,118,412 3.17M/s in 4.9s
2019-01-03 07:05:37 (3.17 MB/s) - “spark-2.4.0.tgz” saved [16118412/16118412]
解压spark源码
[hadoop001@iZuf6dz84cmy6o2y05pb8bZ software]$ tar -zxvf spark-2.4.0.tgz -C ~/app/
- 因为使用的是CDH版本的Hadoop，所以需要在pom.xml添加repository
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
编译方法
用make-distributed 脚本来编译
- 修改dev目录下make-distribution.sh脚本
VERSION=2.4.0 #Spark版本
SCALA_VERSION=2.11 #Scala版本（大版本）
SPARK_HADOOP_VERSION=2.6.0-cdh5.7.0 #Hadoop版本
SPARK_HIVE=1 #是否将Hive打包（非1表示不打包）
-
图示
-
执行编译命令
[hadoop001@iZuf6dz84cmy6o2y05pb8bZ spark-2.4.0]$ ./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --tgz -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver -Dhadoop.version=2.6.0-cdh5.7.0
耐心等待编译完成即可，编成功后生成的部署包位于根目录下。