Differences between Spark download versions
Confused about the difference between the Sparkofficial website Pre-built for Apache Hadoopand Pre-built with user-provided Apache Hadoop.
Why download Sparkso many versions? ? ? This article uses the 2.1.1version as a test.
Pre-built for Apache Hadoop
JavaRun the following command on an installed machine
# 0. 进入测试目录
cd /tmp
# 1. 下载Spark 并解压
wget https://archive.apache.org/dist/spark/spark-2.1.1/spark-2.1.1-bin-hadoop2.7.tgz
tar -xf spark-2.1.1-bin-hadoop2.7.tgz
# 2. 运行Spark
cd /tmp/spark-2.1.1-bin-hadoop2.7 &&\
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
./examples/jars/spark-examples_2.11-2.1.1.jar \
100
# 程序是可以正常运行的且可以看到
# 类似于Pi is roughly 3.1408123140812316 的字样
Pre-built with user-provided Apache Hadoop
JavaRun the following command on an installed machine
# 0. 进入测试目录
cd /tmp
# 1. 下载Spark 并解压
wget https://archive.apache.org/dist/spark/spark-2.1.1/spark-2.1.1-bin-without-hadoop.tgz
tar -xf spark-2.1.1-bin-without-hadoop.tgz
# 2. 运行Spark
cd /tmp/spark-2.1.1-bin-without-hadoop &&\
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
./examples/jars/spark-examples_2.11-2.1.1.jar \
100
# 直接报错，java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
# 3. 与之前的spark-2.1.1-bin-hadoop2.7.tgz 对比以下jar 包
ls spark-2.1.1-bin-hadoop2.7/jars/ > h.txt
ls spark-2.1.1-bin-without-hadoop/jars/ > w.txt
diff -y -W 50 h.txt w.txt # -y 并排对比，-W 列宽
# 看到右边的w.txt 内容中少了很多Hadoop 的包
# 4. 下载Hadoop 并解压
cd /tmp && wget https://archive.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
tar -xf hadoop-2.7.2.tar.gz
# 5. 确认Hadoop 的版本
/tmp/hadoop-2.7.2/bin/hadoop version
# 6. 关联Spark-Without-Hadoop 和Hadoop
cat > /tmp/spark-2.1.1-bin-without-hadoop/conf/spark-env.sh << 'EOF'
#!/usr/bin/env bash
export SPARK_DIST_CLASSPATH=$(/tmp/hadoop-2.7.2/bin/hadoop classpath)
EOF
# 7. 再次运行Spark
cd /tmp/spark-2.1.1-bin-without-hadoop &&\
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
./examples/jars/spark-examples_2.11-2.1.1.jar \
100 2>&1 | grep 'Pi is'
# 成功显示Pi is roughly 3.1424395142439514
to sum up
It can be seen from the above experiment that the [Pre-built with user-provided Apache Hadoop] (# Pre-built with user-provided Apache Hadoop) version needs to modify the configuration file to adapt itself Hadoop. In fact, it is Hadoopthe Jarpackage added in the CLASSPATH during execution , and [Pre-built for Apache Hadoop] (# Pre-built for Apache Hadoop) is out of the box, and Hadoop Jarbundles the corresponding packages in advance . At the same time, because the interfaces such as Hadoop 2.6and Hadoop 2.7are HDFSnot the same, they are Pre-built for Apache Hadoopdivided into two versions. .
reference
Using Spark’s “Hadoop Free” Build
What is the difference between the package types of Spark on the download page?
