Spark 2.4 入门之基于Hadoop 2.7.3环境搭建

准备环境

• 3 台 Linux VM Server RHL6.8 * 3
• Hadoop 2.7.3 集群环境搭建

下载Spark 2.4 版本

网址： https://spark.apache.org/downloads.html

镜像地址 http://mirror.bit.edu.cn/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

下载Scala 2.12.8 版本

下载地址： https://www.scala-lang.org/download/2.12.8.html

创建spark 用户

[root@hadoop01 ~]# useradd spark
[root@hadoop01 ~]# passwd spark
Changing password for user spark.
New password: 
BAD PASSWORD: it is too short
BAD PASSWORD: is too simple
Retype new password: 
passwd: all authentication tokens updated successfully.

解压安装包

[spark@hadoop01 ~]$ tar -xvf spark-2.4.0-bin-hadoop2.7.tgz
[spark@hadoop01 ~]$ tar -xvf scala-2.12.8.tgz

配置环境变量

export JAVA_HOME=/opt/java/jdk1.8.0_144
export HADOOP_HOME=/opt/hadoop/hadoop-2.7.3
export SCALA_HOME=/home/spark/scala-2.12.8
export SPARK_HOME=/home/spark/spark-2.4.0-bin-hadoop2.7
#PATH=$PATH:$HOME/bin
PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH:$HOME/bin
export PATH

修改配合文件 spark-env.sh.template

[spark@hadoop01 conf]$ mv spark-env.sh.template spark-env.sh
[spark@hadoop01 conf]$ vi spark-env.sh
export JAVA_HOME=/opt/java/jdk1.8.0_144
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-2.7.3/etc/hadoop
export SCALA_HOME=/home/spark/scala-2.12.8

启动spark

[spark@hadoop01 bin]$ ./spark-shell 
2018-12-12 06:12:58 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://hadoop01:4040
Spark context available as 'sc' (master = local[*], app id = local-1544623996096).
Spark session available as 'spark'.
Welcome to
    ____              __
   / __/__  ___ _____/ /__
  _\ \/ _ \/ _ `/ __/  '_/
 /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
    /_/
       
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.

scala>