spark-3.0.0安装
官网下载
https://spark.apache.org/downloads.html
将spark.tgz包上传到虚拟机
如图,此版本需要下载scala 2.12.X
解压tgz包到指定目录
tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /new/software/
cd /new/software/spark-3.0.0-bin-hadoop3.2/conf
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
export SCALA_HOME=/new/software/scala-2.12.4
export JAVA_HOME=/new/software/jdk1.8.0_141
export HADOOP_HOME=/new/software/hadoop-3.3.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
SPARK_MASTER_IP=yts1
SPARK_LOCAL_DIRS=/new/software/hadoop-3.3.0
SPARK_DRIVER_MEMORY=512M
cp slaves.template slaves
vim slaves
yts1
yts2
yts3
vim /etc/profile
export SPARK_HOME=/new/software/spark-3.0.0-bin-hadoop3.2
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
source /etc/profile
将spark目录以及/etc/profile同步到其他机器
运行官方样类
run-example SparkPi 10