01 of them installed spark in CentOS7

is a new generation of large data spark assembly comprising sparkcore, sparksql, sparkstreaming, mllib, graphx the like, a great role in the actual production, and so on may interact with HDFS, Hbase, hive. This article explains how to install the spark in CentOS7 in.

1 systems, software and premise constraints

• CentOS 7 64 workstations of the machine ip is 192.168.100.200, host name danji, the reader is set according to their actual situation
• hadoop has been installed and started
  https://www.jianshu.com/p/b7ae3b51e559
• -bin-2.2.1-the Spark hadoop2.7, scala-2.12.2
  the Spark and scala download link: https://pan.baidu.com/s/1c_skDYabCRSkS5hRUB6lFQ
  extraction code: A00t
  the Spark-2.2.1-bin-hadoop2. 7.tgz been uploaded to the / root directory 192.168.100.200 under the
  scala-2.12.2.tgz been uploaded to the / root directory under 192.168.100.200
• Permission to remove the effects of the operation, all operations are carried out in order to root

2 operation

• 1. Log in as root to 192.168.100.200 xshell
• 2. Extract

# 进入家目录
cd
# 解压spark
tar -xvf spark-2.2.1-bin-hadoop2.7.tgz
# 解压scala
tar -xvf scala-2.12.2.tgz

• 3. Set Environment Variables

# 进入家目录
cd
# 修改 .bashrc，保存，退出
export JAVA_HOME=/root/jdk1.8.0_152
export SCALA_HOME=/root/scala-2.12.2
export SPARK_HOME=/root/spark-2.2.1-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$SCALA_HOME/bin:$JAVA_HOME/bin:$PATH
# 使生效
source .bashrc

• 4. Modify spark-env.sh

# 进入spark配置目录
cd /root/spark-2.2.1-bin-hadoop2.7/conf
# 拷贝spark-env.sh
cp spark-env.sh.template spark-env.sh
#修改spark-env.sh，保存，退出
export JAVA_HOME=/root/jdk1.8.0_152
export SPARK_HOME=/root/spark-2.2.1-bin-hadoop2.7
export SPARK_MASTER_IP=danji
export SPARK_EXECUTOR_MEMORY=1G
export SCALA_HOME=/root/scala-2.12.2
export HADOOP_HOME=/root/hadoop-2.5.2
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

• 5. Start spark

# 进入spark的启动目录
cd /root/spark-2.2.1-bin-hadoop2.7/sbin
# 启动
./start-all.sh

• 6. verification

# 查看当前进程
jps
# 结果如下就说明启动成功，其中Master，Worker就是spark的进程，其他的是Hadoop的进程。
10193 NameNode
10515 SecondaryNameNode
10310 DataNode
11447 Worker
11369 Master
10798 NodeManager
15790 Jps
10687 ResourceManager

The above is a spark of the installation process in CentOS7 them.

Reproduced in: https: //www.jianshu.com/p/8384ab76e8d4