Using spark in big data class

9. Install Spark Spark
1. Log in to the cloud server as the hadoop user.
2. Preliminary work: Download the spark and scala installation packages to the local directory /home/hadoop/.
$cd (back to Home directory)
$scp [email protected]:/home/hadoop/scala-2.11.8.tar.gz /home/hadoop/
$scp [email protected]:/home/hadoop/spark- 2.0.1-bin-hadoop2.7.tgz /home/hadoop/
3. Unzip the scala and spark installation packages.
$tar -zxvf scala-2.11.8.tar.gz
$tar -zxvf spark-2.0.1-bin-hadoop2.7.tgz
4. Rename the extracted folders to scala and spark, and migrate to /usr/ local directory.
$sudo mv scala-2.11.8 /usr/local/scala
$sudo mv spark-2.0.1-bin-hadoop2.7 /usr/local/spark
5. Go to the spark configuration directory and modify the parameters.
$cd /usr/local/spark/conf
$cp spark-env.sh.template spark-env.sh
$cp slaves.template slaves
Modify the slaves file and add the same slave node address as the hadoop slaves file.
-------------------------------------------------- -----
6. Copy the scala and spark directories of the master server to the slave1 and slave2 servers.
$cd /usr/local
$scp -r scala slave1:/home/hadoop/ $scp -r spark slave1:
/home/hadoop/
$scp -r scala slave2:/home/hadoop/
$scp -r spark slave2:/ home/hadoop/
7. Switch to slave1 and slave2 servers, and move the scala and spark directories to /usr/local/
(slave1)$sudo mv /home/hadoop/scala /usr/local/
(slave1)$sudo mv /home /hadoop/spark /usr/local/
(slave2)$sudo mv /home/hadoop/scala /usr/local/
(slave2)$sudo mv /home/hadoop/spark /usr/local/
8. Go back to the master server, run spark.
$ cd / usr / local / spark cd / usr / local / sparkcd / usr / local / sparkcd / usr / local / sparkcd / usr / local / spark cd / usr / local / sparkcd / usr / local / sparkcd / usr / local / spark cd / usr / local / sparkcd / usr / local / sparkcd / usr / local / spark cd / usr / local / sparkcd / usr / local /
spark $ sbin / start sbin / startsbin / startsbin / startsbin / start sbin / startsbin / start -all.shall.shall.shall.shall.sh