Install Spark on Linux CentOS 6.8

1. Install JDK

1. First download the jdk corresponding to the CentOS version, here I downloaded jdk-8u131-linux-i586.tar.gz;

2. Download the jdk to the local, and upload it to the temporary directory opt of CentOS 6.8;

3 .Before installing the jdk downloaded by yourself, you should first check whether the CentOS 6.8 system has its own jdk version. If it exists, it is best to uninstall it;

4. Create a new jdk installation directory, here I am in /usr/local/ Create a new java directory under the command as follows:

mkdir /usr/local/java

  Then copy the jdk-8u131-linux-i586.tar.gz compressed package from the opt directory to the /usr/local/java directory. The command is as follows:

cp jdk-8u131-linux-i586.tar.gz /usr/local/java

5. Unzip the package you just downloaded: jdk-8u131-linux-i586.tar.gz, the command is as follows:

tar -zxvf jdk-8u131-linux-i586.tar.gz

6. After the decompression is complete, edit the profile file

cd /etc
 vi profile

  Add at the end of the profile file

export JAVA_HOME=/usr/local/java/jdk1.8.0_131
export JRE_HOME=/usr/local/java/jdk1.8.0_131/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

  Save and exit

7. Enter source /etc/profile to make the profile file take effect

8. Enter java -version to check the java version, the following information indicates that the installation is successful



2. The
installation method of SCALA is similar to installing JDK
1. Download scala, what I downloaded here is scala-2.12.2.tgz, and upload it to the linux server

2. Create a new scala directory: /usr/local/scala

3. Copy scala-2.12.2.tgz to: /usr/local/scala, and decompress it

4. Add in the /etc/profile file:
  SCALA_HOME=/usr/local/scala/scala-2.12.2
  PATH=$PATH:${SCALA_HOME}/bin

5. Enter source /etc/profile to make the profile file effective

6. Enter scala


3.
Install Spark 1. Download Spark, what I downloaded here is spark-2.2.0-bin-hadoop2.7.tgz

2. Create a new spark directory: /usr/local/spark

3. Add spark-2.2 .0-bin-hadoop2.7.tgz is copied to: /usr/local/spark, and decompressed

4. Add in the /etc/profile file:
  SPARK_HOME=/usr/local/spark/spark-2.2.0-bin-hadoop2.7
  PATH=$PATH:${SPARK_HOME}/bin

5. Enter source /etc/profile to make the profile file take effect

6. Modify the spark configuration
  to enter spark -1.2.0-bin-hadoop2.4/conf
  Copy the template file:
  cp spark-env.sh.template spark-env.sh
  cp slaves.template slaves

7. Edit spark-env.sh and add:
  export JAVA_HOME=/usr /local/java/jdk1.8.0_131
  export SCALA_HOME=SCALA_HOME=/usr/local/scala/scala-2.12.2
  export SPARK_MASTER_IP=172.20.0.204
  export SPARK_WORKER_MEMORY=1g
  export HADOOP_CONF_DIR=/usr/local/spark/spark-2.2. 0-bin-hadoop2.7

8. Enter source spark-env.sh to make the spark-env.sh file take effect

9. Try to see if spark is installed successfully
  Enter the spark installation directory and enter: spark-shell

  Try the file read operation: