Install Spark 2.2 stand-alone on CentOS 7

Spark 2.2 needs to run in the java environment, is any version available? not necessarily. I saw on the spark official website that the running environment is jdk8+, so I installed jdk9, but the installation was not successful in the JDK 9 environment. Running spark's ./bin/spark-shell will report that a certain configuration value is missing (it is estimated that it is JDK version issue). During this installation, jdk 8 was selected, and it worked normally after installation. The following is a brief record of the installation process, which is relatively simple.

1. Download Jdk, http://www.oracle.com/technetwork/java/javase/downloads/index.html, select JDK 8 to download

2. Install JDK 8. rpm -ivh jdk-8u151-linux-x64.rpm

3. Set JAVA_HOME. Add the following two lines at the end of /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_151
export PATH=$PATH:$JAVA_HOME/bin

Set the java home according to the actual directory you installed.

4. Download Spark. http://spark.apache.org/downloads.html

Select version 2.2.0, and select the default package type "Pre-built for Apache Hadoop 2.7 and later"

5. Copy to the directory to be installed, unzip

tar zxvf spark-2.2.0-bin-hadoop2.7.tgz

6. Run Spark

./sbin/start-master.sh

Check the logs in logs for errors

7. Run spark-shell

./bin/spark-shell

If there are no errors, the installation is successful.

In addition, if you are using Python, you can install Python and then run ./bin/pyspark. I successfully installed it on Python 3.6.3. The following is a simple operation to install Python 3.6

8. Download Python 3.6

https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tar.xz

9. Install the xz decompression tool

yum install xz

10. Unzip the installation package

xz -d Python-3.6.3.tar.xz
tar xvf Python-3.6.3.tar

11. Compilation and installation can be found in Brother Buwei's blog http://www.cnblogs.com/cloud-80808174-sea/p/6902934.html

It should be noted that Python 2.7 is no longer supported after Spark 2.10, so after installing Python 3.6.3, you need to set 3.6.3 as the default, and how to change it is written in the blog of Brother Not Afraid.