Installation and operation of macOS spark

1. Visit the official website spark https://spark.apache.org/ , then click download connection https://spark.apache.org/downloads.html

2. After entering download interface, select the type of Pre-buil for Apache Hadoop2.7 and later, so do not have a pre-installed Hadoop

3. Click the connection after download spark, download spark-2.4.3-bin-hadoop2.7.tgz file, there are about 230M

4. Open the command line terminal, the spark-2.4.3-bin-hadoop2.7.tgz file to the / usr / local

sudo mv spark-2.4.3-bin-hadoop2.7.tgz /usr/local

5. decompress the file

sudo tar -zvxf spark-2.4.3-bin-hadoop2.7.tgz

6. Set Environment Variables

sudo nano ~/.bash_profile

7. Add the following configuration in the configuration file save and exit

export SPARK_HOME=/usr/local/spark-2.4.3-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin
export PYSPARK_PYTHON=python3

8. take effect profile

source ~/.bash_profile

9. Installation pyspark

pip3 install pyspark

10. Start pyspark, enter the interactive command window

Reference connection:

[1]https://medium.com/luckspark/installing-spark-2-3-0-on-macos-high-sierra-276a127b8b85