This article describes spark set up in the windows of local mode
Spark basic mode of operation can be divided into two types:
Local Mode
Driver program that is run only on the machine
Cluster Mode
That Dirver program will run in a cluster, specific to cluster mode, it can be divided into spark cluster, MESOS, YARN so on.
As a beginner into the pit, it is naturally transferred through the most convenient local mode. Spark installation steps in Mac, Linux does not have to say, basically no additional problems, but the installation of windows or to take note of.
The basic steps:
1. to spark the official website https://spark.apache.org/downloads.html download with hadoop version
Here we must note that the download with hadoop version, that does not need to be installed locally deploy hadoop cluster.
Download, unzip, to the bin directory to perform spark-shell.cmd, will be reported at this time
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': ........
Caused by: java.lang.reflect.InvocationTargetException: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':.................
Error reason is because there is no download Hadoop windows executable file.
2. Download the corresponding version of hadoop binary file
Download Link https://github.com/steveloughran/winutils
After downloading the corresponding version of HADOOP_HOME added to the environment variable, bin also added to the path in
3. At this time will report a The root scratch dir: / tmp / hive on HDFS should be writable Current permissions are:. Rw-rw-rw- (on Windows)
This is the / tmp / hive permissions error, this time to step in 2 downloaded hadoop / bin below, perform winutils.exe chmod 777 E: \ tmp \ hive
Here prefix E: \ express my spark decompression procedures in the E disk. \ Tmp \ hive this is fixed folder that has been created.
This folder is obvious temporary files directory of the spark.
So far, LOCAL mode set up is completed. After the spark-shell.cmd run without error, the IDE spark running program code manner, not being given.
Spark successful run flag:
Original author: of Micro
Original link: https: //www.jianshu.com/p/4dd9d550eeb0