Under ubuntu install Hadoop and Spark Development Environment
surroundings:
Ubuntu16.04.2 64 Wei
JDK 1.8.0_171
scala-2.10.6
spark-1.6.0 hadoop2.6
IntelliJ IDEA 2018.1.4
Note that each software version compatibility issues
IntelliJ IDEA correspondence scala: website https://plugins.jetbrains.com/plugin/1347-scala can view the scala corresponding version, then download it
1. Installation jdk:
Official website http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
In the / usr / lib a new java folder, download the jdk placed in this folder
Extracting tar -zxvf jdk-8u171-linux-x64.tar.gz
2 set environment variables, execute
$sudo vim /etc/profile
In the back of the file with the following contents:
export JAVA_HOME=/usr/lib/java/ jdk1.8.0_171
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=.:${JAVA_HOME}/bin:$PATH
3. Run $ source / etc / profile
4. Test, execute $ java -version
2. Installation scala
Official website http://www.scala-lang.org/download/2.10.6.html
Download: scala-2.10.6.tgz
Installation: 1. Create a scala file in / usr / lib folder, to download the jdk placed in this folder
Decompression sudo tar -zxvf scala-2.10.6.tgz
2. Set the environment variables, execute
$sudo vim /etc/profile
In the back of the file with the following contents:
export SCALA_HOME = / usr / lib / scale / scale-2.10.6
export PATH=${SCALA_HOME}/bin:$PATH
3. Run $ source / etc / profile
4. Test, execute $ scala -version
3. Download the installation package spark
Official website: http://spark.apache.org/downloads.html
Download spark-1.6.0-bin-hadoop2.6.tgz and unzip
Installation: 1. Create a spark files in / usr / lib folder, to download the jdk placed in this folder
Decompression sudo tar -zxvf download spark-1.6.0-bin-hadoop2.6.tgz
2. Set the environment variables, execute
$sudo vim /etc/profile
In the back of the file with the following contents:
export SPARK_HOME=/usr/lib/ spark/spark-1.6.0-bin-hadoop2.6
export PATH=${SPARK_HOME}/bin:$PATH
3. Run $ source / etc / profile
4. The test is performed $ spark-shell
4. Download and install Intellij IDEA community version and install it. When you install scala plugins are installed.
Ubuntu: a command line Open idea, enter the location of the package, enter: bin / idea.sh
Official website: https://www.jetbrains.com/idea/download/#section=linux
Installation: 1. Install scala sdk.
FileàSettingsàPluginsà
Search directly on plugins in Scala, and then install it if you do not have Internet environment, not to force or speed. Also go directly to http://plugins.jetbrains.com/plugin/?idea&id=1347 ZIP package manually download the plugin, download the manual, pay special attention to match the version number intellij idea of the version number, it must be used with this machine, or can not install after downloading. After the download is complete, in the figure above, click on "Install plugin from disk ...", you can select the zip package of plug-ins.
2. Add the spark of the jar package
In the IDE File -> project Structure -> Libraries - next> "+" in the / usr / lib directory, and then enter the spark-XXX-bin-hadoopXX you install spark when unpacked, in the lib directory, select spark- assembly-XXX-hadoopXX.jar
3. Add the scala jar package
In the IDE File -> project Structure -> Libraries -> "+", and then select the location of the installation scala lib folder, and then click Apply, OK.
5. Export jar package
1. Configuration
Intellij idea -->file -->project structure -->artifacts -->+ -->jar -->from ... --->main class选择主类 -->ok
(Spark-assembly.jar package is relatively large, you can delete, big data platform has been installed) -> ok
2.jar packet output
build -> build artifacts -> build -> Output artifacts files in folders inside out