Spark Quick Start SBT Installation

install sbt

The method in this article is a bit cumbersome, you can check the latest update on github : Compile spark source code with sbt

linux版本：CentOS6.7
sbt:    0.13.9

• 1
• 2
• 3

Spark does not come with sbt, you need to install sbt manually , my method is to download sbt-launch.jar, and then change the source to domestic source (aliyun), I choose to install sbt in /usr/local/sbt.

$sudo mkdir /usr/local/sbt
$sudo chown -R hadoop /usr/local/sbt   #username is hadoop.
$cd /usr/local/sbt
$mkdir sbtlaunch   #store sbt-launch.jar

• 1
• 2
• 3
• 4

1. Download sbt-launch.jar and store it in /usr/local/sbt/sbtlaunch

$cd /usr/local/sbt/sbtlaunch
$wget https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar -O ./sbt-launch.jar   #download sbt-launch.jar
$unzip -q ./sbt-launch.jar #解压
$ 

• 1
• 2
• 3
• 4

2. You need to modify the ./sbt/sbt.boot.properties file, and modify the [repositories] to the following:

That is, an image of aliyun-nexus is added.

$cd /usr/local/sbt/sbtlaunch
$vim ./sbt/sbt.boot.properties

• 1
• 2

Modified as follows

[repositories]
  local
  aliyun-nexus: http://maven.aliyun.com/nexus/content/groups/public/
  jcenter: https://jcenter.bintray.com/
  typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
  maven-central

• 1
• 2
• 3
• 4
• 5
• 6
• 7

3. Delete the original sbt-launch. file, and then repackage

$rm ./sbt-launch.jar           #delete the old jar
$jar -cfM ./sbt-launch.jar .   #create new jar 

• 1
• 2

4. Create a sbt script file in the /usr/local/sbt directory and give executable permissions to execute sbt-launch.jar

$cd /usr/lcoal/sbt
$vim ./sbt         #create sbt script

• 1
• 2

Add the following:

SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
java $SBT_OPTS -jar `dirname`/sbt-launch.jar "$@"    #dirname为路径名 

• 1
• 2

my dirname ： /usr/local/sbt/sbtlaunch

5. After saving, add executable permissions to ./sbt:

$chmod u+x ./sbt

• 1

6. Finally check whether sbt is available

$./sbt sbt-version

• 1

Please wait patiently for this step, and don't doubt my internet speed in China. The author waited for about 10 minutes when the first message appeared. As long as I get the following version information, there is no problem:

7. Create a hello world project, and compile, package, and run.

7.1 The directory structure of hello world is:

code show as below:

$mkdir ~/simpleapp
$mkdir ~/simpleapp/src
$mkdir ~/simpleapp/src/main
$mkdir ~/simpleapp/src/main/scala
$vim SimpleApp.scala   #内容如下

• 1
• 2
• 3
• 4
• 5

SimpleApp.scala

object SimpleApp{
        def main(args: Array[String]){
                println("Hello World!")
        } 
}

• 1
• 2
• 3
• 4
• 5

7.2 Creating simple.sbt

$cd ~/simpleapp
$vim ./simple.sbt      #内容如下

• 1
• 2

The content of simple.sbt is as follows: a blank line between each line. scala version and spark version information are written according to the installed spark. The author installed spark2.1.0 and scala 2.11.8.

name :="Simple Project"

version :="1.0"

scalaVersion := "2.11.8"

libraryDependencies +="org.apache.spark" %% "spark-core" % "2.1.0"

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

7.3 Configure environment variables, compile and run.

$vim ~/.bashrc 		#在开头添加如下内容：export PATH=/usr/local/sbt:$PATH
$source ~/.bashrc         #使之生效
$cd ~/simpleapp
$sbt compile           #编译，等待很久，天朝龟速
$sbt package           #打包
$/usr/local/spark/bin/spark-submit --class "SimpleApp" ~/sparkapp/target/scala-2.11/simple-project_2.11-1.0.jar   #将生成的jar包通过spark-summit提交到spark中执行

• 1
• 2
• 3
• 4
• 5
• 6

The compile, package and run interface is as follows:

---