Spark(6)Upgrade to 1.0.2 Version again with YARN
Download the prebuilt version
>wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.2-bin-hadoop2.tgz
Setup and Make hadoop2 running on my VMs
http://sillycat.iteye.com/blog/2090186
Prepare the file on hadoop
>hdfs dfs -mkdir /user/sillycat
>hdfs dfs -put /opt/spark/log.txt /user/sillycat/
Login on the shell.
>MASTER=spark://ubuntu-master1:7077 bin/spark-shell
>val file = sc.textFile("hdfs://ubuntu-master1:9000/user/sillycat/log.txt")
>file.first()
Error Message:
Server IPC version 9 cannot communicate with client version 4
Solution:
Version error, I am using spark-hadoop1 to connect to hadoop 2.4.1
It works in the shell.
Go on and configure the YARN.
>sbin/start-dfs.sh
>sbin/start-yarn.sh
>sbin/mr-jobhistory-daemon.sh start historyserver
YARN is running now. Then we can get info from these URL
http://ubuntu-master1:50070/dfshealth.html#tab-overview
http://ubuntu-master1:8088/cluster/nodes
http://ubuntu-master1:19888/jobhistory
Running Spark Shell on YARN
Change the configuration file of spark
HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
>MASTER=yarn-client bin/spark-shell
Submit the task as follow
>bin/spark-submit --class com.cloudera.sparkwordcount.SparkWordCount --master yarn /Users/carl/work/current/simplesparkapp/target/sparkwordcount-0.0.1-SNAPSHOT.jar hdfs://ubuntu-master1:9000/user/sillycat/log.txt 2
Spark works great on YARN
>bin/spark-submit --class com.sillycat.spark.app.ClusterComplexJob --master yarn /Users/carl/work/sillycat/sillycat-spark/target/scala-2.10/sillycat-spark-assembly-1.0.jar book1
The sample project is in sillycat-spark
Even the standalone cluster is working
>bin/spark-submit --class com.sillycat.spark.app.FindWordJob --master spark://ubuntu-master1:7077 /Users/carl/work/sillycat/sillycat-spark/target/scala-2.10/sillycat-spark-assembly-1.0.jar book1
The command to start the master and slave
>sbin/start-master.sh
>bin/spark-class org.apache.spark.deploy.worker.Worker spark://ubuntu-master1:7077
Configuration on master1
>cat conf/spark-env.sh
#!/usr/bin/env bash
export SPARK_LOCAL_IP=ubuntu-master1
#export SPARK_EXECUTOR_MEMORY=1G
export SPARK_MASTER_IP=ubuntu-master1
export SPARK_WORKER_MEMORY=1024M
Configuration on slave1
>cat conf/spark-env.sh
#!/usr/bin/env bash
export SPARK_LOCAL_IP=ubuntu-slave1
#export SPARK_EXECUTOR_MEMORY=1G
export SPARK_MASTER_IP=ubuntu-master1
export SPARK_WORKER_MEMORY=1024M
Tips
Spark Job runs
>bin/spark-submit --class com.cloudera.sparkwordcount.SparkWordCount --master local /Users/carl/work/current/simplesparkapp/target/sparkwordcount-0.0.1-SNAPSHOT.jar /opt/spark/README.md 2
Error Message
java.lang.OutOfMemoryError: Java heap space
Solution:
Change the Memory configuration from here
>vi bin/spark-class
References:
http://spark.apache.org/docs/latest/running-on-yarn.html
https://github.com/snowplow/spark-example-project
http://blog.cloudera.com/blog/2014/04/how-to-run-a-simple-apache-spark-app-in-cdh-5/
http://parambirs.wordpress.com/2014/05/20/running-spark-1-0-0-snapshot-on-hadoopyarn-2-4-0/
http://parambirs.wordpress.com/2014/05/20/install-hadoopyarn-2-4-0-on-ubuntu-virtualbox/
http://parambirs.wordpress.com/2014/05/20/building-and-running-spark-1-0-0-snapshot-on-ubuntu/