The spark non-HA cluster structures (Standalone Mode)
About sparkHA have a good blog here, you can refer!
The first spark archive into the specified path; then arranged;
The first step: to enter the spark at the conf / directory, there is a configuration file for spark-env.sh.template file, the file name changed to spark-env.sh (default reading is not spark-env.sh to read the template file with the suffix)
mv spark-env.sh.template spark-env.sh
Step 2: Edit the above modified configuration file name of the good; at the bottom, add the following code block (the simplest configuration) e
export JAVA_HOME=$JAVA_HOME (你的jdk路径)
export SPARK_MASTER_HOST=【HostName】（需要开启master的节点的主机名）
export SPARK_MASTER_PORT=7077 (默认的端口)
The third step: will be configured conf file distribution to other nodes spark
scp -r $SPARK_HOME/conf hostName:/$SPARK_HOME/
After the distribution on it; this is the case is to start on the master node master is to start the master host SPARK_MASTER_HOST the configuration items brought back!
Build on the spark HA zookeeper
Based on the above non-HA to build a spark;
Note: First, make sure your host to build on a good zookeeper cluster, as well as JDK environment;
In my example here:
three hosts: master, slave01, slave02
goal:
Master: Master (ALIVE), worker
slave01: Master (STANDBY), worker
slave02: worker
on my three hosts that have installed the zookeeper's ;
or proceed to the next conf / directory to modify configuration files spark-env.sh
首先将以上的SPARK_MASTER_HOST这项配置删除（三台节点都需要删除），然后添加一项配置，如下面所示：
export JAVA_HOME=/usr/local/src/jdk1.8.0_201/
export SPARK_MASTER_PORT=7077
export SPARK_DAEMON_JAVA_OPTS="-Dspark.depoly.recoveryMode=ZOOKEEPER
-Dspark.depoly.zookeeper.url=master:2181,slave01:2181,slave02:2181
-Dspark.depoly.zookeeper.dir=/sparkHA"
Note: -Dspark.depoly.zookeeper.dir behind this one path can be customized
Start spark cluster;
launch execute command on the master node:
bin/start-all.sh (在master节点执行；)
bin/start-master.sh (在备份节点上启动master进程)
Here is complete sparkHA configuration, you can access by 8080 this port to master web interface;
Question: When encountered in building HA spark of over more problems, and sometimes it's time ALIVE STANDBY automatic switching can not switch; sometimes state master web interface on both nodes are ALIVE; but did not find We found just re-build, so here successfully built a good after this one the easiest method for building; the cause of the problem
