1) local mode (single-machine mode)
2) standalone mode (pseudo-distributed mode)
3) yarn-client mode (the driver runs on the client side, the client's running status can be viewed through port 4040, and the client participates in the whole process Scheduling)
4) yarn-cluster mode (the driver runs on the server and needs to be viewed through port 8088 of yarn-cluster, and the client can be closed after submitting the job)
Switch to the SPARK_HOME directory
cd /usr/local/spark-2.1.0-bin-hadoop2.7
1. local mode
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local examples/jars/spark-examples_2.11-2.1.0.jar
2. Standalone mode
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://master.hadoop.zjportdns.gov.cn:7077 examples/jars/spark-examples_2.11-2.1.0.jar
3. In yarn-client mode
3.4, you need to start hadoop and set environment variables.
echo -e "export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.3/etc/hadoop" >> /etc/profile source /etc/profile
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client examples/jars/spark-examples_2.11-2.1.0.jar
Visit: http://master.hadoop.zjportdns.gov.cn:4040/jobs
4. yarn-cluster mode
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster examples/jars/spark-examples_2.11-2.1.0.jar
Visit: http://master.hadoop.zjportdns.gov.cn:8088/cluster
