Yarn mode of Spark's three operating modes

Yarn mode

1. Mode overview

The Spark client directly connects to Yarn, without the need to build a Spark cluster. There are two modes of yarnclient and yarn-cluster,主要区别在于：Driver 程序的运行节点。

yarn-client: The Driver program runs on the client, which is suitable for interaction and debugging. I hope to see the output of the app immediately

yarn-cluster: The Driver program runs on AP (APPMaster) started by RM (ResourceManager) and is suitable for production environments.

Operation mode diagram:
Insert picture description here

Two, installation and use

(1) Modify the hadoop configuration file yarn-site.xml and add the following content:

[atguigu@hadoop102 hadoop]$ vi yarn-site.xml
        <!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
          <property>
                 <name>yarn.nodemanager.pmem-check-enabled</name>                	
                 <value>false</value>
                                        
          </property>
         <!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
          <property>
                 <name>yarn.nodemanager.vmem-check-enabled</name>
                 <value>false</value>
          </property>

(2) Modify spark-env.sh and add the following configuration:

[atguigu@hadoop102 conf]$ vi spark-env.sh
YARN_CONF_DIR=/opt/module/hadoop-2.7.2/etc/hadoop

(3) Distribution configuration file

[atguigu@hadoop102 conf]$ xsync /opt/module/hadoop2.7.2/etc/hadoop/yarn-site.xml
[atguigu@hadoop102 conf]$ xsync spark-env.sh

(4) Execute a program

[atguigu@hadoop102 spark]$ bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
./examples/jars/spark-examples_2.11-2.1.1.jar \
100

note:在提交任务之前需启动 HDFS 以及 YARN 集群。

Three, log view

(1) Modify the configuration file spark-defaults.conf and
add the following content:

spark.yarn.historyServer.address=hadoop101:18080
spark.history.ui.port=4000

(2) Restart the spark history service

[atguigu@hadoop102 spark]$ sbin/stop-history-server.sh
stopping org.apache.spark.deploy.history.HistoryServer
[atguigu@hadoop102 spark]$ sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to
/opt/module/spark/logs/spark-atguiguorg.apache.spark.deploy.history.HistoryServer-1-hadoop102.out

(3) Submit the task to Yarn for execution

[atguigu@hadoop102 spark]$ bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
./examples/jars/spark-examples_2.11-2.1.1.jar \
100

(4) View log on Web page
Insert picture description here