YARN operation mode_environment preparation
The deployment process on YARN is: the client submits the Flink application to Yarn's ResourceManager, and Yarn's ResourceManager will apply for a container from Yarn's NodeManager. On these containers, Flink will deploy instances of JobManager and TaskManager, thereby starting the cluster. Flink will dynamically allocate TaskManager resources according to the number of Slots required by the jobs running on JobManger.
[atguigu@node001 flink-1.17.0]$ source /etc/profile.d/my_env.sh
[atguigu@node001 flink-1.17.0]$ myhadoop.sh s
Input Args Error...
[atguigu@node001 flink-1.17.0]$ myhadoop.sh start
================ 启动 hadoop集群 ================
---------------- 启动 hdfs ----------------
Starting namenodes on [node001]
Starting datanodes
Starting secondary namenodes [node003]
--------------- 启动 yarn ---------------
Starting resourcemanager
Starting nodemanagers
--------------- 启动 historyserver ---------------
[atguigu@node001 flink-1.17.0]$ jpsall
================ node001 ================
9200 JobHistoryServer
8416 NameNode
8580 DataNode
9284 Jps
8983 NodeManager
================ node002 ================
3892 ResourceManager
3690 DataNode
4365 Jps
4015 NodeManager
================ node003 ================
3680 DataNode
3778 SecondaryNameNode
3911 NodeManager
4044 Jps
[atguigu@node001 flink-1.17.0]$
YARN running mode_session mode
This command is a script for starting Apache Flink's YARN session (session). The meaning of each option and parameter is as follows:
yarn-session.sh: This is the script provided by Apache Flink to start a session on YARN.-d: This is an option that starts the session in detached mode. In detached mode, the session runs in the background and the script returns immediately.-nm test: This is another option to specify the name of the session. In this example, the name of the session is set to "test".
Taken together, the purpose of this command is to start an Apache Flink session named "test" on YARN, running in detached mode. Once started, the session runs in the background and the command line prompt returns immediately, allowing you to continue with other operations.
[atguigu@node001 bin]$ ./yarn-session.sh --help
[atguigu@node001 bin]$ ./yarn-session.sh
[atguigu@node001 bin]$ ./yarn-session.sh -d -nm test
YARN operation mode_session mode stop
Single job mode deployment:
In the YARN environment, since there is an external platform for resource scheduling, we can also submit a single job directly to YARN to start a Flink cluster.
Stop job:
