1.1 Overview
The HA high availability mode of Flink on Yarn first relies on Yarn's own high availability mechanism (ResourceManager high availability), and manages the JobManager through Yarn. When the JobManager fails, Yarn will restart the JobManager. Secondly, when restoring Flink Job, it needs to rely on Checkpoint for recovery, and Checkpoint's snapshot depends on the remote storage: HDFS, so HDFS must also be highly available. At the same time, the metadata information of JobManager also relies on the high availability of HDFS (namenode's High availability, and multi-copy mechanism), and the pointer information of JobManager metadata depends on the high availability of Zookeeper.
The binary package deployment method is used here. There is also a parcel package method
1.2 Advantages of Flink on Yarn
Compared with Standalone mode, Yarn mode has the following advantages:
1. Resources are used on demand to improve the resource utilization of the cluster;
2. Tasks have priorities, and jobs are run according to priorities;
3. Based on the Yarn scheduling system, it can automatically handle the Failover of each role:
Both the JobManager process and the TaskManager process are monitored by Yarn NodeManager;
If the JobManager process exits abnormally, Yarn ResourceManager will reschedule the JobManager to other machines;
If the TaskManager process exits abnormally, the JobManager will receive the message and re-apply for resources from the Yarn ResourceManager and restart the TaskManager.
Chapter 2 How to run Flink on Yarn mode
2.1 Application
Application mode: The short answer is to run the job directly. Yarn will allocate a JobManager for each submitted task. After execution, the entire resource will be released, including JobManager and TaskManager.
Application mode is suitable for larger tasks and tasks that take longer to execute.
2.2 Session
Session mode: In Session mode, Dispatcher and ResourceManager can be reused; JobManager will not be released after the job is executed. Session mode is also called multi-threaded mode, and its characteristic is that resources will always exist and will not be released. When using it, start yarn-session first, and then submit the job. Each time a job is submitted, a JobManager will be assigned.
Session mode is suitable for smaller tasks and tasks with shorter execution time. This mode does not require frequent application and release of resources.
So under normal production conditions, we will choose on Yarn to deploy the Application mode.
Without further ado, start the deployment.
Download the installation package from the official website
After unzipping, enter the directory
Open conf/flink-conf.yaml
Modifications or possible modifications are written below.
Flink on yarn will overwrite several parameters:
jobmanager.rpc.address because the running location of jobmanager in the cluster is not determined in advance, it is the address of am;
taskmanager.tmp.dirs uses the temporary directory given by yarn;
parallelism .default will also be overwritten if the number of slots is specified on the command line.
Create flink’s logical data directory on hadoop in advance
jobmanager.rpc.address: 0.0.0.0 #感觉on yarn改这个没什么用处，随便改改
jobmanager.bind-host: 0.0.0.0 #感觉on yarn改这个没什么用处，随便改改
taskmanager.bind-host: 0.0.0.0 #感觉on yarn改这个没什么用处，随便改改
taskmanager.host: 0.0.0.0 #感觉on yarn改这个没什么用处，随便改改
high-availability.storageDir: hdfs:///flink/ha/
state.checkpoints.dir: hdfs://nameservice1/flink-checkpoints
state.savepoints.dir: hdfs://nameservice1/flink-savepoints
rest.address: 0.0.0.0
rest.bind-address: 0.0.0.0
#web.submit.enable: false #允许web提交任务 按需 我不需要
#web.cancel.enable: false #允许web取消任务 按需 我不需要
还有一些针对kafka zookeeoer的kerberos配置 我这边用不着 大同小异。另外这个配置的用处是提交任务之前不用kinit ，我这已经习惯了kinit 所以也用不着 都是字面意思很好配
# security.kerberos.login.use-ticket-cache: true
# security.kerberos.login.keytab: /path/to/kerberos/keytab
# security.kerberos.login.principal: flink-user
jobmanager.archive.fs.dir: hdfs:///flink/completed-jobs/
historyserver.web.address: 0.0.0.0
historyserver.archive.fs.dir: hdfs:///flink/completed-jobs/
添加一行
classloader.check-leaked-classloader: false
Then you can run the command test
kinit your kerberos user. If it is not an hdfs user, you need to configure permissions on /user on hdfs, because temporary files will be output under /user/{username}/.flink/
Set hadoop environment variables from the command line
export HADOOP_CLASSPATH=`hadoop classpath`
I configured the flink environment variable. If you have not configured it, use bin/flink. Run it in the flink installation directory and use the official example to run it.
flink run -m yarn-cluster ./examples/batch/WordCount.jar
does not report a bunch of errors
This means the installation was successful.
You can check flink tasks in the yarn interface and cdh interface.
If you test flink session mode, or run sql-client
A session task must be started first
yarn-session.sh -s 2 -jm 1024 -tm 2048 -nm test1 -d -tm indicates the memory size of each TaskManager
-s indicates the number of slots for each TaskManager
-d indicates running as a background program
Started a flink resident cluster named test1
Some parameters are as follows
-n,--container <arg> 表示分配容器的数量（也就是 TaskManager 的数量）。
-D <arg> 动态属性。
-d,--detached 在后台独立运行。
-jm,--jobManagerMemory <arg>：设置 JobManager 的内存，单位是 MB。
-nm,--name：在 YARN 上为一个自定义的应用设置一个名字。
-q,--query：显示 YARN 中可用的资源（内存、cpu 核数）。
-qu,--queue <arg>：指定 YARN 队列。
-s,--slots <arg>：每个 TaskManager 使用的 Slot 数量。
-tm,--taskManagerMemory <arg>：每个 TaskManager 的内存，单位是 MB。
-z,--zookeeperNamespace <arg>：针对 HA 模式在 ZooKeeper 上创建 NameSpace。
-id,--applicationId <yarnAppId>：指定 YARN 集群上的任务 ID，附着到一个后台独立运行的 yarn session 中Then submit the job
flink run ./example/batch/WordCount.jarThis completes the test in two ways
Don't forget to distribute the environment variables configured in the package and profile to all nodes. Convenient to submit tasks on any node
Finally, start the history web ui. If it is a resident cluster such as session, it will output a web ui of port 8173. It will prompt at the end of the page output.
bin/historyserver.sh start Just access ip:port. The specific port depends on your configuration.
View Flink Yarn Application
yarn application -listStopFlink Yarn Application
yarn application -kill appIDIf you want to use flink sql-client, you must start a resident cluster. Flink sql relies on a running flink cluster. Both standalone and yarn session can be used.
sql-client.sh
建表
跑select等语句即可In the flink web interface, first run a flink real-time program, then view the application in yarn and click in.
When the session is started, the submitted task must specify the type, otherwise it will be submitted directly to the session.
./bin/flink run-application -t yarn-application ./examples/streaming/TopSpeedWindowing.jar
想指定id的话可以用list参数
./bin/flink list -t yarn-application -Dyarn.application.id=application_XXXX_YY
用list就必须指定 不然报错
No cluster id was specified. Please specify a cluster to which you would like to connect