flink on yarn 集群环境部署

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/QXC1281/article/details/85717473

设置yarn的配置目录

vim /etc/profile
添加
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

source /etc/profile

修改配置

jobmanager.rpc.port: 6123
# 高可用模式
high-availability: zookeeper
# zookeeper机器信息,多个,号分割
high-availability.zookeeper.quorum: 10.4.243.134:2181
# zooKeeper节点根目录，其下放置所有集群节点的namespace
high-availability.zookeeper.path.root: /flink
# 集群id
high-availability.cluster-id: /cluster_one
# 恢复一个JobManager挂掉所需的元数据,存放到hdfs中
high-availability.storageDir: hdfs:///flink/recovery/
state.backend: filesystem
state.checkpoints.dir: hdfs:///flink/checkpoints
state.savepoints.dir: hdfs:///flink/savepoints
rest.port: 8081

启动yarn

kafka-demo-1.0-SNAPSHOT.jar 一个可以运行的jar包就可以了
不用启动flink集群，资源管理全部交给yarn

flink run -m yarn-cluster -yn 2 -ynm kafka -d -yjm 1024m -ytm 2048m -p 4 kafka-demo-1.0-SNAPSHOT.jar

效果图

点击ApplicationMaster 查看flink web ui

HA设置

yarn增加配置

    <property>
        <name>yarn.resourcemanager.am.max-attempts</name>
        <value>4</value>
        <description>在重启时，尝试的最大次数</description>
    </property>

设置flink-conf.yml

# jobmanager.rpc.address: 10.4.243.134
jobmanager.rpc.port: 6123
#jobmanager.heap.size: 1024m
#taskmanager.heap.size: 1024m
#taskmanager.numberOfTaskSlots: 4

# 高可用模式
high-availability: zookeeper
# zookeeper机器信息,多个,号分割
high-availability.zookeeper.quorum: 10.4.243.134:2181
# zooKeeper节点根目录，其下放置所有集群节点的namespace
high-availability.zookeeper.path.root: /flink
# 集群id
high-availability.cluster-id: /cluster_one
# 恢复一个JobManager挂掉所需的元数据,存放到hdfs中
high-availability.storageDir: hdfs:///flink/recovery/
state.backend: filesystem
state.checkpoints.dir: hdfs:///flink/checkpoints
state.savepoints.dir: hdfs:///flink/savepoints
rest.port: 8081
io.tmp.dirs: /appdata/flink/io/tmp
# 添加配置重试次数
yarn.application-attempts: 4

效果图

杀死进程

[root@10 target]# jps -l
121434 org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint
[root@10 target]# kill -9 121434

重启了一个

错误记录

内存申请不足

修改yarn-site.xml,添加如下配置

   <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
        <description>Whether virtual memory limits will be enforced for containers</description>
    </property>