Big Data - Play with Data - Installation and Deployment of FLINK (Yarn Mode)

1. Why use Flink on Yarn HA mode

By default, Flink has only one JobManager, which will lead to a single point of failure. Using JobManager HA, the cluster can recover from a single point of failure, thereby avoiding a single point of failure. We can configure Flink cluster HA under Standalone or Flink on Yarn cluster ( high availability). In fact, the high availability of Flink on Yarn is mainly realized by using Yarn's task recovery mechanism. Yarn is recommended for production environments.

First of all, when the cluster is running, there may be many cluster instances including MapReduce, Spark, Flink, etc. If they are all based on on Yarn, resource allocation can be completed, reducing the maintenance of a single instance cluster and improving the utilization of the cluster.

Secondly, Flink is a big data computing framework, not a resource scheduling framework, which is not its strong point; therefore, professional frameworks should be allowed to do professional things, and integration with other resource scheduling frameworks is more reliable. In the current big data ecology, the most widely used resource management platform in China is YARN. How Flink is integrated and deployed on the powerful YARN platform. Overall, the deployment process on YARN is: the client submits the Flink application to Yarn's ResourceManager, and Yarn's ResourceManager will apply for a container from Yarn's NodeManager. On these containers, Flink deploys instances of JobManager and TaskManager, thus starting the cluster. Flink will dynamically allocate TaskManager resources according to the number of Slots required by the jobs running on JobManger.

Moreover, the installation and deployment of Flink on Yarn mode is actually not much to do. The normal steps are: upload the binary package, decompress it, change the file name, and configure environment variables.

Furthermore, Fink on Yarn has two memory management modes.

Centralized memory management mode: Initialize a Flink cluster in Yarn, open up specified resources, and then all the Flink Jons we submit will be in this Flink yarn-session, that is to say, no matter how many jobs are submitted, these jobs will share the resources in yarn at the beginning resources requested in . This Flink cluster will reside in the Yarn cluster unless manually stopped.

Memory job management mode [recommended use]: In Yarn, each time a job is submitted, a new Flink cluster will be created. The tasks are independent of each other, do not affect each other, and are easy to manage. The cluster created after the task execution is completed will also disappear.

2. Flink on Yarn HA mode deployment

2.1. Basic environment construction

2.1.1. Clone a virtual machine and create a hadoop user

Hadoop100, hadoop101, hadoop102, hadoop103, hadoop104 five virtual machines (see virtual machine environment for virtual machine configuration ) and supplements

Log in as the root user, create a hadoop user (useradd hadoop),
modify the hadoop password (passwd hadoop)
and grant root privileges to hadoop users
Modify the /etc/passwd file, find the following line, and change the user ID to 0, as shown below:

admin:x:500:500:admin:/home/tommy:/bin/bash

Change to:

admin:x:0:500:admin:/home/tommy:/bin/bash

2.1.2. Modify the virtual machine hostname

[root@hadoop100 ~]# vi /etc/hostname 

Take effect after reboot

2.1.3. Modify virtual machine hosts

[root@hadoop100 ~]# vi /etc/hosts

2.1.4. Set the password-free login of the virtual machine

Password-free login reference

2.1.5. Turn off the virtual machine firewall

[root@hadoop100 ~]# systemctl disable firewalld

[root@hadoop100 ~]#  chkconfig iptables off

2.1.6. Catalog planning

Planning installation directory: /home/hadoop/apps
Planning data directory: /home/hadoop/data
Note: apps and data folders need to be created separately

2.1.7. Install and configure JDK 1.8

https://www.oracle.com/java/technologies/downloads/#java8
Download jdk-8u371-linux-x64.tar.gz
and upload the installation package to hadoop100 /home/hadoop/apps directory, if not, create it.
decompress

[root@hadoop100 apps]# tar -zxvf jdk-8u371-linux-x64.tar.gz

change name

[root@hadoop100 apps]# mv jdk1.8.0_371 jdk1.8

Configure JAVA_HOME

[root@hadoop100 apps]# vi /etc/profile

add at the end

export JAVA_HOME=/home/hadoop/apps/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin

[root@hadoop100 apps]# source /etc/profile

[root@hadoop100 apps]# java -version

openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

Distribute the directory to the other four nodes and let the environment take effect

[root@hadoop100 apps]# scp -r /home/hadoop/apps/ hadoop101:/home/hadoop/apps/

[root@hadoop100 apps]# scp -r /etc/profile hadoop101:/etc/

[root@hadoop101 ~]# source /etc/profile

2.2. Install hadoop cluster environment

Hadoop nodes: hadoop100, hadoop101, hadoop102

2.2.1. Download the Hadoop installation package

Download address https://hadoop.apache.org/releases.html
Binary has been edited, source needs to
be edited Copy hadoop-2.10.1.tar.gz to the app directory, hadoop users

unzip

[root@hadoop100 apps]# tar -zxvf hadoop-2.10.1.tar.gz

2.2.2. Modify Hadoop related configuration files

[root@hadoop100 apps]# cd hadoop-2.10.1/etc/hadoop/

Configure hadoop-env.sh

[root@hadoop100 hadoop]# vi hadoop-env.sh

export JAVA_HOME=/home/hadoop/apps/jdk1.8
export HADOOP_LOG_DIR=/home/hadoop/data/hadoop_repo/logs/hadoop

configure yarn-env.sh

[root@hadoop100 hadoop]# vi yarn-env.sh

export JAVA_HOME=/home/hadoop/apps/jdk1.8
export YARN_LOG_DIR=/home/hadoop/data/hadoop_repo/logs/yarn

Configure core-site.xml

[root@hadoop100 hadoop]# vi core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop100:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/home/hadoop/data/hadoop_repo</value>
        </property>
</configuration>

Configure hdfs-site.xml

[root@hadoop100 hadoop]# vi hdfs-site.xml

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>2</value>
          <description>HDFS 的数据块的副本存储个数, 默认是3</description>
     </property>
	 <property>
          <name>dfs.secondary.http.address</name>
          <value>hadoop100:50090</value>
          <description>secondarynamenode 运行节点的信息，和 namenode 不同节点</description>
     </property>   
</configuration>

Configure yarn-site.xml

[root@hadoop100 hadoop]# vi yarn-site.xml

<configuration>
       <property>
       <name>yarn.resourcemanager.hostname</name>
             <value>hadoop100</value>
       </property>
        
       <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
            <description>YARN 集群为 MapReduce 程序提供的 shuffle 服务</description>
       </property>
</configuration>

Configure mapred-site.xml
to rename

[root@hadoop100 hadoop]# mv mapred-site.xml.template mapred-site.xml

[root@hadoop100 hadoop]# vi mapred-site.xml

<configuration>
        <property>
             <name>mapreduce.framework.name</name>
             <value>yarn</value>
        </property>
</configuration>

Configure slaves

[root@hadoop100 hadoop]# vi slaves 

hadoop101
hadoop102

2.2.3. Distribution installation directory

Copy the modified Hadoop installation directory on hadoop100 to the other two nodes hadoop101, hadoop102

[root@hadoop100 apps]# scp -r hadoop-2.10.1 hadoop101:/home/hadoop/apps/

2.2.4, HDFS formatting

Format on hadoop100 in
the /home/hadoop/apps/hadoop-2.10.1 directory

[root@hadoop100 hadoop-2.10.1]# bin/hdfs namenode -format

2.2.5, start hadoop

[root@hadoop100 hadoop-2.10.1]# sbin/start-all.sh 

2.2.6. Verification

[root@hadoop100 hadoop-2.10.1]# jps
7248 NameNode
7444 SecondaryNameNode
7597 ResourceManager
7869 Jps

[root@hadoop101 hadoop]# jps
7204 Jps
6667 DataNode
7080 NodeManager

[root@hadoop102 hadoop]# jps
6697 DataNode
6810 NodeManager
6942 Jps

2.2.7. Configure the HADOOP_HOME environment variable

For the convenience of subsequent operations, the HADOOP_HOME configured hadoop100
is added on the basis of the previous JAVA_HOME

export JAVA_HOME=/home/hadoop/apps/jdk1.8
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.10.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

2.3. Install ZooKeeper cluster

Configure ZooKeeper clusters in hadoop100, hadoop101, hadoop102

2.3.1. Download the ZooKeeper installation package

https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.1/apache-zookeeper-3.8.1-bin.tar.gz
needs to download the bin package, apache-zookeeper-3.8.1- bin.tar.gz

Hadoop user login (with root privileges)

[root@hadoop100 apps]#  tar -zxvf apache-zookeeper-3.8.1-bin.tar.gz

2.3.2, configure zookeeper

file rename

[root@hadoop100 apps]# mv apache-zookeeper-3.8.1-bin zookeeper-3.8.1

Configuration file rename

[root@hadoop100 apps]# cd apache-zookeeper-3.8.1-bin/conf

[root@hadoop100 conf]# mv zoo_sample.cfg zoo.cfg 

Configure zoo.cfg
without data, and create a log directory. Note: there can be no spaces after the port

dataDir=/home/hadoop/apps/zookeeper-3.8.1/data
dataLogDir=/home/hadoop/apps/zookeeper-3.8.1/log
server.1=hadoop100:2888:3888
server.2=hadoop101:2888:3888
server.3=hadoop102:2888:3888

Create a myid file and enter the number 0 in the file
to enter the data directory

[root@hadoop100 data]# vi myid
1

For the convenience of management, add zookeeper environment variables on hadoop100 (based on the previous environment variables)

export JAVA_HOME=/home/hadoop/apps/jdk1.8
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.10.1
export ZOOKEEPER_HOME=/home/hadoop/apps/zookeeper-3.8.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin

Environment variables take effect

[root@hadoop100 apps]# source /etc/profile

Copy the configured installation package to hadoop101, hadoop102 nodes

[root@hadoop100 apps]# scp -r zookeeper-3.8.1/  hadoop101:/home/hadoop/apps/

Modify the myid on hadoop101 and hadoop102 nodes
to be 2 and 3 respectively
. Distribute the modified environment variable files to hadoop101 and hadoop102
Note: Change the cluster environment variable to dedicated. If there are other applications installed, be careful that the environment variable is overwritten, you can only use it alone Revise

[root@hadoop100 apps]# scp /etc/profile hadoop101:/etc/

source Let environment variables take effect on hadoop101 and hadoop102

2.3.3. Start the ZooKeeper cluster

Start ZooKeeper on hadoop100, hadoop101, hadoop102 respectively

[root@hadoop100 apps]# zkServer.sh start

5040 SecondaryNameNode
6914 QuorumPeerMain
6949 Jps
4838 NameNode
5196 ResourceManager

2.3.4. Verification

Seeing the QuorumPeerMain process indicates that the startup is successful

[root@hadoop100 conf]# zkServer.sh status

Checked on the three hosts respectively, and found that
ZooKeeper has a leader and two follower nodes

2.4 Installing Flink on Yarn HA

The principle of high availability in YARN mode is different from that in standalone mode (Standalone). In Standalone mode, multiple JobManagers are started at the same time, one is the leader and the others are standby. When the leader hangs up, one of the others will become the leader. The high availability of YARN is to start only one Jobmanager. When the Jobmanager hangs up, YARN will start another one. Therefore, the high availability is actually achieved by using the number of retries of YARN.
Note that when deploying on YARN, Flink manages the high-availability.cluster-id configuration parameter. Flink sets this to the ID of the YARN application by default. You should not override this parameter when deploying HA clusters on YARN. Cluster ID is used to distinguish multiple HA clusters of HA backend (such as Zookeeper). Overriding this configuration parameter will cause multiple YARN clusters to interact with each other

2.4.1. Compared with Yarn's Hadoop environment modification

We first set the maximum number of attempts to submit the application in yarn-site.xml in hadoop100, and set the value of not checking the virtual memory, otherwise an error will be reported
in the /home/hadoop/apps/hadoop-2.10.1/etc/hadoop directory Down

[root@hadoop100 hadoop]# vi yarn-site.xml

<property>
	<name>yarn.resourcemanager.am.max-attempts</name>
	<value>4</value>
	<description>
		The maximum number of application master execution attempts.
	</description>
</property>
<!-- 设置不检查虚拟内存的值，不然内存不够会报错 -->
	<property>
		<name>yarn.nodemanager.pmem-check-enabled</name>
		<value>false</value>
	</property>
	<property>
		<name>yarn.nodemanager.vmem-check-enabled</name>
		<value>false</value>
	</property>

Distribute the modified configuration file to hadoop101, hadoop102

[root@hadoop100 hadoop]# scp yarn-site.xml hadoop101:/home/hadoop/apps/hadoop-2.10.1/etc/hadoop/

Restart hadoop and zookeeper cluster

2.4.1. Download the Flink installation package

https://www.apache.org/dyn/closer.lua/flink/flink-1.17.0/flink-1.17.0-bin-scala_2.12.tgz

2.3.2. Unzip the Flink installation package

First unzip a new Flink installation package on hadoop100

[root@BIGDATA ~] tar -zxvf  flink-1.17.0-bin-scala_2.12.tgz

[root@BIGDATA ~]cd flink-1.17.0/

2.3.3. Configuring Flink

Add environment variables on the basis of the previous

[root@hadoop100 apps]# vi /etc/profile

export JAVA_HOME=/home/hadoop/apps/jdk1.8
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.10.1
export ZOOKEEPER_HOME=/home/hadoop/apps/zookeeper-3.8.1
export HADOOP_CLASSPATH=`hadoop classpath`
export FLINK_HOME=/home/hadoop/apps/flink-1.17.0export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$FLINK_HOME/bin

Environment takes effect

[root@hadoop100 apps]# source /etc/profile

Configure flink-conf.yaml

[root@hadoop100 conf]# vi flink-conf.yaml

high-availability.type: zookeeper

# JobManager元数据保留在文件系统storageDir中 指向此状态的指针存储在ZooKeeper中
high-availability.storageDir: hdfs://hadoop100:9000/flink/ha-yarn
#
# # Zookeeper集群 修改自己的集群
high-availability.zookeeper.quorum: hadoop100:2181,hadoop101:2181
#
# # 在zookeeper下的根目录
high-availability.zookeeper.path.root: /flink_yarn
#
# # zookeeper节点下的集群ID 该节点下放置了集群所需的所有协调数据 多个flink集群连接同一套zookeeper集群需要配置各自不同的集群ID，官方建议这个配置最好去掉，因为在 Yarn（以及Mesos）模式下，cluster-id 如果不配置的话，会配置成 Yarn 上的 Application ID ，从而可以保证唯一性。 (不配，如果配的话，启动两个及以上job会报错)
#high-availability.cluster-id: /default_yarn
#
# # 单个flink job重启次数 必须小于等于yarn-site.xml中Application Master配置的尝试次数
yarn.application-attempts: 4
#
#如果 ZooKeeper 在 Kerberos 的安全模式下运行
#
## default is "zookeeper". If the ZooKeeper quorum is configured
## with a different service name then it can be supplied here.
#
#zookeeper.sasl.service-name: zookeeper
#
## default is "Client". The value needs to match one of the values
## configured in "security.kerberos.login.contexts".
#zookeeper.sasl.login-context-name: Client

Add on the basis of the previous environment variables

export HADOOP_CLASSPATH=`hadoop classpath`

[root@hadoop100 ~]# vi /etc/profile

export HADOOP_CLASSPATH=`hadoop classpath`
export JAVA_HOME=/home/hadoop/apps/jdk1.8
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.10.1
export ZOOKEEPER_HOME=/home/hadoop/apps/zookeeper-3.8.1
export HADOOP_CLASSPATH=`hadoop classpath`
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin

Send the decompressed package after configuration to hadoop101, hadoop102

2.3.4. Start the Flink cluster

There are two main ways to start a Flink on yarn:

Start a Yarn session (initialize a Flink cluster resident inside the yarn cluster and keep running)

Submit and run Flink jobs directly on yarn (each time a job is submitted to the yarn cluster, and the yarn cluster opens up resources to initialize a Flink collection)

Flink provides 3 modes running on yarn, namely Application Mode (1.12), Session-Cluster and Per-Job-Cluster modes.

Application Mode will start the cluster on Yarn, and the main function of the application jar package (the main function of the user class) will be executed on the JobManager. As soon as the application execution ends, the Flink cluster will be shut down immediately. You can also manually stop the cluster. The difference with Per-Job-Cluster: In Application Mode, the user's main function is executed in the cluster (job manager).

Official suggestion:
For production needs, we recommend using Per-job or Application Mode, because they provide better isolation for applications!

A job corresponds to a Flink cluster. Each job submitted will apply to yarn separately for resources according to its own situation until the job execution is completed. The failure of a job will not affect the normal submission and operation of the next job. Exclusive Dispatcher and ResourceManager, accepting resource applications on demand; suitable for large-scale and long-running jobs. Each submission will create a new flink cluster. The tasks are independent of each other and do not affect each other, which is convenient for management. The cluster created after the task execution is completed will also disappear.

The principle of high availability in Yarn mode is different from that in Standalone mode.
In Standalone mode, multiple Jobmanagers are started at the same time, one is the leader and the other is standby. When the leader fails, one of the others will become the leader.
The high availability of yarn is to start only one Jobmanager at the same time. When the Jobmanager hangs up, yarn will start one again. In fact, high availability is achieved by using the number of retries of yarn.

Start the Flink on yarn HA cluster on hadoop100
in the /home/hadoop/apps/flink-1.17.0 directory

[root@hadoop100 flink-1.17.0]# bin/yarn-session.sh -n 2

2.3.5. Verification

http://192.168.220.100:8088/cluster

https://blog.51cto.com/u_12902538/5394081
https://blog.csdn.net/wuxintdrh/article/details/106885847