Flink on Yarn Trilogy Part Two: Deployment and Setup

This article is the second in a series of "Flink on Yarn Trilogy". The previous article "One of Flink on Yarn Trilogy: Preparation" has prepared the required machines and files, and you can deploy CDH and Flink;

Full text link

Execute ansible script to deploy CDH and Flink (ansible computer)

Enter the ~ / playbooks directory of the ansible computer , and after the preparation of the previous article, the directory should be the following content:
Check whether the ansible remote operation of the CDH server is normal. Run the ansible deskmini -a "free -m" command to display the memory information of the CDH server under normal conditions, as shown below:
Execute the command to start deployment: ansible-playbook cm6-cdh5-flink1.7-single-install.yml
The entire deployment process involves time-consuming operations such as online installation and file transfer, so please be patient (about half an hour). If you exit with errors during deployment (such as network problems), you only need to repeat the above command, ansible guarantees the operation Idempotency;
The successful deployment is shown below:

Restart the CDH server

Since the settings of selinux and swap are modified, the operating system needs to be restarted to take effect, so please restart the CDH server;

Execute ansible script to start CDH service (ansible computer)

Wait for the CDH server to restart successfully;
Log in to ansible computer and enter ~ / playbooks directory;
Execute the script to initialize the database and start the CDH: ansible-playbook cdh-single-start.yml
After the startup is complete, the following information is output:
Log in to the CDH server with ssh, and execute this command to observe the start of the CDH service: tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log , when you see the content in the red box below, it means that it is started Finished, you can log in with a browser:

Settings (browser operation)

Now that the CDH service has been started, you can operate it through the browser:

Browser access: http://192.168.50.134:7180, as shown below, the account password is admin :
All the way to the next, select the 60-day trial version on the select version page:
Select the host page to see the CDH server (deskmini):
On the page for selecting the CDH version, please select 5.16.2-1 in the red box below :
Go to the page where Parcel is installed. Since the offline parcle package is uploaded in advance, the download progress becomes 100% instantly. At this time, please wait for the completion of distribution, decompression, and activation:
Next, there are some recommended operations. Here you can skip the red box as shown below:
Next is the page for selecting a service. I chose a custom service, and then selected three items: HDFS, YARN, and Zookeeper, which can meet the needs of running Flink:
On the host selection page, select the CDH server:
Next is the database settings page. The content you fill in must be consistent with the following figure , that is, the host name is localhost , the database, user, and password of Activity Monitor are all amon , and the database, user, and password of Reports Manager are all rman . These The content has been fixed in the ansible script, and the filling here must be consistent:
On the page for setting parameters, please set according to the actual situation of your hard disk. I have enough space under the / home directory, so the storage location is changed to / home directory:
Wait for the service to start:
The start of each service is completed:

YARN settings

The default YARN parameters are very conservative, and some settings need to be made to successfully execute Flink tasks:

Click the red box below to enter the YARN management page:
As shown in the figure below, check the value of the parameter yarn.nodemanager.resource.cpu-vcores , the value must be greater than 1, otherwise YARN does not allocate resources to perform the task after submitting the Flink task, (if your CDH server is a virtual machine, when the CPU only has When single core, this parameter will be set to 1, the solution is to increase the number of virtual machine CPU cores, and then modify this parameter):
yarn.scheduler.minimum-allocation-mb : the minimum memory that can be applied for a single container, I set it to 1G
yarn.scheduler.maximum-allocation-mb : the maximum memory that can be applied for a single container, I set it to 8G
yarn.nodemanager.resource.memory-mb : the maximum available memory of the node, I set it to 8G
The values of the above three parameters are based on the background of my CDH server with 32G memory, please adjust according to your own hardware resources;
After setting up, restart the YARN service, the operation is as shown in the figure below:

At this point, the deployment and settings have been completed, and the Flink on Yarn environment is available. In the next article, we will submit Flink tasks in this environment to experience Flink on Yarn ;