CDH + Kylin Trilogy Part Two: Deployment and Setup

This article is the second in a series of "CDH + Kylin Trilogy". The previous article "CDH + Kylin Trilogy: Preparations" has prepared the required machines and documents, and CDH and Kylin can be deployed;

Execute ansible script to deploy CDH and Kylin (ansible computer)

Enter the ~ / playbooks directory of the ansible computer , and after the preparation of the previous article, the directory should be the following content:
Check whether the ansible remote operation of the CDH server is normal. Run the ansible deskmini -a "free -m" command to display the memory information of the CDH server under normal conditions, as shown below:
Execute the command to start deployment: ansible-playbook cm6-cdh5-kylin264-single-install.yml
The entire deployment process involves time-consuming operations such as online installation and transmission of large files. Please wait patiently (about half an hour). If you exit with errors during deployment (such as network problems), you only need to repeat the above command, ansible guarantees the operation Idempotency;
The successful deployment is shown below:

Restart the CDH server

Since the settings of selinux and swap are modified, the operating system needs to be restarted to take effect, so please restart the CDH server;

Execute ansible script to start CDH service (ansible computer)

Wait for the CDH server to restart successfully;
Log in to ansible computer and enter ~ / playbooks directory;
Execute the script to initialize the database and start the CDH: ansible-playbook cdh-single-start.yml
After the startup is complete, the following information is output:
Log in to the CDH server with ssh, and execute this command to observe the start of the CDH service: tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log , when you see the content in the red box below, it means that it is started Finished, you can log in with a browser:

Settings (browser operation)

Now that the CDH service has been started, you can operate it through the browser:

Browser access: http://192.168.50.134:7180, as shown below, the account password is admin :
All the way to the next, select the 60-day trial version on the select version page:
Select the host page to see the CDH server (deskmini):
On the page for selecting the CDH version, please select 5.16.2-1 in the red box below:
Go to the page where Parcel is installed. Since the offline parcle package is uploaded in advance, the download progress becomes 100% instantly. At this time, please wait for the completion of distribution, decompression, and activation:
Next, there are some recommended operations. Here you can skip the red box as shown below:
Next is the page for selecting a service. I chose a custom service, and then selected HBase, HDFS, Hive, Hue, Oozie, Spark, YARN, Zookeeper, which can meet the needs of running Kylin:
On the host selection page, select the CDH server:
The next page is the database settings that you fill in the content must be consistent with the following figure , the host name is localhost , Hive database, user, password are the hive, Activity Monitor database, user, password are amon, Reports Manager The database, user, and password are all rman, the database, user, and password of Oozie Server are all oozie, and the database, user, and password of Hue are all hue. These contents have been fixed in the ansible script, and the filling here must be consistent :
On the page for setting parameters, please set according to the actual situation of your hard disk. I have enough space under the / home directory, so the storage location is changed to / home directory:
Wait for the service to start:
The start of each service is completed:

HDFS settings

As shown in the red box below, there is a problem with the HDFS service:
Click the red exclamation mark in the picture above to see the details of the problem. The following picture is a common copy problem:
The operation is as shown in the figure below. On the parameter setting page of HDFS, set the value of dfs.replication to 1 (only one data node):
After the above settings, the number of copies has been adjusted to 1, but the number of copies of the existing files has not been synchronized, you need to re-set, SSH login to the CDH server;
Execute the command su-hdfs to switch to the hdfs account, and then execute the following command to complete the copy number setting:

hadoop fs -setrep -R 1 /

Go back to the webpage and restart the HDFS service, as shown below:
After restart, HDFS service is normal:

YARN settings

The default YARN parameters are very conservative, and some settings need to be made to successfully execute Spark tasks:

Enter the YARN management page;
As shown in the figure below, check the value of the parameter yarn.nodemanager.resource.cpu-vcores , the value must be greater than 1, otherwise YARN does not allocate resources to perform the task after submitting the Spark task, (if your CDH server is a virtual machine, when the CPU When single core, this parameter will be set to 1, the solution is to increase the number of virtual machine CPU cores, and then modify this parameter):
yarn.scheduler.minimum-allocation-mb : the minimum memory that can be applied for a single container, I set it to 1G
yarn.scheduler.maximum-allocation-mb : the maximum memory that can be applied for a single container, I set it to 8G
yarn.nodemanager.resource.memory-mb : the maximum available memory of the node, I set it to 8G
The values of the above three parameters are based on the background of my CDH server with 32G memory, please adjust according to your own hardware resources;
After setting, restart the YARN service, the operation is shown in the figure below:

Spark settings (CDH server)

You need to prepare a directory and related jars in the Spark environment, otherwise Kylin will start with an error (prompt spark not found, set SPARK_HOME, or run bin / download-spark.sh ), SSH into the CDH server as root , execute the following command:

mkdir $SPARK_HOME/jars \
&& cp $SPARK_HOME/assembly/lib/*.jar $SPARK_HOME/jars/ \
&& chmod -R 777 $SPARK_HOME/jars

Start Kylin (CDH server)

Log in to the CDH server via SSH and execute su-hdfs to switch to the hdfs account;
According to the official recommendation, first execute the command to check the environment: $ KYLIN_HOME / bin / check-env.sh
If the check passes, the console output is as follows:
启动 Kylin ：$ KYLIN_HOME / bin / kylin.sh start
The console outputs the following content indicating that Kylin is successfully started:

Log in to Kylin

Browser access: http://192.168.50.134:7070/kylin, as shown below, account ADMIN , password KYLIN (account and password are in uppercase):
The login is successful and you can use it:

At this point, the deployment, setup, and startup of CDH and Kylin have been completed, and Kylin is available. In the next article, we will run the official demo of Kylin in this environment to experience Kylin;