Super simple CDH6 deployment and experience (stand-alone version)

Why is it super simple

With the help of ansible, most of the content of CDH6 deployment is simplified, and the probability of manual operation errors is reduced. Today's actual content is to run ansible script on a computer (Apple or Linux operating system) with ansible installed, remote operation A CentOS server, deploy CDH6 on it, and verify whether the deployment is successful.

ansible learning

If you want to understand ansible, please refer to "Ansible2.4 Installation and Experience"

Why deploy standalone CDH6

It is mainly used as a learning and development environment for big data technology and is not suitable for production;

Brief description of actual combat

The content of this actual combat: deployment, startup, verification, the whole process is shown in the following figure:
Insert picture description here

Full text outline

This article consists of the following chapters:

Environmental information
download file;
File placement;
CDH machine settings;
ansible parameter setting;
deploy;
Restart the CDH server
start up;
Set up
Fix the problem
Experience

Environmental information

The operation process of this actual combat is shown in the following figure. Install the ansible2.9 version of the MabBook Pro computer as ansible server, execute the playbook script, remotely operate a CentOS server to complete the deployment and startup of CDH6:
Insert picture description here
blue background above The computer can be an Apple operating system or a Linux operating system. For a computer with a yellow background to run CDH6, it must be CentOS 7.7 operating system (I'm sorry, I have limited conditions, I haven't tried other systems)

The version information of the environment involved in the actual combat is as follows:

ansible server: macOS Catalina 10.15 (CentOS7.7 was also successfully tested)
CDH server: CentOS Linux release 7.7.1908
cm version: 6.1.0
parcel version: 6.1.1
jdk version: 8u191

Download file (ansible server)

All the files used in this actual combat are shown in the following table:

Numbering	file name	Introduction
1	jdk-8u191-linux-x64.tar.gz	Linux version jdk installation package
2	mysql-connector-java-5.1.34.jar	JDBC driver for MySQL
3	cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm	cm server installation package
4	cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm	cmemon installation package
5	cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm	cm agent installation package
6	CDH-6.1.1-1.cdh6.1.1.p0.875250-el7.parcel	CDH application offline installation package
7	CDH-6.1.1-1.cdh6.1.1.p0.875250-el7.parcel.sha	CD verification code for offline installation package of CDH application
8	hosts	The remote host configuration used by ansible, which records the information of the CDH6 server
9	ansible.cfg	Configuration information used by ansible
9	ansible.cfg	Configuration information used by ansible
10	cdh-single-install.yml	Ansible script used when deploying CDH
11	cdh-single-start.yml	The ansible script used when starting CDH for the first time

Download address of the above 11 files:

jdk-8u191-linux-x64.tar.gz : Oracle's official website is available. In addition, I packaged and uploaded jdk-8u191-linux-x64.tar.gz and mysql-connector-java-5.1.34.jar to csdn. Can be downloaded at one time, address: https://download.csdn.net/download/boling_cavalry/12098987
mysql-connector-java-5.1.34.jar : maven central warehouse is available. In addition, I package and upload jdk-8u191-linux-x64.tar.gz and mysql-connector-java-5.1.34.jar to csdn You can download it once, address: https://download.csdn.net/download/boling_cavalry/12098987
cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm：https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm：https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm：https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
CDH-6.1.1-1.cdh6.1.1.p0.875250-el7.parcel：https://archive.cloudera.com/cdh6/6.1.1/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250-el7.parcel
CDH-6.1.1-1.cdh6.1.1.p0.875250-el7.parcel.sha : https://archive.cloudera.com/cdh6/6.1.1/parcels/CDH-6.1.1-1.cdh6. 1.1.p0.875250-el7.parcel.sha256 (After downloading, change the extension from .sha256 to .sha)
hosts, ansible.cfg, cdh-single-install.yml, cdh-single-start.yml : these four files are saved in my GitHub repository, the address is: https://github.com/zq2599/blog_demos, this There are multiple folders in it, and the above files are in the folder named ansible-cdh6-single , as shown in the red box below:

File placement (ansible server)

If you have downloaded the above 11 files, please place them according to the following locations so that the deployment can be completed successfully:

Create a new folder named playbooks under the home directory: mkdir ~ / playbooks
Put these four files into the playbooks folder: hosts, ansible.cfg, cdh-single-install.yml, cdh-single-start.yml
Create a new subfolder named cdh6 in the playbooks folder;
Put these seven files into the cdh6 folder (that is, the remaining seven): jdk-8u191-linux-x64.tar.gz, mysql-connector-java-5.1.34.jar, cloudera-manager-server-6.1. 0-769885.el7.x86_64.rpm, cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm, cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm, CDH-6.1. 1-1.cdh6.1.1.p0.875250-el7.parcel, CDH-6.1.1-1.cdh6.1.1.p0.875250-el7.parcel.sha
After the placement, the directory and files are as shown in the figure below. Again, the folder playbooks must be placed in the home directory (ie: ~ / ):

CDH server settings

In this actual combat, the CDH server hostname is deskmini and the IP address is 192.168.50.134 . The following operations are required:

Please ensure that the CDH server can log in via SSH (user name + password);
SSH to the machine where CDH is deployed;
Check that the / etc / hostname file is correct, as shown below:
Modify the / etc / hosts file, configure your own IP address and hostname, as shown in the red box below (it turns out that this step is very important, if you do not do it, it may cause you to be stuck in the "allocation" stage during deployment, see the agent log Show that the progress of agent download parcel has been zero percent):

ansible parameter settings (ansible server)

The operation setting of ansible parameter setting is very simple, just configure the information of the machine where the CDH is deployed, including the IP address, login account, password, etc., modify the ~ / playbooks / hosts file, the content is as follows, you need to modify it according to your own situation deskmini, ansible_host, ansible_port, ansible_user, ansible_password:

[cdh_group]
deskmini ansible_host=192.168.50.134 ansible_port=22 ansible_user=root ansible_password=888888

Deployment (ansible server)

Enter the ~ / playbooks directory;
Check whether the ansible remote operation of the CDH server is normal. Run the ansible deskmini -a "free -m" command to display the memory information of the CDH server under normal conditions, as shown below:
Execute this command to start deployment: ansible-playbook cdh-single-install.yml
The entire deployment process involves time-consuming operations such as online installation and file transfer, so please be patient (about half an hour). I encountered network problems during deployment and failed to exit. After the network is normal, I can re-execute the above operations, ansible guarantee Idempotency of operation;
The successful deployment is shown below:

Restart the CDH server

Since the settings of selinux and swap are modified, the operating system needs to be restarted to take effect, so please restart the CDH server;

Start (ansible server)

Wait for the CDH server to restart successfully;
Log in to the ansible server and enter the ~ / playbooks directory;
Execute this command to start initializing the database and then start CDH: ansible-playbook cdh-single-start.yml
After the startup is complete, the following information is output:

Settings (Webpage)

CDH has been started, and the CDH server provides web services to the outside world, which can be operated through a browser:

Browser access: http://192.168.50.134:7180, as shown below, the account password is admin :
All the way to the next, select the 60-day trial version on the select version page:
Select the host page to see deskmini:
Select the CDH version in the red box below, because the corresponding offline package has been copied to the local warehouse of CM, no need to download:
The download is completed instantly, waiting for distribution, decompression, and activation:
Select the service page, I chose Data Engineering here, because spark is required:
Select the machine's page, select all deskmini:
The database setting page, please keep the same as the picture below , the database host is localhost , the name, user name and password of each database are the same, namely: hive, amon, rman, oozie, hue
The parameter setting page, please adjust the storage path appropriately according to the situation of the disk. For example, my / home directory has enough space, and I have changed to the / home directory here :
Wait for the startup to complete:
After waiting for the startup to complete, as shown below:

At this point, all services have been started, but there are two minor problems that need to be repaired;

Fix HDFS issues

The overall situation of the service is as shown in the figure below. If there is a problem with the HDFS service, click the icon in the red box:
Click on the red box below:
The details of the fault are shown in the following figure, which is a common problem of insufficient copies:
As shown in the figure below, modify the HDFS configuration dfs.replication from 3 to 1, and then save the changes:
Restart the service:
After the above settings, the number of copies has been adjusted to 1, but the number of copies of the existing files has not been synchronized, you need to re-set up, SSH login to the computer deskmini;
Execute the command vi / etc / passwd to find the configuration of the account hdfs, as shown in the red box below, a shell like / sbin / nologin will cause the switch to the hdfs account to fail:
Change the content of the above red box to / bin / bash , after modification, as shown in the red box below:
Execute the command su-hdfs to switch to the hdfs account, then execute the following command to complete the copy number setting:

hadoop fs -setrep -R 1 /

The service is all normal:

Adjust YARN parameters to avoid spark-shell startup failure

By default, the memory allocated by YARN to the container is too small, which causes the spark-shell to fail to start. You need to adjust the memory parameters related to YARM:
On the YARN configuration page, adjust the values of the two parameters yarn.scheduler.maximum-allocation-mb and yarn.nodemanager.resource.memory-mb . I will change the values of these two parameters to 8G here (please follow your own Adjust the actual hardware configuration of the computer), as shown below:
Restart YARN:
Before executing the spark-shell command, first execute the command su-hdfs to switch to the hdfs account;
This time finally successfully entered the spark-shell interactive mode:

At this point, CDH6 deployment, startup, and settings have been completed, and then experience the big data service;

Experience HDFS and Spark

Next, run a Spark task, the classic WordCount:

Prepare a text file with English content, you can download this file: https://raw.githubusercontent.com/zq2599/blog_demos/master/files/GoneWiththeWind.txt
Log in to SSH and switch to the hdfs account;
Create HDFS folder:

hdfs dfs -mkdir /input

Upload the text file to the / input directory:

hdfs dfs -put ./GoneWiththeWind.txt /input

Run the spark-shell command to start a worker;
Enter the following command to complete a WorkCount task, 192.168.50.134 is the IP address of deskmini:

sc.textFile("hdfs://192.168.50.134:8020/input/GoneWiththeWind.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).saveAsTextFile("hdfs://192.168.50.134:8020/output")

After execution, download the result file:

hdfs dfs -get /output/*

The above command downloads the spark task result files part-00000 and part-00001 to the local machine. Use the vi command to view the file, as shown in the following figure, it can be seen that WorkCount is successfully executed:
View the historical task on the browser, the address is: http://192.168.50.134:18088, you can see the details of this task: So

far, CDH6 deployment, settings, and experience have been completed, if you are setting up your own learning or development Environment, I hope this article can give you some reference.

In-depth customization

Although the entire actual combat avoids a lot of manual operations in the traditional deployment process, the disadvantages are also obvious: all paths, file names, and service versions are fixed, and no settings can be made. Although ansible also supports variables, if there are too many variables, Will cause you trouble, so if you have the need to modify the version or path, it is recommended that you modify the contents of cdh-single-install.yml and cdh-single-start.yml by yourself. All the files and version information are in it.