How to get CDH MapReduce jobs Hive SQL statement being executed or has been completed in

First, the purpose of the preparation of documents

Second, get through xml configuration file YARN execution of the job

Third, to get through Cloudera Manager

Fourth, by Cloudera Manager API interface to obtain

Address reprint: https://cloud.tencent.com/developer/article/1078818

First, the purpose of the preparation of documents

When we submit to the Hive SQL statement after YARN, sometimes if we want to monitor the implementation of a SQL, you need to view specific SQL statement, if the SQL statement is longer, either through the command interface is YARN YARN 8088 can not see the whole the SQL statements.

YARN 8088 interface is as follows:
Here Insert Picture Description view job details SQL statements can not see the entire
yarn application -list also can not see the whole SQL statement as follows:
This article describes three ways to get the Hive SQL statements MapReduce jobs being executed or has been completed in One is available through MapReduce API xml configuration file to execute the job, the other is viewed directly by Cloudera Manager interface, and the third is to get through the Cloudera Manager API.

test environment

1. Operating system RedHat7.3

2.CM and CDH version 5.13.1

3. Cluster Kerberos enabled

Second, get through xml configuration file YARN execution of the job

1. fayson user login hue execute SQL queries
Here Insert Picture Description
2. Get to the configuration information of the job through the SQL ApplicationID

curl -H "Accept: application/json" -X   \
GET http://ip-172-31-16-68.ap-southeast-1.compute.internal:8088/proxy/application_1519613953021_0029/ws/v1/mapreduce/jobs/job_1519613953021_0029/conf > a.xml

Here Insert Picture Description
(Left and right slide)
3. Filter view a.xml property files hive.query.string

interface to obtain operation by obtaining the job Hive Hive can view the SQL statement corresponding to the information to the HDFS / user / $USER/.staging/$ JOBID / job.xml file.

4. If the job execution completion interface API interface to perform JobHistory

curl -H "Accept: application/json" -X   \
GET http://ip-172-31-16-68.ap-southeast-1.compute.internal:19888/ws/v1/history/mapreduce/jobs/job_1519613953021_0029/conf > b.xml

(About slidably)

Here Insert Picture Description For a complete interface to SQL statements executed by JobHistory of Hive job history API, this information corresponds to the HDFS file /user/history/done/2018/02/26/000000/job_1519613953021_0029_conf.xml

Third, to get through Cloudera Manager

1. Go Yarn "Applications" page
Here Insert Picture Description 2. Select "Hive applications" in the "Search" in the
filter screening applications Hive

Here Insert Picture Description 3. Select a Hive job, click on the arrow you can expand to view the complete SQL, query and look at basic statistics

Fourth, by Cloudera Manager API interface to obtain

1. Run the following command to obtain more information on the job command line

[root@ip-172-31-16-68 ~]# curl -u admin:admin "http://ip-172-31-16-68.ap-southeast-1.compute.internal:7180/api/v16/clusters/luster/services/yarn/yarnApplications"

(Slide left or right)
Here Insert Picture Description to view the complete SQL statement we are running by looking hive_query_string property.