一、问题
kylin版本：3.0.0
CDH版本：6.2.0
问题描述：
几分钟内提交8个build cube任务。top命令看到kylin所在节点cpu飙升到了100，找到对应的进程id，cd /proc/id下，发现都是kylin的进程，kill -9 杀死所有进程。重启一下kylin看看，重启后发现下面的日志。重启后，重新构建一个cube，在yarn页面可以看到build的第一个阶段已经完成，但是kylin界面显示仍然在执行Create Intermediate Flat Hive Table。看kylin的kylin.log，kylin.out，未发现error和exception。
重启时报错：
Dump base package locates at:
/opt/software/kylin/apache-kylin-3.0.0-bin-cdh60/bin/../tomcat/temp/diag_job4210256040870725844/base_2020_11_10_11_42_58/base_2020_11_10_11_42_58.zip
========================================
2020-11-10 11:43:24,720 INFO [close-hbase-conn] hbase.HBaseConnection:138 : Closing HBase connections...
2020-11-10 11:43:24,720 INFO [close-hbase-conn] client.ConnectionImplementation:1775 : Closing master protocol: MasterService
2020-11-10 11:43:24,732 INFO [ReadOnlyZKClient-bigdata002:2181,bigdata003:2181,bigdata001:2181@0x71ae31b0] zookeeper.ZooKeeper:693 : Session: 0x175ac793f450811 closed
2020-11-10 11:43:24,733 INFO [ReadOnlyZKClient-bigdata002:2181,bigdata003:2181,bigdata001:2181@0x71ae31b0-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
Exception in thread "Thread-15" org.apache.kylin.job.lock.zookeeper.exception.ZkReleaseLockException: Error while 87121@bigdata003 trying to unlock /job_engine/global_job_engine_lock
at org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.unlock(ZookeeperDistributedLock.java:265)
at org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.unlockJobEngine(ZookeeperDistributedLock.java:359)
at org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock.unlockJobEngine(ZookeeperJobLock.java:91)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler.shutdown(DefaultScheduler.java:188)
at org.apache.kylin.rest.service.JobService$1.run(JobService.java:142)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.lock.zookeeper.exception.ZkReleaseLockException: Error while 87121@bigdata003 trying to purge /job_engine/global_job_engine_lock
at org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.purgeLockInternal(ZookeeperDistributedLock.java:317)
at org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.unlockInternal(ZookeeperDistributedLock.java:288)
at org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.unlock(ZookeeperDistributedLock.java:259)
... 5 more
Caused by: java.lang.IllegalStateException: Client is not started
at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489)
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:250)
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:244)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
at org.apache.kylin.job.lock.zookeeper.ZookeeperDistributedLock.purgeLockInternal(ZookeeperDistributedLock.java:309)
... 7 more
二、解决
cpu飙升到100%通常来讲是死锁或者死循环，死循环的可能性不大，先排查死锁
1.查看报错中对应的kylin的元数据路径
到zookeeper里找到日志里提到路径
sh ./zkCli.sh
[zk: localhost:2181(CONNECTED) 4] ls /kylin/kylin_metadata/job_engine/global_job_engine_lock
[]
发现下面没东西，这条线是断了
2.查看hive中kylin创建的表是否有锁
hive
show locks
default@kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336 EXCLUSIVE
查看锁的时候，发现果然有kylin的表被锁
查看kylin.out日志
hdfs 88412 1.2 0.2 4178460 621944 pts/3 Sl 11:18 0:18 /usr/java/jdk1.8.0_131/bin/java -Dproc_jar -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Xmx2147483648 -Djava.net.preferIPv4Stack=true -Dlog4j.configurationFile=hive-log4j2.properties -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/bin/../conf/parquet-logging.properties -Dyarn.log.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/libexec/../../hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/lib/native -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/lib/hive-cli-2.1.1-cdh6.2.0.jar org.apache.hadoop.hive.cli.CliDriver --hiveconf hive.query.redaction.rules=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/redaction-rules.json --hiveconf hive.exec.query.redactor.hooks=org.cloudera.hadoop.hive.ql.hooks.QueryRedactor --hiveconf hive.aux.jars.path=file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.2.0.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop-compat.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-client.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-protocol.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-common.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop2-compat.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-server.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core-3.2.0-incubating.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core.jar,file:///usr/share/java/mysql-connector-java.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/auxlib/hive-exec-2.1.1-cdh6.2.0-core.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/auxlib/hive-exec-core.jar -e set mapred.job.name='Create Intermediate Flat Hive Table'; USE default; DROP TABLE IF EXISTS kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336 ( `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_AIR_LINE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_POINTNAME` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_DATE` date ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_ID` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTEINDEX` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DEPARTURE_4_CODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ARRIVAL_4_CODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_NUMBER` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_TYPE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CAPTAIN` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISPATCH` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ENGINES1` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_TYPE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_PROPERTY` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_STATUS` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLG_STATE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAX_DEPART_WEIGHT` int ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_SEAT_NUM` int ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_NUMBER` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACQUIS_DATE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISTANCE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLAN_DEPARTURE_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_DEPARTURE_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_ARRIVAL_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_ARRIVAL_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CALIBRATEDAIRSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ELAPSEDTIMEMIN` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELCONSUMPTION` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELFLOW` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_GROUNDSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_HIGHLAYER` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LATITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LONGITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MACH` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAGNETICHEADING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MINIMUMSAFEALTITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALAIRDISTANCE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALOILCONTENT` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATURE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATUREDEVIATION` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_THEREMAININGDISTANCE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TIMEREMAINING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TRUEHEADING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDDIS` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDFUEL` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDTIME` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_VACUUMSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDCOMMPONENT` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDDIRECTION` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_STATUS` int ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_ALT` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_HEADING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_GS` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_STATICAIRTEMP` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINDIR` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINSPD` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TAS` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WITH_WIND` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELATITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELONGITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TOTALAIRTEMP` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_FF1` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_MACH` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_COUNTRYCODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FIRNAME` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_OVERLAP` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTECODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ORGNUM` string ) STORED AS SEQUENCEFILE LOCATION 'hdfs://Aero/kylin/kylin_metadata/kylin-ac12c0af-5d05-d441-b0ce-4d851d504b4a/kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336'; ALTER TABLE kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336 SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE `kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336` SELECT `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`AIR_LINE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_AIR_LINE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`POINTNAME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_POINTNAME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_DATE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_DATE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_ID` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_ID` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ROUTEINDEX` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTEINDEX` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`DEPARTURE_4_CODE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DEPARTURE_4_CODE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ARRIVAL_4_CODE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ARRIVAL_4_CODE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_NUMBER` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_NUMBER` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLANE_TYPE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_TYPE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`CAPTAIN` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CAPTAIN` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`DISPATCH` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISPATCH` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ENGINES1` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ENGINES1` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_TYPE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_TYPE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_PROPERTY` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_PROPERTY` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_STATUS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_STATUS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLG_STATE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLG_STATE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAX_DEPART_WEIGHT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAX_DEPART_WEIGHT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`SEAT_NUM` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_SEAT_NUM` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLANE_NUMBER` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_NUMBER` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ACQUIS_DATE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACQUIS_DATE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`DISTANCE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISTANCE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLAN_DEPARTURE_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLAN_DEPARTURE_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ACTUAL_DEPARTURE_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_DEPARTURE_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLANE_ARRIVAL_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_ARRIVAL_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ACTUAL_ARRIVAL_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_ARRIVAL_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`CALIBRATEDAIRSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CALIBRATEDAIRSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ELAPSEDTIMEMIN` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ELAPSEDTIMEMIN` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FUELCONSUMPTION` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELCONSUMPTION` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FUELFLOW` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELFLOW` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`GROUNDSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_GROUNDSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`HIGHLAYER` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_HIGHLAYER` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`LATITUDE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LATITUDE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`LONGITUDE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LONGITUDE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MACH` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MACH` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAGNETICHEADING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAGNETICHEADING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MINIMUMSAFEALTITUDE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MINIMUMSAFEALTITUDE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`RESIDUALAIRDISTANCE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALAIRDISTANCE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`RESIDUALOILCONTENT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALOILCONTENT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TEMPERATURE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATURE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TEMPERATUREDEVIATION` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATUREDEVIATION` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`THEREMAININGDISTANCE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_THEREMAININGDISTANCE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TIMEREMAINING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TIMEREMAINING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TRUEHEADING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TRUEHEADING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`USEDDIS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDDIS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`USEDFUEL` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDFUEL` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`USEDTIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDTIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`VACUUMSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_VACUUMSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`WINDCOMMPONENT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDCOMMPONENT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`WINDDIRECTION` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDDIRECTION` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`WINDSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`STATUS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_STATUS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_ALT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_ALT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_HEADING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_HEADING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_GS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_GS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_STATICAIRTEMP` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_STAT
hdfs 79607 2.1 0.3 4229972 843120 pts/3 Sl 11:13 0:38 /usr/java/jdk1.8.0_131/bin/java -Dproc_jar -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Xmx2147483648 -Djava.net.preferIPv4Stack=true -Dlog4j.configurationFile=hive-log4j2.properties -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/bin/../conf/parquet-logging.properties -Dyarn.log.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/libexec/../../hadoop-yarn -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/lib/native -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/lib/hive-cli-2.1.1-cdh6.2.0.jar org.apache.hadoop.hive.cli.CliDriver --hiveconf hive.query.redaction.rules=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/redaction-rules.json --hiveconf hive.exec.query.redactor.hooks=org.cloudera.hadoop.hive.ql.hooks.QueryRedactor --hiveconf hive.aux.jars.path=file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/hive-hbase-handler-2.1.1-cdh6.2.0.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop-compat.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-client.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-protocol.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-common.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop2-compat.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-server.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core-3.2.0-incubating.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core.jar,file:///usr/share/java/mysql-connector-java.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/auxlib/hive-exec-2.1.1-cdh6.2.0-core.jar,file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/auxlib/hive-exec-core.jar -e set mapred.job.name='Create Intermediate Flat Hive Table'; USE default; DROP TABLE IF EXISTS kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336 ( `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_AIR_LINE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_POINTNAME` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_DATE` date ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_ID` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTEINDEX` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DEPARTURE_4_CODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ARRIVAL_4_CODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_NUMBER` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_TYPE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CAPTAIN` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISPATCH` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ENGINES1` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_TYPE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_PROPERTY` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_STATUS` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLG_STATE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAX_DEPART_WEIGHT` int ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_SEAT_NUM` int ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_NUMBER` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACQUIS_DATE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISTANCE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLAN_DEPARTURE_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_DEPARTURE_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_ARRIVAL_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_ARRIVAL_TIME` bigint ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CALIBRATEDAIRSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ELAPSEDTIMEMIN` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELCONSUMPTION` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELFLOW` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_GROUNDSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_HIGHLAYER` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LATITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LONGITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MACH` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAGNETICHEADING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MINIMUMSAFEALTITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALAIRDISTANCE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALOILCONTENT` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATURE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATUREDEVIATION` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_THEREMAININGDISTANCE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TIMEREMAINING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TRUEHEADING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDDIS` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDFUEL` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDTIME` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_VACUUMSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDCOMMPONENT` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDDIRECTION` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDSPEED` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_STATUS` int ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_ALT` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_HEADING` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_GS` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_STATICAIRTEMP` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINDIR` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINSPD` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TAS` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WITH_WIND` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELATITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELONGITUDE` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TOTALAIRTEMP` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_FF1` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_MACH` double ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_COUNTRYCODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FIRNAME` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_OVERLAP` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTECODE` string ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ORGNUM` string ) STORED AS SEQUENCEFILE LOCATION 'hdfs://Aero/kylin/kylin_metadata/kylin-ac12c0af-5d05-d441-b0ce-4d851d504b4a/kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336'; ALTER TABLE kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336 SET TBLPROPERTIES('auto.purge'='true'); INSERT OVERWRITE TABLE `kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336` SELECT `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`AIR_LINE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_AIR_LINE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`POINTNAME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_POINTNAME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_DATE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_DATE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_ID` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_ID` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ROUTEINDEX` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTEINDEX` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`DEPARTURE_4_CODE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DEPARTURE_4_CODE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ARRIVAL_4_CODE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ARRIVAL_4_CODE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_NUMBER` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_NUMBER` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLANE_TYPE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_TYPE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`CAPTAIN` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CAPTAIN` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`DISPATCH` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISPATCH` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ENGINES1` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ENGINES1` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_TYPE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_TYPE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_PROPERTY` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_PROPERTY` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLIGHT_STATUS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_STATUS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FLG_STATE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLG_STATE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAX_DEPART_WEIGHT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAX_DEPART_WEIGHT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`SEAT_NUM` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_SEAT_NUM` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLANE_NUMBER` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_NUMBER` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ACQUIS_DATE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACQUIS_DATE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`DISTANCE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISTANCE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLAN_DEPARTURE_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLAN_DEPARTURE_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ACTUAL_DEPARTURE_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_DEPARTURE_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`PLANE_ARRIVAL_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_ARRIVAL_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ACTUAL_ARRIVAL_TIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_ARRIVAL_TIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`CALIBRATEDAIRSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CALIBRATEDAIRSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`ELAPSEDTIMEMIN` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ELAPSEDTIMEMIN` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FUELCONSUMPTION` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELCONSUMPTION` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`FUELFLOW` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELFLOW` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`GROUNDSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_GROUNDSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`HIGHLAYER` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_HIGHLAYER` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`LATITUDE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LATITUDE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`LONGITUDE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LONGITUDE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MACH` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MACH` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAGNETICHEADING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAGNETICHEADING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MINIMUMSAFEALTITUDE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MINIMUMSAFEALTITUDE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`RESIDUALAIRDISTANCE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALAIRDISTANCE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`RESIDUALOILCONTENT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALOILCONTENT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TEMPERATURE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATURE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TEMPERATUREDEVIATION` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATUREDEVIATION` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`THEREMAININGDISTANCE` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_THEREMAININGDISTANCE` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TIMEREMAINING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TIMEREMAINING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`TRUEHEADING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TRUEHEADING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`USEDDIS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDDIS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`USEDFUEL` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDFUEL` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`USEDTIME` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDTIME` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`VACUUMSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_VACUUMSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`WINDCOMMPONENT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDCOMMPONENT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`WINDDIRECTION` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDDIRECTION` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`WINDSPEED` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDSPEED` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`STATUS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_STATUS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_ALT` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_ALT` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_HEADING` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_HEADING` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_GS` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_GS` ,`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810`.`MAP_STATICAIRTEMP` as `VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_STAT
找到两条相关的命令，仔细查看命令发现kylin第一步（Create Intermediate Flat Hive Table）会先删除表然后再创建，这里我手动解锁试试看。
mapred.job.name='Create Intermediate Flat Hive Table'; USE default; DROP TABLE IF EXISTS kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336; CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336……
3 回到hive命令行，解锁一下
hive> unlock table default.kylin_intermediate_cube_ho_flight_route_20200810_63608d88_36bd_f80d_a96e_b7474f730336;
OK
Time taken: 0.121 seconds
去kylin页面构建一个月的看看，执行几分钟后报错，日志如下
java.io.IOException: OS command error exit with return code: 3, error message: WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hive-common-2.1.1-cdh6.2.0.jar!/hive-log4j2.properties Async: false
OK
Time taken: 2.918 seconds
OK
Time taken: 0.071 seconds
OK
Time taken: 0.373 seconds
OK
Time taken: 0.257 seconds
Query ID = hdfs_20201110142329_459001bf-d6c5-40a2-aa59-edab8c4731f0
Total jobs = 2
Stage-3 is selected by condition resolver.
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 10
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
20/11/10 14:23:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm98
Starting Job = job_1604980818051_0001, Tracking URL = http://bigdata004:8088/proxy/application_1604980818051_0001/
Kill Command = /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/bin/hadoop job -kill job_1604980818051_0001
Hadoop job information for Stage-3: number of mappers: 5; number of reducers: 10
2020-11-10 14:23:44,329 Stage-3 map = 0%, reduce = 0%
2020-11-10 14:23:52,625 Stage-3 map = 40%, reduce = 0%, Cumulative CPU 29.39 sec
2020-11-10 14:23:57,787 Stage-3 map = 60%, reduce = 0%, Cumulative CPU 50.17 sec
2020-11-10 14:24:01,909 Stage-3 map = 75%, reduce = 0%, Cumulative CPU 104.75 sec
2020-11-10 14:24:08,092 Stage-3 map = 95%, reduce = 0%, Cumulative CPU 125.93 sec
2020-11-10 14:24:09,126 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 128.09 sec
2020-11-10 14:24:21,456 Stage-3 map = 100%, reduce = 100%, Cumulative CPU 257.54 sec
MapReduce Total cumulative CPU time: 4 minutes 17 seconds 540 msec
Ended Job = job_1604980818051_0001
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2020-11-10 14:25:10 Processing rows: 200000 Hashtable size: 199999 Memory usage: 479813592 percentage: 0.251
2020-11-10 14:25:15 Processing rows: 300000 Hashtable size: 299999 Memory usage: 547358456 percentage: 0.287
2020-11-10 14:25:22 Processing rows: 500000 Hashtable size: 499999 Memory usage: 658565376 percentage: 0.345
2020-11-10 14:25:30 Processing rows: 700000 Hashtable size: 699999 Memory usage: 958387544 percentage: 0.502
2020-11-10 14:25:34 Processing rows: 800000 Hashtable size: 799999 Memory usage: 1016823992 percentage: 0.533
2020-11-10 14:25:47 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 1249201392 percentage: 0.6542020-11-10 14:25:51 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 1403747528 percentage: 0.735
2020-11-10 14:25:55 Processing rows: 1200000 Hashtable size: 1199999 Memory usage: 1520276888 percentage: 0.796
2020-11-10 14:26:22 Processing rows: 1400000 Hashtable size: 1399999 Memory usage: 1536850032 percentage: 0.805
2020-11-10 14:26:47 Processing rows: 1500000 Hashtable size: 1499999 Memory usage: 1723152288 percentage: 0.903
Execution failed with exit status: 3
Obtaining error information
Task failed!
Task ID:
Stage-12
Logs:
/tmp/hdfs/hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
MapReduce Jobs Launched:
Stage-Stage-3: Map: 5 Reduce: 10 Cumulative CPU: 257.54 sec HDFS Read: 646802125 HDFS Write: 650131577 HDFS EC Read: 0 SUCCESS
Total MapReduce CPU Time Spent: 4 minutes 17 seconds 540 msec
The command is:
hive -e "set mapred.job.name='Create Intermediate Flat Hive Table';
USE default;
DROP TABLE IF EXISTS kylin_intermediate_cube_ho_flight_route_20200810_74370ef0_5df6_6b1f_3b03_dfb9b7609f73;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_cube_ho_flight_route_20200810_74370ef0_5df6_6b1f_3b03_dfb9b7609f73
(
\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_AIR_LINE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_POINTNAME\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_DATE\` date
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_ID\` bigint
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTEINDEX\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DEPARTURE_4_CODE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ARRIVAL_4_CODE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_NUMBER\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_TYPE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CAPTAIN\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISPATCH\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ENGINES1\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_TYPE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_PROPERTY\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_STATUS\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLG_STATE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAX_DEPART_WEIGHT\` int
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_SEAT_NUM\` int
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_NUMBER\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACQUIS_DATE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISTANCE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLAN_DEPARTURE_TIME\` bigint
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_DEPARTURE_TIME\` bigint
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_ARRIVAL_TIME\` bigint
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_ARRIVAL_TIME\` bigint
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CALIBRATEDAIRSPEED\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ELAPSEDTIMEMIN\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELCONSUMPTION\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELFLOW\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_GROUNDSPEED\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_HIGHLAYER\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LATITUDE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LONGITUDE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MACH\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAGNETICHEADING\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MINIMUMSAFEALTITUDE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALAIRDISTANCE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALOILCONTENT\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATURE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATUREDEVIATION\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_THEREMAININGDISTANCE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TIMEREMAINING\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TRUEHEADING\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDDIS\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDFUEL\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDTIME\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_VACUUMSPEED\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDCOMMPONENT\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDDIRECTION\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDSPEED\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_STATUS\` int
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_ALT\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_HEADING\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_GS\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_STATICAIRTEMP\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINDIR\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINSPD\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TAS\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WITH_WIND\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELATITUDE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELONGITUDE\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TOTALAIRTEMP\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_FF1\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_MACH\` double
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_COUNTRYCODE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FIRNAME\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_OVERLAP\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTECODE\` string
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ORGNUM\` string
)
STORED AS SEQUENCEFILE
LOCATION 'hdfs://Aero/kylin/kylin_metadata/kylin-6b4ef829-7fad-d610-af6b-400e84632b0a/kylin_intermediate_cube_ho_flight_route_20200810_74370ef0_5df6_6b1f_3b03_dfb9b7609f73';
ALTER TABLE kylin_intermediate_cube_ho_flight_route_20200810_74370ef0_5df6_6b1f_3b03_dfb9b7609f73 SET TBLPROPERTIES('auto.purge'='true');
INSERT OVERWRITE TABLE \`kylin_intermediate_cube_ho_flight_route_20200810_74370ef0_5df6_6b1f_3b03_dfb9b7609f73\` SELECT
\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`AIR_LINE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_AIR_LINE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`POINTNAME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_POINTNAME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_DATE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_DATE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_ID\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_ID\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ROUTEINDEX\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTEINDEX\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`DEPARTURE_4_CODE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DEPARTURE_4_CODE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ARRIVAL_4_CODE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ARRIVAL_4_CODE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_NUMBER\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_NUMBER\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`PLANE_TYPE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_TYPE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`CAPTAIN\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CAPTAIN\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`DISPATCH\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISPATCH\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ENGINES1\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ENGINES1\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_TYPE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_TYPE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_PROPERTY\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_PROPERTY\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_STATUS\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLIGHT_STATUS\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLG_STATE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FLG_STATE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAX_DEPART_WEIGHT\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAX_DEPART_WEIGHT\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`SEAT_NUM\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_SEAT_NUM\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`PLANE_NUMBER\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_NUMBER\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ACQUIS_DATE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACQUIS_DATE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`DISTANCE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_DISTANCE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`PLAN_DEPARTURE_TIME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLAN_DEPARTURE_TIME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ACTUAL_DEPARTURE_TIME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_DEPARTURE_TIME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`PLANE_ARRIVAL_TIME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_PLANE_ARRIVAL_TIME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ACTUAL_ARRIVAL_TIME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ACTUAL_ARRIVAL_TIME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`CALIBRATEDAIRSPEED\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_CALIBRATEDAIRSPEED\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ELAPSEDTIMEMIN\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ELAPSEDTIMEMIN\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FUELCONSUMPTION\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELCONSUMPTION\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FUELFLOW\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FUELFLOW\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`GROUNDSPEED\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_GROUNDSPEED\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`HIGHLAYER\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_HIGHLAYER\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`LATITUDE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LATITUDE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`LONGITUDE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_LONGITUDE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MACH\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MACH\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAGNETICHEADING\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAGNETICHEADING\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MINIMUMSAFEALTITUDE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MINIMUMSAFEALTITUDE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`RESIDUALAIRDISTANCE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALAIRDISTANCE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`RESIDUALOILCONTENT\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_RESIDUALOILCONTENT\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`TEMPERATURE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATURE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`TEMPERATUREDEVIATION\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TEMPERATUREDEVIATION\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`THEREMAININGDISTANCE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_THEREMAININGDISTANCE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`TIMEREMAINING\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TIMEREMAINING\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`TRUEHEADING\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_TRUEHEADING\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`USEDDIS\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDDIS\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`USEDFUEL\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDFUEL\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`USEDTIME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_USEDTIME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`VACUUMSPEED\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_VACUUMSPEED\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`WINDCOMMPONENT\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDCOMMPONENT\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`WINDDIRECTION\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDDIRECTION\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`WINDSPEED\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_WINDSPEED\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`STATUS\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_STATUS\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_ALT\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_ALT\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_HEADING\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_HEADING\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_GS\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_GS\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_STATICAIRTEMP\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_STATICAIRTEMP\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_WINDIR\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINDIR\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_WINSPD\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WINSPD\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_TAS\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TAS\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_WITH_WIND\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_WITH_WIND\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_TRUELATITUDE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELATITUDE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_TRUELONGITUDE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TRUELONGITUDE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_TOTALAIRTEMP\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_TOTALAIRTEMP\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_FF1\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_FF1\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`MAP_MACH\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_MAP_MACH\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`COUNTRYCODE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_COUNTRYCODE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FIRNAME\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_FIRNAME\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`OVERLAP\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_OVERLAP\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ROUTECODE\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ROUTECODE\`
,\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`ORGNUM\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810_ORGNUM\`
FROM \`AERO_HO\`.\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\` as \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`
WHERE 1=1 AND (\`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_DATE\` >= '2018-12-31' AND \`VIEW_CUBE_HO_FLIGHT_ROUTE_20200810\`.\`FLIGHT_DATE\` < '2019-02-01')
;
" --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true
at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.createFlatHiveTable(CreateFlatHiveTableStep.java:59)
at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:97)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
测试姐姐说是不是cube依赖的视图有问题？
4 查询表，发现一直是阻塞状态
可以看到命令行中查询的sql日志和kylin构建的日志有一样的Starting to launch local task to process map join; maximum memory = 1908932608，所以解锁后再次构建cube报错的原因是cube依赖的hive表有问题。
hive> select * from view_cube_ho_flight_route_20200810 limit 10;
Query ID = hdfs_20201110175108_bf5f47ac-5517-4a0d-93d4-5d454d58499a
Total jobs = 2
Stage-2 is selected by condition resolver.
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 11
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
20/11/10 17:51:12 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm98
Starting Job = job_1604980818051_0029, Tracking URL = http://bigdata004:8088/proxy/application_1604980818051_0029/
Kill Command = /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/bin/hadoop job -kill job_1604980818051_0029
Hadoop job information for Stage-2: number of mappers: 5; number of reducers: 11
2020-11-10 17:51:21,351 Stage-2 map = 0%, reduce = 0%
2020-11-10 17:51:30,706 Stage-2 map = 40%, reduce = 0%, Cumulative CPU 39.4 sec
2020-11-10 17:51:37,938 Stage-2 map = 60%, reduce = 0%, Cumulative CPU 65.62 sec
2020-11-10 17:51:38,968 Stage-2 map = 73%, reduce = 0%, Cumulative CPU 117.39 sec
2020-11-10 17:51:50,315 Stage-2 map = 87%, reduce = 0%, Cumulative CPU 143.23 sec
2020-11-10 17:51:51,345 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 152.15 sec
2020-11-10 17:52:01,596 Stage-2 map = 100%, reduce = 18%, Cumulative CPU 178.99 sec
2020-11-10 17:52:02,616 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 303.4 sec
MapReduce Total cumulative CPU time: 5 minutes 3 seconds 400 msec
Ended Job = job_1604980818051_0029
SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2020-11-10 17:52:10 Starting to launch local task to process map join; maximum memory = 1908932608
2020-11-10 17:52:54 Processing rows: 300000 Hashtable size: 299999 Memory usage: 545738152 percentage: 0.286
2020-11-10 17:53:01 Processing rows: 500000 Hashtable size: 499999 Memory usage: 654956920 percentage: 0.343
2020-11-10 17:53:12 Processing rows: 800000 Hashtable size: 799999 Memory usage: 1012034728 percentage: 0.53
2020-11-10 17:53:26 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 1243845424 percentage: 0.652
2020-11-10 17:53:29 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 1398631728 percentage: 0.733
2020-11-10 17:53:37 Processing rows: 1300000 Hashtable size: 1299999 Memory usage: 1492506112 percentage: 0.782
2020-11-10 17:53:59 Processing rows: 1400000 Hashtable size: 1399999 Memory usage: 1537734352 percentage: 0.806
2020-11-10 17:56:29 Processing rows: 1600000 Hashtable size: 1599999 Memory usage: 1668811944 percentage: 0.874
Interrupting... Be patient, this might take some time.
Press Ctrl+C again to kill JVM
Exiting the JVM
^Z
[1]+ Stopped hive
5 修改hive内存
Starting to launch local task to process map join
看上面命令行的日志是在map join，可能是内存不够了，猜想把内存设置大一些或许有用。
这里hive进程的内存默认都是2GB,给它设置为8GB。重新查询视图，发现不会阻塞了，已经可以正常查询。
重新build cube，如下图所示第一阶段已经可以成功了
总结：
问题一共有两个
1.死锁问题（解锁）
2.hive表问题(增加hive进程内存)