Start azkaban
After starting the server and excutor, azkaban open the browser, you will find can not be opened, the log report this error
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) at sun.security.ssl.InputRecord.read(InputRecord.java:527) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2019/09/01 15:18:24.499 +0800 WARN [log] [Azkaban] EXCEPTION javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) at sun.security.ssl.InputRecord.read(InputRecord.java:527) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2019/09/01 15:18:24.680 +0800 WARN [log] [Azkaban] EXCEPTION javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) at sun.security.ssl.InputRecord.read(InputRecord.java:527) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2019/09/01 15:18:24.809 +0800 WARN [log] [Azkaban] EXCEPTION javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) at sun.security.ssl.InputRecord.read(InputRecord.java:527) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2019/09/01 15:26:14.746 +0800 WARN [log] [Azkaban] EXCEPTION javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.InputRecord.read(InputRecord.java:505) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) ... 5 more
In fact, not difficult to solve this problem, we recommend that the browser with Google when open: address is https://192.168.86.131:8443/ (HTTPS: // your ip: 8433)
Attention, until can not be opened because of the use of http, here emphasize, must not use http, must use https, and then select Advanced Options, address operational access unsafe, because this address by default as unsafe
Script writing upload.job
# upload.job type=command command=bash uploadFile2Hdfs.sh
Script writing uploadFile2Hdfs.sh
! # / bin / the bash # SET Java the env Export the JAVA_HOME = / opt / modules / jdk1. . 8 .0_65 Export the JRE_HOME = $ {the JAVA_HOME} / JRE Export the CLASSPATH =:. $ {the JAVA_HOME} / lib: $ {} the JRE_HOME / lib the PATH Export = $ {the JAVA_HOME} / bin: the PATH $ # SET Hadoop the env Export HADOOP_HOME is = / opt / modules / hadoop- 2.6 . 0 Export the PATH = HADOOP_HOME is $ {} / bin: $ {} HADOOP_HOME is / sbin: the PATH $ # version problem 1: # Although uploaded to the Hadoop cluster, but the original file is still there. How to deal with? # Log file names are xxxx.log1, upload the file again, because the hdfs has been in existence, it will complain. How to deal with? How to solve the problem of version # 1 # 1 , first need to upload a file to be uploaded directory # 2 , move to upload directory to be talking about the files, according to a certain format of the same name name # /export/software/hadoop.log1 / Export / the Data / click_log / xxxxx_click_log_ {DATE} # log file storage directory log_src_dir = / Home / Hadoop / logs / log / directory files to be uploaded # stored log_toupload_dir = / Home / Hadoop / logs / toupload / day_01 = -d `DATE ' -1 Day ' + M-%%% Y- d` sYear = `$ = DATE --date day_01% + Y' smonth =` $ = DATE --date day_01 +% M` SDAY = `$ = DATE --date day_01 + % d` #echo $ day_01 #echo $ sYear $ smonth #echo #echo $ SDAY # hdfs log files uploaded to the root path hdfs_root_dir = / the Data / clickLog / $ sYear / $ smonth / $ SDAY hadoop FS -mkdir - the p-$ hdfs_root_dir # environment variable information print echo " Envs: HADOOP_HOME : $ HADOOP_HOME " # directory to read log files, to determine whether there is a need to upload files echo " log_src_dir: " $ log_src_dir LS $ log_src_dir | the while the read fileName do iF [[ " $ fileName " == access.log]]; the then # IF [ " access.log " =" $ FileName" ]; The then DATE =` DATE +% Y_% m_%% D_ H_% M_% S ' # Move the file to be uploaded directory and rename # print information echo " Moving $ log_src_dir $ fileName to $ log_toupload_dir " xxxxx_click_log_ $ fileName " DATE $ " mv $ log_src_dir $ fileName $ log_toupload_dir " xxxxx_click_log_ $ fileName " $ DATE # upload the file path will be written to a file list willDoing echo $ log_toupload_dir " xxxxx_click_log_ $ fileName " $ DATE >> $ log_toupload_dir " willDoing. " $ DATE fi DONE # Find a list of file willDoing LS $ log_toupload_dir | grep by Will | grep -v " _COPY_ " | grep -v " _DONE_ " | the while the Read Line do # print information echo " toupload IS in File: " $ Line # will be uploaded the file list willDoing renamed willDoing_COPY_ mv $ log_toupload_dir log_toupload_dir $ $ $ Line Line " _COPY_ " # read a list of files willDoing_COPY_ content (a file to be uploaded one name), line here is a list of files to be uploaded in the path CAT $ log_toupload_dir Line $ " _COPY_ "| The while the Read Line do # print information echo "puting...$line to hdfs path.....$hdfs_root_dir" hadoop fs -put $line $hdfs_root_dir done mv $log_toupload_dir$line"_COPY_" $log_toupload_dir$line"_DONE_" done
Then these two scripts package
Enter the login page azkaban
Select just packed upflow.zip
This time to upload the log file to the cluster
View HDFS directory, file upload prove successful
The log file is not in the local cluster
Scripting clean.job
# clean.job type=command dependencies=upload command=bash clean.sh
Script writing clean.sh
#!/bin/bash export JAVA_HOME=/opt/modules/jdk1.8.0_65 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH #set hadoop env export HADOOP_HOME=/opt/modules/hadoop-2.6.0 export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH log_local_dir=/home/hadoop/flume/ #log_hdfs_dir=/test/2017/7/ day_01=`date -d'-1 day' +%Y-%m-%d` syear=`date --date=$day_01 +%Y` smonth=`date --date=$day_01 +%m` sday=`date --date=$day_01 +%d` #echo $day_01 #echo $syear #echo $smonth #echo $sday log_hdfs_dir=/data/clickLog/$syear/$smonth/$sday #echo $log_hdfs_dir click_log_clean=com.it19gong.clickLog.AccessLogDriver clean_dir=/cleaup/$syear/$smonth/$sday echo "hadoop jar /home/hadoop/hivedome/hiveaad.jar $click_log_clean $log_hdfs_dir $clean_dir" hadoop fs -rm -r -f $clean_dir hadoop jar /home/hadoop/hivedome/mrclick.jar $click_log_clean $log_hdfs_dir $clean_dir
Script writing upload.job
# upload.job type=command command=bash uploadFile2Hdfs.sh
Then these four scripts package
Upload a data file
Scripting hivesql.sh
#!/bin/bash export JAVA_HOME=/opt/modules/jdk1.8.0_65 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH #set hadoop env export HADOOP_HOME=/opt/modules/hadoop-2.6.0 export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH export HIVE_HOME=/opt/modules/hive export PATH=${HIVE_HOME}/bin:$PATH log_local_dir=/home/hadoop/flume/ #log_hdfs_dir=/test/2017/7/ day_01=`date -d'-1 day' +%Y-%m-%d` syear=`date --date=$day_01 +%Y` smonth=`date --date=$day_01 +%m` sday=`date --date=$day_01 +%d` #echo $day_01 #echo $syear #echo $smonth #echo $sday log_hdfs_dir=/data/clickLog/$syear/$smonth/$sday #echo $log_hdfs_dir click_log_clean=com.it19gong.clickLog.AccessLogDriver clean_dir=/cleaup/$syear/$smonth/$sday HQL_origin="load data inpath '$clean_dir' into table mydb2.access" #HQL_origin="create external table db2.access(ip string,day string,url string,upflow string) row format delimited fields terminated by ',' location '$clean_dir'" #echo $HQL_origin hive -e "$HQL_origin"
Scripting hivesqljob.job
# hivesql.job type=command dependencies=clean command=bash hivesql.sh
Scripting ip.sh
#!/bin/bash export JAVA_HOME=/opt/modules/jdk1.8.0_65 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH #set hadoop env export HADOOP_HOME=/opt/modules/hadoop-2.6.0 export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH export HIVE_HOME=/opt/modules/hive export PATH=${HIVE_HOME}/bin:$PATH log_local_dir=/home/hadoop/flume/ #log_hdfs_dir=/test/2017/7/ day_01=`date -d'-1 day' +%Y-%m-%d` syear=`date --date=$day_01 +%Y` smonth=`date --date=$day_01 +%m` sday=`date --date=$day_01 +%d` #echo $day_01 #echo $syear #echo $smonth #echo $sday log_hdfs_dir=/data/clickLog/$syear/$smonth/$sday #echo $log_hdfs_dir click_log_clean=com.it19gong.clickLog.AccessLogDriver clean_dir=/cleaup/$syear/$smonth/$sday HQL_origin="insert into mydb2.upflow select ip,sum(upflow) as sum from mydb2.access group by ip order by sum desc " #echo $HQL_origin hive -e "$HQL_origin"
Scripting ipjob.job
# ip.job type=command dependencies=hivesqljob command=bash ip.sh
Scripting mysql.sh
#!/bin/bash export JAVA_HOME=/opt/modules/jdk1.8.0_65 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH #set hadoop env export HADOOP_HOME=/opt/modules/hadoop-2.6.0 export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH export HIVE_HOME=/opt/modules/hive export PATH=${HIVE_HOME}/bin:$PATH export SQOOP_HOME=/opt/modules/sqoop export PATH=${SQOOP_HOME}/bin:$PATH sqoop export --connect \ jdbc:mysql://node1:3306/userdb \ --username sqoop --password sqoop --table upflow --export-dir \ /user/hive/warehouse/mydb2.db/upflow --input-fields-terminated-by ','
Scripting mysqljob.job
# mysql.job type=command dependencies=ipjob command=bash mysql.sh
Packaged into job.zip
Upload log files
Open phpsudy local installation
Open the page in a browser http://www.echart.com/
