在CentOS6.9安装Azkaban3.50.0

Azkaban简介
在这里插入图片描述
上图是Azkaban官网提供的架构图,包括三个组件:
Relational Database (MySQL)
AzkabanWebServer
AzkabanExecutorServer

组件分工:
元数据库:用于维护任务流信息以及执行情况信息。
Web服务器:用户交互的门户,同时也是管理任务流、工程,控制调度,任务分配的核心组件。
执行服务器:任务被真正调起的服务器,可扩展。

AzkabanWebServer对数据库的使用:
项目管理
追踪任务流的状态
查看任务执行结果以及历史日志
调度任务
sla

AzkabanExecutorServer 对数据库的使用:
从数据库获取工程的信息
从数据库获取要执行的任务流
储存任务流和任务的日志信息
从数据库获取内置任务流的任务依赖关系

一、环境

操作系统:CentOS6.9
软件版本:Azkaban3.50.0

二、安装

我这里是将Azkaban安装在hadoop用户,所以所有的linux操作都在hadoop用户。

1、编译Azkaban可执行文件

1.1 下载安装包

cd /data/install_pkg
wget https://github.com/azkaban/azkaban/archive/3.50.0.tar.gz

解压安装包:

tar -zxvf 3.50.0.tar.gz

1.2 修改邮件代码

vi /data/install_pkg/azkaban-3.50.0/azkaban-common/src/main/java/azkaban/utils/EmailMessage.java

找到邮件参数配置的地方

:/props

增加以下语句:

props.put("mail.smtp.socketFactory.class", "javax.net.ssl.SSLSocketFactory");

在这里插入图片描述
1.3 安装git

因为使用gradle编译会用到git,所以要安装:

yum install git

1.4 安装JCE

如果不安装JCE,后面编译的时候会报错。安装方法见:
https://blog.csdn.net/andyguan01_2/article/details/88343641

1.5 编译文件

使用Gradle编译文件。Gradle是一个基于Apache Ant和Apache Maven的项目自动化构建工具。

cd /data/install_pkg/azkaban-3.50.0
./gradlew build installDist

编译报错,看着像是测试过程中出错,没找具体到是什么原因。

Starting a Gradle Daemon (subsequent builds will be faster)
Parallel execution with configuration on demand is an incubating feature.

> Task :az-core:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :azkaban-common:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :azkaban-web-server:npmSetup
/data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/bin/npm -> /data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/lib/node_modules/npm/bin/npm-cli.js
/data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/bin/npx -> /data/install_pkg/azkaban-3.50.0/azkaban-web-server/.gradle/npm/npm-v5.6.0/lib/node_modules/npm/bin/npx-cli.js
+ [email protected]
added 476 packages in 14.414s

> Task :azkaban-exec-server:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :azkaban-common:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :azkaban-hadoop-security-plugin:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /data/install_pkg/azkaban-3.50.0/azkaban-hadoop-security-plugin/src/main/java/azkaban/security/HadoopSecurityManager_H_2_0.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :azkaban-common:test

azkaban.utils.UtilsTest > testValidCronExpressionV FAILED
    java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)EmailMessageCreatorTest
        at org.junit.Assert.assertTrue(Assert.java:52)
        at azkaban.utils.UtilsTest.testValidCronExpressionV(UtilsTest.java:63)

> Task :azkaban-web-server:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :azkaban-exec-server:compileTestJava
Note: /data/install_pkg/azkaban-3.50.0/azkaban-exec-server/src/test/java/azkaban/execapp/AzkabanExecutorServerTest.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :azkaban-web-server:npm_install
added 39 packages in 3.23s

> Task :azkaban-common:test

328 tests completed, 1 failed, 18 skipped

> Task :azkaban-solo-server:compileJava
Note: /data/install_pkg/azkaban-3.50.0/azkaban-solo-server/src/main/java/azkaban/soloserver/AzkabanSingleServer.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :az-hdfs-viewer:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :az-reportal:compileJava
Note: /data/install_pkg/azkaban-3.50.0/az-reportal/src/main/java/azkaban/reportal/util/StreamProviderHDFS.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :az-hadoop-jobtype-plugin:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /data/install_pkg/azkaban-3.50.0/az-hadoop-jobtype-plugin/src/main/java/azkaban/jobtype/HadoopSecureSparkWrapper.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':azkaban-common:test'.
> There were failing tests. See the report at: file:///data/install_pkg/azkaban-3.50.0/azkaban-common/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2m 4s
71 actionable tasks: 66 executed, 5 from cache

于是改为使用命令./gradlew build installDist -x test,-x test表示跳过测试。

./gradlew clean
./gradlew build installDist -x test

在这里插入图片描述
1.6 复制可执行文件到安装目录

编译完成后,我们就可以获取到web,exec,db,solo的可执行文件tar包,拷贝到Azkaban安装目录:

ll /data/install_pkg/azkaban-3.50.0/azkaban-*/build/distributions/*.tar.gz
cp /data/install_pkg/azkaban-3.50.0/azkaban-*/build/distributions/*.tar.gz /u01/app/azkaban-3.50.0
ll /u01/app/azkaban-3.50.0

在这里插入图片描述
解压tar包到Azkaban安装目录:

cd /u01/app/azkaban-3.50.0
tar -zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0
tar -zxvf  azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz -C /u01/app/azkaban-3.50.0

2、初始化MySQL

初始化MySQL的库、用户、权限、表、数据:

mysql -u root -p
CREATE DATABASE azkaban;
CREATE USER 'azkaban'@'%' IDENTIFIED BY '填实际密码';
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
flush privileges;
use azkaban;
source /u01/app/azkaban-3.50.0/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql;

3、配置SSL

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

要输入密码的地方,都填相同密码。几个问题的回答,what … country code … ?填CN,Is … correct?填y,其他问题可以直接回车不填。。
在这里插入图片描述
完成上述工作后,将在当前目录生成 keystore 证书文件,将keystore移动到azkaban web服务器根目录中。

mv keystore /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT

4、配置web-server

因为web-server和exec-server都没有默认配置文件,我们需要复制solo的配置文件。

cd /u01/app/azkaban-3.50.0
cp -r azkaban-solo-server-0.1.0-SNAPSHOT/conf azkaban-web-server-0.1.0-SNAPSHOT
cp /data/install_pkg/azkaban-3.50.0/azkaban-web-server/src/test/resources/log4j.properties azkaban-web-server-0.1.0-SNAPSHOT/conf

4.1 配置azkaban.properties

vi /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban.properties

配置以下内容:

# Azkaban Personalization Settings
#服务器UI名称,用于服务器上方显示的名字
azkaban.name=Test
#描述
azkaban.label=My Local Azkaban
#UI颜色
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
#默认根web目录建议最好写绝对路径,以免出现找不到文件错误
web.resource.dir=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/web
#默认时区,已改为亚洲/上海,默认为美国
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
#用户权限管理默认类
user.manager.class=azkaban.user.XmlUserManager
#用户配置文件所在位置
user.manager.xml.file=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
# Loader for projects
# global配置文件所在位置
executor.global.properties=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/global.properties
azkaban.project.dir=projects
#数据库类型
database.type=mysql
#端口号
mysql.port=3306
#数据库连接IP
mysql.host=填实际IP
#数据库名
mysql.database=azkaban
#数据库用户名
mysql.user=azkaban
#数据库密码
mysql.password=填实际密码
#最大连接数
mysql.numconnections=100
# Azkaban Jetty server properties.
# Jetty服务器属性.
velocity.dev.mode=false
#最大线程数
jetty.maxThreads=25
#Jetty SSL端口
jetty.ssl.port=8443
#Jetty端口
jetty.port=8081
#SSL文件名
jetty.keystore=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/keystore
#SSL文件密码
jetty.password=填实际密码
#Jetty主密码 与 keystore文件相同
jetty.keypassword=填实际密码
#SSL文件名
jetty.truststore=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/keystore
# SSL文件密码
jetty.trustpassword=填实际密码
#jetty.use.ssl=false
jetty.maxThreads=25
# Azkaban Executor settings
executor.port=12321

# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy
 between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=

lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes

4.2 配置commonprivate.properties

先创建配置文件:

mkdir -p /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes
vi /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes/commonprivate.properties 

输入以下内容:

azkaban.native.lib=false
execute.as.user=false

4.3 配置azkaban-users.xml

修改用户配置文件azkaban-users.xml:

 vi /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml

增加admin用户:(在第4行)

<azkaban-users>
  <user groups="azkaban" password="填实际密码" roles="admin" username="azkaban"/>
  <user password="填实际密码" roles="metrics" username="metrics"/>
  <user username="admin" password="填实际密码" roles="admin,metrics"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

5、配置executor

将 azkaban-web-server-0.1.0-SNAPSHOT目录下的conf拷贝到 azkaban-exec-server-0.1.0-SNAPSHOT

cd /u01/app/azkaban-3.50.0
cp -r azkaban-web-server-0.1.0-SNAPSHOT/conf azkaban-exec-server-0.1.0-SNAPSHOT
cp -r azkaban-web-server-0.1.0-SNAPSHOT/plugins azkaban-exec-server-0.1.0-SNAPSHOT

6、启动服务

6.1 启动Executor Server

进入azkaban-exec-server-0.1.0-SNAPSHOT下启动(注意不要进入bin目录启动)

cd /u01/app/azkaban-3.50.0/azkaban-exec-server-0.1.0-SNAPSHOT
./bin/start-exec.sh

会在当前目录生成.out的日志文件:

ll /u01/app/azkaban-3.50.0/azkaban-exec-server-0.1.0-SNAPSHOT/*.out
more /u01/app/azkaban-3.50.0/azkaban-exec-server-0.1.0-SNAPSHOT/*.out

日志内容为:

Using Hadoop from /u01/app/hadoop
Using Hive from /u01/app/apache-hive-2.3.4-bin
./bin/internal/../..
.:/usr/java/jdk1.8.0_121/lib/dt.jar:/usr/java/jdk1.8.0_121/lib/tools.jar:/u01/app/hadoop-2.6.5/etc/hadoop:/u01/app/hadoop/share/hado
op/common/lib/*:/u01/app/hadoop/share/hadoop/common/*:/u01/app/hadoop/share/hadoop/hdfs:/u01/app/hadoop/share/hadoop/hdfs/lib/*:/u01
/app/hadoop/share/hadoop/hdfs/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/lib/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/*:/u01/app/had
oop/share/hadoop/mapreduce/lib/*:/u01/app/hadoop/share/hadoop/mapreduce/*:/u01/app/hadoop/contrib/capacity-scheduler/*.jar:./bin/int
ernal/../../lib/activation-1.1.jar:./bin/internal/../../lib/aopalliance-1.0.jar:./bin/internal/../../lib/az-core-0.1.0-SNAPSHOT.jar:
./bin/internal/../../lib/azkaban-common-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-db-0.1.0-SNAPSHOT.jar:./bin/internal/../
../lib/azkaban-exec-server-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.jar:./bin/inter
nal/../../lib/azkaban-spi-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/c3p0-0.9.1.1.jar:./bin/internal/../../lib/commons-codec-1.9.ja
r:./bin/internal/../../lib/commons-collections-3.2.2.jar:./bin/internal/../../lib/commons-dbcp2-2.1.1.jar:./bin/internal/../../lib/c
ommons-dbutils-1.5.jar:./bin/internal/../../lib/commons-fileupload-1.2.1.jar:./bin/internal/../../lib/commons-io-2.4.jar:./bin/inter
nal/../../lib/commons-jexl-2.1.1.jar:./bin/internal/../../lib/commons-lang-2.6.jar:./bin/internal/../../lib/commons-logging-1.2.jar:
./bin/internal/../../lib/commons-math3-3.0.jar:./bin/internal/../../lib/commons-pool2-2.4.2.jar:./bin/internal/../../lib/gson-2.8.1.
jar:./bin/internal/../../lib/guava-21.0.jar:./bin/internal/../../lib/guice-4.1.0.jar:./bin/internal/../../lib/httpclient-4.5.3.jar:.
/bin/internal/../../lib/httpcore-4.4.6.jar:./bin/internal/../../lib/jackson-core-asl-1.9.5.jar:./bin/internal/../../lib/jackson-mapp
er-asl-1.9.5.jar:./bin/internal/../../lib/javax.inject-1.jar:./bin/internal/../../lib/jetty-6.1.26.jar:./bin/internal/../../lib/jett
y-util-6.1.26.jar:./bin/internal/../../lib/joda-time-2.0.jar:./bin/internal/../../lib/jopt-simple-4.3.jar:./bin/internal/../../lib/j
sr305-3.0.2.jar:./bin/internal/../../lib/kafka-clients-0.10.0.0.jar:./bin/internal/../../lib/kafka-log4j-appender-0.10.0.0.jar:./bin
/internal/../../lib/log4j-1.2.17.jar:./bin/internal/../../lib/lz4-1.3.0.jar:./bin/internal/../../lib/mail-1.4.5.jar:./bin/internal/.
./../lib/metrics-core-3.1.0.jar:./bin/internal/../../lib/metrics-jvm-3.1.0.jar:./bin/internal/../../lib/mysql-connector-java-5.1.28.
jar:./bin/internal/../../lib/quartz-2.2.1.jar:./bin/internal/../../lib/servlet-api-2.5-20081211.jar:./bin/internal/../../lib/slf4j-a
pi-1.7.21.jar:./bin/internal/../../lib/slf4j-log4j12-1.7.21.jar:./bin/internal/../../lib/snakeyaml-1.18.jar:./bin/internal/../../lib
/snappy-java-1.1.2.4.jar:./bin/internal/../../lib/velocity-1.7.jar:./bin/internal/../../extlib/*.jar:./bin/internal/../../plugins/*/
*.jar:/u01/app/hadoop/conf:/u01/app/hadoop/*:/u01/app/apache-hive-2.3.4-bin/conf:/u01/app/apache-hive-2.3.4-bin/lib/*
Starting AzkabanExecutorServer on port 12321 ...
2019/03/08 15:01:08.059 +0800 INFO [AzkabanExecutorServer] Starting Jetty Azkaban Executor...
2019/03/08 15:01:08.084 +0800 INFO [AzkabanServer] Loading azkaban settings file from ./bin/internal/../../conf
2019/03/08 15:01:08.084 +0800 INFO [AzkabanServer] Loading azkaban properties file
2019/03/08 15:01:08.094 +0800 ERROR [StdOutErrRedirect] SLF4J: Class path contains multiple SLF4J bindings.
2019/03/08 15:01:08.094 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/hadoop-2.6.5/share/hadoop/common/
lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:01:08.095 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/azkaban-3.50.0/azkaban-exec-serve
r-0.1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:01:08.095 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/apache-hive-2.3.4-bin/lib/log4j-s
lf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:01:08.095 +0800 ERROR [StdOutErrRedirect] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanat
ion.
2019/03/08 15:01:08.097 +0800 ERROR [StdOutErrRedirect] SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019/03/08 15:01:08.607 +0800 WARN [StorageCleaner] Project cleanup disabled. All artifacts will be stored.
2019/03/08 15:01:08.608 +0800 INFO [AzkabanExecServerModule] Event reporter is not enabled
2019/03/08 15:01:08.608 +0800 INFO [FlowRunnerManager] Execution dir retention set to 86400000 ms
2019/03/08 15:01:08.609 +0800 INFO [FlowRunnerManager] useNewThreadPool: false
2019/03/08 15:01:08.611 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:01:08.611 +0800 INFO [FlowRunnerManager] Cleaning recently finished
2019/03/08 15:01:08.612 +0800 INFO [FlowRunnerManager] Cleaning old execution dirs
2019/03/08 15:01:08.613 +0800 INFO [JobTypeManager] Loading plugin default job types
2019/03/08 15:01:08.616 +0800 INFO [JobTypeManager] Job type plugin directory set. Loading extra job types from /u01/app/azkaban-3.5
0.0/azkaban-web-server-0.1.0-SNAPSHOT/plugins/jobtypes
2019/03/08 15:01:08.616 +0800 INFO [JobTypeManager] Common plugin job props file /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SN
APSHOT/plugins/jobtypes/common.properties not found. Using only globals props
2019/03/08 15:01:08.616 +0800 INFO [JobTypeManager] Common plugin load props file /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-S
NAPSHOT/plugins/jobtypes/commonprivate.properties found. Attempt to load.
2019/03/08 15:01:08.626 +0800 INFO [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019/03/08 15:01:08.628 +0800 INFO [ExecJettyServerModule] Setting up connector with stats on: true
2019/03/08 15:01:08.628 +0800 INFO [ExecJettyServerModule] Jetty connector name: 0.0.0.0:12321, default header buffer size: 4096
2019/03/08 15:01:08.628 +0800 INFO [ExecJettyServerModule] Jetty connector name: 0.0.0.0:12321, (if) new header buffer size: 4096
2019/03/08 15:01:08.663 +0800 INFO [JmxJobMBeanManager] Initializing azkaban.execapp.jmx.JmxJobMBeanManager
2019/03/08 15:01:08.663 +0800 INFO [AzkabanExecutorServer] Job callback enabled? true
2019/03/08 15:01:08.665 +0800 INFO [JobCallbackManager] Initializing
2019/03/08 15:01:08.667 +0800 INFO [JobCallbackRequestMaker] responseWaitTimeoutMS: 5000
2019/03/08 15:01:08.668 +0800 INFO [JobCallbackRequestMaker] Global request configuration [expectContinueEnabled=false, proxy=null,
localAddress=null, cookieSpec=null, redirectsEnabled=true, relativeRedirectsAllowed=true, maxRedirects=50, circularRedirectsAllowed=
false, authenticationEnabled=true, targetPreferredAuthSchemes=null, proxyPreferredAuthSchemes=null, connectionRequestTimeout=3000, c
onnectTimeout=3000, socketTimeout=3000, contentCompressionEnabled=true]
2019/03/08 15:01:08.942 +0800 INFO [JobCallbackRequestMaker] Jobcall thread pool size: 10
2019/03/08 15:01:08.943 +0800 INFO [JobCallbackRequestMaker] Initialization for azkaban.execapp.event.JobCallbackRequestMaker is com
pleted
2019/03/08 15:01:08.944 +0800 INFO [JobCallbackManager] Initialization completed azkaban.execapp.event.JobCallbackManager
2019/03/08 15:01:08.944 +0800 INFO [JobCallbackManager] azkabanHostName oracle02.auditonline.prd.df.cn
2019/03/08 15:01:08.945 +0800 INFO [AzkabanExecutorServer] Registering MBeans...
2019/03/08 15:01:08.953 +0800 INFO [AzkabanExecutorServer] Bean azkaban.jmx.JmxJettyServer registered.
2019/03/08 15:01:08.956 +0800 INFO [AzkabanExecutorServer] Bean azkaban.execapp.jmx.JmxFlowRunnerManager registered.
2019/03/08 15:01:08.957 +0800 INFO [AzkabanExecutorServer] Bean azkaban.execapp.jmx.JmxJobMBeanManager registered.
2019/03/08 15:01:08.958 +0800 INFO [AzkabanExecutorServer] Bean azkaban.execapp.jmx.JmxJobCallback registered.
2019/03/08 15:01:08.958 +0800 INFO [AzkabanExecutorServer] No value for property: jmx.attribute.processor.class was found
2019/03/08 15:01:08.958 +0800 INFO [log] jetty-6.1.26
2019/03/08 15:01:08.992 +0800 INFO [log] Started [email protected]:12321
2019/03/08 15:01:09.792 +0800 INFO [AzkabanExecutorServer] Started Executor Server on oracle02.auditonline.prd.df.cn:12321
2019/03/08 15:01:09.831 +0800 INFO [AzkabanExecutorServer] Setting timezone to Asia/Shanghai
2019/03/08 15:02:08.614 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:03:08.614 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:03:08.615 +0800 INFO [FlowRunnerManager] Cleaning recently finished
2019/03/08 15:04:08.615 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:05:08.615 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:05:08.616 +0800 INFO [FlowRunnerManager] Cleaning recently finished
2019/03/08 15:06:08.616 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:07:08.616 +0800 INFO [FlowRunnerManager] # of executing flows: 0
2019/03/08 15:07:08.617 +0800 INFO [FlowRunnerManager] Cleaning recently finished

查看JPS,会出现AzkabanExecutorServer:

jps

在这里插入图片描述
6.2 启动Web Server

进入azkaban-web-server-0.1.0-SNAPSHOT下启动(注意不要进入bin目录启动)

cd /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT
./bin/start-web.sh

会在当前目录生成.out的日志文件:

ll /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/*.out
more /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/*.out

我这边启动后报错:
java.lang.NoSuchMethodError:com.google.common.collect.ImmutableMap.toImmutableMap

解决方法见:
https://blog.csdn.net/andyguan01_2/article/details/88353889

问题解决后,重新启动Web Server,日志内容为:

Using Hadoop from /u01/app/hadoop
Using Hive from /u01/app/apache-hive-2.3.4-bin
./bin/internal/../..
.:/usr/java/jdk1.8.0_121/lib/dt.jar:/usr/java/jdk1.8.0_121/lib/tools.jar:/u01/app/hadoop-2.6.5/etc/hadoop:/u01/app/hadoop/share/hado
op/common/lib/*:/u01/app/hadoop/share/hadoop/common/*:/u01/app/hadoop/share/hadoop/hdfs:/u01/app/hadoop/share/hadoop/hdfs/lib/*:/u01
/app/hadoop/share/hadoop/hdfs/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/lib/*:/u01/app/hadoop-2.6.5/share/hadoop/yarn/*:/u01/app/had
oop/share/hadoop/mapreduce/lib/*:/u01/app/hadoop/share/hadoop/mapreduce/*:/u01/app/hadoop/contrib/capacity-scheduler/*.jar:./bin/int
ernal/../../lib/activation-1.1.jar:./bin/internal/../../lib/antlr-2.7.2.jar:./bin/internal/../../lib/aopalliance-1.0.jar:./bin/inter
nal/../../lib/az-core-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/az-flow-trigger-dependency-plugin-0.1.0-SNAPSHOT.jar:./bin/interna
l/../../lib/azkaban-common-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-db-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaba
n-spi-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/azkaban-web-server-0.1.0-SNAPSHOT.jar:./bin/internal/../../lib/c3p0-0.9.1.1.jar:./
bin/internal/../../lib/cglib-nodep-2.2.jar:./bin/internal/../../lib/commons-beanutils-1.7.0.jar:./bin/internal/../../lib/commons-cha
in-1.1.jar:./bin/internal/../../lib/commons-codec-1.9.jar:./bin/internal/../../lib/commons-collections-3.2.2.jar:./bin/internal/../.
./lib/commons-compress-1.2.jar:./bin/internal/../../lib/commons-dbcp2-2.1.1.jar:./bin/internal/../../lib/commons-dbutils-1.5.jar:./b
in/internal/../../lib/commons-digester-1.8.jar:./bin/internal/../../lib/commons-fileupload-1.2.1.jar:./bin/internal/../../lib/common
s-io-2.4.jar:./bin/internal/../../lib/commons-jexl-2.1.1.jar:./bin/internal/../../lib/commons-lang-2.6.jar:./bin/internal/../../lib/
commons-logging-1.2.jar:./bin/internal/../../lib/commons-math3-3.0.jar:./bin/internal/../../lib/commons-pool2-2.4.2.jar:./bin/intern
al/../../lib/commons-validator-1.3.1.jar:./bin/internal/../../lib/data-1.15.7.jar:./bin/internal/../../lib/data-transform-1.15.7.jar
:./bin/internal/../../lib/dom4j-1.1.jar:./bin/internal/../../lib/gson-2.8.1.jar:./bin/internal/../../lib/guava-21.0.jar:./bin/intern
al/../../lib/guice-4.1.0.jar:./bin/internal/../../lib/httpclient-4.5.3.jar:./bin/internal/../../lib/httpcore-4.4.6.jar:./bin/interna
l/../../lib/jackson-core-2.2.2.jar:./bin/internal/../../lib/jackson-core-asl-1.9.5.jar:./bin/internal/../../lib/jackson-mapper-asl-1
.9.5.jar:./bin/internal/../../lib/javax.inject-1.jar:./bin/internal/../../lib/javax.servlet-api-3.0.1.jar:./bin/internal/../../lib/j
etty-6.1.26.jar:./bin/internal/../../lib/jetty-util-6.1.26.jar:./bin/internal/../../lib/joda-time-2.0.jar:./bin/internal/../../lib/j
opt-simple-4.3.jar:./bin/internal/../../lib/json-20070829.jar:./bin/internal/../../lib/li-jersey-uri-1.15.7.jar:./bin/internal/../..
/lib/log4j-1.2.17.jar:./bin/internal/../../lib/mail-1.4.5.jar:./bin/internal/../../lib/metrics-core-3.1.0.jar:./bin/internal/../../l
ib/metrics-jvm-3.1.0.jar:./bin/internal/../../lib/mina-core-1.1.7.jar:./bin/internal/../../lib/mysql-connector-java-5.1.28.jar:./bin
/internal/../../lib/netty-3.2.3.Final.jar:./bin/internal/../../lib/oro-2.0.8.jar:./bin/internal/../../lib/parseq-1.3.6.jar:./bin/int
ernal/../../lib/pegasus-common-1.15.7.jar:./bin/internal/../../lib/quartz-2.2.1.jar:./bin/internal/../../lib/r2-1.15.7.jar:./bin/int
ernal/../../lib/restli-common-1.15.7.jar:./bin/internal/../../lib/restli-server-1.15.7.jar:./bin/internal/../../lib/servlet-api-2.5-
20081211.jar:./bin/internal/../../lib/slf4j-api-1.7.18.jar:./bin/internal/../../lib/slf4j-log4j12-1.7.18.jar:./bin/internal/../../li
b/snakeyaml-1.18.jar:./bin/internal/../../lib/snappy-0.3.jar:./bin/internal/../../lib/sslext-1.2-0.jar:./bin/internal/../../lib/stru
ts-core-1.3.8.jar:./bin/internal/../../lib/struts-taglib-1.3.8.jar:./bin/internal/../../lib/struts-tiles-1.3.8.jar:./bin/internal/..
/../lib/velocity-1.7.jar:./bin/internal/../../lib/velocity-tools-2.0.jar:./bin/internal/../../extlib/*.jar:./bin/internal/../../plug
ins/*/*.jar:/u01/app/hadoop/conf:/u01/app/hadoop/*:/u01/app/apache-hive-2.3.4-bin/conf:/u01/app/apache-hive-2.3.4-bin/lib/*
2019/03/08 15:02:55.688 +0800 INFO [AzkabanWebServer] Starting Jetty Azkaban Web Server...
2019/03/08 15:02:55.708 +0800 INFO [AzkabanServer] Loading azkaban settings file from ./bin/internal/../../conf
2019/03/08 15:02:55.709 +0800 INFO [AzkabanServer] Loading azkaban properties file
2019/03/08 15:02:55.718 +0800 ERROR [StdOutErrRedirect] SLF4J: Class path contains multiple SLF4J bindings.
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/hadoop-2.6.5/share/hadoop/common/
lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/azkaban-3.50.0/azkaban-web-server
-0.1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.18.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: Found binding in [jar:file:/u01/app/apache-hive-2.3.4-bin/lib/log4j-s
lf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2019/03/08 15:02:55.719 +0800 ERROR [StdOutErrRedirect] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanat
ion.
2019/03/08 15:02:55.722 +0800 ERROR [StdOutErrRedirect] SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019/03/08 15:02:56.157 +0800 INFO [WebServerProvider] Excluded Cipher Suites: []
2019/03/08 15:02:56.161 +0800 INFO [log] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2019/03/08 15:02:56.161 +0800 INFO [WebServerProvider] Setting up connector with stats on: true
2019/03/08 15:02:56.163 +0800 INFO [WebServerProvider] Starting SSL server on port: 8443 # Max threads: 25
2019/03/08 15:02:56.679 +0800 INFO [ExecutorManager] Initializing local executor localhost:12321
2019/03/08 15:02:57.044 +0800 WARN [StorageCleaner] Project cleanup disabled. All artifacts will be stored.
2019/03/08 15:02:57.044 +0800 INFO [AzkabanProjectLoader] Using temp dir: /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/
temp
2019/03/08 15:02:57.044 +0800 INFO [AzkabanProjectLoader] Project version retention is set to 3
2019/03/08 15:02:57.046 +0800 WARN [XmlValidatorManager] Validator directory validators does not exist or is not a directory.
2019/03/08 15:02:57.046 +0800 WARN [XmlValidatorManager] Azkaban properties file does not contain the key project.validators.xml.fil
e
2019/03/08 15:02:57.103 +0800 INFO [TriggerManager] TriggerManager loaded.
2019/03/08 15:02:57.125 +0800 INFO [AzkabanWebServerModule] Loading user manager class azkaban.user.XmlUserManager
2019/03/08 15:02:57.233 +0800 INFO [XmlUserManager] Loading user azkaban
2019/03/08 15:02:57.233 +0800 INFO [XmlUserManager] Loading user metrics
2019/03/08 15:02:57.234 +0800 INFO [XmlUserManager] Loading user admin
2019/03/08 15:02:57.287 +0800 INFO [AzkabanWebServer] Loading built-in checker and action types
2019/03/08 15:02:57.290 +0800 INFO [CheckerTypeLoader] Registering checker BasicTimeChecker
2019/03/08 15:02:57.291 +0800 INFO [CheckerTypeLoader] Registering checker SlaChecker
2019/03/08 15:02:57.291 +0800 INFO [CheckerTypeLoader] Registering checker ExecutionChecker
2019/03/08 15:02:57.291 +0800 INFO [ActionTypeLoader] Registering action ExecuteFlowAction
2019/03/08 15:02:57.291 +0800 INFO [ActionTypeLoader] Registering action KillExecutionAction
2019/03/08 15:02:57.292 +0800 INFO [ActionTypeLoader] Registering action AlertAction
2019/03/08 15:02:57.292 +0800 INFO [ActionTypeLoader] Registering action CreateTriggerAction
2019/03/08 15:02:57.293 +0800 INFO [PluginCheckerAndActionsLoader] Loading plug-in checker and action types
2019/03/08 15:02:57.293 +0800 ERROR [PluginCheckerAndActionsLoader] plugin path plugins/triggers doesn't exist!
2019/03/08 15:02:57.314 +0800 INFO [AzkabanWebServer] Setting timezone to Asia/Shanghai
2019/03/08 15:02:57.314 +0800 INFO [AzkabanWebServer] Registering MBeans...
2019/03/08 15:02:57.325 +0800 INFO [AzkabanWebServer] Bean azkaban.jmx.JmxJettyServer registered.
2019/03/08 15:02:57.328 +0800 INFO [AzkabanWebServer] Bean azkaban.jmx.JmxTriggerManager registered.
2019/03/08 15:02:57.331 +0800 INFO [AzkabanWebServer] Bean azkaban.jmx.JmxExecutorManager registered.
2019/03/08 15:02:57.349 +0800 INFO [AzkabanWebServer] Bean org.apache.log4j.jmx.HierarchyDynamicMBean registered.
2019/03/08 15:02:57.349 +0800 INFO [AzkabanWebServer] ************* loginLoggerObjName is null, make sure there is a logger with nam
e azkaban.webapp.servlet.LoginAbstractAzkabanServlet
2019/03/08 15:02:57.350 +0800 INFO [ExecutorManager] Cleaning old logs from execution_logs
2019/03/08 15:02:57.353 +0800 INFO [AzkabanWebServer] Setting up web resource dir /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-S
NAPSHOT/web
2019/03/08 15:02:57.383 +0800 INFO [ExecutorManager] Cleaning old log files before 2018-12-14T15:02:57.350+08:00
2019/03/08 15:02:57.385 +0800 INFO [ExecutorManager] Cleaned up 0 log entries.
2019/03/08 15:02:57.385 +0800 INFO [ExecutorManager] log clean up time: 0 seconds.
2019/03/08 15:02:57.435 +0800 INFO [JdbcTriggerImpl] Loading all triggers from db.
2019/03/08 15:02:57.437 +0800 INFO [JdbcTriggerImpl] Loaded 0 triggers.
2019/03/08 15:02:57.438 +0800 INFO [log] jetty-6.1.26
2019/03/08 15:02:58.736 +0800 INFO [ProjectManagerServlet] downloadBufferSize: 8192
2019/03/08 15:02:58.879 +0800 INFO [RestLiAnnotationReader] Processed actions resource 'azkaban.restli.UserManagerResource'
2019/03/08 15:02:58.881 +0800 INFO [RestLiAnnotationReader] Processed actions resource 'azkaban.restli.ProjectManagerResource'
2019/03/08 15:02:59.057 +0800 INFO [log] Started [email protected]:8443
2019/03/08 15:02:59.057 +0800 INFO [AzkabanWebServer] Server started

查看JPS,会出现AzkabanWebServer:

jps

在这里插入图片描述
6.3 登录管理界面

打开浏览器,输入https://10.200.4.117:8443,登录admin用户:
在这里插入图片描述
完毕。

猜你喜欢

转载自blog.csdn.net/andyguan01_2/article/details/88173798
0条评论
添加一条新回复