Azkaban 简介及安装(3.70)

1.Azkaban 简介

官网：https://azkaban.github.io/

Azkaban 一个批量工作流任务调度器，使用Java语言开发。用于在一个工作流内以一个特定的顺序运行一组工作和流程。Azkaban使用job配置文件建立任务之间的依赖关系，并提供一个易于使用的web用户界面维护和跟踪你的工作流。

Azkaban 的优点

提供功能清晰，简单易用的Web UI界面
提供job配置文件快速建立任务和任务之间的依赖关系
提供模块化和可插拔的插件机制，原生支持command、Java、Hive、Pig、Hadoop
基于Java开发，代码结构清晰，易于二次开发

Azkaban 安装模式

Azkaban 有三种部署方式：单服务模式、2个服务模式、分布式多服务模式

solo server model（单服务模式）：该模式中 webServer 和 executorServer 运行在同一个进程中，进程名是AzkabanSingleServer。可以使用自带的H2数据库或者配置mysql数据。该模式适用于小规模的使用。此模式合适个人试用使用。它也可以用在小规模的使用案例中。
two server model（2个服务模式）：数据库为mysql，采用主从设置进行备份，管理服务器（webServer）和执行服务器（executorServer）在不同进程中运行，这种模式下，管理服务器和执行服务器互不影响。适用在较重的生成环境中。
multiple-executor（分布式多服务模式）：存放元数据的数据库为mysql，采用主从设置进行备份，管理服务器（webServer）和执行服务器（executorServer）在不同进程中运行。多个 Executor 模式为最重的生产环境。

2.Azkaban 3.x 安装(two-server 模式)

安装前准备

安装JDK1.8
安装MySQL

下载源码

git clone https://github.com/azkaban/azkaban.git

编译源码

cd azkaban; ./gradlew build installDist

编译报错：

> Task :az-crypto:test 

azkaban.crypto.DecryptionTest > testV1_1 FAILED
    java.lang.RuntimeException: java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
        at azkaban.crypto.Crypto.decrypt(Crypto.java:76)
        at azkaban.crypto.DecryptionTest.testV1_1(DecryptionTest.java:35)

        Caused by:
        java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
            at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:57)
            at azkaban.crypto.Crypto.decrypt(Crypto.java:74)
            ... 1 more

            Caused by:
            org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
                at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
                at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1050)
                at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)
                at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:55)
                ... 2 more

azkaban.crypto.EncryptionTest > testEncryption FAILED
    org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.encrypt(StandardPBEByteEncryptor.java:924)
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.encrypt(StandardPBEStringEncryptor.java:642)
        at azkaban.crypto.CryptoV1_1.encrypt(CryptoV1_1.java:42)
        at azkaban.crypto.Crypto.encrypt(Crypto.java:58)
        at azkaban.crypto.EncryptionTest.testEncryption(EncryptionTest.java:28)

5 tests completed, 2 failed

解决方法：

1.下载jre：https://www.oracle.com/technetwork/cn/java/javase/downloads/jce8-download-2133166-zhs.html

2.在/usr/java/jdk1.8.0_141-cloudera/jre/lib/security目录下替换原有的

3.重新编译 ./gradlew build installDist

启停单机版的服务

cd azkaban-solo-server/build/install/azkaban-solo-server
bin/start-solo.sh 
bin/shutdown-solo.sh

数据库设置

# 建用户 建库 授权 刷新权限
CREATE DATABASE azkaban_zxl;
CREATE USER 'azkaban_zxl'@'%' IDENTIFIED BY 'azkaban_zxl';
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban_zxl.* to 'azkaban_zxl'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

# 在/etc/my.cnf中配置Mysql Packet Size的大小
[mysqld]
...
max_allowed_packet=1024M

# 重启mysql
/sbin/service mysqld restart

# 创建Azkaban表
source /root/apps/create-all-sql.sql

配置web-server

vi /root/azkaban/azkaban-web-server/conf/azkaban.properties 

default.timezone.id=Asia/Shanghai
#database.type=h2
#h2.path=./h2
#h2.create.tables=true
database.type=mysql
mysql.port=3306
mysql.host=10.xxx.xxx.xx
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
executor.port=12321
azkaban.use.multiple.executors=true

将azkaban-web-server/conf/azkaban.properties 同步到 /root/azkaban/azkaban-exec-server/conf下

启动分布式多服务模式

多服务模式是多个executorServer分布在不同服务器上，只需要将/root/azkaban/azkaban-exec-server拷贝到不同机器上即可组成分布式。

先启动executorServer bin/start-exec.sh, 再启动webServer bin/start-web.sh。

**Note:**启动executorServer时会将executorServer的host,port同步到mysql的executors表中;

root@localhost|azkaban_zxl>select * from executors;
+----+---------------------------------------+-------+--------+
| id | host                                  | port  | active |
+----+---------------------------------------+-------+--------+
|  1 | 10.xxx.xx.xx 					     | 12321 |      0 |
|  2 | 10.xxx.xx.xx 						 | 12321 |      0 |
+----+---------------------------------------+-------+--------+

启动webServer时会如下错误，是因为自动同步到executors表中的executor默认是未激活，需要在数据库中update：

2019/03/22 10:00:47.210 +0800 INFO [ExecutorManager] [Azkaban] Initializing executors from database.
2019/03/22 10:00:47.418 +0800 ERROR [ExecutorManager] [Azkaban] No active executors found
2019/03/22 10:00:47.418 +0800 ERROR [StdOutErrRedirect] [Azkaban] Exception in thread "main" 
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban] azkaban.executor.ExecutorManagerException: No active executors found
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ActiveExecutors.setupExecutors(ActiveExecutors.java:52)
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ExecutorManager.setupExecutors(ExecutorManager.java:231)
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ExecutorManager.initialize(ExecutorManager.java:155)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.executor.ExecutorManager.start(ExecutorManager.java:169)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.webapp.AzkabanWebServer.launch(AzkabanWebServer.java:235)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban]       at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:228)

root@localhost|azkaban_zxl>update executors set active=1 where id=1;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

root@localhost|azkaban_zxl>update executors set active=1 where id=2;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

shylin