Azkaban笔记

Azkaban笔记
1.下载
https://github.com/azkaban/azkaban/releases
本次下载的是最新3.80.0版本。3.80.0.tar.gz

2.上传服务器，解压
  3.80.0.tar.gz解压重命名/home/hadoop/azkaban

3.修改gradle配置国内镜像
vi azkaban/build.gradle
repositories {
    maven { url 'http://maven.aliyun.com/nexus/content/groups/public/' }
    maven { url 'http://maven.aliyun.com/nexus/content/repositories/jcenter'}
}

4.Mysql配置
vi /home/hadoop/azkaban/azkaban-exec-server/src/main/resources/conf/azkaban.properties
vi /home/hadoop/azkaban/azkaban-web-server/src/main/resources/conf/azkaban.properties
mysql.user=<username>
mysql.password=<password>

5.利用gradle进行安装(要提前安装git)
cd /home/hadoop/azkaban
./gradlew build
./gradlew clean
./gradlew installDist
./gradlew test
./gradlew build -x test

6.创建数据库azkaban，导入sql
mysql > create database azkaban;

cd /home/hadoop/azkaban/azkaban-db
../gradlew build installDist

cd /home/hadoop/azkaban/azkaban-db/build/sql
mysql -h192.168.15.45 -P3306 -uroot -p
mysql > source create-all-sql-0.1.0-SNAPSHOT.sql;

7.启动azkaban-exec-server
cd /home/hadoop/azkaban/azkaban-exec-server/build/install/azkaban-exec-server
./bin/start-exec.sh

8.启动azkaban-web-server
cd /home/hadoop/azkaban/azkaban-web-server/build/install/azkaban-web-server
./bin/start-web.sh

报错azkaban.executor.ExecutorManagerException: No active executors found
没有找到活动的executors，需在MySQL数据库里设置端口为12321的executors表的active为1
进入mysql，update azkaban.executors set active=1;

9.web访问账号配置
azkaban-users.xml
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>
  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

10.示例-1
#bar.job
type=command
dependencies=foo
command=echo bar

#foo.job
type=command
command=echo foo

11.示例-2
#hdfs.job
type=command
command=/home/hadoop/hadoop/bin/hadoop fs -mkdir /azkaban

12.示例-3
#mkdir.job
type=command
command=/home/hadoop/hadoop/bin/hadoop fs -mkdir -p /wordcount/input/

#putfile.job
type=command
dependencies=mkdir
command=/home/hadoop/hadoop/bin/hadoop fs -put someword.txt /wordcount/input/someword.txt

#mrwc.job
type=command
dependencies=putfile
command=/home/hadoop/hadoop/bin/hadoop jar wordcount8.jar com.kizzle.hadoop.mpreduce.wc.WordCount

someword.txt
wordcount8.jar

13.示例-4
#hive.job
type=command
command=/home/hadoop/hive/bin/hive -f "hive_test.hql"

=========>hive_test.hql
use default;
create table if not exists azkaban(
    id int,
    name string)
row format delimited fields terminated by ',';
load data local inpath 'azkaban.txt' overwrite into table azkaban;

========>azkaban.txt
1,zhangsan
2,lisi
3,wangwu
4,zhaoliu
5,qianqi
6,wangba

补充激活executor的端口号，
curl -G "hadoop01:40471/executor?action=activate" && echo

猜你喜欢