Azkaban multi-executor mode refers to deploying executors on multiple nodes in the cluster. In this mode, Azkaban web server will select one of the Executors to execute the task according to the policy.
Since the script we need to hand over to Azkaban to schedule, and the Hive, Sqoop and other applications required by the script are only deployed in hadoop102, in order to ensure the smooth execution of the task, we must choose one of the following two options, 推荐使用方案二.
Option 1: Specify a specific Executor (hadoop102) to execute the task.
1) In the executors table of the azkaban database in MySQL, query the id of the executor on hadoop102.
mysql> use azkaban;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from executors;
±—±----------±------±-------+
| id | host | port | active |
±—±----------±------±-------+
| 1 | hadoop103 | 35985 | 1 |
| 2 | hadoop104 | 36363 | 1 |
|
3| hadoop102 | 12321 | 1 |±—±----------±------±-------+
3 rows in set (0.00 sec)
2) Add the useExecutor attribute when executing the workflow, as follows
Option 2: Deploy the scripts and applications required for the task on all nodes where the Executor is located.
1) Distribution script, hive, sqoop, spark, my_env.sh