Build a Azkaban (keytool SSL certificate generation)

Foreword

Azkaban Linkedin company launched by a batch workflow task scheduler, primarily for operating a set of work and processes to a particular order in a workflow, its configuration is simple key: value pairs manner by dependencies provided to configuration dependencies. Azkaban use of job profiles to establish dependencies between tasks, and provides an easy to use web user interface to maintain and track your workflow.

Preparing the Environment

Since java processes are running on the java virtual machine, you need to configure the java environment! Advice is above 1.8. If it is low, for example, version 1.5, out of memory may occur!

The deployment follows the author

Mysql remote login accounts through the machine 102, the operation performed on the data table! azkaban rely on through the azkaban-sql-script to perform decompression in mysql console table, create a table of mysql established in the create-all-sql-2.5.0.sql script!

Choose the following versions of azkaban

Brief introduction

• AzkabanWebServer: AzkabanWebServer Azkaban is the main manager entire workflow system, which is responsible for user login authentication, project management, timing of execution of the workflow, workflow tracking the progress of the implementation of a series of tasks.
• AzkabanExecutorServer: responsible for specific work flow filed, implementation, to coordinate their tasks are performed by mysql database.
• Relational database (MySQL): Most store execution flow state, AzkabanWebServer and AzkabanExecutorServer need to access the database.

Install

Create a directory
in / opt / module below to create a directory azkaban

mkdir azkaban

executor decompression

Extract to / opt / module / azkaban

tar -zxvf azkaban-executor-server-2.5.0.tar.gz -C /opt/module/azkaban

server unzip

tar -zxvf azkaban-web-server-2.5.0.tar.gz -C /opt/module/azkaban

sql script decompression

tar -zxvf tar -zxvf azkaban-web-server-2.5.0.tar.gz  -C /opt/module/azkaban

Installation and import database sql mysql

Before mysql5.6 installation has been done, and posted a link here. Database install
the following database operations!

1. 103 on the remote connection 102 on the root account! Provided there is a client (and server version must be consistent!)!

mysql -h hadoop102 -P3306 -uroot -p

2. After connecting to execute commands on

source /opt/module/azkaban/azkaban-2.5.0/create-all-sql-2.5.0.sql

3. The quit so that the database transaction is committed!

Configuration azkaban environment

Generated keystore

• Keytool is java certificate data management tool that allows users to manage their own public / private key pairs and associated certificates.
• -keystore specify the key database name and location (there will be all kinds of information generated .keystore file)
• -genkey (or -genkeypair) to generate the key pair
• -alias is the alias for the specified key generation, if there is no default is mykey
• Specified key algorithm RSA -keyalg / DSA default is DSA

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

Directory in which to perform the above command will generate a file where!

Move the file to azkaban web server root directory

mv keystore /opt/module/azkaban/server/

Time synchronization configuration

If you install the system already selected time zone Shanghai, only need to open the ntp service to synchronize time to other servers to perform crontab scheduled tasks can do a synchronization request to the synchronization time machine!
How to configure ntp time synchronization service! Please click! !
If you have a good time synchronization, the following operations may be ignored!

sudo crontab -e

On a good server node to configure the time zone
1) if the time zone configuration file Asia / Shanghai does not exist in / usr / share / zoneinfo / directory, it is necessary to generate a tzselect.

[zhengkw@hadoop103 azkaban]$ tzselect

Please identify a location so that time zone rules can be set
correctly. Please select a continent or ocean. 1) Africa 2) Americas
3) Antarctica 4) Arctic Ocean 5) Asia 6) Atlantic Ocean 7)
Australia 8) Europe 9) Indian Ocean 10) Pacific Ocean 11) none - I
want to specify the time zone using the Posix TZ format.
选择5亚洲

#? 5

Please select a country. 1) Afghanistan 18) Israel
35) Palestine 2) Armenia 19) Japan 36)
Philippines 3) Azerbaijan 20) Jordan 37)
Qatar 4) Bahrain 21) Kazakhstan 38) Russia
5) Bangladesh 22) Korea (North) 39) Saudi Arabia
6) Bhutan 23) Korea (South) 40) Singapore 7)
Brunei 24) Kuwait 41) Sri Lanka 8)
Cambodia 25) Kyrgyzstan 42) Syria 9) China
26) Laos 43) Taiwan 10) Cyprus 27)
Lebanon 44) Tajikistan 11) East Timor 28)
Macau 45) Thailand 12) Georgia 29)
Malaysia 46) Turkmenistan 13) Hong Kong 30)
Mongolia 47) United Arab Emirates 14) India
31) Myanmar (Burma) 48) Uzbekistan 15) Indonesia 32)
Nepal 49) Vietnam 16) Iran 33) Oman
50) Yemen 17) Iraq 34) Pakistan

Select China china!

#? 9

Please select one of the following time zone regions. 1) Beijing Time
2) Xinjiang Time

Select GMT

#? 1

The following information has been given:

    China
    Beijing Time

Therefore TZ=‘Asia/Shanghai’ will be used. Local time is now: Thu
Oct 18 16:24:23 CST 2018. Universal Time is now: Thu Oct 18 08:24:23
UTC 2018. Is the above information OK? 1) Yes 2) No

#? 1

You can make this change permanent for yourself by appending the line
TZ=‘Asia/Shanghai’; export TZ to the file ‘.profile’ in your home directory; then log out and log in again.

Here is that TZ value again, this time on standard output so that you
can use the /usr/bin/tzselect command in shell scripts: Asia/Shanghai

2) Copy the time zone file, the overlay system local time zone configuration

[zhengkw@hadoop103 azkaban]$ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

3) Cluster Time Synchronization (to more three windows)

[zhengkw@hadoop103 azkaban]$ sudo date -s '2018-10-18 16:39:30'

Configuration executor

Open the / opt / module / azkaban / executor / conf

#Azkaban
#时区
default.timezone.id=Asia/Shanghai
# Azkaban JobTypes Plugins
#jobtype 插件所在位置
azkaban.jobtype.plugin.dir=plugins/jobtypes

#Loader for projects
executor.global.properties=/opt/module/azkaban/executor/conf/global.properties
azkaban.project.dir=projects

database.type=mysql
mysql.port=3306
mysql.host=hadoop102
mysql.database=azkaban
mysql.user=root
mysql.password=sa
mysql.numconnections=100

# Azkaban Executor settings
#最大线程数
executor.maxThreads=50
#端口号(如修改,请与web服务中一致)
executor.port=12321
#线程数
executor.flow.threads=30

Configuring server

Modify /opt/module/azkaban/server/conf/azkaban.properties

E-mail notification configuration

Add the following parameters in /opt/module/azkaban/server/conf/azkaban.properties!

host to send mail server
password to a third party authorization code

Obtain authorization code

Modify azkaban-users.xml

azkaban web server installation directory conf directory, according to the following configuration file modified azkaban-users.xml increase administrator user

<azkaban-users>
        <user username="azkaban" password="azkaban" roles="admin" groups="azkaban" />
        <user username="metrics" password="metrics" roles="metrics"/>
        <user username="zhengkw" password="qweasd" roles="admin,metrics" groups="azkaban" />

<role name="admin" permissions="ADMIN" />
        <role name="metrics" permissions="METRICS"/>
</azkaban-users>

Start Service

Start executor

[zhengkw@hadoop103 executor]$ pwd
/opt/module/azkaban/executor
[zhengkw@hadoop103 executor]$ bin/azkaban-executor-start.sh

Start web

[zhengkw@hadoop103 server]$ pwd
/opt/module/azkaban/server
[zhengkw@hadoop103 server]$ bin/azkaban-web-start.sh

See jps

Open a browser test

Here is the configuration of https services, the use of SSL encryption certificates for encryption to ensure the security of packet transmission, because it is generated by keytool certificate, the authority is not generated, so the first time there will be alarm when landing!
Enter https: // hadoop103: 8443
Click inside to continue to visit

Enter the user login page in which users can configure the
successful visit

Close Service

web

bin/azkaban-web-shutdown.sh 

exec

bin/azkaban-executor-shutdown.sh 

to sum up

• Note Time synchronization in order to view the log information!
• Must pay attention to all kinds of information when configuring the connection to the database!
• Users with landing az conf in the web / azkaban-users.xml configure!
• SSL is an alternative way to start, but to be encrypted so that it is generally safe!

Encountered pit

• Because the certificate is a tool of production, will be printed in the console warn level information can be ignored! If you want to shield you can be modified in the print log4j level!
• As the front of the executor in mysql configuration errors caused by not connect to database after startup! But also caused the port occupied!
  

Use the command to view the port 12321 (occupied when exec start port) occupied by the process

lsof -i:12321

Forced to kill!