Preparation : ( If it is already installed , it can be skipped )
A-Basic Command Installation
Since the test machine chooses the minimum installation of centOS7, many Linux commands are not available, so choose to install these commands through yum first
1-yum -y install wget
2-yum -y install cmake
3-yum -y install readline-devel zlib-devel openssl-dev
4-yum install -y gcc
5-yum install -y gcc-c++
6-yum install -y bison.x86_64
7-yum -y install unzip
8-yum -y install git
B-JDK1.8 installation (azkaban strongly depends on jdk1.8 )
Azkaban depends on JDK1.8, so you need to install JDK1.8 first
1- Create a folder to store azkaban and jdk (this custom)
mkdir /software
cd /software
2- Download the jdk installation package through wget (directly copy the content below and press Enter and wait for the download to complete)
wget --no-cookies --no-check-certificate --header "Cookie:gpw_e24=http%3A%2F%2Fwww.oracle.com%2F;oraclelicense=accept-securebackup-cookie""http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz"
3- Unzip the jdk installation package
tar -zxvfjdk-8u141-linux-x64.tar.gz
The file jdk1.8.0_141 is the decompressed jdk file
4- Configure jdk environment variables
vim /etc/profile add the following to the end of the file
export JAVA_HOME=/software/jdk1.8.0_141 #Absolute path of jdk installation file export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/rt.jar
If $JAVA_HOME is not imported in PATH , add : $JAVA_HOME/bin at the end of PATH , don't lose the colon
5- Verify jdk
Execute source /etc/profile to reload environment variables Command line input: java -version
The display version number is 1.8.0_141 .
compile azkaban
1- Download the azkaban source package
cd /software wget https://github.com/azkaban/azkaban/archive/master.zip
2- Unzip the zip installation package you just downloaded
unzip master.zip get the file azkaban-master cd /software/azkaban-master/gradle/wrapper
Run the wget command to download gradle-4.1-all.zip
wget https://services.gradle.org/distributions/gradle-4.1-all.zip
vim /software/azkaban-master/gradle/wrapper/gradle-wrapper.properties
Modify the last item, keep only the installation package name
3- Install npm ( no installation may report an error ) :
curl --silent --locationhttps://rpm.nodesource.com/setup_5.x | bash -
yuminstall -y nodejs
4- Compile
cd /software/azkaban-master [Execute the following command to start compiling, 7-10 minutes] ./gradlew clean ./gradlewbuild installDist -x test
DB installation. (If the machine is already installed, go directly to the third step)
Step 1: Add MariaDB yum repository
First , add MariaDB 's YUM configuration file MariaDB.repo file in the /etc/yum.repos.d/ directory of the CentOS operating system
vi /etc/yum.repos.d/MariaDB.repo
Add the following to the file to save:
[mariadb]
name = MariaDB
baseurl =http://yum.mariadb.org/10.2/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
Step 2: Install MariaDB
Easily install MariaDB via yum command.
yum install MariaDB-server MariaDB-client -y
After MariaDB is installed, start the database service daemon immediately.
systemctl start mariadb
Set MariaDB to start the service automatically after the operating system restarts.
systemctl enable mariadb
Step 3: Create the azkaban library (critical)
Enter mysql (enter directly) to enter mariadb
mysql> CREATE DATABASE azkaban;
Create azkaban user
mysql> CREATE USER'azkaban'@'％'IDENTIFIED BY 'azkaban';
Authorize azkaban user
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ONazkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
Load azkaban's table file
mysql> use azkaban
mysql> source/software/azkaban-master/azkaban-db/build/sql/create-all-sql-0.1.0-SNAPSHOT.sql
Add the ip/domain name and port of the executors in the azkaban library
mysql> insert into executors(host,port)values("ip或者hostname",12321);
mysql> insert into executors(host,port)values("ip或者hostname",12321);
Set the status of all nodes to 1
mysql> update executos set active=1;
install azkaban
1- Copy the compiled installation package
mkdir /software/azkaban cd /software/azkaban
cp/software/azkaban-master/azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz /softwear/azkaban
cp/software/azkaban-master/azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz /softwear/azkaban
cp/software/azkaban-master/azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz/softwear/azkaban
2- Unzip
tar -zxvfazkaban-web-server-0.1.0-SNAPSHOT.tar.gz
tar -zxvfazkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
tar -zxvfazkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
3- After deleting the installation package you get three files
rm -rf *.tar.gz
4- Copy the conf and plugins files in the solo-server directory to web-server and executor-server
cp -r /software/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT/conf/software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/ cp -r /software/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT/plugins/software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/ cp -r /software/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT/plugins/software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/ cp -r /software/azkaban/azkaban-solo-server-0.1.0-SNAPSHOT/conf/software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/
5- Create the folder
mkdir /software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/extlib
mkdir /software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/extlib
6- Download mysql-connector to two extlib directories
cd /software
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar -zxvf mysql-connector-java-5.1.46.tar.gz
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46.jar/software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/extlib/
cp mysql-connector-java-5.1.46.jar/software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT/extlib/
7- Configure jetty ssl for azkaban-web-server
cd /software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT
[a-config jetty ssl]
输入命令: keytool -keystore keystore -alias jetty -genkey -keyalg RSA [The input process is as shown below, all the passwords in it are azkaban] After execution, a keystore file will be generated. Make sure that the file is under /software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT
[b-modify the conf/azkaban.properties configuration file]
cd /software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT/conf mv azkaban.properties azkaban.properties.bak touch azkaban.properties
vim azkaban.properties
[Copy the content below and save]
#cat azkaban.properties # Azkaban Personalization Settings azkaban.name=Test azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir = web / default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=conf/azkaban-users.xml # Loader for projects executor.global.properties=conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.maxThreads=25 jetty.ssl.port=8443 jetty.port=8081 jetty.keystore=keystore jetty.password=azkaban jetty.keypassword=azkaban jetty.truststore=keystore jetty.trustpassword=azkaban # Azkaban Executor settings executor.port=12321 # mail settings mail.sender = mail.host= mail.password= job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir = plugins / jobtypes database.type=mysql mysql.port=3306 mysql.host=172.16.21.23 mysql.database=azkaban mysql.user=root mysql.password=123456 mysql.numconnections=100 #multiple settings azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1
[c-upload log4j file]
(You can also directly copy \software\azkaban-master\azkaban-solo-server\src\main\resources\log4j.properties)
Create a logs folder in the same directory of conf.
mkdir logs
Create log4j.properties file in conf directory
cd conf touch log4j.properties vim log4j.properties
[Enter the following to save and exit]
log4j.rootLogger=DEBUG,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}%-5p %c{1}:%L - %m%n
[d-start azkaban]
cd /software/azkaban/azkaban-web-server-0.1.0-SNAPSHOT bin/start-web.sh
After startup, jps view the process
8-Configure executor-server
[a-config azkaban.properties] cd/software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT mvconf/azkaban.properties conf/azkaban.properties.bak touchconf/azkaban.properties vimconf/azkaban.properties
[Enter the following to save and exit]
# catazkaban.properties # AzkabanPersonalization Settings default.timezone.id=Asia/Shanghai # Loader forprojects executor.global.properties=conf/global.properties azkaban.project.dir=projects # Azkaban pluginsettings azkaban.jobtype.plugin.dir = plugins / jobtypes
#DB settings database.type=mysql mysql.port=3306 mysql.host=172.16.21.23 mysql.database=azkaban mysql.user=root mysql.password=123456 mysql.numconnections=100
# AzkabanExecutor settings executor.maxThreads=50 executor.port=12321 executor.flow.threads=30 azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1
[b-upload log4j.properties file]
(You can also directly copy \software\azkaban-master\azkaban-solo-server\src\main\resources\log4j.properties)
Create a logs folder in the same directory of conf. mkdir logs
Create log4j.properties file in conf directory cd conf touch log4j.properties vim log4j.properties
[Enter the following to save and exit]
log4j.rootLogger=DEBUG,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}%-5p %c{1}:%L - %m%n
[c-start executor]
bin/start-exec.sh
After startup, jps view the process
[d-multi-executor]
Login to another node 172.16.21.24 Create the corresponding directory mkdir -p / cd /software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT
Copy the entire directory of the configured executor through scp
scp -r root:172.16.21.23:/software/azkaban/azkaban-exec-server-0.1.0-SNAPSHOT .
bin/start-exec.sh #Start executor
At this point, the construction is completed. The problems encountered after the construction is completed can basically find solutions on the Internet, and I will not repeat them here.
The only thing that needs to be explained is that if the DB is installed with mysql, it may happen that the azkaban executor cannot change its own active to 1 in the database, resulting in an error at startup: no available executor could be found.
Here are two solutions:
1- Manually set the active value of executor to 1 after running bin/start-exec.sh each time.
Or add a shell statement at the end of the startup script to set its active value to 1
2- Use mariaDB (that's how mine was solved)
Reference documentation:
https://blog.csdn.net/lotusctxcm/article/details/70779296?utm_source=itdadao&utm_medium=referral