oozie4.3.1安装与配置

其他 2018-09-17 08:02:32 阅读次数: 0

1、环境：操作系统aarch64——centos 7.4

2、依赖环境：

       a、Mysql           b、pig                c、Java                d、hadoop

3、安装MySQL：详见另一篇博客https://blog.csdn.net/smart9527_zc/article/details/81153053

      创建oozie数据库，创建oozie用户名和密码，并赋值oozie账号登陆oozie数据库的权限

[root@aarch64vm-centos74-010 mysql]# ./bin/mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.21

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> set password = "smartcore" ;
Query OK, 0 rows affected (0.02 sec)

mysql> create database oozie;
Query OK, 1 row affected (0.02 sec)

mysql>  show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| oozie              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.04 sec)

mysql> grant all privileges on oozie.* to oozie@'%' identified by 'oozie';
Query OK, 0 rows affected, 1 warning (0.02 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

mysql> quit
Bye
[root@aarch64vm-centos74-010 mysql]#

4、安装pig

      （1）下载pig：http://mirrors.shuosc.org/apache/pig/

      （2）解压到/usr/local/pig目录下

$:mkdir -p /usr/local/pig
$:tar -xvf pig-0.17.0.tar.gz -C /usr/local/pig

（3）配置profile文件

$:vim /etc/profile
末尾添加：
        #pig
        export PIG_HOME=/usr/local/pig/pig-0.17.0
        export PIG_CLASSPATH=/usr/local/hadoop/etc/hadoop
        export PATH=$PATH:$PIG_HOME/bin

保存退出重启生效：source   /etc/profile

检测是否安装成功：

$:pig -x local
18/07/27 14:52:23 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
18/07/27 14:52:23 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2018-07-27 14:52:23,843 [main] INFO  org.apache.pig.Main - Apache Pig version 0.17.0 (r1797386) compiled Jun 02 2017, 15:41:58
2018-07-27 14:52:23,846 [main] INFO  org.apache.pig.Main - Logging error messages to: /root/test/zhaochuang/pig_1532674343816.log
2018-07-27 14:52:24,244 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2018-07-27 14:52:29,519 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2018-07-27 14:52:29,648 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2018-07-27 14:52:33,446 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2018-07-27 14:52:33,766 [main] INFO  org.apache.pig.PigServer - Pig Script ID for the session: PIG-default-c070ee49-7069-498a-8e82-ff941aef1e0d
2018-07-27 14:52:33,767 [main] WARN  org.apache.pig.PigServer - ATS is disabled since yarn.timeline-service.enabled set to false
grunt>

5、安装Java

（1）下载Java-1.8.0_141安装包：https://download.csdn.net/download/smart9527_zc/10575966

（2）安装Java

$:mkdir -p /usr/local/java
$:tar -xvf jdk-8u161-linux-arm64-vfp-hflt.tar.gz -C /usr/local/java
$:vim /etc/profile

# set java environment
export JAVA_HOME=/usr/java/jdk1.8.0_161/
export JRE_HOME=/usr/java/jdk1.8.0_161/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

$:source /etc/profile

（3）测试

$:java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

6、下载oozie4.3.1安装包并上传源码到服务器

7、解压编译

$:mkdir -p ~/test/oozie-4.3.1_install
$:tar -xzvf oozie-4.3.0.tar.gz -C ~/test/oozie-4.3.0_install
$:cd ~/test/oozie-4.3.0_install/oozie-4.3.1
$:./bin/mkdistro.sh -Dhadoop.auth.version=2.7.4 -DskipTests

编译报错：

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.2:test (default-test) on project oozie-core: Failure or timeout -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :oozie-core

ERROR, Oozie distro creation failed

原因及解决：

maven-surefire-plugin:2.12.2属于检测文件，第一次编译时后面未加-DskipTests，加上后会跳过检测编译成功！

8、安装oozie-server

（1）oozie-4.3.0-distro.tar.gz存放于/root/test/oozie-4.3.0_install/oozie-4.3.1/distro/target目录下，解压缩包oozie-4.3.0-distro.tar.gz到/usr/local/oozie/目录下。

$:cd /root/test/oozie-4.3.1_install/oozie-4.3.1/distro/target
$:ls
antrun  archive-tmp  classes  maven-archiver  maven-shared-archive-resources  oozie-4.3.1-distro  oozie-4.3.1-distro.tar.gz  oozie-distro-4.3.1.jar  test-classes  tomcat
$: tar -xvf oozie-4.3.1-distro.tar.gz -C /usr/local/oozie/
$:cd /usr/local/oozie/oozie-4.3.1

（2）分别解压缩三个压缩包oozie-client-4.3.0.tar.gz、oozie-examples.tar.gz、oozie-sharelib-4.3.0.tar.gz

$:tar xzvf oozie-examples.tar.gz
$:tar xzvf oozie-sharelib-4.3.0.tar.gz
$:tar xzvf oozie-client-4.3.0.tar.gz

（3）在当前目录创建libext，然后复制hadoop的lib目录下的文件到libext

$:mkdir libext
$:cp ${HADOOP_HOME}/share/hadoop/*/*.jar libext/
$:cp ${HADOOP_HOME}/share/hadoop/*/lib/*.jar libext/

（4）下载并添加ext-2.2.zip和mysql-connector-java-5.1.38.jar驱动包到libext（存放于108服务器的oozie目录下）

（5）将mysql-connector-java-5.1.38.jar放到/usr/local/oozie/oozie-4.3.1/share/lib目录下

（6）修改/usr/local/oozie/oozie-4.3.1/conf/oozie-site.xml

        <property>
                <name>oozie.service.JPAService.jdbc.driver</name>
                <value>com.mysql.jdbc.Driver</value>
        </property>
        <property>
                <name>oozie.service.JPAService.jdbc.url</name>
                <value>jdbc:mysql://localhost:6603/oozie</value>
        </property>
        <property>
                <name>oozie.service.JPAService.jdbc.username</name>
                <value>oozie</value>
        </property>
        <property>
                <name>oozie.service.JPAService.jdbc.username</name>
                <value>root</value>
        </property>
        <property>
                <name>oozie.service.JPAService.jdbc.password</name>
                <value>smartcore</value>
        </property>
        <property>
                <name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name>
                <value>*</value>
        </property>
        <property>
                <name>oozie.service.JPAService.create.db.schema</name>
                <value>false</value>
        </property>
     <property>
                <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
                <value>*=/usr/hadoop/etc/hadoop</value>
        </property>
        <property>
                <name>oozie.service.HadoopAccessorService.action.configurations</name>
                <value>*=/usr/hadoop/etc/hadoop</value>
        </property>
        <property>
                <name>oozie.service.SparkConfigurationService.spark.configurations</name>
                <value>*=/usr/local/spark/spark-1.6.1/conf</value>
        </property>
        <property>
                <name>oozie.service.WorkflowAppService.system.libpath</name>
                <value>/user/root/share/lib</value>
        </property>
        <property>
                <name>oozie.use.system.libpath</name>
                <value>true</value>
        </property>
        <property>
                <name>oozie.subworkflow.classpath.inheritance</name>
                <value>true</value>
        </property>

（8）修改hadoop目录下的/etc/hadoop/core-site.xml，添加

<property>
    <name>hadoop.proxyuser.jingxiao.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.jingxiao.groups</name>
    <value>*</value>
  </property>

（9）配置环境变量

$:vim /etc/profile

#oozie
export OOZIE_HOME=/usr/local/oozie/oozie-4.3.1
export PATH=$PATH:$OOZIE_HOME/bin
export OOZIE_CONFIG=/usr/local/oozie/oozie-4.3.1/conf
export OOZIE_URL=http://192.168.122.149:11000/oozie
$:source /etc/profile

（10）回到oozie目录，创建/user/root目录，然后将share目录上传到hdfs中的/user/root目录。

$:hadoop fs -put ./share/ /user/root

（11）打war包

$:./bin/oozie-setup.sh prepare-war

（12）创建元数据表

bin/ooziedb.sh create -sqlfile oozie.sql -run

9，启动和测试

（1）重新启动hadoop

（2）进入/usr/local/oozie/oozie-4.3.1启动oozie服务

bin/oozied.sh start

（3）登录http://10.2.151.183:11000/oozie，网页显示正常

（4）执行测试

a、将examples这个文件上传到hdfs中的/user/${user.name} 中，我采用的是root这个用户，所以是/user/root。

hdfs dfs -put ./examples/ /user/root

b、配置运行测试的环境examples/apps/demo/job.properties

nameNode=hdfs://10.2.151.183:9000

jobTracker=aarch64vm-centos74-005:8032

修改成为当前环境对应的参数。

c、执行测试

$:oozie job -oozie http://localhost:11000/oozie -config examples/apps/demo/job.properties -run
.....
job: 0000005-180503102542331-oozie-root-W

d、查看测试执行状态

oozie job -oozie http://192.168.122.145:11000/oozie -info 0000005-180503102542331-oozie-root-W

OK！！！！

猜你喜欢

转载自blog.csdn.net/smart9527_zc/article/details/81305061
今日推荐
富文本编辑器 Quill 2.0 重磅发布，特性、可靠性与开发者体验大幅提升
“开源信徒”周鸿祎开源360智脑大模型
周排行
Ubuntu 14.04 下Fuel6.0安装部署
香港一小巴侧翻致1死16伤 警方：未见机件故障
pikachu--XSS盲打
阅读 深入理解JVM虚拟机笔记一
java.sql.SQLException: ORA-00932: 数据类型不一致: 应为 -, 但却获得 CLOB
oracle delete all object under an user
[LeetCode]20 Valid Parentheses 有效的括号
树形DP求树的直径 【模板】
Context propagation over HTTP in Go
【PAT】（B）1053 住房空置率 (20)*
每日归档
更多
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)
2024-04-15(42)
2024-04-14(0)
2024-04-13(119)
2024-04-12(38)
2024-04-11(14)
2024-04-10(68)
2024-04-09(5)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022