Spark2与Oozie整合

背景:

纯粹个人瞎摸索,相关资料太少了,估计大多数公司还没使用spark2吧?

1.在oozie共享目录创建spark2目录

可直接在页面上创建

2.将集群中spark2的jar复制到spark2目录下

 
    hadoop fs -put / /opt/cloudera/parcels/SPARK2- 
    2.2 
    . 
    0 
    .cloudera2- 
    1 
    .cdh5. 
    12.0 
    .p0. 
    232957 
    /lib/spark2/jars /user/oozie/share/lib/lib_20180523180130/spark2  
   

3.将原oozie共享目录中spark的两个与oozie share相关的jar也拷贝到spark2目录下

 
   hadoop fs -cp /user/oozie/share/lib/lib_20180523180130/spark/oozie-sharelib-spark- 
   4.1 
   . 
   0 
   -cdh5. 
   14.0 
   .jar  /user/oozie/share/lib/lib_20180523180130/spark2  
  

 
   hadoop fs -cp /user/oozie/share/lib/lib_20180523180130/spark/oozie-sharelib-spark.jar /user/oozie/share/lib/lib_20180523180130/spark2 
  

其目录结构如下:

4.创建oozie的spark工作流

进入oozie计划工作页面

上图中的Jar/py名称框里只需选中spark-examples_2.11-2.2.0.cloudera2.jar 即可

选中/user/hue/oozie/workspaces/hue-oozie-1527557010.76/lib/spark-examples_2.11-2.2.0.cloudera2.jar 点击选择此文件夹

选完之后自己检查一下看看是不是全选了

图中的例子是官方的例子

 
   hadoop fs -put /opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars/spark-examples_2.11-2.2.0.cloudera2.jar /user/hue/oozie/workspaces/hue-oozie-1527557010.76/lib 
  

Main class :org.apache.spark.examples.SparkPi

点击上图的齿轮设置属性

重点来了:

点击上图中的齿轮

设置spark2 : name=oozie.action.sharelib.for.spark

value=spark2

填写完之后保存提交即可

注意:应该注意一些权限问题

这些可能会用上

 
   sudo -u hdfs hadoop fs -chown -R oozie:oozie /user/oozie/share/lib/lib_20180528183927/spark2 
  

 
   sudo -u hdfs hadoop fs -chown -R hue:hue /user/hue/oozie/workspaces 
  

 
   sudo -u hdfs hadoop fs -chmod 777 /user/hue/oozie/workspaces 
  

参考:https://blog.csdn.net/qq_24908345/article/details/80017660

https://cloud.tencent.com/developer/article/1078092

背景:

猜你喜欢