oozie开启hcatUrl依赖hive的元数据

转载请标明本文地址:http://blog.csdn.net/hblfyla/article/details/78331176

最近在开发基于oozie的调度系统，有调度依赖底层使用的coodinator

有依赖路径，可以在依赖路径生成后程序开始运行，刚开始是等待waiting状态

coodinator配置文件模板后生成可运行代码如下:

<coordinator-app name="aa" frequency="${frequency}" start="${start}" end="${end}" timezone="UTC" xmlns="uri:oozie:coordinator:0.2">
	<controls>
		<timeout>99999</timeout>
		<execution>FIFO</execution>
	</controls>
	<datasets>
		<dataset name="path0" frequency="${coord:days(1)}" initial-instance="2016-01-01T12:28+0800" timezone="Antarctica/Casey">
			<uri-template>${hcatUrl}ods_tinyv/o_wb_xiaodai_user_phone_review_record_d/dt=${YEAR}-${MONTH}-${DAY}</uri-template>
			<done-flag></done-flag>
		</dataset>

		<dataset name="path1" frequency="${coord:days(1)}" initial-instance="2016-01-01T12:28+0800" timezone="Antarctica/Casey">
			<uri-template>${hcatUrl}ods_tinyv_univ/o_univ_xiaodai_user_phone_review_record_d/dt=${YEAR}-${MONTH}-${DAY}</uri-template>
			<done-flag></done-flag>
		</dataset>
	</datasets>

	<input-events>
		<data-in name="path0" dataset="path0">
			<instance>${coord:current(0)}</instance>
		</data-in>

		<data-in name="path1" dataset="path1">
			<instance>${coord:current(0)}</instance>
		</data-in>
	</input-events>

	<action>
		<workflow>
			<app-path>${workflowAppUri}</app-path>
			<configuration>
				<property>
					<name>oozie.use.system.libpath</name>
					<value>true</value>
				</property>
				
				<property>
					<name>start_date</name>
					<value>${start}</value>
				</property>
				
				<property>
					<name>end_date</name>
					<value>${end}</value>
				</property>
				
				<property>
                     <name>queueName</name>
                     <value>${queueName}</value>
                </property>

				<property>
					<name>path0</name>
					<value>${coord:dataIn('path0')}</value>
				</property>
				<property>
					<name>path1</name>
					<value>${coord:dataIn('path1')}</value>
				</property>

				<property>
					<name>dt</name>
					<value>${coord:formatTime((coord:dateOffset(coord:nominalTime(),-1,"DAY")),"yyyy-MM-dd")}</value>
				</property>
				<property>
					<name>a</name>
					<value>yangluan</value>
				</property>

            </configuration>
        </workflow>
    </action>

</coordinator-app>

要想使用依赖路径，先开启hcatURL，开始方式可以参照官方文档API

这里简单整理下:HCatalog Configuration

1.Adding HCatalog jars to Oozie war:

 <property>
    <name>oozie.service.HCatAccessorService.hcat.configuration</name>
    <value>/local/filesystem/path/to/hive-site.xml</value>
  </property>

2. Configure HCatalog URI Handling

<property>
    <name>oozie.service.URIHandlerService.uri.handlers</name>
    <value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
    <description>
        Enlist the different uri handlers supported for data availability checks.
    </description>
  </property>

3. Configure HCatalog services

<property>
    <name>oozie.services.ext</name>
    <value>
        org.apache.oozie.service.JMSAccessorService,
        org.apache.oozie.service.PartitionDependencyManagerService,
        org.apache.oozie.service.HCatAccessorService
      </value>
    <description>
          To add/replace services defined in 'oozie.services' with custom implementations.
          Class names must be separated by commas.
    </description>
  </property>

4.Configure JMS Provider JNDI connection mapping for HCatalog:

 <property>
    <name>oozie.service.HCatAccessorService.jmsconnections</name>
    <value>
      hcat://hcatserver.colo1.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.colo1.com:61616,
      default=java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://broker.colo.com:61616;connectionFactoryNames#ConnectionFactory
    </value>
    <description>
        Specify the map  of endpoints to JMS configuration properties. In general, endpoint
        identifies the HCatalog server URL. "default" is used if no endpoint is mentioned
        in the query. If some JMS property is not defined, the system will use the property
        defined jndi.properties. jndi.properties files is retrieved from the application classpath.
        Mapping rules can also be provided for mapping Hcatalog servers to corresponding JMS providers.
        hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616
    </description>
  </property>

5. Configure HCatalog Polling Frequency:

<property>
    <name>oozie.service.coord.push.check.requeue.interval
        </name>
    <value>600000</value>
    <description>Command re-queue interval for push dependencies (in millisecond).
    </description>
  </property>

以上几个是主要配置文件，其他的不都列举了，具体看官网

我这边的配置所有截图如下

转载请标明本文地址:http://blog.csdn.net/hblfyla/article/details/78331176