使用Springboot+Springbatch+Quartz每隔3秒读取一次文件
4.2.1概述
Spring batch提供了强大的批处理功能，比如ItemReader、ItemProcess、ItemWriter，还有Tasklet，但定时功能不够强大,而Quartz提供了通过JobDetail、Trigger、SchedulerFactory提供了强大的定时器功能，但批处理功能不够强大,鉴于此,Spring对两者做了的整合.
4.2.2 pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mlsama</groupId>
<artifactId>hello-springbatch</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>hello-springbatch</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.3.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<!--Spring batch与Quartz的整合包-->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context-support</artifactId>
</dependency>
<!--Quartz-->
<dependency>
<groupId>org.quartz-scheduler</groupId>
<artifactId>quartz</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!--使用MySQL作为数据库-->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency> <!--c3p0连接池-->
<dependency>
<groupId>com.mchange</groupId>
<artifactId>c3p0</artifactId>
<version>0.9.5.2</version>
</dependency>
<!--整合junit-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-test</artifactId>
<scope>test</scope>
</dependency>
<!--lombok约束-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<!--日记-->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
4.2.3 配置文件(myJobApplication.xml)
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:batch="http://www.springframework.org/schema/batch"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:jdbc="http://www.springframework.org/schema/jdbc"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/batch
http://www.springframework.org/schema/batch/spring-batch.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-4.3.xsd
http://www.springframework.org/schema/jdbc
http://www.springframework.org/schema/jdbc/spring-jdbc-4.3.xsd
http://www.springframework.org/schema/task
http://www.springframework.org/schema/task/spring-task-4.3.xsd">
<!--读取数据-->
<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource" value="classpath:/data/user.txt"></property>
<property name="lineMapper" ref="lineMapper"></property>
<property name="linesToSkip" value="10"></property>
</bean>
<bean id="lineMapper" class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="lineTokenizer"></property>
<property name="fieldSetMapper" ref="fieldSetMapper"></property>
</bean>
<bean id="lineTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="names" value="id,name,age"></property>
</bean>
<bean id="fieldSetMapper" class="com.mlsama.hellospringbatch.config.apart.UserFieldSetMapper"></bean>
<!--核心业务处理-->
<bean id="process" class="com.mlsama.hellospringbatch.config.apart.UserItemProcessor"></bean>
<!--持久化处理-->
<bean id="write" class="com.mlsama.hellospringbatch.config.apart.UserWriter"></bean>
<!-- 配置spring batch的jobRepository，负责与数据库打交道 -->
<bean id="jobRepository" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="dataSource" ref="dataSource"></property>
<property name="transactionManager" ref="transactionManager"></property>
<property name="databaseType" value="MySQL"></property>
</bean>
<!--简单的异步任务执行器-->
<bean id="executor" class="org.springframework.core.task.SimpleAsyncTaskExecutor"></bean>
<!--job执行器-->
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository"></property>
<property name="taskExecutor" ref="executor"></property>
</bean>
<!--事务-->
<bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager"/>
<!--job-->
<batch:job id="jobExample">
<!--id:job的名称,next : 指定下一个要执行的job-->
<batch:step id="stepExample">
<!--任务列表 transaction-manager : 事务-->
<batch:tasklet transaction-manager="transactionManager">
<!--commit-interval : 一次事务提交的数据数量-->
<batch:chunk reader="reader"
processor="process"
writer="write"
commit-interval="3">
</batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
<!--**************与quarter整合调度*******************-->
<!-- 注册job，把job注入到容器中，在jobLauncher启动job时才能从容器中获取要启动的Job -->
<bean id="jobRegistry" class="org.springframework.batch.core.configuration.support.MapJobRegistry"/>
<bean class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry"/>
</bean>
<!--“任务浏览器（JObExplorer）”是“任务存储器（JobRepository）”的只读版本，
像后者一样，它可以通过工厂bean简单的配置生成-->
<bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean">
<property name="dataSource" ref="dataSource"></property>
</bean>
<!--Quartz定时调度配置-->
<bean id="jobDetail" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
<!--记录job的相关数据,启动job时使用-->
<property name="jobDataMap">
<map>
<!--job-->
<entry key="job" value-ref="jobExample"></entry>
<!--jobLauncher-->
<entry key="jobLauncher" value-ref="jobLauncher"></entry>
</map>
</property>
<!--要执行的类,继承QuartzJobBean,在executeInternal方法启动job-->
<property name="jobClass" value="com.mlsama.hellospringbatch.config.apart.QuartzJobLauncher"></property>
</bean>
<!-- 创建Scheduler -->
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<bean id="cronTrigger" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
<!--调度-->
<property name="jobDetail" ref="jobDetail" />
<!--corn表达式-->
<property name="cronExpression" value="0/3 * * * * ?" />
</bean>
</property>
</bean>
</beans>
4.2.4 application.properties
#整合c3p0,要自定义配置类,如:DataSourceConfiguration
spring.datasource.c3p0.driverClass=com.mysql.jdbc.Driver
spring.datasource.c3p0.jdbcUrl=jdbc:mysql://localhost:3306/springbatch
spring.datasource.c3p0.user=root
spring.datasource.c3p0.password=mlsama
spring.datasource.c3p0.maxPoolSize=30
spring.datasource.c3p0.minPoolSize=10
spring.datasource.c3p0.initialPoolSize=10
#关闭项目启动,job自动执行
spring.batch.job.enabled = false
4.2.5 pojo
@Data
public class User {
private String id;
private String name;
private String age;
public User(String id, String name, String age) {
this.id = id;
this.name = name;
this.age = age;
}
@Override
public String toString() {
return "User{" +
"id='" + id + '\'' +
", name='" + name + '\'' +
", age='" + age + '\'' +
'}';
}
}
4.2.6 行映射器
@Slf4j
public class UserFieldSetMapper implements FieldSetMapper<User> {
@Override
public User mapFieldSet(FieldSet fieldSet) throws BindException {
log.info("****************helloWorld************************");
return new User(fieldSet.readString("id"),
fieldSet.readString("name"),
fieldSet.readString("age"));
}
}
4.2.7业务处理
public class UserItemProcessor implements ItemProcessor<User, User> {
@Override
public User process(User item) throws Exception {
if (Integer.parseInt(item.getAge()) % 2 == 0) {
return item;
}
return null;
}
}
4.2.8 持久化
@Slf4j
public class UserWriter implements ItemWriter<User> {
@Override
public void write(List<? extends User> items) throws Exception {
log.info("****************helloWorld************************");
for(User user : items){
log.info("处理的对象是:{}",user);
}
}
}
4.2.9 Quartz调度器
继承QuartzJobBean,重写executeInternal方法,在方法内启动job即可.
@Slf4j
public class QuartzJobLauncher extends QuartzJobBean {
@Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
JobDetail jobDetail = context.getJobDetail();
JobDataMap jobDataMap = jobDetail.getJobDataMap();
Job job = (Job)jobDataMap.get("job");
JobLauncher jobLauncher = (JobLauncher) jobDataMap.get("jobLauncher");
//MapJobRegistry jobRegistry = (MapJobRegistry)jobDataMap.get("jobRegistry");
log.info("job :{}",job);
log.info("jobLauncher :{}",jobLauncher);
//log.info("jobRegistry :{}",jobRegistry);
SimpleDateFormat sf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
String date = sf.format(new Date());
try {
log.info("Current Time :{},job:{}",date,job);
jobLauncher.run(
job,new JobParameters());
//new JobParametersBuilder().addString("date", date).toJobParameters());
log.info("**********************job:{}执行完毕****************",job);
} catch (Exception e) {
log.error("发生异常{}",e);
}
}
}
4.2.10 user.txt
id,name,age
1,mlsama,18
2,ss,20
3,dj,260
4,dn,199
5,jy,212
4.2.11项目启动器
@SpringBootApplication //项目启动
//@EnableBatchProcessing //加载所有的job,与quarter整合调度的时候不能有这个
@ImportResource(locations = "classpath:/myJobApplication.xml")
public class ApartApplication {
public static void main(String[] args) {
SpringApplication.run(ApartApplication.class, args);
}
}
运行main方法即可