springbatch to read the external data mysql

Foreword

In the last springboot integration springbatch, we simply introduce the basic concepts springbatch and use a basic demo. We know the power of at springbatch efficient, fast and can be customized to perform the steps of processing large quantities of data, here is a simple case say how springbatch read external file when its batch data, such as xml, json data or other formats can be resolved into the mysql

Steps

1. Defined job categories
2. Configured to perform the job in step (step)
3. In particular the rearrangement step and itemWriter itemReader
4. itemReader reads the data and parse the data, the data source may be read log file on disk, or redis mysql data or data, and then by itemWriter read data written to the specified storage container

From the map you can see, read and write data sources are diverse, so give bringing the actual use of the convenience of a lot of options, the following to an external csv file, for example, simulate what the process described above, i.e., by reading the file data itemReader, and then written to a table in mysql

1, job category

Note @EnableBatchProcessing must not forget, if easy to forget to put the proposal to start class above

@Configuration
@EnableBatchProcessing
public class JdbcDemoWriterConfig {

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    @Qualifier("jdbcBatchItemWriter")
    private ItemWriter<Customer> itemWriter;

    @Autowired
    @Qualifier("dbOutPutFileReader")
    private ItemReader<Customer> itemReader;

    /**
     * job执行入口
     * @return
     */
    @Bean
    public Job dbOutputJob(){
        return jobBuilderFactory.get("dbOutputJob")
                .start(dbOutputStep())
                .build();
    }

    /**
     * 自定义job的执行步骤
     * @return
     */
    @Bean
    public Step dbOutputStep(){
        return stepBuilderFactory.get("dbOutputStep")
                .<Customer,Customer>chunk(10)   //设置每次操作的数据个数
                .reader(itemReader)
                .writer(itemWriter)
                .build();
    }

}

2, custom itemReader

By concept on an explanation, we know, itemReader can be understood as a data reader, we know by the code, ItemReader only defines an interface, there are many implementation class, it is these rich implementation class, just let the program handle when very free and flexible, you can choose a different reader according to different scenarios,

For example, we are here to read a suffix ".csv" file, you can use FlatFileItemReader this reader, as follows:

/**
 * 自定义读取
 */

@Configuration
public class DbOutputReader {

    @Bean
    public FlatFileItemReader<Customer> dbOutPutFileReader() {
        FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
        reader.setResource(new ClassPathResource("customerInit.csv"));
        //设置是否跳行读取
        reader.setLinesToSkip(1);
        //设置外部文件的字段名
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setNames(new String[]{"id", "firstName", "lastName", "birthdate"});
        DefaultLineMapper<Customer> lineMapper = new DefaultLineMapper();
        lineMapper.setLineTokenizer(tokenizer);
        //这个有点类似jdbc，将读取到的结果集放在一个resultSet中，只需要解析resultSet的字段值就可以拿到具体的值
        lineMapper.setFieldSetMapper((fieldSet) -> {
            Customer customer = new Customer();
            customer.setId(fieldSet.readString("id"));
            customer.setFirstName(fieldSet.readString("firstName"));
            customer.setLastName(fieldSet.readString("lastName"));
            customer.setBirthdate(fieldSet.readString("birthdate"));
            return customer;
        });
        //检查属性设置是否完成
        lineMapper.afterPropertiesSet();
        reader.setLineMapper(lineMapper);
        return reader;
    }

}

This code is logic to customize a csv file reader, read from a csv file to the values ​​parsed package to a collection of objects and returns, in fact, this FlatFileItemReader also help us to do a job and that is how it knows the format of the file and we want to resolve it? This is done in-house work reader, interested partners can be studied source

3, custom itemWriter

The reading by the data reader into a csv file, and encapsulated into an object after the collection, the following data will be written by the itemWriter mysql table, we found the same itemWriter is an interface, there are many implementation class, we as used herein, JdbcBatchItemWriter

this custom code is as follows itemWriter

/**
 * 自定义writer
 */
@Configuration
public class DbOutputWriter {

    @Autowired
    private DataSource dataSource;

    @Bean
    public JdbcBatchItemWriter<Customer> jdbcBatchItemWriter(){
        JdbcBatchItemWriter<Customer> jdbcBatchItemWriter = new JdbcBatchItemWriter();
        //注入是数据源
        jdbcBatchItemWriter.setDataSource(dataSource);
        String sqlStr = "insert into customer(id,firstName,lastName,birthdate) values(:id,:firstName,:lastName,:birthdate)";
        jdbcBatchItemWriter.setSql(sqlStr);
        jdbcBatchItemWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
        return jdbcBatchItemWriter;
    }

}

I am here to advance in the resources directory, prepared a csv file

Interception of part

Create a database in advance with the corresponding customer table

Below start the program, you can quickly see the execution is over, the speed is very fast, according to their own computer configuration, adjust parameters chunk of that number of pieces of data each batch execution

Benpian on this end, and finally Thanks for watching!