Spring Batch Advanced - Multithreading Steps

Table of contents

introduction

Followed by the previous article: Spring Batch ItemWriter component , after understanding the Spring Batch ItemWriter processing component, let’s learn about Spring Batch’s advanced functions-multithreading steps

concept

By default, the steps are basically executed in a single thread, so can it be executed in a multi-threaded environment? The answer is definitely yes, but it should also be noted that the execution of steps in a multi-threaded environment must be cautious. Reason: In a multi-threaded environment, the step is to set non-restartable .

Spring Batch's multi-threaded steps are implemented using Spring's TaskExecutor (task executor). It is agreed that each block starts a thread for independent execution.

the case

Requirement: process user-thread.txt file in 5 blocks

1> Write user-thread.txt file

1#dafei#18
2#xiaofei#16
3#laofei#20
4#zhongfei#19
5#feifei#15
6#zhangsan#14
7#lisi#13
8#wangwu#12
9#zhaoliu#11
10#qianqi#10

2> Define the entity object

@Getter
@Setter
@ToString
public class User {
    private Long id;
    private String name;
    private int age;
}

3> Complete code

package com.langfeiyes.batch._35_step_thread;


import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.task.SimpleAsyncTaskExecutor;

import java.util.List;

@SpringBootApplication
@EnableBatchProcessing
public class ThreadStepJob {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;
    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Bean
    public FlatFileItemReader<User> userItemReader(){

        System.out.println(Thread.currentThread());

        FlatFileItemReader<User> reader = new FlatFileItemReaderBuilder<User>()
                .name("userItemReader")
                .saveState(false) //防止状态被覆盖
                .resource(new ClassPathResource("user-thread.txt"))
                .delimited().delimiter("#")
                .names("id", "name", "age")
                .targetType(User.class)
                .build();

        return reader;
    }

    @Bean
    public ItemWriter<User> itemWriter(){
        return new ItemWriter<User>() {
            @Override
            public void write(List<? extends User> items) throws Exception {
                items.forEach(System.err::println);
            }
        };
    }

    @Bean
    public Step step(){
        return stepBuilderFactory.get("step1")
                .<User, User>chunk(2)
                .reader(userItemReader())
                .writer(itemWriter())
                .taskExecutor(new SimpleAsyncTaskExecutor())
                .build();

    }

    @Bean
    public Job job(){
        return jobBuilderFactory.get("thread-step-job")
                .start(step())
                .build();
    }
    public static void main(String[] args) {
        SpringApplication.run(ThreadStepJob.class, args);
    }
}

4> Results

User(id=2, name=xiaofei, age=16)
User(id=5, name=feifei, age=15)
User(id=4, name=zhongfei, age=19)
User(id=7, name=lisi, age=13)
User(id=1, name=dafei, age=18)
User(id=6, name=zhangsan, age=14)
User(id=3, name=laofei, age=20)
User(id=8, name=wangwu, age=12)
User(id=9, name=zhaoliu, age=11)
User(id=10, name=qianqi, age=10)

analyze

1: userItemReader() plus saveState(false) Spring Batch provides most of the ItemReader is stateful, the job restart basically determines the job stop position through the state, and in a multi-threaded environment, if the object maintenance state is accessed by multiple threads , there may be a problem of mutual state coverage between threads. So setting it to false means that it is closed, but it also means that the job cannot be restarted.

2: The step() method plus .taskExecutor(new SimpleAsyncTaskExecutor()) adds multi-thread processing capabilities to the job steps, taking blocks as units, one thread per block, observing the above results, it is obvious that the output sequence is Out of order. Change the name of the job and execute it again, and you will find that the output data is different every time.

At this point, this article is over. If you want to know what will happen next, please listen to the next chapter to break it down~

Turn to video version

If you are not addicted to reading text, you can switch to the video version: Spring Batch efficient batch processing framework in practice