java batch insert data

Insert data in batches, the common way to use mybatis foreach to insert, the original way and batch processing

1. Common mybatis foreach

xml

<insert id="insertBatch"  parameterType="java.util.List">
		insert into  CODEINFO (CODE_TYPE, CODE, MEAN, STATE, SORT_ID)
		values
		<foreach collection ="records" item="item" separator =",">
			(#{item.codeType}, #{item.code},
			 #{item.remark},  #{item.state}, #{item.sortId})
		</foreach >
	</insert>

mapper:

int insertBatch(@Param("records") List<CodeInfo> records);

For the amount of data is not very large, basically enough. If synchronizing data is particularly slow, consider other methods. Or synchronize the data again in the early morning at night.

2. The original way

batch insert

public void insertBatach(){
        Connection conn=null;
        PreparedStatement ps=null;
        try {
            long start = System.currentTimeMillis();

            conn = JDBCUtils.getConnection();
            conn.setAutoCommit(false);
            String sql="INSERT INTO CODEINFO (CODE_TYPE, CODE, MEAN,STATE, SORT_ID) VALUES (?, ?, ?, ?, ?)";
            ps = conn.prepareStatement(sql);
            for(int i=1;i<=20000;i++){
                ps.setObject(1, "TEST_INSERT_BATCH");
                ps.setObject(2, "0"+i);
                ps.setObject(3, "name_"+i);
                ps.setObject(4, "0SA");
                ps.setObject(5, i);
                //1.sql
                ps.addBatch();

                if(i%500==0){
                    //2.执行batch
                    ps.executeBatch();

                    //3.清空batch
                    ps.clearBatch();
                }
            }

            //提交数据
            conn.commit();
            long end = System.currentTimeMillis();
            System.out.println("批量插入花费的时间为："+(end-start));
        } catch (Exception e) {
            e.printStackTrace();
        } finally{
            JDBCUtils.close(conn, ps);
        }
    }

Database Connectivity:

import java.io.IOException;
import java.io.InputStream;
import java.sql.*;
import java.util.Properties;

public class JDBCUtils {
    private static String url;
    private static String user;
    private static String password;
    private static  Connection conn = null;
    //    静态代码块
    static{
        /* 将外部properties文件放在src文件夹中，用类的加载器读文件，格式：
         * 当前类名.class.getClassLoader().getResourceAsStream("外部文件名");*/
        InputStream in = JDBCUtils.class.getClassLoader().getResourceAsStream("sql.properties");
        Properties p=new Properties();
        try {
            p.load(in);
        } catch (IOException e) {
            e.printStackTrace();
        }
//        读文件给变量赋值
        String driver = p.getProperty("driver");
        url = p.getProperty("url");
        user = p.getProperty("user");
        password = p.getProperty("password");
        try {
            Class.forName(driver);
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }
    }

    //    构造获得数据库链接方法
    public static Connection getConnection() {
        try {
            conn = DriverManager.getConnection(url, user, password);
        } catch (SQLException e) {
            e.printStackTrace();
        }
        return conn;
    }
    //    构造关闭流的方法
    public static void close(Connection conn,Statement stat) {
        if (stat != null) {
            try {
                stat.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
        if (conn != null) {
            try {
                conn.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
    }
    //    重载关闭流的方法
    public static void close(Connection conn,Statement stat, ResultSet rs) {
        if (rs != null) {
            try {
                rs.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
        if (stat != null) {
            try {
                stat.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
        if (conn != null) {
            try {
                conn.close();
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }
    }
}

The original method is more troublesome to write.

3. Batch processing

MybatisGeneralBatchUtils 

import org.apache.ibatis.session.ExecutorType;
import org.apache.ibatis.session.SqlSession;
import org.apache.ibatis.session.SqlSessionFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import org.springframework.transaction.support.TransactionSynchronizationManager;

import java.util.List;
import java.util.function.BiFunction;


@Component
public class MybatisGeneralBatchUtils {
    private static final Logger logger = LoggerFactory.getLogger(MybatisGeneralBatchUtils.class);
    /**
     * 每次处理1000条
     */
    private static final int BATCH_SIZE = 1000;

    /**
     * 批量处理修改或者插入
     *  变成一条一条的数据，然后最后一起执行。并不是 insertBatch那种方式
     * @param data        需要被处理的数据
     * @param mapperClass Mybatis的Mapper类
     * @param function    自定义处理逻辑
     * @return int 影响的总行数
     */
    public <T, U, R> int batchUpdateOrInsert(List<T> data, Class<U> mapperClass, BiFunction<T, U, R> function)  {
        int i = 1;
        SqlSessionFactory sqlSessionFactory = (SqlSessionFactory) SpringUtil.getBean("sqlSessionFactory");
        SqlSession batchSqlSession = sqlSessionFactory.openSession(ExecutorType.BATCH);
        try {
            U mapper = batchSqlSession.getMapper(mapperClass);
            int size = data.size();
            for (T element : data) {
                function.apply(element, mapper);
                if ((i % BATCH_SIZE == 0) || i == size) {
                    batchSqlSession.flushStatements();
                }
                i++;
            }
            // 非事务环境下强制commit，事务情况下该commit相当于无效
            batchSqlSession.commit(!TransactionSynchronizationManager.isSynchronizationActive());
        } catch (Exception e) {
            batchSqlSession.rollback();
            logger.error("batchUpdateOrInsert", e);
        } finally {
            batchSqlSession.close();
        }
        return i - 1;
    }

}

SpringUtil 

import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.stereotype.Component;

@Component
public class SpringUtil implements ApplicationContextAware {

    private static ApplicationContext applicationContext;

    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
         SpringUtil.applicationContext = applicationContext;
    }

    public static Object getBean(String name) {
        return applicationContext.getBean(name);
    }

    public static <T> T getBean(Class<T> clazz) {
        return applicationContext.getBean(clazz);
    }
}

transfer:

mapper:

int insertSelective(CodeInfo codeInfo);

xml:

<insert id="insertSelective" parameterType="com.web.dict.entity.CodeInfo">
		insert into CODEINFO
		<trim prefix="(" suffix=")" suffixOverrides=",">
			<if test="codeType != null">
				CODE_TYPE,
			</if>
			<if test="code != null">
				CODE,
			</if>
			<if test="mean != null">
				MEAN,
			</if>
			<if test="state != null">
				STATE,
			</if>
			<if test="sortId != null">
				SORT_ID,
			</if>
		</trim>
		<trim prefix="values (" suffix=")" suffixOverrides=",">
			<if test="codeType != null">
				#{codeType,jdbcType=VARCHAR},
			</if>
			<if test="code != null">
				#{code,jdbcType=VARCHAR},
			</if>
			<if test="mean != null">
				#{mean,jdbcType=VARCHAR},
			</if>
			<if test="state != null">
				#{state,jdbcType=VARCHAR},
			</if>
			<if test="sortId != null">
				#{sortId,jdbcType=VARCHAR},
			</if>
		</trim>
	</insert>

service:

    @Resource
    private MybatisGeneralBatchUtils mybatisGeneralBatchUtils;

    public int batchInsertData(List<CodeInfo> codeInfos){
        return mybatisGeneralBatchUtils.batchUpdateOrInsert(codeInfos, CodeInfoMapper.class,
                (item, codeInfoMapper) -> codeInfoMapper.insertSelective(item));
    }

This method seems to be more general, but if I test it myself, the speed is slower. It may be because the simulated fields and data are relatively small; if you encounter a large amount of data later, perform a comparison.

The method recommended by the official website:

When writing batch inserts in MyBatis documents , it is recommended to use another method.  Batch Insert Support The content in the title

try(SqlSession session = sqlSessionFactory.openSession(ExecutorType.BATCH)) {
        SimpleTableMapper mapper = session.getMapper(SimpleTableMapper.class);
        List<SimpleTableRecord> records = getRecordsToInsert(); // not shown

        BatchInsert<SimpleTableRecord> batchInsert = insert(records)
                .into(simpleTable)
                .map(id).toProperty("id")
                .map(firstName).toProperty("firstName")
                .map(lastName).toProperty("lastName")
                .map(birthDate).toProperty("birthDate")
                .map(employed).toProperty("employed")
                .map(occupation).toProperty("occupation")
                .build()
                .render(RenderingStrategies.MYBATIS3);

        batchInsert.insertStatements().forEach(mapper::insert);

        session.commit();
    }

Summarize:

     If the amount of data is not large, the first one is enough. If there are many data contents and many fields, try other methods to see if the efficiency is faster. Synchronizing data is still suitable for running at night with a timer.