多线程抽取数据库数据，数据迁移

关键代码

2000万数据同步，每次查询20000分页，一分钟分钟之内全部塞入到队列里等待

    log.info("房屋数仓数据同步调度开始！"); 
   Date yesterday = null;
   HouseFeedbackCount hfb = getHouseFeedbackCountTime(21);
    if (hfb != null){
       yesterday = DateUtils.addDays(hfb.getEndDate(),1);
    }else{
       yesterday= DateUtils.addDays(new Date(),-1);//今天
    } 
   Date today= DateUtils.addDays(new Date(),0);//今天
   String beginTime = DateUtils.toString(yesterday,DateUtils.YYYY_MM_DD)+" 00:00:00";
   String endTime = DateUtils.toString(today,DateUtils.YYYY_MM_DD)+" 23:59:59";
    System.out.println("生成任务开始"+beginTime+endTime);
    int pageSize = 20000;
    int size = houseService.getHouseTotal(beginTime, endTime);
   if(size > pageSize){
   int page = size%pageSize==0?(size/pageSize):(size/pageSize)+1;
   System.out.println(page); 
   int begin = 0;
   int end = 0;
   for (int i = 0; i < page; i++) {
      begin = pageSize*i+1;
      end = pageSize*(i+1);
      houseService.updateHouse(beginTime, endTime, begin, end);
}
   }else{
      houseService.updateHouse(beginTime, endTime, 1, size);
   }
   hfb = new HouseFeedbackCount();
   hfb.setBizname("房屋数仓数据同步定时生成任务");

HouseServiceImpl里面的updateHouse 加上注解@Async

@Async
   @Transactional
@Override
public void updateHouse(String beginTime, String endTime, int begin, int end) {
//你的迁移代码
}

配置 apring类扫描文件中配置

<task:scheduler id="yuncaiCheduler" pool-size="10"/><!-- 定时器线程数 -->
<task:annotation-driven scheduler="yuncaiCheduler"/><!-- 开启定时器注解 -->

假如你的线程池配置类，确保你的类能被扫描到
import java.util.concurrent.Executor;

import org.springframework.aop.interceptor.AsyncUncaughtExceptionHandler;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.AsyncConfigurer;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;

/**
 * @Title:  TaskExecutorConfig.java   
 * @Package com.yuncai.scheduled   
 * @Description: 异步任务支持配置 
 * @author: LGH   
 * @date: 2018年8月10日 上午10:28:22
 */
@Configuration
@ComponentScan("com.yuncai.scheduled")
@EnableAsync //利用@EnableAsync注解开启异步任务支持
public class TaskExecutorConfig implements AsyncConfigurer{
    //配置类实现AsyncConfigurer接口并重写getAsyncExcutor方法，并返回一个ThreadPoolTaskExevutor
    //这样我们就获得了一个基于线程池的TaskExecutor
    @Override
    public Executor getAsyncExecutor() {
        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        taskExecutor.setCorePoolSize(10);//线程池维护线程的最少数量
        taskExecutor.setMaxPoolSize(35);//线程池维护线程的最大数量
        taskExecutor.initialize();
        return taskExecutor;
    }

   @Override
   public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
      return null;
   }

}

然后你的上面的程序就可以开跑了。十个线程跑调度，主要是cpu的问题

多线程抽取数据库数据，数据迁移

猜你喜欢