For large data storage solutions to storage problems

During development, sometimes encounter this situation, when the main thread to open the N sub-thread, and each child thread in operation particularly large amount of data analysis, if conducted in sub-threads for such large quantities of data storage, then the connection will cause a lot of waste of resources, will also result in blocking the database affect program execution efficiency. At this time there are two solutions:

• The unified data sub-thread receiving the main thread, a storage operation in the main thread.

But this contention on the program execution is completed, a small amount of data, the case will not cause a memory overflow, if the amount of data in the case of very large, it will cause the program to crash.

• Re-open a thread, which is dedicated to data storage warehousing.

Since each thread of execution speed is not the same, taking into account the amount of data execution time is rather long, so that the child thread executed the result is added to the data storage queue, and the data storage of the thread to get the queues to bulk storage. Here with JAVA realize what this program:

public class MainClass {
    public static void main(String[] args) {
        new Thread(new DataThread()).start();//模拟其它线程正在写数据
        new Thread(new DBThread()).start();//开启数据入库线程
    }
}

/**
 * 模拟子线程添加数据
 * @author libing
 *
 */
public class DataThread implements Runnable {

    @Override
    public void run() {
        while(true) {
            //每100ms添加当前时间戳
            DataQueue.addObject(System.currentTimeMillis());
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

/**
 * 数据入库的操作线程
 * @author libing
 *
 */
public class DBThread implements Runnable {
    private static Logger logger = LoggerFactory.getLogger(DBThread.class);
    private static int MAX_OBJECT_NUM=10;//每批次最大的数据大小
    List<Object> objectList=new ArrayList<>();//
    DbDao dbDao=new DbDao();//数据库操作

    @Override
    public void run() {
        while(true) {
            try {
                if(objectList.size()<MAX_OBJECT_NUM) {
                    //当前入库的数据小于最大的要求
                    Object object = DataQueue.getObject();
                    if(object==null) {
                        //执行保存操作
                        dbDao.save(objectList);
                        objectList.clear();
                        Thread.sleep(5000);
                    }else {
                        //添加进列表
                        objectList.add(object);
                    }
                }else {
                    //执行保存操作
                    dbDao.save(objectList);
                    objectList.clear();
                    Thread.sleep(5000);
                }
            } catch (Exception e) {
                logger.error("出错",e);
            }
        }
    }
}