A long time did not write the article, because now engaged in system development JAVA direction, so the task of writing a basic framework of the implementation of its own, depending on the desired application scenarios purpose is to customize the system to achieve the task. With the framework here have thought the company's implementation. Ado, directly to the question.
Since the purposes of this framework is to high intensity for each function can achieve its mission of custom extensions. So the design idea is as follows:
-
Running two threads open
the first thread is a database query every 10s whether the task to be performed, if there is added to the task queue, the second thread is a task execution thread, which directly create a thread pool to perform the task.
Thread 1:/** * 任务查询线程 * @author libing * */ public class TaskThread implements Runnable { private static Logger logger = LoggerFactory.getLogger(TaskThread.class); @Override public void run() { while(true) { //添加任务到任务队列 TaskDBUtil.addTaskQueue(); try { logger.debug("准备休眠10秒后取数据库任务"); //休眠10秒 Thread.sleep(10000); } catch (InterruptedException e) { logger.error("休眠失败",e); } } } }Thread 2:
/** * 具体任务的执行线程 * @author libing * */ public class BusinessThread implements Runnable { private static Logger logger = LoggerFactory.getLogger(BusinessThread.class); ThreadPoolExecutor executor = new ThreadPoolExecutor(GlobalData.taskMaxNum, GlobalData.taskMaxNum*2, 200, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(GlobalData.taskMaxNum)); @Override public void run() { while(true) { try { //判断任务队列中等待执行的任务数量不可大于线程池中 if(executor.getQueue().size()<GlobalData.taskMaxNum){ TaskBean task = TaskQueue.getTask(); if(task!=null) { //利用反射创建这个任务 //得到这个任务的taskGc Map<String, String> taskGc = TaskGcGlobal.getTaskGc(task.getExecId()); executor.execute((Runnable) Class.forName(taskGc.get("controller")).getConstructor(TaskBean.class).newInstance(task)); logger.debug("线程池中线程数目："+executor.getPoolSize()+"，队列中等待执行的任务数目："+ executor.getQueue().size()+"，已执行完别的任务数目："+executor.getCompletedTaskCount()); } }else { logger.error("任务队列已满，休眠3秒"); //休眠3秒再去取任务 Thread.sleep(3000); } } catch (Exception e) { logger.error("任务执行异常，停止任务！",e); executor.shutdown(); } } } }When the task execution queue of the pending tasks, the task execution thread taken added to the thread pool, for the corresponding operation achieved by the object is reflected by the key tasks of the task.
- The second step, when the task is turned to run, it creates a corresponding execution object. I'm here with a mission crawling sister diagram of an example. This figure is climbing from page to page by page crawl, if you want to crawl resources of the whole station, if this task with a single thread to run a very long time to be consumed, so here is still considered open a thread pool to work. The basic implementation code as follows:
/**
* 爬图任务实现控制器
* @author libing
*
*/
public class CrawlerController implements Runnable {
private static Logger logger = LoggerFactory.getLogger(CrawlerController.class);
ThreadPoolExecutor executor = new ThreadPoolExecutor(GlobalData.taskThreadMaxNum, GlobalData.taskThreadMaxNum*2, 200, TimeUnit.MILLISECONDS,
new ArrayBlockingQueue<Runnable>(GlobalData.taskThreadMaxNum));
private TaskBean task;
private CrawlerDBUtil dbUtil=new CrawlerDBUtil();
public CrawlerController(TaskBean task) {
this.task = task;
}
@Override
public void run() {
//查询爬图配置表
List<CrawlerConfigBean> configList = dbUtil.allCrawlerConfigList();
for(int i=0;i<configList.size();) {
if(executor.getQueue().size()<GlobalData.taskThreadMaxNum){
CrawlerWorker worker=new CrawlerWorker(configList.get(i),dbUtil);
executor.execute(worker);
i++;
}
}
while(true) {
if(executor.getCompletedTaskCount()==configList.size()) {
logger.debug("爬图任务执行完毕!");
//修改当前执行配置的列表当前页数
dbUtil.updateCrawlerConfig(configList);
//修改当前任务为完成run_flag=0
TaskDBUtil.completeTask(this.task);
break;
}
}
}
}/**
* 爬图具体实现worker（每个页面一个worker）
* @author libing
*
*/
public class CrawlerWorker implements Runnable {
private static Logger logger = LoggerFactory.getLogger(CrawlerWorker.class);
private CrawlerConfigBean crawlerConfig;
private CrawlerDBUtil dbUtil;
public CrawlerWorker(CrawlerConfigBean crawlerConfig, CrawlerDBUtil dbUtil) {
this.crawlerConfig = crawlerConfig;
this.dbUtil = dbUtil;
}
@Override
public void run() {
// 循环当次采集的总页数
for (int i = 1; i <= crawlerConfig.getDayPage(); i++) {
// 当前查看的地址是当次的此页数+采集过的页数
// http://www.mzitu.com/zipai/ comment-page-1/#comments
List<String> imgUrlList = CrawlerUtil.findImgUrlListByUrl(
crawlerConfig.getUrl().replace("@", (i + crawlerConfig.getCurrentPage()) + ""));
dbUtil.saveCrawlerImgResult(imgUrlList);
for (String url : imgUrlList) {
// 下载图片到本地
try {
CrawlerUtil.download(url, GlobalData.crawlerImgPath + crawlerConfig.getIdent() + "/"
+ (crawlerConfig.getCurrentPage() + i));
} catch (Exception e) {
logger.error("当前图片下载失败："+e);
}
}
}
}
}
So far, the basic framework to achieve this is the case, if there are other tasks customization, you can still follow this chart climb to achieve the task. Great God welcome you advice.