nutch的配置文件我们可以从Crawl.java中看起,在main函数中, 首先加载配置类: Configuration conf = NutchConfiguration.createCrawlConfiguration();,其中createCrawlConfiguration()类为
public static Configuration createCrawlConfiguration() { Configuration conf = new Configuration(); addNutchResources(conf, true); return conf; }
Configuration加载hadoop的配置文件,NutchResources加载nutch-default.xml,crawl-tool.xml和nutch-site.xml,越后面加载优先级越高。
private static Configuration addNutchResources(Configuration conf, boolean crawlConfiguration) { conf.addResource("nutch-default.xml"); if (crawlConfiguration) { conf.addResource("crawl-tool.xml"); } conf.addResource("nutch-site.xml"); return conf; }
在nutch-default.xml中有parse.plugin.file项,用来定义content-type和parsers的联系,映射到parse-plugin.xml文件。
parse-plugin.xml文件定义了contentType="text/html"和id parse-html对应,而parse-html的值为org.apache.nutch.parse.html.HtmlParser类,即利用这个类来处理html网页的解析。