通过脚本启动爬虫:
main.py from scrapy.cmdline import execute #可执行scrapy 脚本 import sys import os main_path = os.path.abspath(__file__) sys.path.append(os.path.dirname(main_path)) execute(["scrapy","crawl","jobbole"])
依据传入的值crawl,会调用scrapy.commands中Command.run()方法
scrapy.commands.crawl.py: def run(self, args, opts): if len(args) < 1: raise UsageError() elif len(args) > 1: raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported") spname = args[0] self.crawler_process.crawl(spname, **opts.spargs)#生成一个crawler对象 self.crawler_process.start()