Scrapy of CloseSpider extension will automatically terminate crawlers in meeting the conditions. After may be provided CLOSESPIDER_TIMEOUT (s), CLOSESPIDER_ITEMCOUNT, CLOSESPIDER_PAGECOUNT, CLOSESPIDER_ERRORCOUNT representing after a specified time after a specified number of gripping Item, in response to a specified number of received, after a specified number of error has occurred is terminated crawlers . Typically the command line may be provided:
$ scrapy crawl fast -s CLOSESPIDER_TIMEOUT=10
$ scrapy crawl fast -s CLOSESPIDER_ITEMCOUNT=10
$ scrapy crawl fast -s CLOSESPIDER_PAGECOUNT=10
$ scrapy crawl fast -s CLOSESPIDER_ERRORCOUNT=10