scrapy frame custom command

Written after their projects reptiles, reptiles can run their own custom commands.

A single reptile

New items in the root directory of a file py, as named start.py, write the following code:

from scrapy.cmdline import execute

if __name__ == "__main__":
    execute(["scrapy", "crawl", "chouti", "--nolog"])

Start.py to run.

Second, and more reptiles run

1, create a folder in the same directory spiders, such as commands;

2, created under the new folder a py file, such as name crawlall.py, write code:

from scrapy.commands Import ScrapyCommand 


class the Command (ScrapyCommand): 
    requires_project = True 

    DEF syntax (Self):
         return  " [Options] " 

    DEF SHORT_DESC (Self):
         return  " the Run All of Spiders The "   # custom command description 

    DEF RUN (Self , args, the opts): 
        spider_list = self.crawler_process.spiders.list ()   # get a list of reptiles 
        for name in spider_list:   # circulation list for each reptile crawling. You can also filter the list of reptiles, according to their needs crawling want
            self.crawler_process.crawl(name, **opts.__dict__)
        self.crawler_process.start()

3, add the configuration in the settings.py: COMMANDS_MODULE = ". Project name directory name"

如：COMMANDS_MODULE = "myspider.spiders.commands"

4, enter in a terminal: scrapy crawlall --nolog to run (crawlall in step 2 is your new file name py)

Directory Structure

 └─myspider
        │  items.py
        │  middlewares.py
        │  pipelines.py
        │  settings.py
        │  __init__.py
        │
        ├─spiders
        │  │  zhihu.py
        │  │  __init__.py
        │  │
        │  ├─commands
        │  │  │  crawlall.py
        │  │  │
        │  │  └─__pycache__
        │  │          crawlall.cpython-36.pyc
        │  │
        │  └─__pycache__
        │          zhihu.cpython-36.pyc
        │          __init__.cpython-36.pyc
        │
        └─__pycache__
                items.cpython-36.pyc
                pipelines.cpython-36.pyc
                settings.cpython-36.pyc
                __init__.cpython-36.pyc