scrapy how debug breakpoint debugging

scrapy how debug breakpoint debugging
1, add a file main.py in the folder of the project (scrapy.cfg its class)
main.py file

from scrapy.cmdline import execute
import os
import sys
if __name__ == '__main__':

    sys.path.append(os.path.dirname(os.path.abspath(__file__)))
    execute(['scrapy','crawl','scrapy_name'])

Interpreted as follows
1, os.path.abspath (path) returns the absolute path
2, os.path.dirname (path) Returns the path to the folder
3, sys.path return path search module, using the value of environment variable PYTHONPATH initialization, sys Road King .path.append current file directory on the execution added to python
Road King inside
4, file address this document
file parsing more specific reference https://blog.csdn.net/huangwencai123/article/details/89879951 address
this lines of code to get the current parent directory py file, eliminating the need for a direct copy of our time and energy path, but also to help us put the project up to run other platforms, will not report a wrong path.
5, execute (execute function is embedded in the scrapy, call this function can be called crawlers script Scrapy the project directly, perform this function needs to be inside the parameters .execute function in the parent directory of the project is actually the reptiles Run Scrapy crawler split, and then were placed in the manner of a character in an array.

The execution of the command can not write scrpy

2, debugging breakpoints
set breakpoints in the spider scrapy, the main.py by the debug can.

Guess you like

Origin blog.csdn.net/huangwencai123/article/details/89890940