[Scrapy Framework] "Version 2.4.0 Source Code" Exception Operation (Exceptions) Detailed Chapter

All source code analysis article index directory portal

[Scrapy Framework] Version 2.4.0 Source Code: All Configuration Directory Index

Introduction

This is a list of all the exceptions included in Scrapy and their usage.

Routine abnormal operation

Close spider

exception scrapy.exceptions.CloseSpider(reason=‘cancelled’)

def parse_page(self, response):
    if 'Bandwidth exceeded' in response.body:
        raise CloseSpider('bandwidth_exceeded')

Do not close spider

exception scrapy.exceptions.DontCloseSpider

This exception can be raised in the spider_idle signal handler to prevent the spider web from being closed.

Abandon Item

exception scrapy.exceptions.DropItem

The pipline phase of the Item must raise an exception to stop processing the Item.

Ignore request

exception scrapy.exceptions.IgnoreRequest

The scheduler or any downloader middleware can raise this exception to indicate that the request should be ignored.

Not configured

exception scrapy.exceptions.NotConfigured

Certain components can raise this exception to indicate that they will remain disabled. These components include

1. Extensions
2. Item pipelines
3. Downloader middlewares
4. Spider middlewares

not support

exception scrapy.exceptions.NotSupported

This exception is raised to indicate an unsupported feature.

Stop download

exception scrapy.exceptions.StopDownload(fail=True)

Sent from the bytes_received signal handler, indicating that no more bytes should be downloaded in response.

1. If fail=True (default), call request errback. The response object can be used as the response attribute of the exception, and the StopDownload exception is stored as the attribute of the value receiving the Failure object. This means that in the errback defined as, although the response can be accessed.

def errback(self, failure)：
	failure.value.response

2. If fail=False, call the request callback.