[scrapy] DEBUG:Filtered duplicate request:<GET:xxxx>-no more duplicates will be shown
It does not display more duplicates,
In fact, this problem is, CrawlSpider combination LinkExtractor \ Rule, when the extract links and send link, there have been repeated connection, repeat the request, the emergence of this DEBUG
Or yield scrapy.Request (xxxurl, callback = self.xxxx) has duplicate request
In fact, there is the default filter itself scrapy repeat request
DEBUG not let this occur, there can be dont_filter = True, add can be resolved in the Request
yield scrapy.Request(xxxurl,callback=self.xxxx,dont_filter=True)
From: https://blog.csdn.net/sinat_41701878/article/details/80302357