Is there a way to trigger a method in a Spider class just before it terminates?
I can terminate the spider myself, like this:
#Config stuff goes here...
#Do some stuff...
raise CloseSpider('MySpider is quitting now.')
def my_parser(self, response):
#Parsing stuff goes here...
It looks like you can register a signal listener through
I would try something like:
from scrapy import signals from scrapy.xlib.pydispatch import dispatcher class MySpider(CrawlSpider): def __init__(self): dispatcher.connect(self.spider_closed, signals.spider_closed) def spider_closed(self, spider): # second param is instance of spder about to be closed.