scrapy: Call a function when a spider quits
Just to update, you can just call closed
function like this:
class MySpider(CrawlSpider):
def closed(self, reason):
do-something()
It looks like you can register a signal listener through dispatcher
.
I would try something like:
from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher
class MySpider(CrawlSpider):
def __init__(self):
dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
# second param is instance of spder about to be closed.
In the newer version of scrapy scrapy.xlib.pydispatch
is deprecated. instead you can use from pydispatch import dispatcher
.