Signals

Scrapy uses signals extensively to notify when certain events occur. You cancatch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapyto add functionality not provided out of the box.

Even though signals provide several arguments, the handlers that catch themdon’t need to accept all of them - the signal dispatching mechanism will onlydeliver the arguments that the handler receives.

You can connect to signals (or send your own) through theSignals API.

Here is a simple example showing how you can catch signals and perform some action:

from scrapy import signals
from scrapy import Spider
 
 
class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
    ]
 
 
    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
        return spider
 
 
    def spider_closed(self, spider):
        spider.logger.info('Spider closed: %s', spider.name)
 
 
    def parse(self, response):
        pass

Deferred signal handlers

Some signals support returning Deferredobjects from their handlers, see the Built-in signals reference below to knowwhich ones.

Built-in signals reference

Here’s the list of Scrapy built-in signals and their meaning.

engine_started

• scrapy.signals.engine_started()
• Sent when the Scrapy engine has started crawling.

This signal supports returning deferreds from their handlers.

Note

This signal may be fired after the spider_opened signal,depending on how the spider was started. So don’t rely on this signalgetting fired before spider_opened.

engine_stopped

• scrapy.signals.engine_stopped()
• Sent when the Scrapy engine is stopped (for example, when a crawlingprocess has finished).

This signal supports returning deferreds from their handlers.

item_scraped

• scrapy.signals.itemscraped(_item, response, spider)
• Sent when an item has been scraped, after it has passed all theItem Pipeline stages (without being dropped).

This signal supports returning deferreds from their handlers.

Parameters:

• item (dict or Item object) – the item scraped
• spider (Spider object) – the spider which scraped the item
• response (Response object) – the response from where the item was scraped

item_dropped

• scrapy.signals.itemdropped(_item, response, exception, spider)
• Sent after an item has been dropped from the Item Pipelinewhen some stage raised a DropItem exception.

This signal supports returning deferreds from their handlers.

Parameters:

• item (dict or Item object) – the item dropped from the Item Pipeline
• spider (Spider object) – the spider which scraped the item
• response (Response object) – the response from where the item was dropped
• exception (DropItem exception) – the exception (which must be aDropItem subclass) which caused the itemto be dropped

item_error

• scrapy.signals.itemerror(_item, response, spider, failure)
• Sent when a Item Pipeline generates an error (i.e. raisesan exception), except DropItem exception.

This signal supports returning deferreds from their handlers.

Parameters:

• item (dict or Item object) – the item dropped from the Item Pipeline
• response (Response object) – the response being processed when the exception was raised
• spider (Spider object) – the spider which raised the exception
• failure (twisted.python.failure.Failure) – the exception raised

spider_closed

• scrapy.signals.spiderclosed(_spider, reason)
• Sent after a spider has been closed. This can be used to release per-spiderresources reserved on spider_opened.

This signal supports returning deferreds from their handlers.

Parameters:

• spider (Spider object) – the spider which has been closed
• reason (str) – a string which describes the reason why the spider was closed. Ifit was closed because the spider has completed scraping, the reasonis 'finished'. Otherwise, if the spider was manually closed bycalling the close_spider engine method, then the reason is the onepassed in the reason argument of that method (which defaults to'cancelled'). If the engine was shutdown (for example, by hittingCtrl-C to stop it) the reason will be 'shutdown'.

spider_opened

• scrapy.signals.spideropened(_spider)
• Sent after a spider has been opened for crawling. This is typically used toreserve per-spider resources, but can be used for any task that needs to beperformed when a spider is opened.

This signal supports returning deferreds from their handlers.

Parameters:spider (Spider object) – the spider which has been opened

spider_idle

• scrapy.signals.spideridle(_spider)
• Sent when a spider has gone idle, which means the spider has no further:

• requests waiting to be downloaded
• requests scheduled
• items being processed in the item pipeline

If the idle state persists after all handlers of this signal have finished,the engine starts closing the spider. After the spider has finishedclosing, the spider_closed signal is sent.

You may raise a DontCloseSpider exception toprevent the spider from being closed.

This signal does not support returning deferreds from their handlers.

Parameters:spider (Spider object) – the spider which has gone idle

Note

Scheduling some requests in your spider_idle handler doesnot guarantee that it can prevent the spider from being closed,although it sometimes can. That’s because the spider may still remain idleif all the scheduled requests are rejected by the scheduler (e.g. filtereddue to duplication).

spider_error

• scrapy.signals.spidererror(_failure, response, spider)
• Sent when a spider callback generates an error (i.e. raises an exception).

This signal does not support returning deferreds from their handlers.

Parameters:

• failure (twisted.python.failure.Failure) – the exception raised
• response (Response object) – the response being processed when the exception was raised
• spider (Spider object) – the spider which raised the exception

request_scheduled

• scrapy.signals.requestscheduled(_request, spider)
• Sent when the engine schedules a Request, to bedownloaded later.

The signal does not support returning deferreds from their handlers.

Parameters:

• request (Request object) – the request that reached the scheduler
• spider (Spider object) – the spider that yielded the request

request_dropped

• scrapy.signals.requestdropped(_request, spider)
• Sent when a Request, scheduled by the engine to bedownloaded later, is rejected by the scheduler.

The signal does not support returning deferreds from their handlers.

Parameters:

• request (Request object) – the request that reached the scheduler
• spider (Spider object) – the spider that yielded the request

request_reached_downloader

• scrapy.signals.requestreached_downloader(_request, spider)
• Sent when a Request reached downloader.

The signal does not support returning deferreds from their handlers.

Parameters:

• request (Request object) – the request that reached downloader
• spider (Spider object) – the spider that yielded the request

request_left_downloader

• scrapy.signals.requestleft_downloader(_request, spider)

New in version 2.0.

Sent when a Request leaves the downloader, even in case offailure.

This signal does not support returning deferreds from its handlers.

Parameters:

• request (Request object) – the request that reached the downloader
• spider (Spider object) – the spider that yielded the request

response_received

• scrapy.signals.responsereceived(_response, request, spider)
• Sent when the engine receives a new Response from thedownloader.

This signal does not support returning deferreds from their handlers.

Parameters:

• response (Response object) – the response received
• request (Request object) – the request that generated the response
• spider (Spider object) – the spider for which the response is intended

response_downloaded

• scrapy.signals.responsedownloaded(_response, request, spider)
• Sent by the downloader right after a HTTPResponse is downloaded.

This signal does not support returning deferreds from their handlers.

Parameters:

• response (Response object) – the response downloaded
• request (Request object) – the request that generated the response
• spider (Spider object) – the spider for which the response is intended