Signals
- Deferred signal handlers
- Built-in signals reference

Signals

Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box.

Even though signals provide several arguments, the handlers that catch them don’t need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives.

You can connect to signals (or send your own) through the Signals API.

Here is a simple example showing how you can catch signals and perform some action:

from scrapy import signals
from scrapy import Spider
class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
    ]
    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
        return spider
    def spider_closed(self, spider):
        spider.logger.info('Spider closed: %s', spider.name)
    def parse(self, response):
        pass

Deferred signal handlers

Some signals support returning Deferred objects from their handlers, allowing you to run asynchronous code that does not block Scrapy. If a signal handler returns a Deferred, Scrapy waits for that Deferred to fire.

Let’s take an example:

class SignalSpider(scrapy.Spider):
    name = 'signals'
    start_urls = ['http://quotes.toscrape.com/page/1/']
    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
        return spider
    def item_scraped(self, item):
        # Send the scraped item to the server
        d = treq.post(
            'http://example.com/post',
            json.dumps(item).encode('ascii'),
            headers={b'Content-Type': [b'application/json']}
        )
        # The next item will be scraped only after
        # deferred (d) is fired
        return d
    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

See the Built-in signals reference below to know which signals support Deferred.

Built-in signals reference

Here’s the list of Scrapy built-in signals and their meaning.

Engine signals

engine_started

scrapy.signals.``engine_started()

Sent when the Scrapy engine has started crawling.

This signal supports returning deferreds from their handlers.

Note

This signal may be fired after the spider_opened signal, depending on how the spider was started. So don’t rely on this signal getting fired before spider_opened.

engine_stopped

scrapy.signals.``engine_stopped()

Sent when the Scrapy engine is stopped (for example, when a crawling process has finished).

This signal supports returning deferreds from their handlers.

Item signals

Note

As at max CONCURRENT_ITEMS items are processed in parallel, many deferreds are fired together using DeferredList. Hence the next batch waits for the DeferredList to fire and then runs the respective item signal handler for the next batch of scraped items.

item_scraped

scrapy.signals.``item_scraped(item, response, spider)

Sent when an item has been scraped, after it has passed all the Item Pipeline stages (without being dropped).

This signal supports returning deferreds from their handlers.

Parameters
- item (dict or Item object) – the item scraped
- spider (Spider object) – the spider which scraped the item
- response (Response object) – the response from where the item was scraped

item_dropped

scrapy.signals.``item_dropped(item, response, exception, spider)

Sent after an item has been dropped from the Item Pipeline when some stage raised a DropItem exception.

This signal supports returning deferreds from their handlers.

Parameters
- item (dict or Item object) – the item dropped from the Item Pipeline
- spider (Spider object) – the spider which scraped the item
- response (Response object) – the response from where the item was dropped
- exception (DropItem exception) – the exception (which must be a DropItem subclass) which caused the item to be dropped

item_error

scrapy.signals.``item_error(item, response, spider, failure)

Sent when a Item Pipeline generates an error (i.e. raises an exception), except DropItem exception.

This signal supports returning deferreds from their handlers.

Parameters
- item (dict or Item object) – the item dropped from the Item Pipeline
- response (Response object) – the response being processed when the exception was raised
- spider (Spider object) – the spider which raised the exception
- failure (twisted.python.failure.Failure) – the exception raised

Spider signals

spider_closed

scrapy.signals.``spider_closed(spider, reason)

Sent after a spider has been closed. This can be used to release per-spider resources reserved on spider_opened.

This signal supports returning deferreds from their handlers.

Parameters
- spider (Spider object) – the spider which has been closed
- reason (str) – a string which describes the reason why the spider was closed. If it was closed because the spider has completed scraping, the reason is 'finished'. Otherwise, if the spider was manually closed by calling the close_spider engine method, then the reason is the one passed in the reason argument of that method (which defaults to 'cancelled'). If the engine was shutdown (for example, by hitting Ctrl-C to stop it) the reason will be 'shutdown'.

spider_opened

scrapy.signals.``spider_opened(spider)

Sent after a spider has been opened for crawling. This is typically used to reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened.

This signal supports returning deferreds from their handlers.

Parameters

spider (Spider object) – the spider which has been opened

spider_idle

scrapy.signals.``spider_idle(spider)

Sent when a spider has gone idle, which means the spider has no further:

requests waiting to be downloaded

requests scheduled

items being processed in the item pipeline

If the idle state persists after all handlers of this signal have finished, the engine starts closing the spider. After the spider has finished closing, the spider_closed signal is sent.

You may raise a DontCloseSpider exception to prevent the spider from being closed.

This signal does not support returning deferreds from their handlers.

Parameters

spider (Spider object) – the spider which has gone idle

Note

Scheduling some requests in your spider_idle handler does not guarantee that it can prevent the spider from being closed, although it sometimes can. That’s because the spider may still remain idle if all the scheduled requests are rejected by the scheduler (e.g. filtered due to duplication).

spider_error

scrapy.signals.``spider_error(failure, response, spider)

Sent when a spider callback generates an error (i.e. raises an exception).

This signal does not support returning deferreds from their handlers.

Parameters
- failure (twisted.python.failure.Failure) – the exception raised
- response (Response object) – the response being processed when the exception was raised
- spider (Spider object) – the spider which raised the exception

Request signals

request_scheduled

scrapy.signals.``request_scheduled(request, spider)

Sent when the engine schedules a Request, to be downloaded later.

The signal does not support returning deferreds from their handlers.

Parameters
- request (Request object) – the request that reached the scheduler
- spider (Spider object) – the spider that yielded the request

request_dropped

scrapy.signals.``request_dropped(request, spider)

Sent when a Request, scheduled by the engine to be downloaded later, is rejected by the scheduler.

The signal does not support returning deferreds from their handlers.

Parameters
- request (Request object) – the request that reached the scheduler
- spider (Spider object) – the spider that yielded the request

request_reached_downloader

scrapy.signals.``request_reached_downloader(request, spider)

Sent when a Request reached downloader.

The signal does not support returning deferreds from their handlers.

Parameters
- request (Request object) – the request that reached downloader
- spider (Spider object) – the spider that yielded the request

request_left_downloader

scrapy.signals.``request_left_downloader(request, spider)

New in version 2.0.

Sent when a Request leaves the downloader, even in case of failure.

This signal does not support returning deferreds from its handlers.

Parameters
- request (Request object) – the request that reached the downloader
- spider (Spider object) – the spider that yielded the request

Response signals

response_received

scrapy.signals.``response_received(response, request, spider)

Sent when the engine receives a new Response from the downloader.

This signal does not support returning deferreds from their handlers.

Parameters
- response (Response object) – the response received
- request (Request object) – the request that generated the response
- spider (Spider object) – the spider for which the response is intended

response_downloaded

scrapy.signals.``response_downloaded(response, request, spider)

Sent by the downloader right after a HTTPResponse is downloaded.

This signal does not support returning deferreds from their handlers.

Parameters
- response (Response object) – the response downloaded
- request (Request object) – the request that generated the response
- spider (Spider object) – the spider for which the response is intended