Spider Middleware

The spider middleware is a framework of hooks into Scrapy’s spider processingmechanism where you can plug custom functionality to process the responses thatare sent to Spiders for processing and to process the requestsand items that are generated from spiders.

Activating a spider middleware

To activate a spider middleware component, add it to theSPIDER_MIDDLEWARES setting, which is a dict whose keys are themiddleware class path and their values are the middleware orders.

Here’s an example:

SPIDER_MIDDLEWARES = {
    'myproject.middlewares.CustomSpiderMiddleware': 543,
}

The SPIDER_MIDDLEWARES setting is merged with theSPIDER_MIDDLEWARES_BASE setting defined in Scrapy (and not meant tobe overridden) and then sorted by order to get the final sorted list of enabledmiddlewares: the first middleware is the one closer to the engine and the lastis the one closer to the spider. In other words,the process_spider_input()method of each middleware will be invoked in increasingmiddleware order (100, 200, 300, …), and theprocess_spider_output() methodof each middleware will be invoked in decreasing order.

To decide which order to assign to your middleware see theSPIDER_MIDDLEWARES_BASE setting and pick a value according to whereyou want to insert the middleware. The order does matter because eachmiddleware performs a different action and your middleware could depend on someprevious (or subsequent) middleware being applied.

If you want to disable a builtin middleware (the ones defined inSPIDER_MIDDLEWARES_BASE, and enabled by default) you must define itin your project SPIDER_MIDDLEWARES setting and assign None as itsvalue. For example, if you want to disable the off-site middleware:

SPIDER_MIDDLEWARES = {
    'myproject.middlewares.CustomSpiderMiddleware': 543,
    'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,
}

Finally, keep in mind that some middlewares may need to be enabled through aparticular setting. See each middleware documentation for more info.

Writing your own spider middleware

Each spider middleware is a Python class that defines one or more of themethods defined below.

The main entry point is the from_crawler class method, which receives aCrawler instance. The Crawlerobject gives you access, for example, to the settings.

• class scrapy.spidermiddlewares.SpiderMiddleware
• • processspider_input(_response, spider)
  • This method is called for each response that goes through the spidermiddleware and into the spider, for processing.

process_spider_input() should return None or raise anexception.

If it returns None, Scrapy will continue processing this response,executing all other middlewares until, finally, the response is handedto the spider for processing.

If it raises an exception, Scrapy won’t bother calling any other spidermiddleware process_spider_input() and will call the requesterrback if there is one, otherwise it will start the process_spider_exception()chain. The output of the errback is chained back in the otherdirection for process_spider_output() to process it, orprocess_spider_exception() if it raised an exception.

Parameters:

- **response** ([<code>Response</code>]($a8ff621e6a7e7cdf.md#scrapy.http.Response) object) – the response being processed
- **spider** ([<code>Spider</code>]($fb832e20e85f228c.md#scrapy.spiders.Spider) object) – the spider for which this response is intended

• processspider_output(_response, result, spider)
• This method is called with the results returned from the Spider, afterit has processed the response.

process_spider_output() must return an iterable ofRequest, dict or Itemobjects.

Parameters:

- **response** ([<code>Response</code>]($a8ff621e6a7e7cdf.md#scrapy.http.Response) object) – the response which generated this output from thespider
- **result** (an iterable of [<code>Request</code>]($a8ff621e6a7e7cdf.md#scrapy.http.Request), dictor [<code>Item</code>]($03ea72809515384c.md#scrapy.item.Item) objects) – the result returned by the spider
- **spider** ([<code>Spider</code>]($fb832e20e85f228c.md#scrapy.spiders.Spider) object) – the spider whose result is being processed

• processspider_exception(_response, exception, spider)
• This method is called when a spider or process_spider_output()method (from a previous spider middleware) raises an exception.

process_spider_exception() should return either None or aniterable of Request, dict orItem objects.

If it returns None, Scrapy will continue processing this exception,executing any other process_spider_exception() in the followingmiddleware components, until no middleware components are left and theexception reaches the engine (where it’s logged and discarded).

If it returns an iterable the process_spider_output() pipelinekicks in, starting from the next spider middleware, and no otherprocess_spider_exception() will be called.

Parameters:

- **response** ([<code>Response</code>]($a8ff621e6a7e7cdf.md#scrapy.http.Response) object) – the response being processed when the exception wasraised
- **exception** ([Exception](https://docs.python.org/2/library/exceptions.html#exceptions.Exception) object) – the exception raised
- **spider** ([<code>Spider</code>]($fb832e20e85f228c.md#scrapy.spiders.Spider) object) – the spider which raised the exception

• processstart_requests(_start_requests, spider)

New in version 0.15.

This method is called with the start requests of the spider, and workssimilarly to the process_spider_output() method, except that itdoesn’t have a response associated and must return only requests (notitems).

It receives an iterable (in the start_requests parameter) and mustreturn another iterable of Request objects.

Note

When implementing this method in your spider middleware, youshould always return an iterable (that follows the input one) andnot consume all start_requests iterator because it can be verylarge (or even unbounded) and cause a memory overflow. The Scrapyengine is designed to pull start requests while it has capacity toprocess them, so the start requests iterator can be effectivelyendless where there is some other condition for stopping the spider(like a time limit or item/page count).

Parameters:

- **start_requests** (an iterable of [<code>Request</code>]($a8ff621e6a7e7cdf.md#scrapy.http.Request)) – the start requests
- **spider** ([<code>Spider</code>]($fb832e20e85f228c.md#scrapy.spiders.Spider) object) – the spider to whom the start requests belong

• fromcrawler(_cls, crawler)
• If present, this classmethod is called to create a middleware instancefrom a Crawler. It must return a new instanceof the middleware. Crawler object provides access to all Scrapy corecomponents like settings and signals; it is a way for middleware toaccess them and hook its functionality into Scrapy.

Parameters:crawler (Crawler object) – crawler that uses this middleware

Built-in spider middleware reference

This page describes all spider middleware components that come with Scrapy. Forinformation on how to use them and how to write your own spider middleware, seethe spider middleware usage guide.

For a list of the components enabled by default (and their orders) see theSPIDER_MIDDLEWARES_BASE setting.

DepthMiddleware

• class scrapy.spidermiddlewares.depth.DepthMiddleware[source]
• DepthMiddleware is used for tracking the depth of each Request inside thesite being scraped. It works by setting request.meta['depth'] = 0 wheneverthere is no value previously set (usually just the first Request) andincrementing it by 1 otherwise.

It can be used to limit the maximum depth to scrape, control Requestpriority based on their depth, and things like that.

The DepthMiddleware can be configured through the followingsettings (see the settings documentation for more info):

• DEPTH_LIMIT - The maximum depth that will be allowed tocrawl for any site. If zero, no limit will be imposed.
• DEPTH_STATS_VERBOSE - Whether to collect the number ofrequests for each depth.
• DEPTH_PRIORITY - Whether to prioritize the requests based ontheir depth.

HttpErrorMiddleware

• class scrapy.spidermiddlewares.httperror.HttpErrorMiddleware[source]
• Filter out unsuccessful (erroneous) HTTP responses so that spiders don’thave to deal with them, which (most of the time) imposes an overhead,consumes more resources, and makes the spider logic more complex.

According to the HTTP standard, successful responses are those whosestatus codes are in the 200-300 range.

If you still want to process response codes outside that range, you canspecify which response codes the spider is able to handle using thehandle_httpstatus_list spider attribute orHTTPERROR_ALLOWED_CODES setting.

For example, if you want your spider to handle 404 responses you can dothis:

class MySpider(CrawlSpider):
    handle_httpstatus_list = [404]

The handle_httpstatus_list key of Request.meta can also be used to specify which response codes toallow on a per-request basis. You can also set the meta key handle_httpstatus_allto True if you want to allow any response code for a request.

Keep in mind, however, that it’s usually a bad idea to handle non-200responses, unless you really know what you’re doing.

For more information see: HTTP Status Code Definitions.

HttpErrorMiddleware settings

HTTPERROR_ALLOWED_CODES

Default: []

Pass all responses with non-200 status codes contained in this list.

HTTPERROR_ALLOW_ALL

Default: False

Pass all responses, regardless of its status code.

OffsiteMiddleware

• class scrapy.spidermiddlewares.offsite.OffsiteMiddleware[source]
• Filters out Requests for URLs outside the domains covered by the spider.

This middleware filters out every request whose host names aren’t in thespider’s allowed_domains attribute.All subdomains of any domain in the list are also allowed.E.g. the rule www.example.org will also allow bob.www.example.orgbut not www2.example.com nor example.com.

When your spider returns a request for a domain not belonging to thosecovered by the spider, this middleware will log a debug message similar tothis one:

DEBUG: Filtered offsite request to 'www.othersite.com': <GET http://www.othersite.com/some/page.html>

To avoid filling the log with too much noise, it will only print one ofthese messages for each new domain filtered. So, for example, if anotherrequest for www.othersite.com is filtered, no log message will beprinted. But if a request for someothersite.com is filtered, a messagewill be printed (but only for the first request filtered).

If the spider doesn’t define anallowed_domains attribute, or theattribute is empty, the offsite middleware will allow all requests.

If the request has the dont_filter attributeset, the offsite middleware will allow the request even if its domain is notlisted in allowed domains.

RefererMiddleware

• class scrapy.spidermiddlewares.referer.RefererMiddleware[source]
• Populates Request Referer header, based on the URL of the Response whichgenerated it.

RefererMiddleware settings

REFERER_ENABLED

New in version 0.15.

Default: True

Whether to enable referer middleware.

REFERRER_POLICY

New in version 1.4.

Default: 'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'

Referrer Policy to apply when populating Request “Referer” header.

Note

You can also set the Referrer Policy per request,using the special "referrer_policy" Request.meta key,with the same acceptable values as for the REFERRER_POLICY setting.

Acceptable values for REFERRER_POLICY

• either a path to a scrapy.spidermiddlewares.referer.ReferrerPolicysubclass — a custom policy or one of the built-in ones (see classes below),
• or one of the standard W3C-defined string values,
• or the special "scrapy-default".

• class scrapy.spidermiddlewares.referer.DefaultReferrerPolicy[source]
• A variant of “no-referrer-when-downgrade”,with the addition that “Referer” is not sent if the parent request wasusing file:// or s3:// scheme.

Warning

Scrapy’s default referrer policy — just like “no-referrer-when-downgrade”,the W3C-recommended value for browsers — will send a non-empty“Referer” header from any http(s):// to any https:// URL,even if the domain is different.

“same-origin” may be a better choice if you want to remove referrerinformation for cross-domain requests.

• class scrapy.spidermiddlewares.referer.NoReferrerPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer

The simplest policy is “no-referrer”, which specifies that no referrer informationis to be sent along with requests made from a particular request client to any origin.The header will be omitted entirely.

• class scrapy.spidermiddlewares.referer.NoReferrerWhenDowngradePolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade

The “no-referrer-when-downgrade” policy sends a full URL along with requestsfrom a TLS-protected environment settings object to a potentially trustworthy URL,and requests from clients which are not TLS-protected to any origin.

Requests from TLS-protected clients to non-potentially trustworthy URLs,on the other hand, will contain no referrer information.A Referer HTTP header will not be sent.

This is a user agent’s default behavior, if no policy is otherwise specified.

Note

“no-referrer-when-downgrade” policy is the W3C-recommended default,and is used by major web browsers.

However, it is NOT Scrapy’s default referrer policy (see DefaultReferrerPolicy).

• class scrapy.spidermiddlewares.referer.SameOriginPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin

The “same-origin” policy specifies that a full URL, stripped for use as a referrer,is sent as referrer information when making same-origin requests from a particular request client.

Cross-origin requests, on the other hand, will contain no referrer information.A Referer HTTP header will not be sent.

• class scrapy.spidermiddlewares.referer.OriginPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-origin

The “origin” policy specifies that only the ASCII serializationof the origin of the request client is sent as referrer informationwhen making both same-origin requests and cross-origin requestsfrom a particular request client.

• class scrapy.spidermiddlewares.referer.StrictOriginPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin

The “strict-origin” policy sends the ASCII serializationof the origin of the request client when making requests:- from a TLS-protected environment settings object to a potentially trustworthy URL, and- from non-TLS-protected environment settings objects to any origin.

Requests from TLS-protected request clients to non- potentially trustworthy URLs,on the other hand, will contain no referrer information.A Referer HTTP header will not be sent.

• class scrapy.spidermiddlewares.referer.OriginWhenCrossOriginPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin

The “origin-when-cross-origin” policy specifies that a full URL,stripped for use as a referrer, is sent as referrer informationwhen making same-origin requests from a particular request client,and only the ASCII serialization of the origin of the request clientis sent as referrer information when making cross-origin requestsfrom a particular request client.

• class scrapy.spidermiddlewares.referer.StrictOriginWhenCrossOriginPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin

The “strict-origin-when-cross-origin” policy specifies that a full URL,stripped for use as a referrer, is sent as referrer informationwhen making same-origin requests from a particular request client,and only the ASCII serialization of the origin of the request clientwhen making cross-origin requests:

• from a TLS-protected environment settings object to a potentially trustworthy URL, and
• from non-TLS-protected environment settings objects to any origin.Requests from TLS-protected clients to non- potentially trustworthy URLs,on the other hand, will contain no referrer information.A Referer HTTP header will not be sent.

• class scrapy.spidermiddlewares.referer.UnsafeUrlPolicy[source]
• https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url

The “unsafe-url” policy specifies that a full URL, stripped for use as a referrer,is sent along with both cross-origin requestsand same-origin requests made from a particular request client.

Note: The policy’s name doesn’t lie; it is unsafe.This policy will leak origins and paths from TLS-protected resourcesto insecure origins.Carefully consider the impact of setting such a policy for potentially sensitive documents.

Warning

“unsafe-url” policy is NOT recommended.

UrlLengthMiddleware

• class scrapy.spidermiddlewares.urllength.UrlLengthMiddleware[source]
• Filters out requests with URLs longer than URLLENGTH_LIMIT

The UrlLengthMiddleware can be configured through the followingsettings (see the settings documentation for more info):

• URLLENGTH_LIMIT - The maximum URL length to allow for crawled URLs.