Selectors

When you’re scraping web pages, the most common task you need to perform isto extract data from the HTML source. There are several libraries available toachieve this, such as:

  • BeautifulSoup is a very popular web scraping library among Pythonprogrammers which constructs a Python object based on the structure of theHTML code and also deals with bad markup reasonably well, but it has onedrawback: it’s slow.
  • lxml is an XML parsing library (which also parses HTML) with a pythonicAPI based on ElementTree. (lxml is not part of the Python standardlibrary.)

Scrapy comes with its own mechanism for extracting data. They’re calledselectors because they “select” certain parts of the HTML document specifiedeither by XPath or CSS expressions.

XPath is a language for selecting nodes in XML documents, which can also beused with HTML. CSS is a language for applying styles to HTML documents. Itdefines selectors to associate those styles with specific HTML elements.

Note

Scrapy Selectors is a thin wrapper around parsel library; the purpose ofthis wrapper is to provide better integration with Scrapy Response objects.

parsel is a stand-alone web scraping library which can be used withoutScrapy. It uses lxml library under the hood, and implements aneasy API on top of lxml API. It means Scrapy selectors are very similarin speed and parsing accuracy to lxml.

Using selectors

Constructing selectors

Response objects expose a Selector instanceon .selector attribute:

  1. >>> response.selector.xpath('//span/text()').get()
  2. 'good'

Querying responses using XPath and CSS is so common that responses include twomore shortcuts: response.xpath() and response.css():

  1. >>> response.xpath('//span/text()').get()
  2. 'good'
  3. >>> response.css('span::text').get()
  4. 'good'

Scrapy selectors are instances of Selector classconstructed by passing either TextResponse object ormarkup as an unicode string (in text argument).Usually there is no need to construct Scrapy selectors manually:response object is available in Spider callbacks, so in most casesit is more convenient to use response.css() and response.xpath()shortcuts. By using response.selector or one of these shortcutsyou can also ensure the response body is parsed only once.

But if required, it is possible to use Selector directly.Constructing from text:

  1. >>> from scrapy.selector import Selector
  2. >>> body = '<html><body><span>good</span></body></html>'
  3. >>> Selector(text=body).xpath('//span/text()').get()
  4. 'good'

Constructing from response - HtmlResponse is one ofTextResponse subclasses:

  1. >>> from scrapy.selector import Selector
  2. >>> from scrapy.http import HtmlResponse
  3. >>> response = HtmlResponse(url='http://example.com', body=body)
  4. >>> Selector(response=response).xpath('//span/text()').get()
  5. 'good'

Selector automatically chooses the best parsing rules(XML vs HTML) based on input type.

Using selectors

To explain how to use the selectors we’ll use the Scrapy shell (whichprovides interactive testing) and an example page located in the Scrapydocumentation server:

For the sake of completeness, here’s its full HTML code:

  1. <html>
  2.  <head>
  3.   <base href='http://example.com/' />
  4.   <title>Example website</title>
  5.  </head>
  6.  <body>
  7.   <div id='images'>
  8.    <a href='image1.html'>Name: My image 1 <br /><img src='image1_thumb.jpg' /></a>
  9.    <a href='image2.html'>Name: My image 2 <br /><img src='image2_thumb.jpg' /></a>
  10.    <a href='image3.html'>Name: My image 3 <br /><img src='image3_thumb.jpg' /></a>
  11.    <a href='image4.html'>Name: My image 4 <br /><img src='image4_thumb.jpg' /></a>
  12.    <a href='image5.html'>Name: My image 5 <br /><img src='image5_thumb.jpg' /></a>
  13.   </div>
  14.  </body>
  15. </html>

First, let’s open the shell:

  1. scrapy shell https://docs.scrapy.org/en/latest/_static/selectors-sample1.html

Then, after the shell loads, you’ll have the response available as responseshell variable, and its attached selector in response.selector attribute.

Since we’re dealing with HTML, the selector will automatically use an HTML parser.

So, by looking at the HTML code of thatpage, let’s construct an XPath for selecting the text inside the title tag:

  1. >>> response.xpath('//title/text()')
  2. [<Selector xpath='//title/text()' data='Example website'>]

To actually extract the textual data, you must call the selector .get()or .getall() methods, as follows:

  1. >>> response.xpath('//title/text()').getall()
  2. ['Example website']
  3. >>> response.xpath('//title/text()').get()
  4. 'Example website'

.get() always returns a single result; if there are several matches,content of a first match is returned; if there are no matches, Noneis returned. .getall() returns a list with all results.

Notice that CSS selectors can select text or attribute nodes using CSS3pseudo-elements:

  1. >>> response.css('title::text').get()
  2. 'Example website'

As you can see, .xpath() and .css() methods return aSelectorList instance, which is a list of newselectors. This API can be used for quickly selecting nested data:

  1. >>> response.css('img').xpath('@src').getall()
  2. ['image1_thumb.jpg',
  3.  'image2_thumb.jpg',
  4.  'image3_thumb.jpg',
  5.  'image4_thumb.jpg',
  6.  'image5_thumb.jpg']

If you want to extract only the first matched element, you can call theselector .get() (or its alias .extract_first() commonly used inprevious Scrapy versions):

  1. >>> response.xpath('//div[@id="images"]/a/text()').get()
  2. 'Name: My image 1 '

It returns None if no element was found:

  1. >>> response.xpath('//div[@id="not-exists"]/text()').get() is None
  2. True

A default return value can be provided as an argument, to be used insteadof None:

  1. >>> response.xpath('//div[@id="not-exists"]/text()').get(default='not-found')
  2. 'not-found'

Instead of using e.g. '@src' XPath it is possible to query for attributesusing .attrib property of a Selector:

  1. >>> [img.attrib['src'] for img in response.css('img')]
  2. ['image1_thumb.jpg',
  3.  'image2_thumb.jpg',
  4.  'image3_thumb.jpg',
  5.  'image4_thumb.jpg',
  6.  'image5_thumb.jpg']

As a shortcut, .attrib is also available on SelectorList directly;it returns attributes for the first matching element:

  1. >>> response.css('img').attrib['src']
  2. 'image1_thumb.jpg'

This is most useful when only a single result is expected, e.g. when selectingby id, or selecting unique elements on a web page:

  1. >>> response.css('base').attrib['href']
  2. 'http://example.com/'

Now we’re going to get the base URL and some image links:

  1. >>> response.xpath('//base/@href').get()
  2. 'http://example.com/'
  1. >>> response.css('base::attr(href)').get()
  2. 'http://example.com/'
  1. >>> response.css('base').attrib['href']
  2. 'http://example.com/'
  1. >>> response.xpath('//a[contains(@href, "image")]/@href').getall()
  2. ['image1.html',
  3.  'image2.html',
  4.  'image3.html',
  5.  'image4.html',
  6.  'image5.html']
  1. >>> response.css('a[href*=image]::attr(href)').getall()
  2. ['image1.html',
  3.  'image2.html',
  4.  'image3.html',
  5.  'image4.html',
  6.  'image5.html']
  1. >>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()
  2. ['image1_thumb.jpg',
  3.  'image2_thumb.jpg',
  4.  'image3_thumb.jpg',
  5.  'image4_thumb.jpg',
  6.  'image5_thumb.jpg']
  1. >>> response.css('a[href*=image] img::attr(src)').getall()
  2. ['image1_thumb.jpg',
  3.  'image2_thumb.jpg',
  4.  'image3_thumb.jpg',
  5.  'image4_thumb.jpg',
  6.  'image5_thumb.jpg']

Extensions to CSS Selectors

Per W3C standards, CSS selectors do not support selecting text nodesor attribute values.But selecting these is so essential in a web scraping contextthat Scrapy (parsel) implements a couple of non-standard pseudo-elements:

  • to select text nodes, use ::text
  • to select attribute values, use ::attr(name) where name is thename of the attribute that you want the value of

Warning

These pseudo-elements are Scrapy-/Parsel-specific.They will most probably not work with other libraries likelxml or PyQuery.

Examples:

  • title::text selects children text nodes of a descendant <title> element:
  1. >>> response.css('title::text').get()
  2. 'Example website'
  • *::text selects all descendant text nodes of the current selector context:
  1. >>> response.css('#images *::text').getall()
  2. ['\n   ',
  3.  'Name: My image 1 ',
  4.  '\n   ',
  5.  'Name: My image 2 ',
  6.  '\n   ',
  7.  'Name: My image 3 ',
  8.  '\n   ',
  9.  'Name: My image 4 ',
  10.  '\n   ',
  11.  'Name: My image 5 ',
  12.  '\n  ']
  • foo::text returns no results if foo element exists, but containsno text (i.e. text is empty):
  1. >>> response.css('img::text').getall()
  2. []
This means .css('foo::text').get() could return None even if an elementexists. Use default='' if you always want a string:
  1. >>> response.css('img::text').get()
  2. >>> response.css('img::text').get(default='')
  3. ''
  • a::attr(href) selects the href attribute value of descendant links:
  1. >>> response.css('a::attr(href)').getall()
  2. ['image1.html',
  3.  'image2.html',
  4.  'image3.html',
  5.  'image4.html',
  6.  'image5.html']

Note

See also: Selecting element attributes.

Note

You cannot chain these pseudo-elements. But in practice it would notmake much sense: text nodes do not have attributes, and attribute valuesare string values already and do not have children nodes.

Nesting selectors

The selection methods (.xpath() or .css()) return a list of selectorsof the same type, so you can call the selection methods for those selectorstoo. Here’s an example:

  1. >>> links = response.xpath('//a[contains(@href, "image")]')
  2. >>> links.getall()
  3. ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
  4.  '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
  5.  '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
  6.  '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
  7.  '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
  1. >>> for index, link in enumerate(links):
  2. ...     args = (index, link.xpath('@href').get(), link.xpath('img/@src').get())
  3. ...     print('Link number %d points to url %r and image %r' % args)
  4. Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'
  5. Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'
  6. Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'
  7. Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'
  8. Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'

Selecting element attributes

There are several ways to get a value of an attribute. First, one can useXPath syntax:

  1. >>> response.xpath("//a/@href").getall()
  2. ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']

XPath syntax has a few advantages: it is a standard XPath feature, and@attributes can be used in other parts of an XPath expression - e.g.it is possible to filter by attribute value.

Scrapy also provides an extension to CSS selectors (::attr(…))which allows to get attribute values:

  1. >>> response.css('a::attr(href)').getall()
  2. ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']

In addition to that, there is a .attrib property of Selector.You can use it if you prefer to lookup attributes in Pythoncode, without using XPaths or CSS extensions:

  1. >>> [a.attrib['href'] for a in response.css('a')]
  2. ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']

This property is also available on SelectorList; it returns a dictionarywith attributes of a first matching element. It is convenient to use whena selector is expected to give a single result (e.g. when selecting by elementID, or when selecting an unique element on a page):

  1. >>> response.css('base').attrib
  2. {'href': 'http://example.com/'}
  3. >>> response.css('base').attrib['href']
  4. 'http://example.com/'

.attrib property of an empty SelectorList is empty:

  1. >>> response.css('foo').attrib
  2. {}

Using selectors with regular expressions

Selector also has a .re() method for extractingdata using regular expressions. However, unlike using .xpath() or.css() methods, .re() returns a list of unicode strings. So youcan’t construct nested .re() calls.

Here’s an example used to extract image names from the HTML code above:

  1. >>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')
  2. ['My image 1',
  3.  'My image 2',
  4.  'My image 3',
  5.  'My image 4',
  6.  'My image 5']

There’s an additional helper reciprocating .get() (and itsalias .extract_first()) for .re(), named .re_first().Use it to extract just the first matching string:

  1. >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)')
  2. 'My image 1'

extract() and extract_first()

If you’re a long-time Scrapy user, you’re probably familiarwith .extract() and .extract_first() selector methods. Many blog postsand tutorials are using them as well. These methods are still supportedby Scrapy, there are no plans to deprecate them.

However, Scrapy usage docs are now written using .get() and.getall() methods. We feel that these new methods result in a more conciseand readable code.

The following examples show how these methods map to each other.

  • SelectorList.get() is the same as SelectorList.extract_first():
  1. >>> response.css('a::attr(href)').get()
  2. 'image1.html'
  3. >>> response.css('a::attr(href)').extract_first()
  4. 'image1.html'
  • SelectorList.getall() is the same as SelectorList.extract():
  1. >>> response.css('a::attr(href)').getall()
  2. ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
  3. >>> response.css('a::attr(href)').extract()
  4. ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
  • Selector.get() is the same as Selector.extract():
  1. >>> response.css('a::attr(href)')[0].get()
  2. 'image1.html'
  3. >>> response.css('a::attr(href)')[0].extract()
  4. 'image1.html'
  • For consistency, there is also Selector.getall(), which returns a list:
  1. >>> response.css('a::attr(href)')[0].getall()
  2. ['image1.html']

So, the main difference is that output of .get() and .getall() methodsis more predictable: .get() always returns a single result, .getall()always returns a list of all extracted results. With .extract() methodit was not always obvious if a result is a list or not; to get a singleresult either .extract() or .extract_first() should be called.

Working with XPaths

Here are some tips which may help you to use XPath with Scrapy selectorseffectively. If you are not much familiar with XPath yet,you may want to take a look first at this XPath tutorial.

Note

Some of the tips are based on this post from ScrapingHub’s blog.

Working with relative XPaths

Keep in mind that if you are nesting selectors and use an XPath that startswith /, that XPath will be absolute to the document and not relative to theSelector you’re calling it from.

For example, suppose you want to extract all <p> elements inside <div>elements. First, you would get all <div> elements:

  1. >>> divs = response.xpath('//div')

At first, you may be tempted to use the following approach, which is wrong, asit actually extracts all <p> elements from the document, not only thoseinside <div> elements:

  1. >>> for p in divs.xpath('//p'):  # this is wrong - gets all <p> from the whole document
  2. ...     print(p.get())

This is the proper way to do it (note the dot prefixing the .//p XPath):

  1. >>> for p in divs.xpath('.//p'):  # extracts all <p> inside
  2. ...     print(p.get())

Another common case would be to extract all direct <p> children:

  1. >>> for p in divs.xpath('p'):
  2. ...     print(p.get())

For more details about relative XPaths see the Location Paths section in theXPath specification.

When querying by class, consider using CSS

Because an element can contain multiple CSS classes, the XPath way to select elementsby class is the rather verbose:

  1. *[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')]

If you use @class='someclass' you may end up missing elements that haveother classes, and if you just use contains(@class, 'someclass') to make upfor that you may end up with more elements that you want, if they have a differentclass name that shares the string someclass.

As it turns out, Scrapy selectors allow you to chain selectors, so most of the timeyou can just select by class using CSS and then switch to XPath when needed:

  1. >>> from scrapy import Selector
  2. >>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>')
  3. >>> sel.css('.shout').xpath('./time/@datetime').getall()
  4. ['2014-07-23 19:00']

This is cleaner than using the verbose XPath trick shown above. Just rememberto use the . in the XPath expressions that will follow.

Beware of the difference between //node[1] and (//node)[1]

//node[1] selects all the nodes occurring first under their respective parents.

(//node)[1] selects all the nodes in the document, and then gets only the first of them.

Example:

  1. >>> from scrapy import Selector
  2. >>> sel = Selector(text="""
  3. ....:     <ul class="list">
  4. ....:         <li>1</li>
  5. ....:         <li>2</li>
  6. ....:         <li>3</li>
  7. ....:     </ul>
  8. ....:     <ul class="list">
  9. ....:         <li>4</li>
  10. ....:         <li>5</li>
  11. ....:         <li>6</li>
  12. ....:     </ul>""")
  13. >>> xp = lambda x: sel.xpath(x).getall()

This gets all first <li> elements under whatever it is its parent:

  1. >>> xp("//li[1]")
  2. ['<li>1</li>', '<li>4</li>']

And this gets the first <li> element in the whole document:

  1. >>> xp("(//li)[1]")
  2. ['<li>1</li>']

This gets all first <li> elements under an <ul> parent:

  1. >>> xp("//ul/li[1]")
  2. ['<li>1</li>', '<li>4</li>']

And this gets the first <li> element under an <ul> parent in the whole document:

  1. >>> xp("(//ul/li)[1]")
  2. ['<li>1</li>']

Using text nodes in a condition

When you need to use the text content as argument to an XPath string function,avoid using .//text() and use just . instead.

This is because the expression .//text() yields a collection of text elements – a node-set.And when a node-set is converted to a string, which happens when it is passed as argument toa string function like contains() or starts-with(), it results in the text for the first element only.

Example:

  1. >>> from scrapy import Selector
  2. >>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')

Converting a node-set to string:

  1. >>> sel.xpath('//a//text()').getall() # take a peek at the node-set
  2. ['Click here to go to the ', 'Next Page']
  3. >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
  4. ['Click here to go to the ']

A node converted to a string, however, puts together the text of itself plus of all its descendants:

  1. >>> sel.xpath("//a[1]").getall() # select the first node
  2. ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
  3. >>> sel.xpath("string(//a[1])").getall() # convert it to string
  4. ['Click here to go to the Next Page']

So, using the .//text() node-set won’t select anything in this case:

  1. >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
  2. []

But using the . to mean the node, works:

  1. >>> sel.xpath("//a[contains(., 'Next Page')]").getall()
  2. ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']

Variables in XPath expressions

XPath allows you to reference variables in your XPath expressions, usingthe $somevariable syntax. This is somewhat similar to parameterizedqueries or prepared statements in the SQL world where you replacesome arguments in your queries with placeholders like ?,which are then substituted with values passed with the query.

Here’s an example to match an element based on its “id” attribute value,without hard-coding it (that was shown previously):

  1. >>> # `$val` used in the expression, a `val` argument needs to be passed
  2. >>> response.xpath('//div[@id=$val]/a/text()', val='images').get()
  3. 'Name: My image 1 '

Here’s another example, to find the “id” attribute of a <div> tag containingfive <a> children (here we pass the value 5 as an integer):

  1. >>> response.xpath('//div[count(a)=$cnt]/@id', cnt=5).get()
  2. 'images'

All variable references must have a binding value when calling .xpath()(otherwise you’ll get a ValueError: XPath error: exception).This is done by passing as many named arguments as necessary.

parsel, the library powering Scrapy selectors, has more details and exampleson XPath variables.

Removing namespaces

When dealing with scraping projects, it is often quite convenient to get rid ofnamespaces altogether and just work with element names, to write moresimple/convenient XPaths. You can use theSelector.remove_namespaces() method for that.

Let’s show an example that illustrates this with the Python Insider blog atom feed.

First, we open the shell with the url we want to scrape:

  1. $ scrapy shell https://feeds.feedburner.com/PythonInsider

This is how the file starts:

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet ...
  3. <feed xmlns="http://www.w3.org/2005/Atom"
  4.       xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/"
  5.       xmlns:blogger="http://schemas.google.com/blogger/2008"
  6.       xmlns:georss="http://www.georss.org/georss"
  7.       xmlns:gd="http://schemas.google.com/g/2005"
  8.       xmlns:thr="http://purl.org/syndication/thread/1.0"
  9.       xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0">
  10.   ...

You can see several namespace declarations including a default“http://www.w3.org/2005/Atom” and another one using the “gd:” prefix for“http://schemas.google.com/g/2005”.

Once in the shell we can try selecting all <link> objects and see that itdoesn’t work (because the Atom XML namespace is obfuscating those nodes):

  1. >>> response.xpath("//link")
  2. []

But once we call the Selector.remove_namespaces() method, allnodes can be accessed directly by their names:

  1. >>> response.selector.remove_namespaces()
  2. >>> response.xpath("//link")
  3. [<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
  4.     <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
  5.     ...

If you wonder why the namespace removal procedure isn’t always called by defaultinstead of having to call it manually, this is because of two reasons, which, in orderof relevance, are:

  • Removing namespaces requires to iterate and modify all nodes in thedocument, which is a reasonably expensive operation to perform by defaultfor all documents crawled by Scrapy
  • There could be some cases where using namespaces is actually required, incase some element names clash between namespaces. These cases are very rarethough.

Using EXSLT extensions

Being built atop lxml, Scrapy selectors support some EXSLT extensionsand come with these pre-registered namespaces to use in XPath expressions:

prefixnamespaceusage
rehttp://exslt.org/regular-expressionsregular expressions
sethttp://exslt.org/setsset manipulation

Regular expressions

The test() function, for example, can prove quite useful when XPath’sstarts-with() or contains() are not sufficient.

Example selecting links in list item with a “class” attribute ending with a digit:

  1. >>> from scrapy import Selector
  2. >>> doc = u"""
  3. ... <div>
  4. ...     <ul>
  5. ...         <li class="item-0"><a href="link1.html">first item</a></li>
  6. ...         <li class="item-1"><a href="link2.html">second item</a></li>
  7. ...         <li class="item-inactive"><a href="link3.html">third item</a></li>
  8. ...         <li class="item-1"><a href="link4.html">fourth item</a></li>
  9. ...         <li class="item-0"><a href="link5.html">fifth item</a></li>
  10. ...     </ul>
  11. ... </div>
  12. ... """
  13. >>> sel = Selector(text=doc, type="html")
  14. >>> sel.xpath('//li//@href').getall()
  15. ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']
  16. >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()
  17. ['link1.html', 'link2.html', 'link4.html', 'link5.html']

Warning

C library libxslt doesn’t natively support EXSLT regularexpressions so lxml’s implementation uses hooks to Python’s re module.Thus, using regexp functions in your XPath expressions may add a smallperformance penalty.

Set operations

These can be handy for excluding parts of a document tree beforeextracting text elements for example.

Example extracting microdata (sample content taken from https://schema.org/Product)with groups of itemscopes and corresponding itemprops:

  1. >>> doc = u"""
  2. ... <div itemscope itemtype="http://schema.org/Product">
  3. ...   <span itemprop="name">Kenmore White 17" Microwave</span>
  4. ...   <img src="kenmore-microwave-17in.jpg" alt='Kenmore 17" Microwave' />
  5. ...   <div itemprop="aggregateRating"
  6. ...     itemscope itemtype="http://schema.org/AggregateRating">
  7. ...    Rated <span itemprop="ratingValue">3.5</span>/5
  8. ...    based on <span itemprop="reviewCount">11</span> customer reviews
  9. ...   </div>
  10. ...
  11. ...   <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">
  12. ...     <span itemprop="price">$55.00</span>
  13. ...     <link itemprop="availability" href="http://schema.org/InStock" />In stock
  14. ...   </div>
  15. ...
  16. ...   Product description:
  17. ...   <span itemprop="description">0.7 cubic feet countertop microwave.
  18. ...   Has six preset cooking categories and convenience features like
  19. ...   Add-A-Minute and Child Lock.</span>
  20. ...
  21. ...   Customer reviews:
  22. ...
  23. ...   <div itemprop="review" itemscope itemtype="http://schema.org/Review">
  24. ...     <span itemprop="name">Not a happy camper</span> -
  25. ...     by <span itemprop="author">Ellie</span>,
  26. ...     <meta itemprop="datePublished" content="2011-04-01">April 1, 2011
  27. ...     <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating">
  28. ...       <meta itemprop="worstRating" content = "1">
  29. ...       <span itemprop="ratingValue">1</span>/
  30. ...       <span itemprop="bestRating">5</span>stars
  31. ...     </div>
  32. ...     <span itemprop="description">The lamp burned out and now I have to replace
  33. ...     it. </span>
  34. ...   </div>
  35. ...
  36. ...   <div itemprop="review" itemscope itemtype="http://schema.org/Review">
  37. ...     <span itemprop="name">Value purchase</span> -
  38. ...     by <span itemprop="author">Lucas</span>,
  39. ...     <meta itemprop="datePublished" content="2011-03-25">March 25, 2011
  40. ...     <div itemprop="reviewRating" itemscope itemtype="http://schema.org/Rating">
  41. ...       <meta itemprop="worstRating" content = "1"/>
  42. ...       <span itemprop="ratingValue">4</span>/
  43. ...       <span itemprop="bestRating">5</span>stars
  44. ...     </div>
  45. ...     <span itemprop="description">Great microwave for the price. It is small and
  46. ...     fits in my apartment.</span>
  47. ...   </div>
  48. ...   ...
  49. ... </div>
  50. ... """
  51. >>> sel = Selector(text=doc, type="html")
  52. >>> for scope in sel.xpath('//div[@itemscope]'):
  53. ...     print("current scope:", scope.xpath('@itemtype').getall())
  54. ...     props = scope.xpath('''
  55. ...                 set:difference(./descendant::*/@itemprop,
  56. ...                                .//*[@itemscope]/*/@itemprop)''')
  57. ...     print("    properties: %s" % (props.getall()))
  58. ...     print("")
  59.  
  60. current scope: ['http://schema.org/Product']
  61.     properties: ['name', 'aggregateRating', 'offers', 'description', 'review', 'review']
  62.  
  63. current scope: ['http://schema.org/AggregateRating']
  64.     properties: ['ratingValue', 'reviewCount']
  65.  
  66. current scope: ['http://schema.org/Offer']
  67.     properties: ['price', 'availability']
  68.  
  69. current scope: ['http://schema.org/Review']
  70.     properties: ['name', 'author', 'datePublished', 'reviewRating', 'description']
  71.  
  72. current scope: ['http://schema.org/Rating']
  73.     properties: ['worstRating', 'ratingValue', 'bestRating']
  74.  
  75. current scope: ['http://schema.org/Review']
  76.     properties: ['name', 'author', 'datePublished', 'reviewRating', 'description']
  77.  
  78. current scope: ['http://schema.org/Rating']
  79.     properties: ['worstRating', 'ratingValue', 'bestRating']

Here we first iterate over itemscope elements, and for each one,we look for all itemprops elements and exclude those that are themselvesinside another itemscope.

Other XPath extensions

Scrapy selectors also provide a sorely missed XPath extension functionhas-class that returns True for nodes that have all of the specifiedHTML classes.

For the following HTML:

<p class="foo bar-baz">First</p>
<p class="foo">Second</p>
<p class="bar">Third</p>
<p>Fourth</p>

You can use it like this:

>>> response.xpath('//p[has-class("foo")]')
[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
 <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
>>> response.xpath('//p[has-class("foo", "bar-baz")]')
[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
>>> response.xpath('//p[has-class("foo", "bar")]')
[]

So XPath //p[has-class("foo", "bar-baz")] is roughly equivalent to CSSp.foo.bar-baz. Please note, that it is slower in most of the cases,because it’s a pure-Python function that’s invoked for every node in questionwhereas the CSS lookup is translated into XPath and thus runs more efficiently,so performance-wise its uses are limited to situations that are not easilydescribed with CSS selectors.

Parsel also simplifies adding your own XPath extensions.

  • parsel.xpathfuncs.setxpathfunc(_fname, func)[source]
  • Register a custom extension function to use in XPath expressions.

The function func registered under fname identifier will be calledfor every matching node, being passed a context parameter as well asany parameters passed from the corresponding XPath expression.

If func is None, the extension function will be removed.

See more in lxml documentation.

Built-in Selectors reference

Selector objects

  • class scrapy.selector.Selector(response=None, text=None, type=None, root=None, **kwargs)[source]
  • An instance of Selector is a wrapper over response to selectcertain parts of its content.

response is an HtmlResponse or anXmlResponse object that will be used for selectingand extracting data.

text is a unicode string or utf-8 encoded text for cases when aresponse isn’t available. Using text and response together isundefined behavior.

type defines the selector type, it can be "html", "xml"or None (default).

If type is None, the selector automatically chooses the best typebased on response type (see below), or defaults to "html" in case itis used together with text.

If type is None and a response is passed, the selector type isinferred from the response type as follows:

  • "html" for HtmlResponse type
  • "xml" for XmlResponse type

  • "html" for anything elseOtherwise, if type is set, the selector type will be forced and nodetection will occur.

  • xpath(query, namespaces=None, **kwargs)[source]

  • Find nodes matching the xpath query and return the result as aSelectorList instance with all elements flattened. Listelements implement Selector interface too.

query is a string containing the XPATH query to apply.

namespaces is an optional prefix: namespace-uri mapping (dict)for additional prefixes to those registered with register_namespace(prefix, uri).Contrary to register_namespace(), these prefixes are notsaved for future calls.

Any additional named arguments can be used to pass values for XPathvariables in the XPath expression, e.g.:

selector.xpath('//a[href=$url]', url="http://www.example.com")

Note

For convenience, this method can be called as response.xpath()

query is a string containing the CSS selector to apply.

In the background, CSS queries are translated into XPath queries usingcssselect library and run .xpath() method.

Note

For convenience, this method can be called as response.css()

  • get()[source]
  • Serialize and return the matched nodes in a single unicode string.Percent encoded content is unquoted.

See also: extract() and extract_first()

  • attrib
  • Return the attributes dictionary for underlying element.

See also: Selecting element attributes.

  • re(regex, replace_entities=True)[source]
  • Apply the given regex and return a list of unicode strings with thematches.

regex can be either a compiled regular expression or a string whichwill be compiled to a regular expression using re.compile(regex).

By default, character entity references are replaced by theircorresponding character (except for &amp; and &lt;).Passing replace_entities as False switches off thesereplacements.

  • refirst(_regex, default=None, replace_entities=True)[source]
  • Apply the given regex and return the first unicode string whichmatches. If there is no match, return the default value (None ifthe argument is not provided).

By default, character entity references are replaced by theircorresponding character (except for &amp; and &lt;).Passing replace_entities as False switches off thesereplacements.

  • registernamespace(_prefix, uri)[source]

  • Register the given namespace to be used in this Selector.Without registering namespaces you can’t select or extract data fromnon-standard namespaces. See Selector examples on XML response.

  • remove_namespaces()[source]

  • Remove all namespaces, allowing to traverse the document usingnamespace-less xpaths. See Removing namespaces.

  • bool()[source]

  • Return True if there is any real content selected or Falseotherwise. In other words, the boolean value of a Selector isgiven by the contents it selects.

  • getall()[source]

  • Serialize and return the matched node in a 1-element list of unicode strings.

This method is added to Selector for consistency; it is more usefulwith SelectorList. See also: extract() and extract_first()

SelectorList objects

  • class scrapy.selector.SelectorList[source]

  • The SelectorList class is a subclass of the builtin listclass, which provides a few additional methods.

    • xpath(xpath, namespaces=None, **kwargs)[source]
    • Call the .xpath() method for each element in this list and returntheir results flattened as another SelectorList.

query is the same argument as the one in Selector.xpath()

namespaces is an optional prefix: namespace-uri mapping (dict)for additional prefixes to those registered with register_namespace(prefix, uri).Contrary to register_namespace(), these prefixes are notsaved for future calls.

Any additional named arguments can be used to pass values for XPathvariables in the XPath expression, e.g.:

selector.xpath('//a[href=$url]', url="http://www.example.com")
  • css(query)[source]
  • Call the .css() method for each element in this list and returntheir results flattened as another SelectorList.

query is the same argument as the one in Selector.css()

  • getall()[source]
  • Call the .get() method for each element is this list and returntheir results flattened, as a list of unicode strings.

See also: extract() and extract_first()

  • get(default=None)[source]
  • Return the result of .get() for the first element in this list.If the list is empty, return the default value.

See also: extract() and extract_first()

  • re(regex, replace_entities=True)[source]
  • Call the .re() method for each element in this list and returntheir results flattened, as a list of unicode strings.

By default, character entity references are replaced by theircorresponding character (except for &amp; and &lt;.Passing replace_entities as False switches off thesereplacements.

  • refirst(_regex, default=None, replace_entities=True)[source]
  • Call the .re() method for the first element in this list andreturn the result in an unicode string. If the list is empty or theregex doesn’t match anything, return the default value (None ifthe argument is not provided).

By default, character entity references are replaced by theircorresponding character (except for &amp; and &lt;.Passing replace_entities as False switches off thesereplacements.

  • attrib
  • Return the attributes dictionary for the first element.If the list is empty, return an empty dict.

See also: Selecting element attributes.

Examples

Selector examples on HTML response

Here are some Selector examples to illustrate several concepts.In all cases, we assume there is already a Selector instantiated witha HtmlResponse object like this:

sel = Selector(html_response)
  • Select all <h1> elements from an HTML response body, returning a list ofSelector objects (i.e. a SelectorList object):
sel.xpath("//h1")
  • Extract the text of all <h1> elements from an HTML response body,returning a list of unicode strings:
sel.xpath("//h1").getall()         # this includes the h1 tag
sel.xpath("//h1/text()").getall()  # this excludes the h1 tag
  • Iterate over all <p> tags and print their class attribute:
for node in sel.xpath("//p"):
    print(node.attrib['class'])

Selector examples on XML response

Here are some examples to illustrate concepts for Selector objectsinstantiated with an XmlResponse object:

sel = Selector(xml_response)
  • Select all <product> elements from an XML response body, returning a listof Selector objects (i.e. a SelectorList object):
sel.xpath("//product")

sel.register_namespace("g", "http://base.google.com/ns/1.0")
sel.xpath("//g:price").getall()