Installation guide

Installing Scrapy

Scrapy runs on Python 3.5 or above under CPython (default Pythonimplementation) and PyPy (starting with PyPy 5.9).

If you’re using Anaconda or Miniconda, you can install the package fromthe conda-forge channel, which has up-to-date packages for Linux, Windowsand macOS.

To install Scrapy using conda, run:

  1. conda install -c conda-forge scrapy

Alternatively, if you’re already familiar with installation of Python packages,you can install Scrapy and its dependencies from PyPI with:

  1. pip install Scrapy

Note that sometimes this may require solving compilation issues for some Scrapydependencies depending on your operating system, so be sure to check thePlatform specific installation notes.

We strongly recommend that you install Scrapy in a dedicated virtualenv,to avoid conflicting with your system packages.

For more detailed and platform specifics instructions, as well astroubleshooting information, read on.

Things that are good to know

Scrapy is written in pure Python and depends on a few key Python packages (among others):

  • lxml, an efficient XML and HTML parser
  • parsel, an HTML/XML data extraction library written on top of lxml,
  • w3lib, a multi-purpose helper for dealing with URLs and web page encodings
  • twisted, an asynchronous networking framework
  • cryptography and pyOpenSSL, to deal with various network-level security needs

The minimal versions which Scrapy is tested against are:

  • Twisted 14.0
  • lxml 3.4
  • pyOpenSSL 0.14

Scrapy may work with older versions of these packagesbut it is not guaranteed it will continue workingbecause it’s not being tested against them.

Some of these packages themselves depends on non-Python packagesthat might require additional installation steps depending on your platform.Please check platform-specific guides below.

In case of any trouble related to these dependencies,please refer to their respective installation instructions:

Using a virtual environment (recommended)

TL;DR: We recommend installing Scrapy inside a virtual environmenton all platforms.

Python packages can be installed either globally (a.k.a system wide),or in user-space. We do not recommend installing Scrapy system wide.

Instead, we recommend that you install Scrapy within a so-called“virtual environment” (venv).Virtual environments allow you to not conflict with already-installed Pythonsystem packages (which could break some of your system tools and scripts),and still install packages normally with pip (without sudo and the likes).

See Virtual Environments and Packages on how to create your virtual environment.

Once you have created a virtual environment, you can install Scrapy inside it with pip,just like any other Python package.(See platform-specific guidesbelow for non-Python dependencies that you may need to install beforehand).

Platform specific installation notes

Windows

Though it’s possible to install Scrapy on Windows using pip, we recommend youto install Anaconda or Miniconda and use the package from theconda-forge channel, which will avoid most installation issues.

Once you’ve installed Anaconda or Miniconda, install Scrapy with:

  1. conda install -c conda-forge scrapy

Ubuntu 14.04 or above

Scrapy is currently tested with recent-enough versions of lxml,twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions.But it should support older versions of Ubuntu too, like Ubuntu 14.04,albeit with potential issues with TLS connections.

Don’t use the python-scrapy package provided by Ubuntu, they aretypically too old and slow to catch up with latest Scrapy.

To install Scrapy on Ubuntu (or Ubuntu-based) systems, you need to installthese dependencies:

  1. sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
  • python3-dev, zlib1g-dev, libxml2-dev and libxslt1-devare required for lxml
  • libssl-dev and libffi-dev are required for cryptography

Inside a virtualenv,you can install Scrapy with pip after that:

  1. pip install scrapy

Note

The same non-Python dependencies can be used to install Scrapy in DebianJessie (8.0) and above.

macOS

Building Scrapy’s dependencies requires the presence of a C compiler anddevelopment headers. On macOS this is typically provided by Apple’s Xcodedevelopment tools. To install the Xcode command line tools open a terminalwindow and run:

  1. xcode-select --install

There’s a known issue thatprevents pip from updating system packages. This has to be addressed tosuccessfully install Scrapy and its dependencies. Here are some proposedsolutions:

  • (Recommended)Don’t use system python, install a new, updated versionthat doesn’t conflict with the rest of your system. Here’s how to do it usingthe homebrew package manager:

    • Install homebrew following the instructions in https://brew.sh/

    • Update your PATH variable to state that homebrew packages should beused before system packages (Change .bashrc to .zshrc accordantlyif you’re using zsh as default shell):

  1. echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
  • Reload .bashrc to ensure the changes have taken place:
  1. source ~/.bashrc
  • Install python:
  1. brew install python
  • Latest versions of python have pip bundled with them so you won’t needto install it separately. If this is not the case, upgrade python:
  1. brew update; brew upgrade python
This method is a workaround for the above macOS issue, but it’s an overallgood practice for managing dependencies and can complement the first method.

After any of these workarounds you should be able to install Scrapy:

  1. pip install Scrapy

PyPy

We recommend using the latest PyPy version. The version tested is 5.9.0.For PyPy3, only Linux installation was tested.

Most Scrapy dependencides now have binary wheels for CPython, but not for PyPy.This means that these dependecies will be built during installation.On macOS, you are likely to face an issue with building Cryptography dependency,solution to this problem is describedhere,that is to brew install openssl and then export the flags that this commandrecommends (only needed when installing Scrapy). Installing on Linux has no specialissues besides installing build dependencies.Installing Scrapy with PyPy on Windows is not tested.

You can check that Scrapy is installed correctly by running scrapy bench.If this command gives errors such asTypeError: … got 2 unexpected keyword arguments, this meansthat setuptools was unable to pick up one PyPy-specific dependency.To fix this issue, run pip install 'PyPyDispatcher>=2.1.0'.

Troubleshooting

AttributeError: ‘module’ object has no attribute ‘OP_NO_TLSv1_1’

After you install or upgrade Scrapy, Twisted or pyOpenSSL, you may get anexception with the following traceback:

  1. […]
  2.   File "[…]/site-packages/twisted/protocols/tls.py", line 63, in <module>
  3.     from twisted.internet._sslverify import _setAcceptableProtocols
  4.   File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in <module>
  5.     TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
  6. AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'

The reason you get this exception is that your system or virtual environmenthas a version of pyOpenSSL that your version of Twisted does not support.

To install a version of pyOpenSSL that your version of Twisted supports,reinstall Twisted with the tls extra option:

  1. pip install twisted[tls]

For details, see Issue #2473.