Building PyPy from Source

For building PyPy, we recommend installing a pre-built PyPy first (seeDownloading and Installing PyPy). It is possible to build PyPy with CPython, but it will take alot longer to run – depending on your architecture, between two and threetimes as long.

Even when using PyPy to build PyPy, translation is time-consuming – 30minutes on a fast machine – and RAM-hungry. You will need at least 2 GBof memory on a 32-bit machine and 4GB on a 64-bit machine.

Before you start

Our normal development workflow avoids a full translation by using test-drivendevelopment. You can read more about how to develop PyPy here, and latesttranslated (hopefully functional) binary packages are available on ourbuildbot’s nightly builds

You will need the build dependencies below to run the tests.

Clone the repository

If you prefer to compile your own PyPy, or if you want to modify it, youwill need to obtain a copy of the sources. This can be done either bydownloading them from the download page or by checking them out from therepository using mercurial. We suggest using mercurial if you want to accessthe current development.

You must issue the following command on yourcommand line, DOS box, or terminal:

  1. hg clone http://foss.heptapod.net/pypy/pypy pypy

This will clone the repository and place it into a directorynamed pypy, and will get you the PyPy source in pypy/pypy anddocumentation files in pypy/pypy/doc.We try to ensure that the tip is always stable, but it mightoccasionally be broken. You may want to check out our nightly tests:find a revision (12-chars alphanumeric string, e.g. “963e808156b3”)that passed at least the{linux32} tests (corresponding to a + sign on theline success) and then, in your cloned repository, switch to this revisionusing:

  1. hg up -r XXXXX

where XXXXX is the revision id.

Install build-time dependencies

(Note: for some hints on how to translate the Python interpreter underWindows, see the windows document .

The host Python needs to have CFFI installed. If translating on PyPy, CFFI isalready installed. If translating on CPython, you need to install it, e.g.using python -mpip install cffi.

To build PyPy on Unix using the C translation backend, you need at least a Ccompiler and make installed. Further, some optional modules have additionaldependencies:

  • cffi, ctypes
  • libffi, pkg-config
  • zlib
  • libz
  • bz2
  • libbz2
  • pyexpat
  • libexpat1
  • _vmprof
  • libunwind (optional, loaded dynamically at runtime)

Make sure to have these libraries (with development headers) installedbefore building PyPy, otherwise the resulting binary will not containthese modules. Furthermore, the following libraries should be presentafter building PyPy, otherwise the corresponding CFFI modules are notbuilt (you can run or re-run pypy/tool/release/package.py to retryto build them; you don’t need to re-translate the whole PyPy):

  • sqlite3
  • libsqlite3
  • _ssl, _hashlib
  • libssl
  • curses
  • libncurses-dev (for PyPy2)libncursesw-dev (for PyPy3)
  • gdbm
  • libgdbm-dev
  • tk
  • tk-dev
  • lzma (PyPy3 only)
  • liblzma or libxz, version 5 and up

To run untranslated tests, you need the Boehm garbage collector libgc, version7.4 and up

On Debian and Ubuntu (16.04 onwards), this is the command to installall build-time dependencies:

  1. apt-get install gcc make libffi-dev pkg-config zlib1g-dev libbz2-dev \
  2. libsqlite3-dev libncurses5-dev libexpat1-dev libssl-dev libgdbm-dev \
  3. tk-dev libgc-dev python-cffi \
  4. liblzma-dev libncursesw5-dev     # these two only needed on PyPy3

On Fedora:

  1. dnf install gcc make libffi-devel pkgconfig zlib-devel bzip2-devel \
  2. sqlite-devel ncurses-devel expat-devel openssl-devel tk-devel \
  3. gdbm-devel python-cffi gc-devel\
  4. xz-devel  # For lzma on PyPy3.

On SLES11:

  1. zypper install gcc make python-devel pkg-config \
  2. zlib-devel libopenssl-devel libbz2-devel sqlite3-devel \
  3. libexpat-devel libffi-devel python-curses python-cffi \
  4. xz-devel # For lzma on PyPy3.
  5. (XXX plus the SLES11 version of libgdbm-dev and tk-dev)

On Mac OS X:

Most of these build-time dependencies are installed alongsidethe Developer Tools. However, note that in order for the installation tofind them you may need to run:

  1. xcode-select --install

An exception is OpenSSL, which is no longer provided with the operatingsystem. It can be obtained via Homebrew (with $ brew install openssl),but it will not be available on the system path by default. The easiestway to enable it for building pypy is to set an environment variable:

  1. export PKG_CONFIG_PATH=$(brew --prefix)/opt/openssl/lib/pkgconfig

After setting this, translation (described next) will find the OpenSSL libsas expected.

Run the translation

We usually translate in the pypy/goal directory, so all the followingcommands assume your $pwd is there.

Translate with JIT:

  1. pypy ../../rpython/bin/rpython --opt=jit

Translate without JIT:

  1. pypy ../../rpython/bin/rpython --opt=2

Note this translates pypy via the targetpypystandalone.py file, so theseare shorthand for:

  1. pypy ../../rpython/bin/rpython <rpython options> targetpypystandalone.py <pypy options>

More help is availabe via —help at either option position, and more infocan be found in the Configuration Options for PyPy section.

(You can use python instead of pypy here, which will take longerbut works too.)

If everything works correctly this will:

  • Run the rpython translation chain, producing a database of theentire pypy interpreter. This step is currently singe threaded, and RAMhungry. As part of this step, the chain creates a large number of C codefiles and a Makefile to compile them in adirectory controlled by the PYPY_USESSION_DIR environment variable.
  • Create an executable pypy-c by running the Makefile. This step canutilize all possible cores on the machine.
  • Copy the needed binaries to the current directory.
  • Generate c-extension modules for any cffi-based stdlib modules. The resulting executable behaves mostly like a normal Pythoninterpreter (see Differences between PyPy and CPython), and is ready for testing, foruse as a base interpreter for a new virtualenv, or for packaging into a binarysuitable for installation on another machine running the same OS as the buildmachine.

Note that step 4 is merely done as a convenience, any of the steps may be rerunwithout rerunning the previous steps.

Making a debug build of PyPy

Rerun the Makefile with the make lldebug or make lldebug0 target,which will build in a way that running under a debugger makes sense.Appropriate compilation flags are added to add debug info, and for lldebug0compiler optimizations are fully disabled. If you stop in a debugger, you willsee the very wordy machine-generated C code from the rpython translation step,which takes a little bit of reading to relate back to the rpython code.

Build cffi import libraries for the stdlib

Various stdlib modules require a separate build step to create the cffiimport libraries in the out-of-line API mode. This is done by the followingcommand:

  1. cd pypy/goal
  2. PYTHONPATH=../.. ./pypy-c ../../lib_pypy/tools/build_cffi_imports.py

Packaging (preparing for installation)

Packaging is required if you want to install PyPy system-wide, even toinstall on the same machine. The reason is that doing so prepares anumber of extra features that cannot be done lazily on a root-installedPyPy, because the normal users don’t have write access. This concernsmostly libraries that would normally be compiled if and when they areimported the first time.

  1. cd pypy/tool/release
  2. ./package.py --archive-name=pypy-VER-PLATFORM

This creates a clean and prepared hierarchy, as well as a .tar.bz2with the same content; both are found by default in/tmp/usession-YOURNAME/build/. You can then either move the filehierarchy or unpack the .tar.bz2 at the correct place.

It is recommended to use package.py because custom scripts willinvariably become out-of-date. If you want to write custom scriptsanyway, note an easy-to-miss point: some modules are written with CFFI,and require some compilation. If you install PyPy as root withoutpre-compiling them, normal users will get errors:

  • PyPy 2.5.1 or earlier: normal users would see permission errors.Installers need to run pypy -c "import gdbm" and other similarcommands at install time; the exact list is inpypy/tool/release/package.py. Usersseeing a broken installation of PyPy can fix it after-the-fact if theyhave sudo rights, by running once e.g. sudo pypy -c "import gdbm.
  • PyPy 2.6 and later: anyone would get ImportError: no module named _gdbm_cffi. Installers need to run pypy _gdbm_build.py in thelib_pypy directory during the installation process (plus others;see the exact list in pypy/tool/release/package.py).Users seeing a brokeninstallation of PyPy can fix it after-the-fact, by running pypy /path/to/lib_pypy/_gdbm_build.py. This command produces a filecalled _gdbm_cffi.pypy-41.so locally, which is a C extensionmodule for PyPy. You can move it at any place where modules arenormally found: e.g. in your project’s main directory, or in adirectory that you add to the env var PYTHONPATH.

Installation

PyPy dynamically finds the location of its libraries depending on the locationof the executable. The directory hierarchy of a typical PyPy installationlooks like this:

  1. ./bin/pypy
  2. ./include/
  3. ./lib_pypy/
  4. ./lib-python/2.7
  5. ./site-packages/

The hierarchy shown above is relative to a PREFIX directory. PREFIX iscomputed by starting from the directory where the executable resides, and“walking up” the filesystem until we find a directory containing lib_pypyand lib-python/2.7.

To install PyPy system wide on unix-like systems, it is recommended to put thewhole hierarchy alone (e.g. in /opt/pypy) and put a symlink to thepypy executable into /usr/bin or /usr/local/bin.

If the executable fails to find suitable libraries, it will report debug: WARNING: library path not found, using compiled-in sys.path and then attemptto continue normally. If the default path is usable, most code will be fine.However, the sys.prefix will be unset and some existing libraries assumethat this is never the case.