Translating on Windows

RPython is supported on Windows platforms, starting with Windows 2000.The following text gives some hints about how to translate a interpreterwritten in RPython, using PyPy as an example.

PyPy supports only being translated as a 32bit program, even on64bit Windows. See at the end of this page for what is missingfor a full 64bit translation.

To build pypy-c you need a working python environment, and a C compiler.It is possible to translate with a CPython 2.6 or later, but this is notthe preferred way, because it will take a lot longer to run – dependingon your architecture, between two and three times as long. So head toour downloads and get the latest stable version.

Microsoft Visual Studio is preferred as a compiler, but there are reportsof success with the mingw32 port of gcc.

What Compiler to use and How to find it?

The first stumbling block when building something for Python on windows ishow to discover the path to the compiler, headers, and libraries. One caninstall many versions of the MSVC compiler tools, from stand-alone buildtools to full blown Visual Studio IDE installations. Each of these use casesput the compiler at different locations, and the layout changes from time totime.

The distutils package, located in the stdlib, is the natural place to putthis discovery code, but it is frozen by the python version. The pip-installable setuptools can move faster to adapt to new tools. So the firstthing that will happen after building PyPy is it will install pip and downloadsetuptools, then it will build the cffi modules used in stdlib.PyPy has a chicken and egg problem: in order to compile something we needsetuptools, but in order to get setuptools we need pip which requires_ssl, and _ssl must be compiled. So PyPy vendors in a copy ofmsvc.py in rpython/tools/setuptools_msvc.py.

PyPy will prefer to compile with the latest MSVC compiler it can find, which isa departure from CPython’s desire to compile with Visual Studio 9.

Translating PyPy with Visual Studio

We routinely test translation of PyPy using Visual Studio 2019, MSVC160.Other configurations may work as well.

The translation scripts will set up the appropriate environment variablesfor the compiler, so you do not need to run vcvars before translation.They will pick the most recent Visual Studiocompiler they can find. In addition, the target architecture(32 bits, 64 bits) is automatically selected. A 32 bit build can only be builtusing a 32 bit Python and vice versa. By default the interpreter is built usingthe Multi-threaded DLL (/MD) runtime environment.

If you wish to override this detection method to use a different compiler(mingw or a different version of MSVC):

  • set up the PATH and other environment variables as needed
  • set the CC environment variable to compiler exe to be used,for a different version of MSVC SET CC=cl.exe.

Note: The RPython translator does currently not support 64 bit Python, andtranslation will fail in this case.

Python and a C compiler are all you need to build pypy, but it will miss somemodules that relies on third-party libraries. See below how to getand build them.

Please see the non-windows instructions for more information, especially notethat translation is RAM-hungry. A standard translation requires around 4GB, sospecial preparations are necessary, or you may want to use the method in thenotes of the build instructions to reduce memory usage at the price of aslower translation:

  1. set PYPY_GC_MAX_DELTA=200MB
  2. pypy --jit loop_longevity=300 ../../rpython/bin/rpython -Ojit targetpypystandalone
  3. set PYPY_GC_MAX_DELTA=
  4. # This is done as part of translation
  5. PYTHONPATH=../.. ./pypy-c ../../lib_pypy/tools/build_cffi_imports.py

Preparing Windows for the large build

Normally 32bit programs are limited to 2GB of memory on Windows. It ispossible to raise this limit, to 3GB on Windows 32bit, and almost 4GBon Windows 64bit.

On Windows 32bit, it is necessary to modify the system: followhttp://usa.autodesk.com/adsk/servlet/ps/dl/item?siteID=123112&id=9583842&linkID=9240617to enable the “3GB” feature, and reboot. This step is not necessary onWindows 64bit.

Then you need to execute:

  1. <path-to-visual>\vc\vcvars.bat
  2. editbin /largeaddressaware translator.exe

where translator.exe is the pypy.exe or cpython.exe you will use totranslate with. This is done by default during PyPy translation, so it shouldJust Work.

Installing external packages

We uses a subrepository inside pypy to hold binary compiled versions of thebuild dependencies for windows. As part of the rpython setup stage, environmentvariables will be set to use these dependencies. The repository has a READMEfile on how to replicate, and a branch for each supported platform. You may runthe get_externals.py utility to checkout the proper branch for your platformand PyPy version.

Using the mingw compiler

You can compile an RPython program with the mingw compiler, using the–cc=mingw32 option; gcc.exe must be on the PATH. If the -cc flag does notbegin with “ming”, it should be the name of a valid gcc-derivative compiler,i.e. x86_64-w64-mingw32-gcc for the 64 bit compiler creating a 64 bit target.

You probably want to set the CPATH, LIBRARY_PATH, and PATH environmentvariables to the header files, lib or dlls, and dlls respectively of thelocally installed packages if they are not in the mingw directory heirarchy.

libffi for the mingw compiler

To enable the _rawffi (and ctypes) module, you need to compile a mingwversion of libffi. Here is one way to do this, wich should allow you to tryto build for win64 or win32:

  • Download and unzip a mingw32 build or mingw64 build, say into c:mingw

  • If you do not use cygwin, you will need msys to provide make,autoconf tools and other goodies.

  1. Download and unzip a msys for mingw, say into c:msys
  2. Edit the c:msysetcfstab file to mount c:mingw

  • Download and unzip the libffi source files, and extractthem in the base directory.

  • Run c:msysmsys.bat or a cygwin shell which should make youfeel better since it is a shell prompt with shell tools.

  • From inside the shell, cd to the libffi directory and do:

  1. sh ./configure
  2. make
  3. cp .libs/libffi-5.dll <somewhere on the PATH>

If you can’t find the dll, and the libtool issued a warning about“undefined symbols not allowed”, you will need to edit the libffiMakefile in the toplevel directory. Add the flag -no-undefined tothe definition of libffi_la_LDFLAGS

If you wish to experiment with win64, you must run configure with flags:

  1. sh ./configure --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32

or such, depending on your mingw64 download.

hacking on PyPy with the mingw compiler

Since hacking on PyPy means running tests, you will need a way to specifythe mingw compiler when hacking (as opposed to translating). As ofMarch 2012, –cc is not a valid option for pytest.py. However if you set anenvironment variable CC to the compiler exe, testing will use it.

What is missing for a full 64-bit translation

The main blocker is that we assume that the integer type of RPython islarge enough to (occasionally) contain a pointer value cast to aninteger. The simplest fix is to make sure that it is so, but it willgive the following incompatibility between CPython and PyPy on Win64:

CPython: sys.maxint == 231-1, sys.maxsize == 263-1

PyPy: sys.maxint == sys.maxsize == 2**63-1

…and, correspondingly, PyPy supports ints up to the larger value ofsys.maxint before they are converted to long. The first decisionthat someone needs to make is if this incompatibility is reasonable.

Assuming that it is, the first thing to do is probably to hack _CPython_until it fits this model: replace the field in PyIntObject with a long long field, and change the value of sys.maxint. This might justwork, even if half-brokenly: I’m sure you can crash it because of theprecision loss that undoubtedly occurs everywhere, but try not to. :-)

Such a hacked CPython is what you’ll use in the next steps. We’ll callit CPython64/64.

It is probably not too much work if the goal is only to get a translatedPyPy executable, and to run all tests before translation. But you needto start somewhere, and you should start with some tests inrpython/translator/c/test/, like test_standalone.py andtest_newgc.py: try to have them pass on top of CPython64/64.

Keep in mind that this runs small translations, and some details may gowrong. The most obvious one is to check that it produces C files thatuse the integer type Signed — but what is Signed defined to?It should be equal to long on every other platform, but on Win64 itshould be something like long long.

What is more generally needed is to review all the C files inrpython/translator/c/src for the word long, because this means a32-bit integer even on Win64. Replace it with Signed most of thetimes. You can replace one with the other without breaking anything onany other platform, so feel free to.

Then, these two C types have corresponding RPython types: rffi.LONGand lltype.Signed respectively. The first should really correspondto the C long. Add tests that check that integers cast to onetype or the other really have 32 and 64 bits respectively, on Win64.

Once these basic tests work, you need to review rpython/rlib/ foruses of rffi.LONG versus lltype.Signed. The goal would be tofix some more LONG-versus-Signed issues, by fixing the tests — asalways run on top of CPython64/64. Note that there was some early workdone in rpython/rlib/rarithmetic with the goal of running all thetests on Win64 on the regular CPython, but I think by now that it’s abad idea. Look only at CPython64/64.

The major intermediate goal is to get a translation of PyPy with -O2with a minimal set of modules, starting with —no-allworkingmodules;you need to use CPython64/64 to run this translation too. Checkcarefully the warnings of the C compiler at the end. By default, MSVCreports a lot of mismatches of integer sizes as warnings instead oferrors.

Then you need to review pypy/module/*/ for LONG-versus-Signedissues. At some time during this review, we get a working translatedPyPy on Windows 64 that includes all —translationmodules, i.e.everything needed to run translations. Once we have that, the hackedCPython64/64 becomes much less important, because we can run futuretranslations on top of this translated PyPy. As soon as we get there,please distribute the translated PyPy. It’s an essential componentfor anyone else that wants to work on Win64! We end up with a strangekind of dependency — we need a translated PyPy in order to translate aPyPy —, but I believe it’s ok here, as Windows executables aresupposed to never be broken by newer versions of Windows.

Happy hacking :-)