Pure Python Mode

In some cases, it’s desirable to speed up Python code without losing theability to run it with the Python interpreter. While pure Python scriptscan be compiled with Cython, it usually results only in a speed gain ofabout 20%-50%.

To go beyond that, Cython provides language constructs to add static typingand cythonic functionalities to a Python module to make it run much fasterwhen compiled, while still allowing it to be interpreted.This is accomplished via an augmenting .pxd file, via Pythontype annotations (followingPEP 484 andPEP 526), and/orvia special functions and decorators available after importing the magiccython module. All three ways can be combined at need, althoughprojects would commonly decide on a specific way to keep the static typeinformation easy to manage.

Although it is not typically recommended over writing straight Cython codein a .pyx file, there are legitimate reasons to do this - easiertesting and debugging, collaboration with pure Python developers, etc.In pure mode, you are more or less restricted to code that can be expressed(or at least emulated) in Python, plus static type declarations. Anythingbeyond that can only be done in .pyx files with extended language syntax,because it depends on features of the Cython compiler.

Augmenting .pxd

Using an augmenting .pxd allows to let the original .py filecompletely untouched. On the other hand, one needs to maintain both the.pxd and the .py to keep them in sync.

While declarations in a .pyx file must correspond exactly with thoseof a .pxd file with the same name (and any contradiction results ina compile time error, see pxd files), the untyped definitions in a.py file can be overridden and augmented with static types by the morespecific ones present in a .pxd.

If a .pxd file is found with the same name as the .py filebeing compiled, it will be searched for cdef classes andcdef/cpdef functions and methods. The compiler willthen convert the corresponding classes/functions/methods in the .pyfile to be of the declared type. Thus if one has a file A.py:

  1. def myfunction(x, y=2):
  2.     a = x - y
  3.     return a + x * y
  4.  
  5. def _helper(a):
  6.     return a + 1
  7.  
  8. class A:
  9.     def __init__(self, b=0):
  10.         self.a = 3
  11.         self.b = b
  12.  
  13.     def foo(self, x):
  14.         print(x + _helper(1.0))

and adds A.pxd:

  1. cpdef int myfunction(int x, int y=*)
  2. cdef double _helper(double a)
  3.  
  4. cdef class A:
  5.     cdef public int a, b
  6.     cpdef foo(self, double x)

then Cython will compile the A.py as if it had been written as follows:

  1. cpdef int myfunction(int x, int y=2):
  2.     a = x - y
  3.     return a + x * y
  4.  
  5. cdef double _helper(double a):
  6.     return a + 1
  7.  
  8. cdef class A:
  9.     cdef public int a, b
  10.     def __init__(self, b=0):
  11.         self.a = 3
  12.         self.b = b
  13.  
  14.     cpdef foo(self, double x):
  15.         print(x + _helper(1.0))

Notice how in order to provide the Python wrappers to the definitionsin the .pxd, that is, to be accessible from Python,

  • Python visible function signatures must be declared as cpdef (with defaultarguments replaced by a * to avoid repetition):
  1. cpdef int myfunction(int x, int y=*)
  • C function signatures of internal functions can be declared as cdef:
  1. cdef double _helper(double a)

  • cdef classes (extension types) are declared as cdef class;

  • cdef class attributes must be declared as cdef public if read/writePython access is needed, cdef readonly for read-only Python access, orplain cdef for internal C level attributes;

  • cdef class methods must be declared as cpdef for Python visiblemethods or cdef for internal C methods.

In the example above, the type of the local variable a in _myfunction()_is not fixed and will thus be a Python object. To statically type it, onecan use Cython’s @cython.locals decorator (see Magic Attributes,and Magic Attributes within the .pxd).

Normal Python (def) functions cannot be declared in .pxdfiles. It is therefore currently impossible to override the types of plainPython functions in .pxd files, e.g. to override types of their localvariables. In most cases, declaring them as cpdef will work as expected.

Magic Attributes

Special decorators are available from the magic cython module that canbe used to add static typing within the Python file, while being ignoredby the interpreter.

This option adds the cython module dependency to the original code, butdoes not require to maintain a supplementary .pxd file. Cythonprovides a fake version of this module as Cython.Shadow, which is availableas cython.py when Cython is installed, but can be copied to be used by othermodules when Cython is not installed.

“Compiled” switch

  • compiled is a special variable which is set to True when the compilerruns, and False in the interpreter. Thus, the code
  1. import cython
  2.  
  3. if cython.compiled:
  4.     print("Yep, I'm compiled.")
  5. else:
  6.     print("Just a lowly interpreted script.")

will behave differently depending on whether or not the code is executed as acompiled extension (.so/.pyd) module or a plain .pyfile.

Static typing

  • cython.declare declares a typed variable in the current scope, which can beused in place of the cdef type var [= value] construct. This has two forms,the first as an assignment (useful as it creates a declaration in interpretedmode as well):
  1. import cython
  2.  
  3. x = cython.declare(cython.int)              # cdef int x
  4. y = cython.declare(cython.double, 0.57721)  # cdef double y = 0.57721

and the second mode as a simple function call:

  1. import cython
  2.  
  3. cython.declare(x=cython.int, y=cython.double)  # cdef int x; cdef double y

It can also be used to define extension type private, readonly and public attributes:

  1. import cython
  2.  
  3.  
  4. @cython.cclass
  5. class A:
  6.     cython.declare(a=cython.int, b=cython.int)
  7.     c = cython.declare(cython.int, visibility='public')
  8.     d = cython.declare(cython.int)  # private by default.
  9.     e = cython.declare(cython.int, visibility='readonly')
  10.  
  11.     def __init__(self, a, b, c, d=5, e=3):
  12.         self.a = a
  13.         self.b = b
  14.         self.c = c
  15.         self.d = d
  16.         self.e = e
  • @cython.locals is a decorator that is used to specify the types of localvariables in the function body (including the arguments):
  1. import cython
  2.  
  3. @cython.locals(a=cython.long, b=cython.long, n=cython.longlong)
  4. def foo(a, b, x, y):
  5.     n = a * b
  6.     # ...

  • @cython.returns(<type>) specifies the function’s return type.

  • @cython.exceptval(value=None, *, check=False) specifies the function’s exceptionreturn value and exception check semantics as follows:

  1. @exceptval(-1)               # cdef int func() except -1:
    @exceptval(-1, check=False)  # cdef int func() except -1:
    @exceptval(check=True)       # cdef int func() except *:
    @exceptval(-1, check=True)   # cdef int func() except? -1:

  • Python annotations can be used to declare argument types, as shown in thefollowing example. To avoid conflicts with other kinds of annotationusages, this can be disabled with the directive annotation_typing=False.
  1. import cython
  2.  
  3. def func(foo: dict, bar: cython.int) -> tuple:
  4.     foo["hello world"] = 3 + bar
  5.     return foo, 5

This can be combined with the @cython.exceptval() decorator for non-Pythonreturn types:

  1. import cython
  2.  
  3. @cython.exceptval(-1)
  4. def func(x: cython.int) -> cython.int:
  5.     if x < 0:
  6.         raise ValueError("need integer >= 0")
  7.     return x + 1

Since version 0.27, Cython also supports the variable annotations definedin PEP 526. This allows todeclare types of variables in a Python 3.6 compatible way as follows:

  1. import cython
  2.  
  3. def func():
  4.     # Cython types are evaluated as for cdef declarations
  5.     x: cython.int               # cdef int x
  6.     y: cython.double = 0.57721  # cdef double y = 0.57721
  7.     z: cython.float = 0.57721   # cdef float z  = 0.57721
  8.  
  9.     # Python types shadow Cython types for compatibility reasons
  10.     a: float = 0.54321          # cdef double a = 0.54321
  11.     b: int = 5                  # cdef object b = 5
  12.     c: long = 6                 # cdef object c = 6
  13.     pass
  14.  
  15. @cython.cclass
  16. class A:
  17.     a: cython.int
  18.     b: cython.int
  19.  
  20.     def __init__(self, b=0):
  21.         self.a = 3
  22.         self.b = b

There is currently no way to express the visibility of object attributes.

C types

There are numerous types built into the Cython module. It provides all thestandard C types, namely char, short, int, long, longlongas well as their unsigned versions uchar, ushort, uint, ulong,ulonglong. The special bint type is used for C boolean values andPy_ssize_t for (signed) sizes of Python containers.

For each type, there are pointer types p_int, pp_int, etc., up tothree levels deep in interpreted mode, and infinitely deep in compiled mode.Further pointer types can be constructed with cython.pointer(cython.int),and arrays as cython.int[10]. A limited attempt is made to emulate thesemore complex types, but only so much can be done from the Python language.

The Python types int, long and bool are interpreted as C int, longand bint respectively. Also, the Python builtin types list, dict,tuple, etc. may be used, as well as any user defined types.

Typed C-tuples can be declared as a tuple of C types.

Extension types and cdef functions

  • The class decorator @cython.cclass creates a cdef class.
  • The function/method decorator @cython.cfunc creates a cdef function.
  • @cython.ccall creates a cpdef function, i.e. one that Cython codecan call at the C level.
  • @cython.locals declares local variables (see above). It can also be used todeclare types for arguments, i.e. the local variables that are used in thesignature.
  • @cython.inline is the equivalent of the C inline modifier.
  • @cython.final terminates the inheritance chain by preventing a type frombeing used as a base class, or a method from being overridden in subtypes.This enables certain optimisations such as inlined method calls.

Here is an example of a cdef function:

  1. @cython.cfunc
    @cython.returns(cython.bint)
    @cython.locals(a=cython.int, b=cython.int)
    def c_compare(a,b):
        return a == b

Further Cython functions and declarations

  • address is used in place of the & operator:
  1. cython.declare(x=cython.int, x_ptr=cython.p_int)
  2. x_ptr = cython.address(x)
  • sizeof emulates the sizeof operator. It can take both types andexpressions.
  1. cython.declare(n=cython.longlong)
  2. print(cython.sizeof(cython.longlong))
  3. print(cython.sizeof(n))
  • struct can be used to create struct types.:
  1. MyStruct = cython.struct(x=cython.int, y=cython.int, data=cython.double)
  2. a = cython.declare(MyStruct)

is equivalent to the code:

  1. cdef struct MyStruct:
  2.     int x
  3.     int y
  4.     double data
  5.  
  6. cdef MyStruct a

  • union creates union types with exactly the same syntax as struct.

  • typedef defines a type under a given name:

  1. T = cython.typedef(cython.p_int)   # ctypedef int* T
  • cast will (unsafely) reinterpret an expression type. cython.cast(T, t)is equivalent to <T>t. The first attribute must be a type, the second isthe expression to cast. Specifying the optional keyword argumenttypecheck=True has the semantics of <T?>t.
  1. t1 = cython.cast(T, t)
  2. t2 = cython.cast(T, t, typecheck=True)

Magic Attributes within the .pxd

The special cython module can also be imported and used within the augmenting.pxd file. For example, the following Python file dostuff.py:

  1. def dostuff(n):
  2.     t = 0
  3.     for i in range(n):
  4.         t += i
  5.     return t

can be augmented with the following .pxd file dostuff.pxd:

  1. import cython
  2.  
  3. @cython.locals(t=cython.int, i=cython.int)
  4. cpdef int dostuff(int n)

The cython.declare() function can be used to specify types for globalvariables in the augmenting .pxd file.

Tips and Tricks

Calling C functions

Normally, it isn’t possible to call C functions in pure Python mode as thereis no general way to support it in normal (uncompiled) Python. However, incases where an equivalent Python function exists, this can be achieved bycombining C function coercion with a conditional import as follows:

  1. # mymodule.pxd
  2.  
  3. # declare a C function as "cpdef" to export it to the module
  4. cdef extern from "math.h":
  5.     cpdef double sin(double x)
  1. # mymodule.py
  2.  
  3. import cython
  4.  
  5. # override with Python import if not in compiled code
  6. if not cython.compiled:
  7.     from math import sin
  8.  
  9. # calls sin() from math.h when compiled with Cython and math.sin() in Python
  10. print(sin(0))

Note that the “sin” function will show up in the module namespace of “mymodule”here (i.e. there will be a mymodule.sin() function). You can mark it as aninternal name according to Python conventions by renaming it to “_sin” in the.pxd file as follows:

  1. cdef extern from "math.h":
  2.     cpdef double _sin "sin" (double x)

You would then also change the Python import to from math import sin as _sinto make the names match again.

Using C arrays for fixed size lists

C arrays can automatically coerce to Python lists or tuples.This can be exploited to replace fixed size Python lists in Python code by Carrays when compiled. An example:

  1. import cython
  2.  
  3.  
  4. @cython.locals(counts=cython.int[10], digit=cython.int)
  5. def count_digits(digits):
  6.     """
  7.     >>> digits = '01112222333334445667788899'
  8.     >>> count_digits(map(int, digits))
  9.     [1, 3, 4, 5, 3, 1, 2, 2, 3, 2]
  10.     """
  11.     counts = [0] * 10
  12.     for digit in digits:
  13.         assert 0 <= digit <= 9
  14.         counts[digit] += 1
  15.     return counts

In normal Python, this will use a Python list to collect the counts, whereasCython will generate C code that uses a C array of C ints.