Theano at a Glance

Theano at a Glance

Theano is a Python library that lets you to define, optimize, and evaluatemathematical expressions, especially ones with multi-dimensional arrays(numpy.ndarray). Using Theano it ispossible to attain speeds rivaling hand-crafted C implementations for problemsinvolving large amounts of data. It can also surpass C on a CPU by many ordersof magnitude by taking advantage of recent GPUs.

Theano combines aspects of a computer algebra system (CAS) with aspects of anoptimizing compiler. It can also generate customized C code for manymathematical operations. This combination of CAS with optimizing compilationis particularly useful for tasks in which complicated mathematical expressionsare evaluated repeatedly and evaluation speed is critical. For situationswhere many different expressions are each evaluated once Theano can minimizethe amount of compilation/analysis overhead, but still provide symbolicfeatures such as automatic differentiation.

Theano’s compiler applies many optimizations of varying complexity tothese symbolic expressions. These optimizations include, but are notlimited to:

use of GPU for computations
constant folding
merging of similar subgraphs, to avoid redundant calculation
arithmetic simplification (e.g. x*y/x -> y, —x -> x)
inserting efficient BLAS operations (e.g. GEMM) in a variety ofcontexts
using memory aliasing to avoid calculation
using inplace operations wherever it does not interfere with aliasing
loop fusion for elementwise sub-expressions
improvements to numerical stability (e.g. $\log(1+\exp(x))$ and $\log(\sum_i \exp(x[i]))$ )
for a complete list, see Optimizations

Theano was written at the LISA lab to support rapid development ofefficient machine learning algorithms. Theano isnamed after the Greek mathematician), who may have been Pythagoras’wife. Theano is released under a BSD license (link).

Sneak peek

Here is an example of how to use Theano. It doesn’t show off many ofTheano’s features, but it illustrates concretely what Theano is.

import theano
from theano import tensor
 
# declare two symbolic floating-point scalars
a = tensor.dscalar()
b = tensor.dscalar()
 
# create a simple expression
c = a + b
 
# convert the expression into a callable object that takes (a,b)
# values as input and computes a value for c
f = theano.function([a,b], c)
 
# bind 1.5 to 'a', 2.5 to 'b', and evaluate 'c'
assert 4.0 == f(1.5, 2.5)

Theano is not a programming language in the normal sense because youwrite a program in Python that builds expressions for Theano. Still itis like a programming language in the sense that you have to

declare variables (a,b) and give their types
build expressions for how to put those variables together
compile expression graphs to functions in order to use them for computation.

It is good to think of theano.function as the interface to acompiler which builds a callable object from a purely symbolic graph.One of Theano’s most important features is that theano.functioncan optimize a graph and even compile some or all of it into nativemachine instructions.

What does it do that they don’t?

Theano is a Python library and optimizing compiler for manipulatingand evaluating expressions, especially matrix-valuedones. Manipulation of matrices is typically done using the numpypackage, so what does Theano do that Python and numpy do not?

execution speed optimizations: Theano can use g++ or nvcc to compileparts your expression graph into CPU or GPU instructions, which runmuch faster than pure Python.
symbolic differentiation: Theano can automatically build symbolic graphsfor computing gradients.
stability optimizations: Theano can recognize [some] numerically unstableexpressions and compute them with more stable algorithms.

The closest Python package to Theano is sympy.Theano focuses more on tensor expressions than Sympy, and has more machineryfor compilation. Sympy has more sophisticated algebra rules and canhandle a wider variety of mathematical operations (such as series, limits, and integrals).

If numpy is to be compared to MATLAB and sympy to Mathematica,Theano is a sort of hybrid of the two which tries to combine the best ofboth worlds.

Getting started

Installing Theano
Instructions to download and install Theano on your system.
Tutorial
Getting started with Theano’s basic features. Go here if you arenew!
API Documentation
Details of what Theano provides. It is recommended to go throughthe Tutorial first though.

A PDF version of the online documentation may be found here.

Theano Vision

This is the vision we have for Theano. This is give people an idea of what toexpect in the future of Theano, but we can’t promise to implement allof it. This should also help you to understand where Theano fits in relationto other computational tools.

Support tensor and sparse operations
Support linear algebra operations
- Graph Transformations
- - Differentiation/higher order differentiation
  - ‘R’ and ‘L’ differential operators
  - Speed/memory optimizations
  - Numerical stability optimizations
Can use many compiled languages, instructions sets: C/C++, CUDA, OpenCL, PTX, CAL, AVX, …
Lazy evaluation
Loop
Parallel execution (SIMD, multi-core, multi-node on cluster,multi-node distributed)
Support all NumPy/basic SciPy functionality
Easy wrapping of library functions in Theano

Note: There is no short term plan to support multi-node computation.

Theano Vision State

Here is the state of that vision as of December 3th, 2013 (after Theano release0.6):

We support tensors using the numpy.ndarray object and we support many operations on them.
We support sparse types by using the scipy.{csc,csr,bsr}_matrix object and support some operations on them.
We have started implementing/wrapping more advanced linear algebra operations.
We have many graph transformations that cover the 4 categories listed above.
We can improve the graph transformation with better storage optimizationand instruction selection.
- Similar to auto-tuning during the optimization phase, but thisdoesn’t apply to only 1 op.
- Example of use: Determine if we should move computation to theGPU or not depending on the input size.
- Possible implementation note: allow Theano Variable in the fgraph tohave more than 1 owner.
We support Python 2 and Python 3.
We have a CUDA backend for tensors of type float32 only.
Efforts have begun towards a generic GPU ndarray (GPU tensor) (started in thelibgpuarray project)
- Move GPU backend outside of Theano.
- Will provide better support for GPU on Windows and support an OpenCL backend on CPU.
Loops work, but not all related optimizations are currently done.
The cvm linker allows lazy evaluation. It is the current default linker.
- How to have DebugMode check it? Right now, DebugMode checks the computation non-lazily.
SIMD parallelism on the CPU comes from the compiler.
Multi-core parallelism support limited.If the external BLAS implementation supports it,many dot are parallelized via gemm, gemv and ger.Also, element-wise operation are supported. See Multi cores support in Theano.
No multi-node support.
Most, but not all NumPy functions/aliases are implemented.* https://github.com/Theano/Theano/issues/1080
Wrapping an existing Python function in easy and documented.
We know how to separate the shared variable memorystorage location from its object type (tensor, sparse, dtype, broadcastflags), but we need to do it.

Contact us

Discussion about Theano takes place in the theano-dev andtheano-users mailing lists. People interested in development ofTheano should check the former, while the latter is reserved forissues that concern the end users.

Questions, comments, praise, criticism as well as bug reports shouldbe submitted to these mailing lists.

We welcome all kinds of contributions. If you have any questionsregarding how to extend Theano, please feel free to ask on thetheano-dev mailing list.