How Shape Information is Handled by Theano

It is not possible to strictly enforce the shape of a Theano variable whenbuilding a graph since the particular value provided at run-time for a parameter of aTheano function may condition the shape of the Theano variables in its graph.

Currently, information regarding shape is used in two ways in Theano:

  • To generate faster C code for the 2d convolution on the CPU and the GPU,when the exact output shape is known in advance.

  • To remove computations in the graph when we only want to know theshape, but not the actual value of a variable. This is done with theOp.infer_shapemethod.

Example:

  1. >>> import theano
  2. >>> x = theano.tensor.matrix('x')
  3. >>> f = theano.function([x], (x ** 2).shape)
  4. >>> theano.printing.debugprint(f) 
  5. MakeVector{dtype='int64'} [id A] ''   2
  6.  |Shape_i{0} [id B] ''   1
  7.  | |x [id C]
  8.  |Shape_i{1} [id D] ''   0
  9.    |x [id C]

The output of this compiled function does not contain any multiplicationor power. Theano has removed them to compute directly the shape of theoutput.

Shape Inference Problem

Theano propagates information about shape in the graph. Sometimes thiscan lead to errors. Consider this example:

  1. >>> import numpy
  2. >>> import theano
  3. >>> x = theano.tensor.matrix('x')
  4. >>> y = theano.tensor.matrix('y')
  5. >>> z = theano.tensor.join(0, x, y)
  6. >>> xv = numpy.random.rand(5, 4)
  7. >>> yv = numpy.random.rand(3, 3)
  1. >>> f = theano.function([x, y], z.shape)
  2. >>> theano.printing.debugprint(f) 
  3. MakeVector{dtype='int64'} [id A] ''   4
  4.  |Elemwise{Add}[(0, 0)] [id B] ''   3
  5.  | |Shape_i{0} [id C] ''   1
  6.  | | |x [id D]
  7.  | |Shape_i{0} [id E] ''   2
  8.  |   |y [id F]
  9.  |Shape_i{1} [id G] ''   0
  10.    |x [id D]
  1. >>> f(xv, yv) # DOES NOT RAISE AN ERROR AS SHOULD BE.
  2. array([8, 4])
  1. >>> f = theano.function([x,y], z)# Do not take the shape.
  2. >>> theano.printing.debugprint(f) 
  3. Join [id A] ''   0
  4.  |TensorConstant{0} [id B]
  5.  |x [id C]
  6.  |y [id D]
  1. >>> f(xv, yv)  
  2. Traceback (most recent call last):
  3.   ...
  4. ValueError: ...

As you can see, when asking only for the shape of some computation (join in theexample), an inferred shape is computed directly, without executingthe computation itself (there is no join in the first output or debugprint).

This makes the computation of the shape faster, but it can also hide errors. Inthis example, the computation of the shape of the output of join is done onlybased on the first input Theano variable, which leads to an error.

This might happen with other ops such as elemwise and dot, for example.Indeed, to perform some optimizations (for speed or stability, for instance),Theano assumes that the computation is correct and consistentin the first place, as it does here.

You can detect those problems by running the code without thisoptimization, using the Theano flagoptimizer_excluding=local_shape_to_shape_i. You can also obtain thesame effect by running in the modes FAST_COMPILE (it will not apply thisoptimization, nor most other optimizations) or DebugMode (it will testbefore and after all optimizations (much slower)).

Specifing Exact Shape

Currently, specifying a shape is not as easy and flexible as we wish and we plan someupgrade. Here is the current state of what can be done:

  • You can pass the shape info directly to the ConvOp createdwhen calling conv2d. You simply set the parameters image_shapeand filter_shape inside the call. They must be tuples of 4elements. For example:
  1. theano.tensor.nnet.conv2d(..., image_shape=(7, 3, 5, 5), filter_shape=(2, 3, 4, 4))
  • You can use the SpecifyShape op to add shape information anywhere in thegraph. This allows to perform some optimizations. In the following example,this makes it possible to precompute the Theano function to a constant.
  1. >>> import theano
  2. >>> x = theano.tensor.matrix()
  3. >>> x_specify_shape = theano.tensor.specify_shape(x, (2, 2))
  4. >>> f = theano.function([x], (x_specify_shape ** 2).shape)
  5. >>> theano.printing.debugprint(f) 
  6. DeepCopyOp [id A] ''   0
  7.  |TensorConstant{(2,) of 2} [id B]

Future Plans

The parameter “constant shape” will be added to theano.shared(). This is probably the most frequent occurrence with shared variables. It will make the code simpler and will make it possible to check that the shape does not change when updating the shared variable.