tensor.opt – Tensor Optimizations

tensor.opt – Tensor Optimizations

class theano.tensor.opt.Assert(msg='Theano Assert failed!')[source]
Implements assertion in a computational graph.

Returns the first parameter if the condition is true, otherwise, triggersAssertionError.

Notes

This Op is a debugging feature. It can be removed from the graphbecause of optimizations, and can hide some possible optimizations tothe optimizer. Specifically, removing happens if it can be determinedthat condition will always be true. Also, the output of the Op must beused in the function computing the graph, but it doesn’t have to bereturned.

Examples

>>> import theano
>>> T = theano.tensor
>>> x = T.vector('x')
>>> assert_op = T.opt.Assert()
>>> func = theano.function([x], assert_op(x, x.size<2))

class theano.tensor.opt.Canonizer(main, inverse, reciprocal, calculate, use_reciprocal=True)[source]
Simplification tool. The variable is a local_optimizer. It is best usedwith a TopoOptimizer in in_to_out order.

Usage: Canonizer(main, inverse, reciprocal, calculate)

Parameters:

main – A suitable Op class that is commutative, associative andtakes one to an arbitrary number of inputs, e.g. add ormul
inverse – An Op class such that inverse(main(x, y), y) == xe.g. sub or true_div
reciprocal – A function such that main(x, reciprocal(y)) == inverse(x, y)e.g. neg or inv
calculate – Function that takes a list of numpy.ndarray instancesfor the numerator, another list for the denumerator,and calculates inverse(main(num), main(denum)). Ittakes a keyword argument, aslist. If True, the valueshould be returned as a list of one element, unlessthe value is such that value = main(). In that case,the return value should be an empty list.

Examples

>>> import theano.tensor as T
>>> from theano.tensor.opt import Canonizer
>>> add_canonizer = Canonizer(T.add, T.sub, T.neg, \
...                           lambda n, d: sum(n) - sum(d))
>>> mul_canonizer = Canonizer(T.mul, T.true_div, T.inv, \
...                           lambda n, d: prod(n) / prod(d))

Examples of optimizations mul_canonizer can perform:

x / x -> 1

(x * y) / x -> y

x / y / x -> 1 / y

x / y / z -> x / (y * z)

x / (y / z) -> (x * z) / y

(a / b) (b / c) (c / d) -> a / d

(2.0 x) / (4.0 y) -> (0.5 * x) / y

2 * x / 2 -> x

x y z -> Elemwise(T.mul){x,y,z} #only one pass over the memory.

!-> Elemwise(T.mul){x,Elemwise(T.mul){y,z}}

static getconstant(_v)[source]

Returns:A numeric constant if v is a Constant or, well, anumeric constant. If v is a plain Variable, returns None.Return type:object

getnum_denum(_input)[source]
This extract two lists, num and denum, such that the input is:self.inverse(self.main(num), self.main(denum)). It returnsthe two lists in a (num, denum) pair.

For example, for main, inverse and reciprocal = *, / and inv(),

input -> returned value (num, denum)

x*y -> ([x, y], [])

inv(x) -> ([], [x])

inv(x) * inv(y) -> ([], [x, y])

x*y/z -> ([x, y], [z])

log(x) / y * (z + x) / y -> ([log(x), z + x], [y, y])

(((a / b) * c) / d) -> ([a, c], [b, d])

a / (b / c) -> ([a, c], [b])

log(x) -> ([log(x)], [])

xy -> ([xy], [])

x y z -> ([x, y, z], [])

mergenum_denum(_num, denum)[source]
Utility function which takes two lists, num and denum, andreturns something which is equivalent to inverse(main(num),main(denum)), but depends on the length of num and the lengthof denum (in order to minimize the number of operations).

Let n = len(num) and d = len(denum):

n=0, d=0: neutral element (given by self.calculate([], []))

(for example, this would be 0 if main is addition

and 1 if main is multiplication)

n=1, d=0: num[0]

n=0, d=1: reciprocal(denum[0])

n=1, d=1: inverse(num[0], denum[0])

n=0, d>1: reciprocal(main(*denum))

n>1, d=0: main(*num)

n=1, d>1: inverse(num[0], main(*denum))

n>1, d=1: inverse(main(*num), denum[0])

n>1, d>1: inverse(main(num), main(denum))

Given the values of n and d to which they are associated, allof the above are equivalent to:inverse(main(num), main(denum))

simplify(num, denum, out_type)[source]
Shorthand for:

self.simplify_constants(*self.simplify_factors(num, denum))

simplifyconstants(_orig_num, orig_denum, out_type=None)[source]
Find all constants and put them together into a single constant.

Finds all constants in orig_num and orig_denum (usingget_constant) and puts them together into a singleconstant. The constant is inserted as the first element of thenumerator. If the constant is the neutral element, it isremoved from the numerator.

Examples

Let main be multiplication:

[2, 3, x], [] -> [6, x], []

[x, y, 2], [4, z] -> [0.5, x, y], [z]

[x, 2, y], [z, 2] -> [x, y], [z]

simplifyfactors(_num, denum)[source]
For any Variable r which is both in num and denum, removes itfrom both lists. Modifies the lists inplace. Returns themodified lists. For example:

[x], [x] -> [], []

[x, y], [x] -> [y], []

[a, b], [c, d] -> [a, b], [c, d]

class theano.tensor.opt.FusionOptimizer(local_optimizer)[source]
Graph optimizer for Fusion of elemwise operations.

class theano.tensor.opt.InplaceElemwiseOptimizer(OP)[source]
We parametrise it to make it work for Elemwise and GpuElemwise op.
- apply(fgraph)[source]
- Usage: InplaceElemwiseOptimizer(op).optimize(fgraph)

Attempts to replace all Broadcast ops by versions of themthat operate inplace. It operates greedily: for each BroadcastOp that is encountered, for each output, tries each input tosee if it can operate inplace on that input. If so, makes thechange and go to the next output or Broadcast Op.

Examples

x + y + z -> x += y += z

(x + y) (x y) -> (x += y) = (x y) or (x + y) = (x = y)

class theano.tensor.opt.MakeVector(dtype='int64')[source]
Concatenate a number of scalars together into a vector.

This is a simple version of stack() that introduces far less cruftinto the graph. Should work with 0 inputs. The constant_foldingoptimization will remove it.

class theano.tensor.opt.ShapeFeature[source]
Graph optimizer for removing all calls to shape().

This optimizer replaces all Shapes and Subtensors of Shapes withShape_i and MakeVector Ops.

This optimizer has several goals:

to ‘lift’ Shapes to as close to the inputs as possible.
to infer the shape of every node in the graph in terms of theinput shapes.
remove all fills (T.second, T.fill) from the graph Lifting shapes as close to the inputs as possible is important forcanonicalization because it is very bad form to have to computesomething just to know how big it will be. Firstly, it is a wasteof time to compute such outputs. But it is important to get ridof these outputs as early as possible in the compilation processbecause the extra computations make it appear as if many internalgraph nodes have multiple clients. Many optimizations refuse towork on nodes with multiple clients.

Lifting is done by using an .infer_shape function if one ispresent, or else using a conservative default. An Op thatsupports shape-lifting should define a infer_shape(self, node,input_shapes) function. The argument input_shapes is a tuple oftuples… there is an interior tuple for each input to the node.The tuple has as many elements as dimensions. The element inposition i of tuple j represents the i’th shape component of thej’th input. The function should return a tuple of tuples. Oneoutput tuple for each node.output. Again, the i’th element of thej’th output tuple represents the output[j].shape[i] of thefunction. If an output is not a TensorType, then None should bereturned instead of a tuple for that output.

For example the infer_shape for a matrix-matrix product would acceptinput_shapes=((x0,x1), (y0,y1)) and return ((x0, y1),).

Inferring the shape of internal nodes in the graph is importantfor doing size-driven optimizations. If we know how big variousintermediate results will be, we can estimate the cost of many Opsaccurately, and generate c-code that is specific [e.g. unrolled]to particular sizes.

In cases where you cannot figure out the shape, raise a ShapeError.

Notes

Right now there is only the ConvOp that could really takeadvantage of this shape inference, but it is worth it evenjust for the ConvOp. All that’s necessary to do shapeinference is 1) to mark shared inputs as having a particularshape, either via a .tag or some similar hacking; and 2) toadd an optional In() argument to promise that inputs willhave a certain shape (or even to have certain shapes incertain dimensions). We can’t automatically infer the shape ofshared variables as they can change of shape during theexecution by default. (NOT IMPLEMENTED YET, BUT IS IN TRAC)

Using Shape information in Optimizations

To use this shape information in OPTIMIZATIONS, use theshape_of dictionary.

For example:

try:
    shape_of = node.fgraph.shape_feature.shape_of
except AttributeError:
    # This can happen when the mode doesn't include the ShapeFeature.
    return
 
shape_of_output_zero = shape_of[node.output[0]]

The shape_of_output_zero symbol will contain a tuple, whoseelements are either integers or symbolic integers.

TODO: check to see if the symbols are necessarilynon-constant… or are integer literals sometimes Theanoconstants?? That would be confusing.

defaultinfer_shape(_node, i_shapes)[source]
Return a list of shape tuple or None for the outputs of node.

This function is used for Ops that don’t implement infer_shape.Ops that do implement infer_shape should use the i_shapes parameter,but this default implementation ignores it.

getshape(_var, idx)[source]
Optimization can call this to get the current shape_i

It is better to call this then use directly shape_of[var][idx]as this method should update shape_of if needed.

TODO: Up to now, we don’t update it in all cases. Update in all cases.

initr(_r)[source]
Register r’s shape in the shape_of dictionary.
sameshape(_x, y, dim_x=None, dim_y=None)[source]
Return True if we are able to assert that x and y have thesame shape.

dim_x and dim_y are optional. If used, they should be an indexto compare only 1 dimension of x and y.

setshape(_r, s, override=False)[source]
Assign the shape s to previously un-shaped variable r.

Parameters:

- **r** (_a variable_) – 
- **s** (_None__ or __a tuple of symbolic integers_) – 
- **override** (_If False__, __it mean r is a new object in the fgraph._) – If True, it mean r is already in the fgraph and we want tooverride its shape.

setshape_i(_r, i, s_i)[source]
Replace element i of shape_of[r] by s_i
shapeir(_i, r)[source]
Return symbolic r.shape[i] for tensor variable r, int i.
shapetuple(_r)[source]
Return a tuple of symbolic shape vars for tensor variable r.
unpack(s_i, var)[source]
Return a symbolic integer scalar for the shape element s_i.

The s_i argument was produced by the infer_shape() of an Op subclass.

var: the variable that correspond to s_i. This is just forerror reporting.

updateshape(_r, other_r)[source]
Replace shape of r by shape of other_r.

If, on some dimensions, the shape of other_r is not informative,keep the shape of r on those dimensions.

class theano.tensor.opt.ShapeOptimizer[source]
Optimizer that serves to add ShapeFeature as an fgraph feature.

class theano.tensor.opt.UnShapeOptimizer[source]
Optimizer remove ShapeFeature as an fgraph feature.

theano.tensor.opt.applyrebroadcast_opt(_rval)[source]
Apply as many times as required the optimization local_useless_rebroadcastand local_rebroadcast_lift.

Parameters:rval (a Variable) – Returns:Return type:A Variable (the same if no optimization can be applied)

theano.tensor.opt.broadcastlike(_value, template, fgraph, dtype=None)[source]
Return a Variable with the same shape and dtype as the template,filled by broadcasting value through it. value will be cast asnecessary.

theano.tensor.opt.checkfor_x_over_absX(_numerators, denominators)[source]
Convert x/abs(x) into sign(x).

theano.tensor.opt.encompassesbroadcastable(_b1, b2)[source]

Parameters:

b1 – The broadcastable attribute of a tensor type.
b2 – The broadcastable attribute of a tensor type.Returns: True if the broadcastable patterns b1 and b2 are such that b2 isbroadcasted to b1’s shape and not the opposite. Return type: bool

theano.tensor.opt.getclients(_node)[source]
Used by erf/erfc opt to track less frequent op.

theano.tensor.opt.getclients2(_node)[source]
Used by erf/erfc opt to track less frequent op.

theano.tensor.opt.isan_upcast(_type1, type2)[source]
Given two data types (as strings), check if converting totype2 from type1 constitutes an upcast.Differs from theano.scalar.upcast

theano.tensor.opt.isinverse_pair(_node_op, prev_op, inv_pair)[source]
Given two consecutive operations, check if they are theprovided pair of inverse functions.

theano.tensor.opt.localadd_mul_fusion(_node)[source]
Fuse consecutive add or mul in one such node with more inputs.

It is better to fuse add/mul that way then in a Composite node asthis make the inner graph of the Composite smaller. This allow toput more computation in a Composite before hitting the maxrecusion limit when pickling Composite.

theano.tensor.opt.localelemwise_fusion(_node)[source]
As part of specialization, we fuse two consecutive elemwise Ops of thesame shape.

For mixed dtype, we let the Composite op do the cast. It lets the Ccompiler do the cast.The number of dimensions is validated at call time by theano itself.

theano.tensor.opt.localelemwise_fusion_op(_OP, max_input_fct=>, maker=None)[source]
We parametrize it to make it work for Elemwise and GpuElemwise op.

Parameters:

OP – GpuElemwise or Elemwise class (the one that we want to fuse)
max_input_fct – A function that returns the maximum number of inputsthat this elemwise can take (useful for GpuElemwise).GPU kernel currently has a limit of 256 bytes forthe size of all parameters passed to it. As currentlywe pass many information only by parameter, we mustlimit how many ops we fuse together to avoid bustingthat 256 limit.

On the CPU we limit to 32 input variablessince that is the maximum numpy support.

theano.tensor.opt.mergetwo_slices(_slice1, len1, slice2, len2)[source]

This function merges two slices into a single slice. The code works on the assumption that:

slice1 is actually a slice and not an index, while slice2 can be just an index.

the two slices have been applied consecutively on the same tensor

The output slice is not in canonical form, but actually just a slicethat can be applied to a tensor to produce the same output as applyingthe two consecutive slices.len1 is the length of the tensor before applying the first slice,while len2 is the length after applying the first slice.

theano.tensor.opt.scalarconstsrest(_inputs, elemwise=True, only_process_constants=False)[source]
Partition a list of variables into two kinds:scalar constants, and the rest.