Compound Tasks

This is a fairly advanced topic, which you should only tackle once you have mastered the basics of Jug.

A compound task is a function that builds up a potentially-complex task. Because this function might generate many intermediate tasks which are not very meaningful, using a compound task construct allows us to throw them away once they have run.

For example:

  1. @TaskGenerator
  2. def poly(a,b,c):
  3.     return a*b+c
  5. @TaskGenerator
  6. def max3(values):
  7.     'Return the maximum 3 values'
  8.     values.sort()
  9.     return tuple(values[-3:])
  11. def buildup(numbers):
  12.     results = []
  13.     for n in numbers:
  14.         results.append(poly(n, 2*n, poly(n,n,n)))
  15.     return max3(results)
  17. # Work in Python2 & Python3
  18. numbers = list(range(192))
  19. intermediate = buildup(numbers)
  20. ...

We have a fairly complex function buildup, which takes the list of N numbers and generated 2*N+1 Tasks. These are a lot of Tasks and may make jug run slower. On the other hand, you may not want to simply have a single Task do everything:

  1. def poly(a,b,c):
  2.     return a*b+c
  4. def max3(values):
  5.     'Return the maximum 3 values'
  6.     values.sort()
  7.     return tuple(values[-3:])
  9. @TaskGenerator
  10. def buildup(numbers):
  11.     results = []
  12.     for n in numbers:
  13.         results.append(poly(n, 2*n, poly(n,n,n)))
  14.     return max3(results)

Because, this way, you will have lost any ability to have the different calls to poly be run in parallel.

A compound task behaves like the first example until all necessary tasks have been computed:

  1. @TaskGenerator
  2. def poly(a,b,c):
  3.     return a*b+c
  5. @TaskGenerator
  6. def max3(values):
  7.     'Return the maximum 3 values'
  8.     values.sort()
  9.     return tuple(values[-3:])
  11. @CompoundTaskGenerator
  12. def buildup(numbers):
  13.     results = []
  14.     for n in numbers:
  15.         results.append(poly(n, 2*n, poly(n,n,n)))
  16.     return max3(results)
  18. # Work in Python2 & Python3
  19. numbers = list(range(192))
  20. intermediate = buildup(numbers)
  21. ...

Basically, when Jug sees the CompoundTask, it asks is the result of all of this already available? If yes, then it is just the current result; otherwise, the function is called immediately (not at execution time, but every time the jugfile is loaded).

See Also

See also the section on mapreduce.