First-class iterators

There are 2 kinds of iterators in Nim: inline and closure iterators. An inline iterator is an iterator that’s always inlined by the compiler leading to zero overhead for the abstraction, but may result in a heavy increase in code size.

Caution: the body of a for loop over an inline iterator is inlined into each yield statement appearing in the iterator code, so ideally the code should be refactored to contain a single yield when possible to avoid code bloat.

Inline iterators are second class citizens; They can be passed as parameters only to other inlining code facilities like templates, macros, and other inline iterators.

In contrast to that, a closure iterator can be passed around more freely:

  1. iterator count0(): int {.closure.} =
  2.   yield 0
  4. iterator count2(): int {.closure.} =
  5.   var x = 1
  6.   yield x
  7.   inc x
  8.   yield x
  10. proc invoke(iter: iterator(): int {.closure.}) =
  11.   for x in iter(): echo x
  13. invoke(count0)
  14. invoke(count2)

Closure iterators and inline iterators have some restrictions:

  1. For now, a closure iterator cannot be executed at compile time.
  2. return is allowed in a closure iterator but not in an inline iterator (but rarely useful) and ends the iteration.
  3. Inline iterators cannot be recursive.
  4. Neither inline nor closure iterators have the special result variable.
  5. Closure iterators are not supported by the JS backend.

Iterators that are neither marked {.closure.} nor {.inline.} explicitly default to being inline, but this may change in future versions of the implementation.

The iterator type is always of the calling convention closure implicitly; the following example shows how to use iterators to implement a collaborative tasking system:

  1. # simple tasking:
  2. type
  3.   Task = iterator (ticker: int)
  5. iterator a1(ticker: int) {.closure.} =
  6.   echo "a1: A"
  7.   yield
  8.   echo "a1: B"
  9.   yield
  10.   echo "a1: C"
  11.   yield
  12.   echo "a1: D"
  14. iterator a2(ticker: int) {.closure.} =
  15.   echo "a2: A"
  16.   yield
  17.   echo "a2: B"
  18.   yield
  19.   echo "a2: C"
  21. proc runTasks(t: varargs[Task]) =
  22.   var ticker = 0
  23.   while true:
  24.     let x = t[ticker mod t.len]
  25.     if finished(x): break
  26.     x(ticker)
  27.     inc ticker
  29. runTasks(a1, a2)

The builtin system.finished can be used to determine if an iterator has finished its operation; no exception is raised on an attempt to invoke an iterator that has already finished its work.

Note that system.finished is error-prone to use because it only returns true one iteration after the iterator has finished:

  1. iterator mycount(a, b: int): int {.closure.} =
  2.   var x = a
  3.   while x <= b:
  4.     yield x
  5.     inc x
  7. var c = mycount # instantiate the iterator
  8. while not finished(c):
  9.   echo c(1, 3)
  11. # Produces
  12. 1
  13. 2
  14. 3
  15. 0

Instead, this code has to be used:

  1. var c = mycount # instantiate the iterator
  2. while true:
  3.   let value = c(1, 3)
  4.   if finished(c): break # and discard 'value'!
  5.   echo value

It helps to think that the iterator actually returns a pair (value, done) and finished is used to access the hidden done field.

Closure iterators are resumable functions and so one has to provide the arguments to every call. To get around this limitation one can capture parameters of an outer factory proc:

  1. proc mycount(a, b: int): iterator (): int =
  2.   result = iterator (): int =
  3.     var x = a
  4.     while x <= b:
  5.       yield x
  6.       inc x
  8. let foo = mycount(1, 4)
  10. for f in foo():
  11.   echo f

The call can be made more like an inline iterator with a for loop macro:

  1. import std/macros
  2. macro toItr(x: ForLoopStmt): untyped =
  3.   let expr = x[0]
  4.   let call = x[1][1] # Get foo out of toItr(foo)
  5.   let body = x[2]
  6.   result = quote do:
  7.     block:
  8.       let itr = `call`
  9.       for `expr` in itr():
  10.           `body`
  12. for f in toItr(mycount(1, 4)): # using early `proc mycount`
  13.   echo f

Because of full backend function call apparatus involvement, closure iterator invocation is typically higher cost than inline iterators. Adornment by a macro wrapper at the call site like this is a possibly useful reminder.

The factory proc, as an ordinary procedure, can be recursive. The above macro allows such recursion to look much like a recursive iterator would. For example:

  1. proc recCountDown(n: int): iterator(): int =
  2.   result = iterator(): int =
  3.     if n > 0:
  4.       yield n
  5.       for e in toItr(recCountDown(n - 1)):
  6.         yield e
  8. for i in toItr(recCountDown(6)): # Emits: 6 5 4 3 2 1
  9.   echo i

See also iterable for passing iterators to templates and macros.