Iterators

An iterator is a structured pattern for pulling information from a source in one-at-a-time fashion. This pattern has been around programming for a long time. And to be sure, JS developers have been ad hoc designing and implementing iterators in JS programs since before anyone can remember, so it’s not at all a new topic.

What ES6 has done is introduce an implicit standardized interface for iterators. Many of the built-in data structures in JavaScript will now expose an iterator implementing this standard. And you can also construct your own iterators adhering to the same standard, for maximal interoperability.

Iterators are a way of organizing ordered, sequential, pull-based consumption of data.

For example, you may implement a utility that produces a new unique identifier each time it’s requested. Or you may produce an infinite series of values that rotate through a fixed list, in round-robin fashion. Or you could attach an iterator to a database query result to pull out new rows one at a time.

Although they have not commonly been used in JS in such a manner, iterators can also be thought of as controlling behavior one step at a time. This can be illustrated quite clearly when considering generators (see “Generators” later in this chapter), though you can certainly do the same without generators.

Interfaces

At the time of this writing, ES6 section 25.1.1.2 (https://people.mozilla.org/~jorendorff/es6-draft.html#sec-iterator-interface) details the Iterator interface as having the following requirement:

  1. Iterator [required]
  2.     next() {method}: retrieves next IteratorResult

There are two optional members that some iterators are extended with:

  1. Iterator [optional]
  2.     return() {method}: stops iterator and returns IteratorResult
  3.     throw() {method}: signals error and returns IteratorResult

The IteratorResult interface is specified as:

  1. IteratorResult
  2.     value {property}: current iteration value or final return value
  3.         (optional if `undefined`)
  4.     done {property}: boolean, indicates completion status

Note: I call these interfaces implicit not because they’re not explicitly called out in the specification — they are! — but because they’re not exposed as direct objects accessible to code. JavaScript does not, in ES6, support any notion of “interfaces,” so adherence for your own code is purely conventional. However, wherever JS expects an iterator — a for..of loop, for instance — what you provide must adhere to these interfaces or the code will fail.

There’s also an Iterable interface, which describes objects that must be able to produce iterators:

  1. Iterable
  2.     @@iterator() {method}: produces an Iterator

If you recall from “Built-In Symbols” in Chapter 2, @@iterator is the special built-in symbol representing the method that can produce iterator(s) for the object.

IteratorResult

The IteratorResult interface specifies that the return value from any iterator operation will be an object of the form:

  1. { value: .. , done: true / false }

Built-in iterators will always return values of this form, but more properties are, of course, allowed to be present on the return value, as necessary.

For example, a custom iterator may add additional metadata to the result object (e.g., where the data came from, how long it took to retrieve, cache expiration length, frequency for the appropriate next request, etc.).

Note: Technically, value is optional if it would otherwise be considered absent or unset, such as in the case of the value undefined. Because accessing res.value will produce undefined whether it’s present with that value or absent entirely, the presence/absence of the property is more an implementation detail or an optimization (or both), rather than a functional issue.

next() Iteration

Let’s look at an array, which is an iterable, and the iterator it can produce to consume its values:

  1. var arr = [1,2,3];
  3. var it = arr[Symbol.iterator]();
  5. it.next();        // { value: 1, done: false }
  6. it.next();        // { value: 2, done: false }
  7. it.next();        // { value: 3, done: false }
  9. it.next();        // { value: undefined, done: true }

Each time the method located at Symbol.iterator (see Chapters 2 and 7) is invoked on this arr value, it will produce a new fresh iterator. Most structures will do the same, including all the built-in data structures in JS.

However, a structure like an event queue consumer might only ever produce a single iterator (singleton pattern). Or a structure might only allow one unique iterator at a time, requiring the current one to be completed before a new one can be created.

The it iterator in the previous snippet doesn’t report done: true when you receive the 3 value. You have to call next() again, in essence going beyond the end of the array’s values, to get the complete signal done: true. It may not be clear why until later in this section, but that design decision will typically be considered a best practice.

Primitive string values are also iterables by default:

  1. var greeting = "hello world";
  3. var it = greeting[Symbol.iterator]();
  5. it.next();        // { value: "h", done: false }
  6. it.next();        // { value: "e", done: false }
  7. ..

Note: Technically, the primitive value itself isn’t iterable, but thanks to “boxing”, "hello world" is coerced/converted to its String object wrapper form, which is an iterable. See the Types & Grammar title of this series for more information.

ES6 also includes several new data structures, called collections (see Chapter 5). These collections are not only iterables themselves, but they also provide API method(s) to generate an iterator, such as:

  1. var m = new Map();
  2. m.set( "foo", 42 );
  3. m.set( { cool: true }, "hello world" );
  5. var it1 = m[Symbol.iterator]();
  6. var it2 = m.entries();
  8. it1.next();        // { value: [ "foo", 42 ], done: false }
  9. it2.next();        // { value: [ "foo", 42 ], done: false }
  10. ..

The next(..) method of an iterator can optionally take one or more arguments. The built-in iterators mostly do not exercise this capability, though a generator’s iterator definitely does (see “Generators” later in this chapter).

By general convention, including all the built-in iterators, calling next(..) on an iterator that’s already been exhausted is not an error, but will simply continue to return the result { value: undefined, done: true }.

Optional: return(..) and throw(..)

The optional methods on the iterator interface — return(..) and throw(..) — are not implemented on most of the built-in iterators. However, they definitely do mean something in the context of generators, so see “Generators” for more specific information.

return(..) is defined as sending a signal to an iterator that the consuming code is complete and will not be pulling any more values from it. This signal can be used to notify the producer (the iterator responding to next(..) calls) to perform any cleanup it may need to do, such as releasing/closing network, database, or file handle resources.

If an iterator has a return(..) present and any condition occurs that can automatically be interpreted as abnormal or early termination of consuming the iterator, return(..) will automatically be called. You can call return(..) manually as well.

return(..) will return an IteratorResult object just like next(..) does. In general, the optional value you send to return(..) would be sent back as value in this IteratorResult, though there are nuanced cases where that might not be true.

throw(..) is used to signal an exception/error to an iterator, which possibly may be used differently by the iterator than the completion signal implied by return(..). It does not necessarily imply a complete stop of the iterator as return(..) generally does.

For example, with generator iterators, throw(..) actually injects a thrown exception into the generator’s paused execution context, which can be caught with a try..catch. An uncaught throw(..) exception would end up abnormally aborting the generator’s iterator.

Note: By general convention, an iterator should not produce any more results after having called return(..) or throw(..).

Iterator Loop

As we covered in the “for..of“ section in Chapter 2, the ES6 for..of loop directly consumes a conforming iterable.

If an iterator is also an iterable, it can be used directly with the for..of loop. You make an iterator an iterable by giving it a Symbol.iterator method that simply returns the iterator itself:

  1. var it = {
  2.     // make the `it` iterator an iterable
  3.     [Symbol.iterator]() { return this; },
  5.     next() { .. },
  6.     ..
  7. };
  9. it[Symbol.iterator]() === it;        // true

Now we can consume the it iterator with a for..of loop:

  1. for (var v of it) {
  2.     console.log( v );
  3. }

To fully understand how such a loop works, recall the for equivalent of a for..of loop from Chapter 2:

  1. for (var v, res; (res = it.next()) && !res.done; ) {
  2.     v = res.value;
  3.     console.log( v );
  4. }

If you look closely, you’ll see that it.next() is called before each iteration, and then res.done is consulted. If res.done is true, the expression evaluates to false and the iteration doesn’t occur.

Recall earlier that we suggested iterators should in general not return done: true along with the final intended value from the iterator. Now you can see why.

If an iterator returned { done: true, value: 42 }, the for..of loop would completely discard the 42 value and it’d be lost. For this reason, assuming that your iterator may be consumed by patterns like the for..of loop or its manual for equivalent, you should probably wait to return done: true for signaling completion until after you’ve already returned all relevant iteration values.

Warning: You can, of course, intentionally design your iterator to return some relevant value at the same time as returning done: true. But don’t do this unless you’ve documented that as the case, and thus implicitly forced consumers of your iterator to use a different pattern for iteration than is implied by for..of or its manual equivalent we depicted.

Custom Iterators

In addition to the standard built-in iterators, you can make your own! All it takes to make them interoperate with ES6’s consumption facilities (e.g., the for..of loop and the ... operator) is to adhere to the proper interface(s).

Let’s try constructing an iterator that produces the infinite series of numbers in the Fibonacci sequence:

  1. var Fib = {
  2.     [Symbol.iterator]() {
  3.         var n1 = 1, n2 = 1;
  5.         return {
  6.             // make the iterator an iterable
  7.             [Symbol.iterator]() { return this; },
  9.             next() {
  10.                 var current = n2;
  11.                 n2 = n1;
  12.                 n1 = n1 + current;
  13.                 return { value: current, done: false };
  14.             },
  16.             return(v) {
  17.                 console.log(
  18.                     "Fibonacci sequence abandoned."
  19.                 );
  20.                 return { value: v, done: true };
  21.             }
  22.         };
  23.     }
  24. };
  26. for (var v of Fib) {
  27.     console.log( v );
  29.     if (v > 50) break;
  30. }
  31. // 1 1 2 3 5 8 13 21 34 55
  32. // Fibonacci sequence abandoned.

Warning: If we hadn’t inserted the break condition, this for..of loop would have run forever, which is probably not the desired result in terms of breaking your program!

The Fib[Symbol.iterator]() method when called returns the iterator object with next() and return(..) methods on it. State is maintained via n1 and n2 variables, which are kept by the closure.

Let’s next consider an iterator that is designed to run through a series (aka a queue) of actions, one item at a time:

  1. var tasks = {
  2.     [Symbol.iterator]() {
  3.         var steps = this.actions.slice();
  5.         return {
  6.             // make the iterator an iterable
  7.             [Symbol.iterator]() { return this; },
  9.             next(...args) {
  10.                 if (steps.length > 0) {
  11.                     let res = steps.shift()( ...args );
  12.                     return { value: res, done: false };
  13.                 }
  14.                 else {
  15.                     return { done: true }
  16.                 }
  17.             },
  19.             return(v) {
  20.                 steps.length = 0;
  21.                 return { value: v, done: true };
  22.             }
  23.         };
  24.     },
  25.     actions: []
  26. };

The iterator on tasks steps through functions found in the actions array property, if any, and executes them one at a time, passing in whatever arguments you pass to next(..), and returning any return value to you in the standard IteratorResult object.

Here’s how we could use this tasks queue:

  1. tasks.actions.push(
  2.     function step1(x){
  3.         console.log( "step 1:", x );
  4.         return x * 2;
  5.     },
  6.     function step2(x,y){
  7.         console.log( "step 2:", x, y );
  8.         return x + (y * 2);
  9.     },
  10.     function step3(x,y,z){
  11.         console.log( "step 3:", x, y, z );
  12.         return (x * y) + z;
  13.     }
  14. );
  16. var it = tasks[Symbol.iterator]();
  18. it.next( 10 );            // step 1: 10
  19.                         // { value:   20, done: false }
  21. it.next( 20, 50 );        // step 2: 20 50
  22.                         // { value:  120, done: false }
  24. it.next( 20, 50, 120 );    // step 3: 20 50 120
  25.                         // { value: 1120, done: false }
  27. it.next();                // { done: true }

This particular usage reinforces that iterators can be a pattern for organizing functionality, not just data. It’s also reminiscent of what we’ll see with generators in the next section.

You could even get creative and define an iterator that represents meta operations on a single piece of data. For example, we could define an iterator for numbers that by default ranges from 0 up to (or down to, for negative numbers) the number in question.

Consider:

  1. if (!Number.prototype[Symbol.iterator]) {
  2.     Object.defineProperty(
  3.         Number.prototype,
  4.         Symbol.iterator,
  5.         {
  6.             writable: true,
  7.             configurable: true,
  8.             enumerable: false,
  9.             value: function iterator(){
  10.                 var i, inc, done = false, top = +this;
  12.                 // iterate positively or negatively?
  13.                 inc = 1 * (top < 0 ? -1 : 1);
  15.                 return {
  16.                     // make the iterator itself an iterable!
  17.                     [Symbol.iterator](){ return this; },
  19.                     next() {
  20.                         if (!done) {
  21.                             // initial iteration always 0
  22.                             if (i == null) {
  23.                                 i = 0;
  24.                             }
  25.                             // iterating positively
  26.                             else if (top >= 0) {
  27.                                 i = Math.min(top,i + inc);
  28.                             }
  29.                             // iterating negatively
  30.                             else {
  31.                                 i = Math.max(top,i + inc);
  32.                             }
  34.                             // done after this iteration?
  35.                             if (i == top) done = true;
  37.                             return { value: i, done: false };
  38.                         }
  39.                         else {
  40.                             return { done: true };
  41.                         }
  42.                     }
  43.                 };
  44.             }
  45.         }
  46.     );
  47. }

Now, what tricks does this creativity afford us?

  1. for (var i of 3) {
  2.     console.log( i );
  3. }
  4. // 0 1 2 3
  6. [...-3];                // [0,-1,-2,-3]

Those are some fun tricks, though the practical utility is somewhat debatable. But then again, one might wonder why ES6 didn’t just ship with such a minor but delightful feature easter egg!?

I’d be remiss if I didn’t at least remind you that extending native prototypes as I’m doing in the previous snippet is something you should only do with caution and awareness of potential hazards.

In this case, the chances that you’ll have a collision with other code or even a future JS feature is probably exceedingly low. But just beware of the slight possibility. And document what you’re doing verbosely for posterity’s sake.

Note: I’ve expounded on this particular technique in this blog post (http://blog.getify.com/iterating-es6-numbers/) if you want more details. And this comment (http://blog.getify.com/iterating-es6-numbers/comment-page-1/#comment-535294) even suggests a similar trick but for making string character ranges.

Iterator Consumption

We’ve already shown consuming an iterator item by item with the for..of loop. But there are other ES6 structures that can consume iterators.

Let’s consider the iterator attached to this array (though any iterator we choose would have the following behaviors):

  1. var a = [1,2,3,4,5];

The ... spread operator fully exhausts an iterator. Consider:

  1. function foo(x,y,z,w,p) {
  2.     console.log( x + y + z + w + p );
  3. }
  5. foo( ...a );            // 15

... can also spread an iterator inside an array:

  1. var b = [ 0, ...a, 6 ];
  2. b;                        // [0,1,2,3,4,5,6]

Array destructuring (see “Destructuring” in Chapter 2) can partially or completely (if paired with a ... rest/gather operator) consume an iterator:

  1. var it = a[Symbol.iterator]();
  3. var [x,y] = it;            // take just the first two elements from `it`
  4. var [z, ...w] = it;        // take the third, then the rest all at once
  6. // is `it` fully exhausted? Yep.
  7. it.next();                // { value: undefined, done: true }
  9. x;                        // 1
  10. y;                        // 2
  11. z;                        // 3
  12. w;                        // [4,5]