Generator’ing Values

In the previous section, we mentioned an interesting use for generators, as a way to produce values. This is not the main focus in this chapter, but we’d be remiss if we didn’t cover the basics, especially because this use case is essentially the origin of the name: generators.

We’re going to take a slight diversion into the topic of iterators for a bit, but we’ll circle back to how they relate to generators and using a generator to generate values.

Producers and Iterators

Imagine you’re producing a series of values where each value has a definable relationship to the previous value. To do this, you’re going to need a stateful producer that remembers the last value it gave out.

You can implement something like that straightforwardly using a function closure (see the Scope & Closures title of this series):

  1. var gimmeSomething = (function(){
  2.     var nextVal;
  4.     return function(){
  5.         if (nextVal === undefined) {
  6.             nextVal = 1;
  7.         }
  8.         else {
  9.             nextVal = (3 * nextVal) + 6;
  10.         }
  12.         return nextVal;
  13.     };
  14. })();
  16. gimmeSomething();        // 1
  17. gimmeSomething();        // 9
  18. gimmeSomething();        // 33
  19. gimmeSomething();        // 105

Note: The nextVal computation logic here could have been simplified, but conceptually, we don’t want to calculate the next value (aka nextVal) until the next gimmeSomething() call happens, because in general that could be a resource-leaky design for producers of more persistent or resource-limited values than simple numbers.

Generating an arbitrary number series isn’t a terribly realistic example. But what if you were generating records from a data source? You could imagine much the same code.

In fact, this task is a very common design pattern, usually solved by iterators. An iterator is a well-defined interface for stepping through a series of values from a producer. The JS interface for iterators, as it is in most languages, is to call next() each time you want the next value from the producer.

We could implement the standard iterator interface for our number series producer:

  1. var something = (function(){
  2.     var nextVal;
  4.     return {
  5.         // needed for `for..of` loops
  6.         [Symbol.iterator]: function(){ return this; },
  8.         // standard iterator interface method
  9.         next: function(){
  10.             if (nextVal === undefined) {
  11.                 nextVal = 1;
  12.             }
  13.             else {
  14.                 nextVal = (3 * nextVal) + 6;
  15.             }
  17.             return { done:false, value:nextVal };
  18.         }
  19.     };
  20. })();
  22. something.next().value;        // 1
  23. something.next().value;        // 9
  24. something.next().value;        // 33
  25. something.next().value;        // 105

Note: We’ll explain why we need the [Symbol.iterator]: .. part of this code snippet in the “Iterables” section. Syntactically though, two ES6 features are at play. First, the [ .. ] syntax is called a computed property name (see the this & Object Prototypes title of this series). It’s a way in an object literal definition to specify an expression and use the result of that expression as the name for the property. Next, Symbol.iterator is one of ES6’s predefined special Symbol values (see the ES6 & Beyond title of this book series).

The next() call returns an object with two properties: done is a boolean value signaling the iterator’s complete status; value holds the iteration value.

ES6 also adds the for..of loop, which means that a standard iterator can automatically be consumed with native loop syntax:

  1. for (var v of something) {
  2.     console.log( v );
  4.     // don't let the loop run forever!
  5.     if (v > 500) {
  6.         break;
  7.     }
  8. }
  9. // 1 9 33 105 321 969

Note: Because our something iterator always returns done:false, this for..of loop would run forever, which is why we put the break conditional in. It’s totally OK for iterators to be never-ending, but there are also cases where the iterator will run over a finite set of values and eventually return a done:true.

The for..of loop automatically calls next() for each iteration — it doesn’t pass any values in to the next() — and it will automatically terminate on receiving a done:true. It’s quite handy for looping over a set of data.

Of course, you could manually loop over iterators, calling next() and checking for the done:true condition to know when to stop:

  1. for (
  2.     var ret;
  3.     (ret = something.next()) && !ret.done;
  4. ) {
  5.     console.log( ret.value );
  7.     // don't let the loop run forever!
  8.     if (ret.value > 500) {
  9.         break;
  10.     }
  11. }
  12. // 1 9 33 105 321 969

Note: This manual for approach is certainly uglier than the ES6 for..of loop syntax, but its advantage is that it affords you the opportunity to pass in values to the next(..) calls if necessary.

In addition to making your own iterators, many built-in data structures in JS (as of ES6), like arrays, also have default iterators:

  1. var a = [1,3,5,7,9];
  3. for (var v of a) {
  4.     console.log( v );
  5. }
  6. // 1 3 5 7 9

The for..of loop asks a for its iterator, and automatically uses it to iterate over a‘s values.

Note: It may seem a strange omission by ES6, but regular objects intentionally do not come with a default iterator the way arrays do. The reasons go deeper than we will cover here. If all you want is to iterate over the properties of an object (with no particular guarantee of ordering), Object.keys(..) returns an array, which can then be used like for (var k of Object.keys(obj)) { ... Such a for..of loop over an object’s keys would be similar to a for..in loop, except that Object.keys(..) does not include properties from the [[Prototype]] chain while for..in does (see the this & Object Prototypes title of this series).

Iterables

The something object in our running example is called an iterator, as it has the next() method on its interface. But a closely related term is iterable, which is an object that contains an iterator that can iterate over its values.

As of ES6, the way to retrieve an iterator from an iterable is that the iterable must have a function on it, with the name being the special ES6 symbol value Symbol.iterator. When this function is called, it returns an iterator. Though not required, generally each call should return a fresh new iterator.

a in the previous snippet is an iterable. The for..of loop automatically calls its Symbol.iterator function to construct an iterator. But we could of course call the function manually, and use the iterator it returns:

  1. var a = [1,3,5,7,9];
  3. var it = a[Symbol.iterator]();
  5. it.next().value;    // 1
  6. it.next().value;    // 3
  7. it.next().value;    // 5
  8. ..

In the previous code listing that defined something, you may have noticed this line:

  1. [Symbol.iterator]: function(){ return this; }

That little bit of confusing code is making the something value — the interface of the something iterator — also an iterable; it’s now both an iterable and an iterator. Then, we pass something to the for..of loop:

  1. for (var v of something) {
  2.     ..
  3. }

The for..of loop expects something to be an iterable, so it looks for and calls its Symbol.iterator function. We defined that function to simply return this, so it just gives itself back, and the for..of loop is none the wiser.

Generator Iterator

Let’s turn our attention back to generators, in the context of iterators. A generator can be treated as a producer of values that we extract one at a time through an iterator interface’s next() calls.

So, a generator itself is not technically an iterable, though it’s very similar — when you execute the generator, you get an iterator back:

  1. function *foo(){ .. }
  3. var it = foo();

We can implement the something infinite number series producer from earlier with a generator, like this:

  1. function *something() {
  2.     var nextVal;
  4.     while (true) {
  5.         if (nextVal === undefined) {
  6.             nextVal = 1;
  7.         }
  8.         else {
  9.             nextVal = (3 * nextVal) + 6;
  10.         }
  12.         yield nextVal;
  13.     }
  14. }

Note: A while..true loop would normally be a very bad thing to include in a real JS program, at least if it doesn’t have a break or return in it, as it would likely run forever, synchronously, and block/lock-up the browser UI. However, in a generator, such a loop is generally totally OK if it has a yield in it, as the generator will pause at each iteration, yielding back to the main program and/or to the event loop queue. To put it glibly, “generators put the while..true back in JS programming!”

That’s a fair bit cleaner and simpler, right? Because the generator pauses at each yield, the state (scope) of the function *something() is kept around, meaning there’s no need for the closure boilerplate to preserve variable state across calls.

Not only is it simpler code — we don’t have to make our own iterator interface — it actually is more reason-able code, because it more clearly expresses the intent. For example, the while..true loop tells us the generator is intended to run forever — to keep generating values as long as we keep asking for them.

And now we can use our shiny new *something() generator with a for..of loop, and you’ll see it works basically identically:

  1. for (var v of something()) {
  2.     console.log( v );
  4.     // don't let the loop run forever!
  5.     if (v > 500) {
  6.         break;
  7.     }
  8. }
  9. // 1 9 33 105 321 969

But don’t skip over for (var v of something()) ..! We didn’t just reference something as a value like in earlier examples, but instead called the *something() generator to get its iterator for the for..of loop to use.

If you’re paying close attention, two questions may arise from this interaction between the generator and the loop:

  • Why couldn’t we say for (var v of something) ..? Because something here is a generator, which is not an iterable. We have to call something() to construct a producer for the for..of loop to iterate over.
  • The something() call produces an iterator, but the for..of loop wants an iterable, right? Yep. The generator’s iterator also has a Symbol.iterator function on it, which basically does a return this, just like the something iterable we defined earlier. In other words, a generator’s iterator is also an iterable!

Stopping the Generator

In the previous example, it would appear the iterator instance for the *something() generator was basically left in a suspended state forever after the break in the loop was called.

But there’s a hidden behavior that takes care of that for you. “Abnormal completion” (i.e., “early termination”) of the for..of loop — generally caused by a break, return, or an uncaught exception — sends a signal to the generator’s iterator for it to terminate.

Note: Technically, the for..of loop also sends this signal to the iterator at the normal completion of the loop. For a generator, that’s essentially a moot operation, as the generator’s iterator had to complete first so the for..of loop completed. However, custom iterators might desire to receive this additional signal from for..of loop consumers.

While a for..of loop will automatically send this signal, you may wish to send the signal manually to an iterator; you do this by calling return(..).

If you specify a try..finally clause inside the generator, it will always be run even when the generator is externally completed. This is useful if you need to clean up resources (database connections, etc.):

  1. function *something() {
  2.     try {
  3.         var nextVal;
  5.         while (true) {
  6.             if (nextVal === undefined) {
  7.                 nextVal = 1;
  8.             }
  9.             else {
  10.                 nextVal = (3 * nextVal) + 6;
  11.             }
  13.             yield nextVal;
  14.         }
  15.     }
  16.     // cleanup clause
  17.     finally {
  18.         console.log( "cleaning up!" );
  19.     }
  20. }

The earlier example with break in the for..of loop will trigger the finally clause. But you could instead manually terminate the generator’s iterator instance from the outside with return(..):

  1. var it = something();
  2. for (var v of it) {
  3.     console.log( v );
  5.     // don't let the loop run forever!
  6.     if (v > 500) {
  7.         console.log(
  8.             // complete the generator's iterator
  9.             it.return( "Hello World" ).value
  10.         );
  11.         // no `break` needed here
  12.     }
  13. }
  14. // 1 9 33 105 321 969
  15. // cleaning up!
  16. // Hello World

When we call it.return(..), it immediately terminates the generator, which of course runs the finally clause. Also, it sets the returned value to whatever you passed in to return(..), which is how "Hello World" comes right back out. We also don’t need to include a break now because the generator’s iterator is set to done:true, so the for..of loop will terminate on its next iteration.

Generators owe their namesake mostly to this consuming produced values use. But again, that’s just one of the uses for generators, and frankly not even the main one we’re concerned with in the context of this book.

But now that we more fully understand some of the mechanics of how they work, we can next turn our attention to how generators apply to async concurrency.