Please support this book: buy it (PDF, EPUB, MOBI) or donate

22. Generators

22.1 Overview

22.1.1 What are generators?

You can think of generators as processes (pieces of code) that you can pause and resume:

function* genFunc() {
    // (A)
    console.log('First');
    yield;
    console.log('Second');
}

Note the new syntax: function* is a new “keyword” for generator functions (there are also generator methods). yield is an operator with which a generator can pause itself. Additionally, generators can also receive input and send output via yield.

When you call a generator function genFunc(), you get a generator object genObj that you can use to control the process:

const genObj = genFunc();

The process is initially paused in line A. genObj.next() resumes execution, a yield inside genFunc() pauses execution:

genObj.next();
// Output: First
genObj.next();
// output: Second

22.1.2 Kinds of generators

There are four kinds of generators:

• Generator function declarations:

 function* genFunc() { ··· }
 const genObj = genFunc();

• Generator function expressions:

 const genFunc = function* () { ··· };
 const genObj = genFunc();

• Generator method definitions in object literals:

 const obj = {
     * generatorMethod() {
         ···
     }
 };
 const genObj = obj.generatorMethod();

• Generator method definitions in class definitions (class declarations or class expressions):

 class MyClass {
     * generatorMethod() {
         ···
     }
 }
 const myInst = new MyClass();
 const genObj = myInst.generatorMethod();

22.1.3 Use case: implementing iterables

The objects returned by generators are iterable; each yield contributes to the sequence of iterated values. Therefore, you can use generators to implement iterables, which can be consumed by various ES6 language mechanisms: for-of loop, spread operator (…), etc.

The following function returns an iterable over the properties of an object, one [key, value] pair per property:

function* objectEntries(obj) {
    const propKeys = Reflect.ownKeys(obj);
 
    for (const propKey of propKeys) {
        // `yield` returns a value and then pauses
        // the generator. Later, execution continues
        // where it was previously paused.
        yield [propKey, obj[propKey]];
    }
}

objectEntries() is used like this:

const jane = { first: 'Jane', last: 'Doe' };
for (const [key,value] of objectEntries(jane)) {
    console.log(`${key}: ${value}`);
}
// Output:
// first: Jane
// last: Doe

How exactly objectEntries() works is explained in a dedicated section. Implementing the same functionality without generators is much more work.

22.1.4 Use case: simpler asynchronous code

You can use generators to tremendously simplify working with Promises. Let’s look at a Promise-based function fetchJson() and how it can be improved via generators.

function fetchJson(url) {
    return fetch(url)
    .then(request => request.text())
    .then(text => {
        return JSON.parse(text);
    })
    .catch(error => {
        console.log(`ERROR: ${error.stack}`);
    });
}

With the library co and a generator, this asynchronous code looks synchronous:

const fetchJson = co.wrap(function* (url) {
    try {
        let request = yield fetch(url);
        let text = yield request.text();
        return JSON.parse(text);
    }
    catch (error) {
        console.log(`ERROR: ${error.stack}`);
    }
});

ECMAScript 2017 will have async functions which are internally based on generators. With them, the code looks like this:

async function fetchJson(url) {
    try {
        let request = await fetch(url);
        let text = await request.text();
        return JSON.parse(text);
    }
    catch (error) {
        console.log(`ERROR: ${error.stack}`);
    }
}

All versions can be invoked like this:

fetchJson('http://example.com/some_file.json')
.then(obj => console.log(obj));

22.1.5 Use case: receiving asynchronous data

Generators can receive input from next() via yield. That means that you can wake up a generator whenever new data arrives asynchronously and to the generator it feels like it receives the data synchronously.

22.2 What are generators?

Generators are functions that can be paused and resumed (think cooperative multitasking or coroutines), which enables a variety of applications.

As a first example, consider the following generator function whose name is genFunc:

function* genFunc() {
    // (A)
    console.log('First');
    yield; // (B)
    console.log('Second'); // (C)
}

Two things distinguish genFunc from a normal function declaration:

• It starts with the “keyword” function*.
• It can pause itself, via yield (line B). Calling genFunc does not execute its body. Instead, you get a so-called generator object, with which you can control the execution of the body:

> const genObj = genFunc();

genFunc() is initially suspended before the body (line A). The method call genObj.next() continues execution until the next yield:

> genObj.next()
First
{ value: undefined, done: false }

As you can see in the last line, genObj.next() also returns an object. Let’s ignore that for now. It will matter later.

genFunc is now paused in line B. If we call next() again, execution resumes and line C is executed:

> genObj.next()
Second
{ value: undefined, done: true }

Afterwards, the function is finished, execution has left the body and further calls of genObj.next() have no effect.

22.2.1 Roles played by generators

Generators can play three roles:

• Iterators (data producers): Each yield can return a value via next(), which means that generators can produce sequences of values via loops and recursion. Due to generator objects implementing the interface Iterable (which is explained in the chapter on iteration), these sequences can be processed by any ECMAScript 6 construct that supports iterables. Two examples are: for-of loops and the spread operator (…).
• Observers (data consumers): yield can also receive a value from next() (via a parameter). That means that generators become data consumers that pause until a new value is pushed into them via next().
• Coroutines (data producers and consumers): Given that generators are pausable and can be both data producers and data consumers, not much work is needed to turn them into coroutines (cooperatively multitasked tasks). The next sections provide deeper explanations of these roles.

22.3 Generators as iterators (data production)

interface Iterable {
    Symbol.iterator : Iterator;
}
interface Iterator {
    next() : IteratorResult;
}
interface IteratorResult {
    value : any;
    done : boolean;
}

function genFunc() {
    yield 'a';
    yield 'b';
}

> const genObj = genFunc();
> genObj.next()
{ value: 'a', done: false }
> genObj.next()
{ value: 'b', done: false }
> genObj.next() // done: true => end of sequence
{ value: undefined, done: true }

for (const x of genFunc()) {
    console.log(x);
}
// Output:
// a
// b

const arr = […genFunc()]; // ['a', 'b']

> const [x, y] = genFunc();
> x
'a'
> y
'b'

function genFuncWithReturn() {
    yield 'a';
    yield 'b';
    return 'result';
}

> const genObjWithReturn = genFuncWithReturn();
> genObjWithReturn.next()
{ value: 'a', done: false }
> genObjWithReturn.next()
{ value: 'b', done: false }
> genObjWithReturn.next()
{ value: 'result', done: true }

for (const x of genFuncWithReturn()) {
    console.log(x);
}
// Output:
// a
// b
 
const arr = […genFuncWithReturn()]; // ['a', 'b']

function genFunc() {
    throw new Error('Problem!');
}
const genObj = genFunc();
genObj.next(); // Error: Problem!

function objectEntries(obj) {
    // In ES6, you can use strings or symbols as property keys,
    // Reflect.ownKeys() retrieves both
    const propKeys = Reflect.ownKeys(obj);
    for (const propKey of propKeys) {
        yield [propKey, obj[propKey]];
    }
}

const jane = { first: 'Jane', last: 'Doe' };
for (const [key,value] of objectEntries(jane)) {
    console.log(</code><code>${</code><code>key</code><code>}</code><code>: </code><code>${</code><code>value</code><code>}</code><code>);
}
// Output:
// first: Jane
// last: Doe

function objectEntries(obj) {
    let index = 0;
    let propKeys = Reflect.ownKeys(obj);
 
    return {
        Symbol.iterator {
            return this;
        },
        next() {
            if (index < propKeys.length) {
                let key = propKeys[index];
                index++;
                return { value: [key, obj[key]] };
            } else {
                return { done: true };
            }
        }
    };
}

function genFunc() {
    ['a', 'b'].forEach(x => yield x); // SyntaxError
}

function genFunc() {
    for (const x of ['a', 'b']) {
        yield x; // OK
    }
}

function foo() {
    yield 'a';
    yield 'b';
}

function bar() {
    yield 'x';
    foo(); // does nothing!
    yield 'y';
}

function bar() {
    yield 'x';
    yield foo();
    yield 'y';
}
// Collect all values yielded by bar() in an array
const arr = […bar()];
    // ['x', 'a', 'b', 'y']

function bar() {
    yield 'x';
    for (const value of foo()) {
        yield value;
    }
    yield 'y';
}

function bla() {
    yield 'sequence';
    yield ['of', 'yielded'];
    yield 'values';
}
 
const arr = […bla()];
    // ['sequence', 'of', 'yielded', 'values']

function genFuncWithReturn() {
    yield 'a';
    yield 'b';
    return 'The result';
}
function logReturned(genObj) {
    const result = yield genObj;
    console.log(result); // (A)
}

> […logReturned(genFuncWithReturn())]
The result
[ 'a', 'b' ]

class BinaryTree {
    constructor(value, left=null, right=null) {
        this.value = value;
        this.left = left;
        this.right = right;
    }
    /** Prefix iteration /
     Symbol.iterator {
        yield this.value;
        if (this.left) {
            yield this.left;
            // Short for: yield this.leftSymbol.iterator
        }
        if (this.right) {
            yield this.right;
        }
    }
}

const tree = new BinaryTree('a',
    new BinaryTree('b',
        new BinaryTree('c'),
        new BinaryTree('d')),
    new BinaryTree('e'));
 
for (const x of tree) {
    console.log(x);
}
// Output:
// a
// b
// c
// d
// e

interface Observer {
    next(value? : any) : void;
    return(value? : any) : void;
    throw(error) : void;
}

function dataConsumer() {
    console.log('Started');
    console.log(1. </code><code>${</code><code>yield</code><code>}</code><code>); // (A)
    console.log(2. </code><code>${</code><code>yield</code><code>}</code><code>);
    return 'result';
}

> const genObj = dataConsumer();

> genObj.next()
Started
{ value: undefined, done: false }

> genObj.next('a')
1. a
{ value: undefined, done: false }
 
> genObj.next('b')
2. b
{ value: 'result', done: true }

function gen() {
    // (A)
    while (true) {
        const input = yield; // (B)
        console.log(input);
    }
}
const obj = gen();
obj.next('a');
obj.next('b');
 
// Output:
// b

/*
  Returns a function that, when called,
  returns a generator object that is immediately
  ready for input via next()
 /
function coroutine(generatorFunction) {
    return function (…args) {
        const generatorObject = generatorFunction(…args);
        generatorObject.next();
        return generatorObject;
    };
}

const wrapped = coroutine(function () {
    console.log(First input: </code><code>${</code><code>yield</code><code>}</code><code>);
    return 'DONE';
});
const normal = function () {
    console.log(First input: </code><code>${</code><code>yield</code><code>}</code><code>);
    return 'DONE';
};

> wrapped().next('hello!')
First input: hello!

> const genObj = normal();
> genObj.next()
{ value: undefined, done: false }
> genObj.next('hello!')
First input: hello!
{ value: 'DONE', done: true }

yield a + b + c;

yield (a + b + c);

(yield a) + b + c;

console.log('Hello' + yield); // SyntaxError
console.log('Hello' + yield 123); // SyntaxError
 
console.log('Hello' + (yield)); // OK
console.log('Hello' + (yield 123)); // OK

foo(yield 'a', yield 'b');

const input = yield;

Expression :
    AssignmentExpression
    Expression , AssignmentExpression
AssignmentExpression :
    ConditionalExpression
    YieldExpression
    ArrowFunction
    LeftHandSideExpression = AssignmentExpression
    LeftHandSideExpression AssignmentOperator AssignmentExpression
 
···
 
AdditiveExpression :
    MultiplicativeExpression
    AdditiveExpression + MultiplicativeExpression
    AdditiveExpression - MultiplicativeExpression
MultiplicativeExpression :
    UnaryExpression
    MultiplicativeExpression MultiplicativeOperator UnaryExpression
 
···
 
PrimaryExpression :
    this
    IdentifierReference
    Literal
    ArrayLiteral
    ObjectLiteral
    FunctionExpression
    ClassExpression
    GeneratorExpression
    RegularExpressionLiteral
    TemplateLiteral
    ParenthesizedExpression
ParenthesizedExpression :
    ( Expression )

function genFunc1() {
    try {
        console.log('Started');
        yield; // (A)
    } finally {
        console.log('Exiting');
    }
}

> const genObj1 = genFunc1();
> genObj1.next()
Started
{ value: undefined, done: false }
> genObj1.return('Result')
Exiting
{ value: 'Result', done: true }

function genFunc2() {
    try {
        console.log('Started');
        yield;
    } finally {
        yield 'Not done, yet!';
    }
}

> const genObj2 = genFunc2();
 
> genObj2.next()
Started
{ value: undefined, done: false }
 
> genObj2.return('Result')
{ value: 'Not done, yet!', done: false }

> genObj2.next()
{ value: 'Result', done: true }

> function genFunc() {}
> genFunc().return('yes')
{ value: 'yes', done: true }

22.4.5 throw() signals an error

throw() throws an exception at the location of the yield that led to the last suspension of the generator. Let’s examine how that works via the following generator function.

function* genFunc1() {
    try {
        console.log('Started');
        yield; // (A)
    } catch (error) {
        console.log('Caught: ' + error);
    }
}

In the following interaction, we first use next() to start the generator and proceed until the yield in line A. Then we throw an exception from that location.

> const genObj1 = genFunc1();
 
> genObj1.next()
Started
{ value: undefined, done: false }
 
> genObj1.throw(new Error('Problem!'))
Caught: Error: Problem!
{ value: undefined, done: true }

The result of throw() (shown in the last line) stems from us leaving the function with an implicit return.

22.4.5.1 Throwing from a newborn generator

Throwing an exception in a newborn generator (that hasn’t started yet) is allowed:

> function* genFunc() {}
> genFunc().throw(new Error('Problem!'))
Error: Problem!

22.4.6 Example: processing asynchronously pushed data

The fact that generators-as-observers pause while they wait for input makes them perfect for on-demand processing of data that is received asynchronously. The pattern for setting up a chain of generators for processing is as follows:

• Each member of the chain of generators (except the last one) has a parameter target. It receives data via yield and sends data via target.next().
• The last member of the chain of generators has no parameter target and only receives data. The whole chain is prefixed by a non-generator function that makes an asynchronous request and pushes the results into the chain of generators via next().

As an example, let’s chain generators to process a file that is read asynchronously.

readFile(fileName, splitLines(numberLines(printLines())));

import {createReadStream} from 'fs';
 
/
  Creates an asynchronous ReadStream for the file whose name
  is fileName and feeds it to the generator object target.
 
  @see ReadStream https://nodejs.org/api/fs.html#fs_class_fs_readstream
 */
function readFile(fileName, target) {
    const readStream = createReadStream(fileName,
        { encoding: 'utf8', bufferSize: 1024 });
    readStream.on('data', buffer => {
        const str = buffer.toString('utf8');
        target.next(str);
    });
    readStream.on('end', () => {
        // Signal end of output sequence
        target.return();
    });
}

/
  Turns a sequence of text chunks into a sequence of lines
  (where lines are separated by newlines)
 /
const splitLines = coroutine(function (target) {
    let previous = '';
    try {
        while (true) {
            previous += yield;
            let eolIndex;
            while ((eolIndex = previous.indexOf('\n')) >= 0) {
                const line = previous.slice(0, eolIndex);
                target.next(line);
                previous = previous.slice(eolIndex+1);
            }
        }
    } finally {
        // Handle the end of the input sequence
        // (signaled via return())
        if (previous.length > 0) {
            target.next(previous);
        }
        // Signal end of output sequence
        target.return();
    }
});

//
  Prefixes numbers to a sequence of lines
 /
const numberLines = coroutine(function* (target) {
    try {
        for (const lineNo = 0; ; lineNo++) {
            const line = yield;
            target.next(</code><code>${</code><code>lineNo</code><code>}</code><code>: </code><code>${</code><code>line</code><code>}</code><code>);
        }
    } finally {
        // Signal end of output sequence
        target.return();
    }
});

/
  Receives a sequence of lines (without newlines)
  and logs them (adding newlines).
 /
const printLines = coroutine(function () {
    while (true) {
        const line = yield;
        console.log(line);
    }
});

function callee() {
    console.log('callee: ' + (yield));
}
function caller() {
    while (true) {
        yield callee();
    }
}

> const callerObj = caller();
 
> callerObj.next() // start
{ value: undefined, done: false }
 
> callerObj.next('a')
callee: a
{ value: undefined, done: false }
 
> callerObj.next('b')
callee: b
{ value: undefined, done: false }

let yieldStarResult = yield calleeFunc();

let yieldStarResult;
 
const calleeObj = calleeFunc();
let prevReceived = undefined;
while (true) {
    try {
        // Forward input previously received
        const {value,done} = calleeObj.next(prevReceived);
        if (done) {
            yieldStarResult = value;
            break;
        }
        prevReceived = yield value;
    } catch (e) {
        // Pretend return can be caught like an exception
        if (e instanceof Return) {
            // Forward input received via return()
            calleeObj.return(e.returnedValue);
            return e.returnedValue; // “re-throw”
        } else {
            // Forward input received via throw()
            calleeObj.throw(e); // may throw
        }
    }
}

interface Generator {
    next(value? : any) : IteratorResult;
    throw(value? : any) : IteratorResult;
    return(value? : any) : IteratorResult;
}
interface IteratorResult {
    value : any;
    done : boolean;
}

interface Iterator { // data producer
    next() : IteratorResult;
    return?(value? : any) : IteratorResult;
}
 
interface Observer { // data consumer
    next(value? : any) : void;
    return(value? : any) : void;
    throw(error) : void;
}

import fetch from 'isomorphic-fetch';
const co = require('co');

function getFile(url) {
    return fetch(url)
        .then(request => request.text());
}

co(function () {
    try {
        const [croftStr, bondStr] = yield Promise.all([  // (A)
            getFile('http://localhost:8000/croft.json'),
            getFile('http://localhost:8000/bond.json'),
        ]);
        const croftJson = JSON.parse(croftStr);
        const bondJson = JSON.parse(bondStr);
 
        console.log(croftJson);
        console.log(bondJson);
    } catch (e) {
        console.log('Failure to read: ' + e);
    }
});

function co(genFunc) {
    const genObj = genFunc();
    step(genObj.next());
 
    function step({value,done}) {
        if (!done) {
            // A Promise was yielded
            value
            .then(result => {
                step(genObj.next(result)); // (A)
            })
            .catch(error => {
                step(genObj.throw(error)); // (B)
            });
        }
    }
}

step1(sharedData);
step2(sharedData);
lastStep(sharedData);

22.6.1 Implementing iterables via generators

In the chapter on iteration, I implemented several iterables “by hand”. In this section, I use generators, instead.

22.6.1.1 The iterable combinator take()

take() converts a (potentially infinite) sequence of iterated values into a sequence of length n:

function* take(n, iterable) {
    for (const x of iterable) {
        if (n <= 0) return;
        n--;
        yield x;
    }
}

The following is an example of using it:

const arr = ['a', 'b', 'c', 'd'];
for (const x of take(2, arr)) {
    console.log(x);
}
// Output:
// a
// b

An implementation of take() without generators is more complicated:

function take(n, iterable) {
    const iter = iterable[Symbol.iterator]();
    return {
        [Symbol.iterator]() {
            return this;
        },
        next() {
            if (n > 0) {
                n--;
                return iter.next();
            } else {
                maybeCloseIterator(iter);
                return { done: true };
            }
        },
        return() {
            n = 0;
            maybeCloseIterator(iter);
        }
    };
}
function maybeCloseIterator(iterator) {
    if (typeof iterator.return === 'function') {
        iterator.return();
    }
}