Scanner

The source code for the TypeScript scanner is located entirely in scanner.ts. Scanner is controlled internally by the Parser to convert the source code to an AST. Here is what the desired outcome is.

  1. SourceCode ~~ scanner ~~> Token Stream ~~ parser ~~> AST

Usage by Parser

There is a singleton scanner created in parser.ts to avoid the cost of creating scanners over and over again. This scanner is then primed by the parser on demand using the initializeState function.

Here is a simplied version of the actual code in the parser that you can run demonstrating this concept:

code/compiler/scanner/runScanner.ts

  1. import * as ts from "ntypescript";
  3. // TypeScript has a singelton scanner
  4. const scanner = ts.createScanner(ts.ScriptTarget.Latest, /*skipTrivia*/ true);
  6. // That is initialized using a function `initializeState` similar to
  7. function initializeState(text: string) {
  8.     scanner.setText(text);
  9.     scanner.setOnError((message: ts.DiagnosticMessage, length: number) => {
  10.         console.error(message);
  11.     });
  12.     scanner.setScriptTarget(ts.ScriptTarget.ES5);
  13.     scanner.setLanguageVariant(ts.LanguageVariant.Standard);
  14. }
  16. // Sample usage
  17. initializeState(`
  18. var foo = 123;
  19. `.trim());
  21. // Start the scanning
  22. var token = scanner.scan();
  23. while (token != ts.SyntaxKind.EndOfFileToken) {
  24.     console.log(ts.formatSyntaxKind(token));
  25.     token = scanner.scan();
  26. }

This will print out the following :

  1. VarKeyword
  2. Identifier
  3. FirstAssignment
  4. FirstLiteralToken
  5. SemicolonToken

Scanner State

After you call scan the scanner updates its local state (position in the scan, current token details etc). The scanner provides a bunch of utility functions to get the current scanner state. In the below sample we create a scanner and then use it to identify the tokens as well as their positions in the code.

code/compiler/scanner/runScannerWithPosition.ts

  1. // Sample usage
  2. initializeState(`
  3. var foo = 123;
  4. `.trim());
  6. // Start the scanning
  7. var token = scanner.scan();
  8. while (token != ts.SyntaxKind.EndOfFileToken) {
  9.     let currentToken = ts.formatSyntaxKind(token);
  10.     let tokenStart = scanner.getStartPos();
  11.     token = scanner.scan();
  12.     let tokenEnd = scanner.getStartPos();
  13.     console.log(currentToken, tokenStart, tokenEnd);
  14. }

This will print out the following:

  1. VarKeyword 0 3
  2. Identifier 3 7
  3. FirstAssignment 7 9
  4. FirstLiteralToken 9 13
  5. SemicolonToken 13 14

Standalone scanner

Even though the typescript parser has a singleton scanner you can create a standalone scanner using createScanner and use its setText/setTextPos to scan at different points in a file for your amusement.