Source Edit

This module implements a base object of a lexer with efficient buffer handling. Only at line endings checks are necessary if the buffer needs refilling.

Imports

strutils, streams

Types

  1. BaseLexer = object of RootObj
  2.   bufpos*: int               ## the current position within the buffer
  3.   buf*: string               ## the buffer itself
  4.   ## the input stream
  5.   lineNumber*: int           ## the current line number
  6.   offsetBase*: int

the base lexer. Inherit your lexer from this object. Source Edit

Consts

  1. EndOfFile = '\x00'

end of file marker Source Edit 

  1. NewLines = {'\r', '\n'}

Source Edit

Procs

  1. proc close(L: var BaseLexer) {....raises: [IOError, OSError],
  2.                                tags: [WriteIOEffect], forbids: [].}

closes the base lexer. This closes L’s associated stream too. Source Edit 

  1. proc getColNumber(L: BaseLexer; pos: int): int {....raises: [], tags: [],
  2.     forbids: [].}

retrieves the current column. Source Edit 

  1. proc getCurrentLine(L: BaseLexer; marker: bool = true): string {....raises: [],
  2.     tags: [], forbids: [].}

retrieves the current line. Source Edit 

  1. proc handleCR(L: var BaseLexer; pos: int): int {....raises: [IOError, OSError],
  2.     tags: [ReadIOEffect], forbids: [].}

Call this if you scanned over ‘c’ in the buffer; it returns the position to continue the scanning from. pos must be the position of the ‘c’. Source Edit 

  1. proc handleLF(L: var BaseLexer; pos: int): int {....raises: [IOError, OSError],
  2.     tags: [ReadIOEffect], forbids: [].}

Call this if you scanned over ‘L’ in the buffer; it returns the position to continue the scanning from. pos must be the position of the ‘L’. Source Edit 

  1. proc handleRefillChar(L: var BaseLexer; pos: int): int {.
  2.     ...raises: [IOError, OSError], tags: [ReadIOEffect], forbids: [].}

Call this if a terminator character other than a new line is scanned at pos; it returns the position to continue the scanning from. Source Edit 

  1. proc open(L: var BaseLexer; input: Stream; bufLen: int = 8192;
  2.           refillChars: set[char] = NewLines) {....raises: [IOError, OSError],
  3.     tags: [ReadIOEffect], forbids: [].}

inits the BaseLexer with a stream to read from. Source Edit