Source Edit

This module contains a scanf macro that can be used for extracting substrings from an input string. This is often easier than regular expressions. Some examples as an appetizer:

# check if input string matches a triple of integers:
const input = "(1,2,4)"
var x, y, z: int
if scanf(input, "($i,$i,$i)", x, y, z):
  echo "matches and x is ", x, " y is ", y, " z is ", z
# check if input string matches an ISO date followed by an identifier followed
# by whitespace and a floating point number:
var year, month, day: int
var identifier: string
var myfloat: float
if scanf(input, "$i-$i-$i $w$s$f", year, month, day, identifier, myfloat):
  echo "yes, we have a match!"

As can be seen from the examples, strings are matched verbatim except for substrings starting with $. These constructions are available:

Even though $* and $+ look similar to the regular expressions .* and .+, they work quite differently. There is no non-deterministic state machine involved and the matches are non-greedy. [$*] matches [xyz] via parseutils.parseUntil.

Furthermore no backtracking is performed, if parsing fails after a value has already been bound to a matched subexpression this value is not restored to its original value. This rarely causes problems in practice and if it does for you, it’s easy enough to bind to a temporary variable first.

Startswith vs full match

scanf returns true if the input string starts with the specified pattern. If instead it should only return true if there is also nothing left in the input, append $. to your pattern.

User definable matchers

One very nice advantage over regular expressions is that scanf is extensible with ordinary Nim procs. The proc is either enclosed in ${} or in $[]. ${} matches and binds the result to a variable (that was passed to the scanf macro) while $[] merely matches optional tokens without any result binding.

In this example, we define a helper proc someSep that skips some separators which we then use in our scanf pattern to help us in the matching process:

proc someSep(input: string; start: int; seps: set[char] = {':','-','.'}): int =
  # Note: The parameters and return value must match to what ``scanf`` requires
  result = 0
  while start+result < input.len and input[start+result] in seps: inc result
if scanf(input, "$w$[someSep]$w", key, value):
  ...

It also possible to pass arguments to a user definable matcher:

proc ndigits(input: string; intVal: var int; start: int; n: int): int =
  # matches exactly ``n`` digits. Matchers need to return 0 if nothing
  # matched or otherwise the number of processed chars.
  var x = 0
  var i = 0
  while i < n and i+start < input.len and input[i+start] in {'0'..'9'}:
    x = x * 10 + input[i+start].ord - '0'.ord
    inc i
  # only overwrite if we had a match
  if i == n:
    result = n
    intVal = x
# match an ISO date extracting year, month, day at the same time.
# Also ensure the input ends after the ISO date:
var year, month, day: int
if scanf("2013-01-03", "${ndigits(4)}-${ndigits(2)}-${ndigits(2)}$.", year, month, day):
  ...

The scanp macro

This module also implements a scanp macro, which syntax somewhat resembles an EBNF or PEG grammar, except that it uses Nim’s expression syntax and so has to use prefix instead of postfix operators.

Note that unordered or ordered choice operators (/, |) are not implemented.

Simple example that parses the /etc/passwd file line by line:

const
  etc_passwd = """root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
messagebus:x:103:107::/var/run/dbus:/bin/false
"""
proc parsePasswd(content: string): seq[string] =
  result = @[]
  var idx = 0
  while true:
    var entry = ""
    if scanp(content, idx, +(~{'\L', '\0'} -> entry.add($_)), '\L'):
      result.add entry
    else:
      break

The scanp maps the grammar code into Nim code that performs the parsing. The parsing is performed with the help of 3 helper templates that that can be implemented for a custom type.

These templates need to be named atom and nxt. atom should be overloaded to handle both single characters and sets of character.

import std/streams
template atom(input: Stream; idx: int; c: char): bool =
  ## Used in scanp for the matching of atoms (usually chars).
  peekChar(input) == c
template atom(input: Stream; idx: int; s: set[char]): bool =
  peekChar(input) in s
template nxt(input: Stream; idx, step: int = 1) =
  inc(idx, step)
  setPosition(input, idx)
if scanp(content, idx, +( ~{'\L', '\0'} -> entry.add(peekChar($input))), '\L'):
  result.add entry

Calling ordinary Nim procs inside the macro is possible:

proc digits(s: string; intVal: var int; start: int): int =
  var x = 0
  while result+start < s.len and s[result+start] in {'0'..'9'} and s[result+start] != ':':
    x = x * 10 + s[result+start].ord - '0'.ord
    inc result
  intVal = x
proc extractUsers(content: string): seq[string] =
  # Extracts the username and home directory
  # of each entry (with UID greater than 1000)
  const
    digits = {'0'..'9'}
  result = @[]
  var idx = 0
  while true:
    var login = ""
    var uid = 0
    var homedir = ""
    if scanp(content, idx, *(~ {':', '\0'}) -> login.add($_), ':', * ~ ':', ':',
            digits($input, uid, $index), ':', *`digits`, ':', * ~ ':', ':',
            *('/', * ~{':', '/'}) -> homedir.add($_), ':', *('/', * ~{'\L', '/'}), '\L'):
      if uid >= 1000:
        result.add login & " " & homedir
    else:
      break

When used for matching, keep in mind that likewise scanf, no backtracking is performed.

proc skipUntil(s: string; until: string; unless = '\0'; start: int): int =
  # Skips all characters until the string `until` is found. Returns 0
  # if the char `unless` is found first or the end is reached.
  var i = start
  var u = 0
  while true:
    if i >= s.len or s[i] == unless:
      return 0
    elif s[i] == until[0]:
      u = 1
      while i+u < s.len and u < until.len and s[i+u] == until[u]:
        inc u
      if u >= until.len: break
    inc(i)
  result = i+u-start
iterator collectLinks(s: string): string =
  const quote = {'\'', '"'}
  var idx, old = 0
  var res = ""
  while idx < s.len:
    old = idx
    if scanp(s, idx, "<a", skipUntil($input, "href=", '>', $index),
            `quote`, *( ~`quote`) -> res.add($_)):
      yield res
      res = ""
    idx = old + 1
for r in collectLinks(body):
  echo r

In this example both macros are combined seamlessly in order to maximise efficiency and perform different checks.

iterator parseIps*(soup: string): string =
  ## ipv4 only!
  const digits = {'0'..'9'}
  var a, b, c, d: int
  var buf = ""
  var idx = 0
  while idx < soup.len:
    if scanp(soup, idx, (`digits`{1,3}, '.', `digits`{1,3}, '.',
             `digits`{1,3}, '.', `digits`{1,3}) -> buf.add($_)):
      discard buf.scanf("$i.$i.$i.$i", a, b, c, d)
      if (a >= 0 and a <= 254) and
         (b >= 0 and b <= 254) and
         (c >= 0 and c <= 254) and
         (d >= 0 and d <= 254):
        yield buf
    buf.setLen(0) # need to clear `buf` each time, cause it might contain garbage
    idx.inc

Imports

macros, parseutils, since

Macros

macro scanf(input: string; pattern: static[string]; results: varargs[typed]): bool

See top level documentation of this module about how scanf works. Source Edit

macro scanp(input, idx: typed; pattern: varargs[untyped]): bool

See top level documentation of this module about how scanp works. Source Edit

macro scanTuple(input: untyped; pattern: static[string];
                matcherTypes: varargs[untyped]): untyped

Works identically as scanf, but instead of predeclaring variables it returns a tuple. Tuple is started with a bool which indicates if the scan was successful followed by the requested data. If using a user defined matcher, provide the types in order they appear after pattern: line.scanTuple(“${yourMatcher()}”, int)

Example:

let (success, year, month, day, time) = scanTuple("1000-01-01 00:00:00", "$i-$i-$i$s$+")
if success:
  assert year == 1000
  assert month == 1
  assert day == 1
  assert time == "00:00:00"

Source Edit

Templates

template atom(input: string; idx: int; c: char): bool

Used in scanp for the matching of atoms (usually chars). EOF is matched as ‘\0’. Source Edit

template atom(input: string; idx: int; s: set[char]): bool

Source Edit

template hasNxt(input: string; idx: int): bool

Source Edit

template nxt(input: string; idx, step: int = 1)

Source Edit

template success(x: int): bool

Source Edit