Source Edit

Regular expression support for Nim.

This module is implemented by providing a wrapper around the PCRE (Perl-Compatible Regular Expressions) C library. This means that your application will depend on the PCRE library’s licence when using this module, which should not be a problem though.

Note: There are also alternative nimble packages such as tinyre and regex.

PCRE’s licence follows:

Licence of the PCRE library

PCRE is a library of functions to support regular expressions whose syntax and semantics are as close as possible to those of the Perl 5 language.

Written by Philip Hazel
Copyright (c) 1997-2005 University of Cambridge


Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of the University of Cambridge nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Regular expression syntax and semantics

As the regular expressions supported by this module are enormous, the reader is referred to http://perldoc.perl.org/perlre.html for the full documentation of Perl’s regular expressions.

Because the backslash \ is a meta character both in the Nim programming language and in regular expressions, it is strongly recommended that one uses the raw strings of Nim, so that backslashes are interpreted by the regular expression engine:

  1. r"\S" # matches any character that is not whitespace

A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern:

  1. The quick brown fox

matches a portion of a subject string that is identical to itself. The power of regular expressions comes from the ability to include alternatives and repetitions in the pattern. These are encoded in the pattern by the use of metacharacters, which do not stand for themselves but instead are interpreted in some special way.

There are two different sets of metacharacters: those that are recognized anywhere in the pattern except within square brackets, and those that are recognized in square brackets. Outside square brackets, the metacharacters are as follows:

meta charactermeaning
\general escape character with several uses
^assert start of string (or line, in multiline mode)
$assert end of string (or line, in multiline mode)
.match any character except newline (by default)
[start character class definition
|start of alternative branch
(start subpattern
)end subpattern
{start min/max quantifier
?extends the meaning of (

also 0 or 1 quantifier (equal to {0,1})
also quantifier minimizer

*0 or more quantifier (equal to {0,})
+1 or more quantifier (equal to {1,})

also “possessive quantifier”

Part of a pattern that is in square brackets is called a “character class”. In a character class the only metacharacters are:

meta charactermeaning
\general escape character
^negate the class, but only if the first character
-indicates character range
[POSIX character class (only if followed by POSIX syntax)
]terminates the character class

The following sections describe the use of each of the metacharacters.

Backslash

The backslash character has several uses. Firstly, if it is followed by a non-alphanumeric character, it takes away any special meaning that character may have. This use of backslash as an escape character applies both inside and outside character classes.

For example, if you want to match a * character, you write \* in the pattern. This escaping action applies whether or not the following character would otherwise be interpreted as a metacharacter, so it is always safe to precede a non-alphanumeric with backslash to specify that it stands for itself. In particular, if you want to match a backslash, you write \\.

Non-printing characters

A second use of backslash provides a way of encoding non-printing characters in patterns in a visible manner. There is no restriction on the appearance of non-printing characters, apart from the binary zero that terminates a pattern, but when a pattern is being prepared by text editing, it is usually easier to use one of the following escape sequences than the binary character it represents:

charactermeaning
\aalarm, that is, the BEL character (hex 07)
\eescape (hex 1B)
\fformfeed (hex 0C)
\nnewline (hex 0A)
\rcarriage return (hex 0D)
\ttab (hex 09)
\dddcharacter with octal code ddd, or backreference
\xhhcharacter with hex code hh

After \x, from zero to two hexadecimal digits are read (letters can be in upper or lower case). In UTF-8 mode, any number of hexadecimal digits may appear between \x{ and }, but the value of the character code must be less than 2^31 (that is, the maximum hexadecimal value is 7FFFFFFF). If characters other than hexadecimal digits appear between \x{ and }, or if there is no terminating }, this form of escape is not recognized. Instead, the initial \x will be interpreted as a basic hexadecimal escape, with no following digits, giving a character whose value is zero.

After \0 up to two further octal digits are read. In both cases, if there are fewer than two digits, just those that are present are used. Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL character (code value 7). Make sure you supply two digits after the initial zero if the pattern character that follows is itself an octal digit.

The handling of a backslash followed by a digit other than 0 is complicated. Outside a character class, PCRE reads it and any following digits as a decimal number. If the number is less than 10, or if there have been at least that many previous capturing left parentheses in the expression, the entire sequence is taken as a back reference. A description of how this works is given later, following the discussion of parenthesized subpatterns.

Inside a character class, or if the decimal number is greater than 9 and there have not been that many capturing subpatterns, PCRE re-reads up to three octal digits following the backslash, and generates a single byte from the least significant 8 bits of the value. Any subsequent digits stand for themselves. For example:

examplemeaning
\040is another way of writing a space
\40is the same, provided there are fewer than 40 previous capturing subpatterns
\7is always a back reference
\11might be a back reference, or another way of writing a tab
\011is always a tab
\0113is a tab followed by the character “3”
\113might be a back reference, otherwise the character with octal code 113
\377might be a back reference, otherwise the byte consisting entirely of 1 bits
\81is either a back reference, or a binary zero followed by the two characters “8” and “1”

Note that octal values of 100 or greater must not be introduced by a leading zero, because no more than three octal digits are ever read.

All the sequences that define a single byte value or a single UTF-8 character (in UTF-8 mode) can be used both inside and outside character classes. In addition, inside a character class, the sequence \b is interpreted as the backspace character (hex 08), and the sequence \X is interpreted as the character “X”. Outside a character class, these sequences have different meanings (see below).

Generic character types

The third use of backslash is for specifying generic character types. The following are always recognized:

character typemeaning
\dany decimal digit
\Dany character that is not a decimal digit
\sany whitespace character
\Sany character that is not a whitespace character
\wany “word” character
\Wany “non-word” character

Each pair of escape sequences partitions the complete set of characters into two disjoint sets. Any given character matches one, and only one, of each pair.

These character type sequences can appear both inside and outside character classes. They each match one character of the appropriate type. If the current matching point is at the end of the subject string, all of them fail, since there is no character to match.

For compatibility with Perl, \s does not match the VT character (code 11). This makes it different from the POSIX “space” class. The \s characters are HT (9), LF (10), FF (12), CR (13), and space (32).

A “word” character is an underscore or any character less than 256 that is a letter or digit. The definition of letters and digits is controlled by PCRE’s low-valued character tables, and may vary if locale-specific matching is taking place (see “Locale support” in the pcreapi page). For example, in the “fr_FR” (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.

In UTF-8 mode, characters with values greater than 128 never match \d, \s, or \w, and always match \D, \S, and \W. This is true even when Unicode character property support is available.

Simple assertions

The fourth use of backslash is for certain simple assertions. An assertion specifies a condition that has to be met at a particular point in a match, without consuming any characters from the subject string. The use of subpatterns for more complicated assertions is described below. The backslashed assertions are:

assertionmeaning
\bmatches at a word boundary
\Bmatches when not at a word boundary
\Amatches at start of subject
\Zmatches at end of subject or before newline at end
\zmatches at end of subject
\Gmatches at first matching position in subject

These assertions may not appear in character classes (but note that \b has a different meaning, namely the backspace character, inside a character class).

A word boundary is a position in the subject string where the current character and the previous character do not both match \w or \W (i.e. one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively.

The \A, \Z, and \z assertions differ from the traditional circumflex and dollar in that they only ever match at the very start and end of the subject string, whatever options are set. The difference between \Z and \z is that \Z matches before a newline that is the last character of the string as well as at the end of the string, whereas \z matches only at the end.

Example:

  1. import std/re
  2. ## Unless specified otherwise, `start` parameter in each proc indicates
  3. ## where the scan starts, but outputs are relative to the start of the input
  4. ## string, not to `start`:
  5. doAssert find("uxabc", re"(?<=x|y)ab", start = 1) == 2 # lookbehind assertion
  6. doAssert find("uxabc", re"ab", start = 3) == -1 # we're past `start` => not found
  7. doAssert not match("xabc", re"^abc$", start = 1)
  8. # can't match start of string since we're starting at 1

Imports

pcre, strutils, rtarrays

Types

  1. Regex = ref RegexDesc

a compiled regular expression Source Edit

  1. RegexError = object of ValueError

is raised if the pattern is no valid regular expression. Source Edit

  1. RegexFlag = enum
  2. reIgnoreCase = 0, ## do caseless matching
  3. reMultiLine = 1, ## `^` and `$` match newlines within data
  4. reDotAll = 2, ## `.` matches anything including NL
  5. reExtended = 3, ## ignore whitespace and `#` comments
  6. reStudy = 4 ## study the expression (may be omitted if the
  7. ## expression will be used only once)

options for regular expressions Source Edit

Consts

  1. MaxReBufSize = 2147483647'i32

Maximum PCRE (API 1) buffer start/size equal to high(cint), which even for 64-bit systems can be either 231-1 or 263-1. Source Edit

  1. MaxSubpatterns = 20

defines the maximum number of subpatterns that can be captured. This limit still exists for replacef and parallelReplace. Source Edit

Procs

  1. proc contains(s: string; pattern: Regex; matches: var openArray[string];
  2. start = 0): bool {.inline, ...raises: [], tags: [], forbids: [].}

same as find(s, pattern, matches, start) >= 0

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc contains(s: string; pattern: Regex; start = 0): bool {.inline, ...raises: [],
  2. tags: [], forbids: [].}

same as find(s, pattern, start) >= 0 Source Edit

  1. proc endsWith(s: string; suffix: Regex): bool {.inline, ...raises: [], tags: [],
  2. forbids: [].}

returns true if s ends with the pattern suffix Source Edit

  1. proc escapeRe(s: string): string {....raises: [], tags: [], forbids: [].}

escapes s so that it is matched verbatim when used as a regular expression. Source Edit

  1. proc find(buf: cstring; pattern: Regex; matches: var openArray[string];
  2. start = 0; bufSize: int): int {....raises: [], tags: [], forbids: [].}

returns the starting position of pattern in buf and the captured substrings in the array matches. If it does not match, nothing is written into matches and -1 is returned. buf has length bufSize (not necessarily ‘\0’ terminated).

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc find(buf: cstring; pattern: Regex; start = 0; bufSize: int): int {.
  2. ...raises: [], tags: [], forbids: [].}

returns the starting position of pattern in buf, where buf has length bufSize (not necessarily ‘\0’ terminated). If it does not match, -1 is returned. Source Edit

  1. proc find(s: string; pattern: Regex; matches: var openArray[string]; start = 0): int {.
  2. inline, ...raises: [], tags: [], forbids: [].}

returns the starting position of pattern in s and the captured substrings in the array matches. If it does not match, nothing is written into matches and -1 is returned.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc find(s: string; pattern: Regex; start = 0): int {.inline, ...raises: [],
  2. tags: [], forbids: [].}

returns the starting position of pattern in s. If it does not match, -1 is returned. We start the scan at start.

Example:

  1. doAssert find("abcdefg", re"cde") == 2
  2. doAssert find("abcdefg", re"abc") == 0
  3. doAssert find("abcdefg", re"zz") == -1 # not found
  4. doAssert find("abcdefg", re"cde", start = 2) == 2 # still 2
  5. doAssert find("abcdefg", re"cde", start = 3) == -1 # we're past the start position
  6. doAssert find("xabc", re"(?<=x|y)abc", start = 1) == 1
  7. # lookbehind assertion `(?<=x|y)` can look behind `start`

Source Edit

  1. proc findAll(s: string; pattern: Regex; start = 0): seq[string] {.inline,
  2. ...raises: [], tags: [], forbids: [].}

returns all matching substrings of s that match pattern. If it does not match, @[] is returned. Source Edit

  1. proc findBounds(buf: cstring; pattern: Regex;
  2. matches: var openArray[tuple[first, last: int]]; start = 0;
  3. bufSize: int): tuple[first, last: int] {....raises: [], tags: [],
  4. forbids: [].}

returns the starting position and end position of pattern in buf (where buf has length bufSize and is not necessarily ‘\0’ terminated), and the captured substrings in the array matches. If it does not match, nothing is written into matches and (-1,0) is returned.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc findBounds(buf: cstring; pattern: Regex; matches: var openArray[string];
  2. start = 0; bufSize: int): tuple[first, last: int] {....raises: [],
  3. tags: [], forbids: [].}

returns the starting position and end position of pattern in buf (where buf has length bufSize and is not necessarily ‘\0’ terminated), and the captured substrings in the array matches. If it does not match, nothing is written into matches and (-1,0) is returned.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc findBounds(buf: cstring; pattern: Regex; start = 0; bufSize: int): tuple[
  2. first, last: int] {....raises: [], tags: [], forbids: [].}

returns the first and last position of pattern in buf, where buf has length bufSize (not necessarily ‘\0’ terminated). If it does not match, (-1,0) is returned. Source Edit

  1. proc findBounds(s: string; pattern: Regex;
  2. matches: var openArray[tuple[first, last: int]]; start = 0): tuple[
  3. first, last: int] {.inline, ...raises: [], tags: [], forbids: [].}

returns the starting position and end position of pattern in s and the captured substrings in the array matches. If it does not match, nothing is written into matches and (-1,0) is returned.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Example:

  1. var matches = newSeq[tuple[first, last: int]](1)
  2. let (first, last) = findBounds("Hello World", re"(\w+)", matches)
  3. doAssert first == 0
  4. doAssert last == 4
  5. doAssert matches[0] == (0, 4)

Source Edit

  1. proc findBounds(s: string; pattern: Regex; matches: var openArray[string];
  2. start = 0): tuple[first, last: int] {.inline, ...raises: [],
  3. tags: [], forbids: [].}

returns the starting position and end position of pattern in s and the captured substrings in the array matches. If it does not match, nothing is written into matches and (-1,0) is returned.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Example:

  1. var matches = newSeq[string](1)
  2. let (first, last) = findBounds("Hello World", re"(W\w+)", matches)
  3. doAssert first == 6
  4. doAssert last == 10
  5. doAssert matches[0] == "World"

Source Edit

  1. proc findBounds(s: string; pattern: Regex; start = 0): tuple[first, last: int] {.
  2. inline, ...raises: [], tags: [], forbids: [].}

returns the first and last position of pattern in s. If it does not match, (-1,0) is returned.

Note: there is a speed improvement if the matches do not need to be captured.

Example:

  1. assert findBounds("01234abc89", re"abc") == (5,7)

Source Edit

  1. proc match(buf: cstring; pattern: Regex; matches: var openArray[string];
  2. start = 0; bufSize: int): bool {.inline, ...raises: [], tags: [],
  3. forbids: [].}

returns true if buf[start..<bufSize] matches the pattern and the captured substrings in the array matches. If it does not match, nothing is written into matches and false is returned. buf has length bufSize (not necessarily ‘\0’ terminated).

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc match(s: string; pattern: Regex; matches: var openArray[string]; start = 0): bool {.
  2. inline, ...raises: [], tags: [], forbids: [].}

returns true if s[start..] matches the pattern and the captured substrings in the array matches. If it does not match, nothing is written into matches and false is returned.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Example:

  1. import std/sequtils
  2. var matches: array[2, string]
  3. if match("abcdefg", re"c(d)ef(g)", matches, 2):
  4. doAssert toSeq(matches) == @["d", "g"]

Source Edit

  1. proc match(s: string; pattern: Regex; start = 0): bool {.inline, ...raises: [],
  2. tags: [], forbids: [].}

returns true if s[start..] matches the pattern. Source Edit

  1. proc matchLen(buf: cstring; pattern: Regex; matches: var openArray[string];
  2. start = 0; bufSize: int): int {.inline, ...raises: [], tags: [],
  3. forbids: [].}

the same as match, but it returns the length of the match, if there is no match, -1 is returned. Note that a match length of zero can happen.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc matchLen(buf: cstring; pattern: Regex; start = 0; bufSize: int): int {.
  2. inline, ...raises: [], tags: [], forbids: [].}

the same as match, but it returns the length of the match, if there is no match, -1 is returned. Note that a match length of zero can happen. Source Edit

  1. proc matchLen(s: string; pattern: Regex; matches: var openArray[string];
  2. start = 0): int {.inline, ...raises: [], tags: [], forbids: [].}

the same as match, but it returns the length of the match, if there is no match, -1 is returned. Note that a match length of zero can happen.

Note: The memory for matches needs to be allocated before this function is called, otherwise it will just remain empty.

Source Edit

  1. proc matchLen(s: string; pattern: Regex; start = 0): int {.inline, ...raises: [],
  2. tags: [], forbids: [].}

the same as match, but it returns the length of the match, if there is no match, -1 is returned. Note that a match length of zero can happen.

Example:

  1. doAssert matchLen("abcdefg", re"cde", 2) == 3
  2. doAssert matchLen("abcdefg", re"abcde") == 5
  3. doAssert matchLen("abcdefg", re"cde") == -1

Source Edit

  1. proc multiReplace(s: string;
  2. subs: openArray[tuple[pattern: Regex, repl: string]]): string {.
  3. ...raises: [ValueError], tags: [], forbids: [].}

Returns a modified copy of s with the substitutions in subs applied in parallel. Source Edit

  1. proc re(s: string; flags = {reStudy}): Regex {....raises: [RegexError], tags: [],
  2. forbids: [].}

Constructor of regular expressions.

Note that Nim’s extended raw string literals support the syntax re”[abc]“ as a short form for re(r”[abc]“). Also note that since this compiles the regular expression, which is expensive, you should avoid putting it directly in the arguments of the functions like the examples show below if you plan to use it a lot of times, as this will hurt performance immensely. (e.g. outside a loop, …)

Source Edit

  1. proc replace(s: string; sub: Regex; by = ""): string {....raises: [], tags: [],
  2. forbids: [].}

Replaces sub in s by the string by. Captures cannot be accessed in by.

Example:

  1. doAssert "var1=key; var2=key2".replace(re"(\w+)=(\w+)") == "; "
  2. doAssert "var1=key; var2=key2".replace(re"(\w+)=(\w+)", "?") == "?; ?"

Source Edit

  1. proc replacef(s: string; sub: Regex; by: string): string {....raises: [ValueError],
  2. tags: [], forbids: [].}

Replaces sub in s by the string by. Captures can be accessed in by with the notation $i and $# (see strutils.`%`).

Example:

  1. doAssert "var1=key; var2=key2".replacef(re"(\w+)=(\w+)", "$1<-$2$2") ==
  2. "var1<-keykey; var2<-key2key2"

Source Edit

  1. proc rex(s: string; flags = {reStudy, reExtended}): Regex {.
  2. ...raises: [RegexError], tags: [], forbids: [].}

Constructor for extended regular expressions.

The extended means that comments starting with # and whitespace are ignored.

Source Edit

  1. proc split(s: string; sep: Regex; maxsplit = -1): seq[string] {.inline,
  2. ...raises: [], tags: [], forbids: [].}

Splits the string s into a seq of substrings.

The portion matched by sep is not returned.

Source Edit

  1. proc startsWith(s: string; prefix: Regex): bool {.inline, ...raises: [], tags: [],
  2. forbids: [].}

returns true if s starts with the pattern prefix Source Edit

  1. proc transformFile(infile, outfile: string;
  2. subs: openArray[tuple[pattern: Regex, repl: string]]) {.
  3. ...raises: [IOError, ValueError], tags: [ReadIOEffect, WriteIOEffect],
  4. forbids: [].}

reads in the file infile, performs a parallel replacement (calls parallelReplace) and writes back to outfile. Raises IOError if an error occurs. This is supposed to be used for quick scripting. Source Edit

Iterators

  1. iterator findAll(buf: cstring; pattern: Regex; start = 0; bufSize: int): string {.
  2. ...raises: [], tags: [], forbids: [].}

Yields all matching substrings of s that match pattern.

Note that since this is an iterator you should not modify the string you are iterating over: bad things could happen.

Source Edit

  1. iterator findAll(s: string; pattern: Regex; start = 0): string {....raises: [],
  2. tags: [], forbids: [].}

Yields all matching substrings of s that match pattern.

Note that since this is an iterator you should not modify the string you are iterating over: bad things could happen.

Source Edit

  1. iterator split(s: string; sep: Regex; maxsplit = -1): string {....raises: [],
  2. tags: [], forbids: [].}

Splits the string s into substrings.

Substrings are separated by the regular expression sep (and the portion matched by sep is not returned).

Example:

  1. import std/sequtils
  2. doAssert toSeq(split("00232this02939is39an22example111", re"\d+")) ==
  3. @["", "this", "is", "an", "example", ""]

Source Edit

Templates

  1. template `=~`(s: string; pattern: Regex): untyped

This calls match with an implicit declared matches array that can be used in the scope of the \=~ call:

Example:

  1. proc parse(line: string): string =
  2. if line =~ re"\s*(\w+)\s*\=\s*(\w+)": # matches a key=value pair:
  3. result = $(matches[0], matches[1])
  4. elif line =~ re"\s*(\#.*)": # matches a comment
  5. # note that the implicit `matches` array is different from 1st branch
  6. result = $(matches[0],)
  7. else: doAssert false
  8. doAssert not declared(matches)
  9. doAssert parse("NAME = LENA") == """("NAME", "LENA")"""
  10. doAssert parse(" # comment ... ") == """("# comment ... ",)"""

Source Edit