7.3 Regexp

Regular Expressions (“Regexp”) is a complicated but powerful tool for pattern matching and text manipulation. Although it does not perform as well as pure text matching, it’s more flexible. Based on its syntax, you can filter almost any kind of text from your source content. If you need to collect data in web development, it’s not difficult to use Regexp to retrieve meaningful data.

Go has the regexp package, which provides official support for regexp. If you’ve already used regexp in other programming languages, you should be familiar with it. Note that Go implemented RE2 standard except for \C. For more details, follow this link: http://code.google.com/p/re2/wiki/Syntax.

Go’s strings package can actually do many jobs like searching (Contains, Index), replacing (Replace), parsing (Split, Join), etc., and it’s faster than Regexp. However, these are all trivial operations. If you want to search a case insensitive string, Regexp should be your best choice. So, if the strings package is sufficient for your needs, just use it since it’s easy to use and read; if you need to perform more advanced operations, use Regexp.

If you recall form validation from previous sections, we used Regexp to verify the validity of user input information. Be aware that all characters are UTF-8. Let’s learn more about the Go regexp package!

Match

The regexp package has 3 functions to match: if it matches a pattern, then it returns true, returning false otherwise.

  1. func Match(pattern string, b []byte) (matched bool, error error)
  2. func MatchReader(pattern string, r io.RuneReader) (matched bool, error error)
  3. func MatchString(pattern string, s string) (matched bool, error error)

All 3 functions check if pattern matches the input source, returning true if it matches. However if your Regex has syntax errors, it will return an error. The 3 input sources of these functions are slice of byte, RuneReader and string.

Here is an example of how to verify an IP address:

  1. func IsIP(ip string) (b bool) {
  2.     if m, _ := regexp.MatchString("^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}$", ip); !m {
  3.         return false
  4.     }
  5.     return true
  6. }

As you can see, using pattern in the regexp package is not that different. Here’s one more example on verifying whether user input is valid:

  1. func main() {
  2.     if len(os.Args) == 1 {
  3.         fmt.Println("Usage: regexp [string]")
  4.         os.Exit(1)
  5.     } else if m, _ := regexp.MatchString("^[0-9]+$", os.Args[1]); m {
  6.         fmt.Println("Number")
  7.     } else {
  8.         fmt.Println("Not number")
  9.     }
  10. }

In the above examples, we use Match(Reader|String) to check if content is valid, but they are all easy to use.

Filter

Match mode can verify content but it cannot cut, filter or collect data from it. If you want to do that, you have to use the complex mode of Regexp.

Let’s say we need to write a crawler. Here is an example for when you must use Regexp to filter and cut data.

  1. package main
  3. import (
  4.     "fmt"
  5.     "io/ioutil"
  6.     "net/http"
  7.     "regexp"
  8.     "strings"
  9. )
  11. func main() {
  12.     resp, err := http.Get("http://www.baidu.com")
  13.     if err != nil {
  14.         fmt.Println("http get error.")
  15.     }
  16.     defer resp.Body.Close()
  17.     body, err := ioutil.ReadAll(resp.Body)
  18.     if err != nil {
  19.         fmt.Println("http read error")
  20.         return
  21.     }
  23.     src := string(body)
  25.     // Convert HTML tags to lower case.
  26.     re, _ := regexp.Compile("\\<[\\S\\s]+?\\>")
  27.     src = re.ReplaceAllStringFunc(src, strings.ToLower)
  29.     // Remove STYLE.
  30.     re, _ = regexp.Compile("\\<style[\\S\\s]+?\\</style\\>")
  31.     src = re.ReplaceAllString(src, "")
  33.     // Remove SCRIPT.
  34.     re, _ = regexp.Compile("\\<script[\\S\\s]+?\\</script\\>")
  35.     src = re.ReplaceAllString(src, "")
  37.     // Remove all HTML code in angle brackets, and replace with newline.
  38.     re, _ = regexp.Compile("\\<[\\S\\s]+?\\>")
  39.     src = re.ReplaceAllString(src, "\n")
  41.     // Remove continuous newline.
  42.     re, _ = regexp.Compile("\\s{2,}")
  43.     src = re.ReplaceAllString(src, "\n")
  45.     fmt.Println(strings.TrimSpace(src))
  46. }

In this example, we use Compile as the first step for complex mode. It verifies that your Regex syntax is correct, then returns a Regexp for parsing content in other operations.

Here are some functions to parse your Regexp syntax:

  1. func Compile(expr string) (*Regexp, error)
  2. func CompilePOSIX(expr string) (*Regexp, error)
  3. func MustCompile(str string) *Regexp
  4. func MustCompilePOSIX(str string) *Regexp

The difference between ComplePOSIX and Compile is that the former has to use POSIX syntax which is leftmost longest search, and the latter is only leftmost search. For instance, for Regexp [a-z]{2,4} and content "aa09aaa88aaaa", CompilePOSIX returns aaaa but Compile returns aa. Must prefix means panic when the Regexp syntax is not correct, returning error otherwise.

Now that we know how to create a new Regexp, let’s see how the methods provided by this struct can help us to operate on content:

  1. func (re *Regexp) Find(b []byte) []byte
  2. func (re *Regexp) FindAll(b []byte, n int) [][]byte
  3. func (re *Regexp) FindAllIndex(b []byte, n int) [][]int
  4. func (re *Regexp) FindAllString(s string, n int) []string
  5. func (re *Regexp) FindAllStringIndex(s string, n int) [][]int
  6. func (re *Regexp) FindAllStringSubmatch(s string, n int) [][]string
  7. func (re *Regexp) FindAllStringSubmatchIndex(s string, n int) [][]int
  8. func (re *Regexp) FindAllSubmatch(b []byte, n int) [][][]byte
  9. func (re *Regexp) FindAllSubmatchIndex(b []byte, n int) [][]int
  10. func (re *Regexp) FindIndex(b []byte) (loc []int)
  11. func (re *Regexp) FindReaderIndex(r io.RuneReader) (loc []int)
  12. func (re *Regexp) FindReaderSubmatchIndex(r io.RuneReader) []int
  13. func (re *Regexp) FindString(s string) string
  14. func (re *Regexp) FindStringIndex(s string) (loc []int)
  15. func (re *Regexp) FindStringSubmatch(s string) []string
  16. func (re *Regexp) FindStringSubmatchIndex(s string) []int
  17. func (re *Regexp) FindSubmatch(b []byte) [][]byte
  18. func (re *Regexp) FindSubmatchIndex(b []byte) []int

These 18 methods include identical functions for different input sources (byte slice, string and io.RuneReader), so we can really simplify this list by ignoring input sources as follows:

  1. func (re *Regexp) Find(b []byte) []byte
  2. func (re *Regexp) FindAll(b []byte, n int) [][]byte
  3. func (re *Regexp) FindAllIndex(b []byte, n int) [][]int
  4. func (re *Regexp) FindAllSubmatch(b []byte, n int) [][][]byte
  5. func (re *Regexp) FindAllSubmatchIndex(b []byte, n int) [][]int
  6. func (re *Regexp) FindIndex(b []byte) (loc []int)
  7. func (re *Regexp) FindSubmatch(b []byte) [][]byte
  8. func (re *Regexp) FindSubmatchIndex(b []byte) []int

Code sample:

  1. package main
  3. import (
  4.     "fmt"
  5.     "regexp"
  6. )
  8. func main() {
  9.     a := "I am learning Go language"
  11.     re, _ := regexp.Compile("[a-z]{2,4}")
  13.     // Find the first match.
  14.     one := re.Find([]byte(a))
  15.     fmt.Println("Find:", string(one))
  17.     // Find all matches and save to a slice, n less than 0 means return all matches, indicates length of slice if it's greater than 0.
  18.     all := re.FindAll([]byte(a), -1)
  19.     fmt.Println("FindAll", all)
  21.     // Find index of first match, start and end position.
  22.     index := re.FindIndex([]byte(a))
  23.     fmt.Println("FindIndex", index)
  25.     // Find index of all matches, the n does same job as above.
  26.     allindex := re.FindAllIndex([]byte(a), -1)
  27.     fmt.Println("FindAllIndex", allindex)
  29.     re2, _ := regexp.Compile("am(.*)lang(.*)")
  31.     // Find first submatch and return array, the first element contains all elements, the second element contains the result of first (), the third element contains the result of second ().
  32.     // Output: 
  33.     // the first element: "am learning Go language"
  34.     // the second element: " learning Go ", notice spaces will be outputed as well.
  35.     // the third element: "uage"
  36.     submatch := re2.FindSubmatch([]byte(a))
  37.     fmt.Println("FindSubmatch", submatch)
  38.     for _, v := range submatch {
  39.         fmt.Println(string(v))
  40.     }
  42.     // Same as FindIndex().
  43.     submatchindex := re2.FindSubmatchIndex([]byte(a))
  44.     fmt.Println(submatchindex)
  46.     // FindAllSubmatch, find all submatches.
  47.     submatchall := re2.FindAllSubmatch([]byte(a), -1)
  48.     fmt.Println(submatchall)
  50.     // FindAllSubmatchIndex,find index of all submatches.
  51.     submatchallindex := re2.FindAllSubmatchIndex([]byte(a), -1)
  52.     fmt.Println(submatchallindex)
  53. }

As we’ve previously mentioned, Regexp also has 3 methods for matching. They do the exact same thing as the exported functions. In fact, those exported functions actually call these methods under the hood:

  1. func (re *Regexp) Match(b []byte) bool
  2. func (re *Regexp) MatchReader(r io.RuneReader) bool
  3. func (re *Regexp) MatchString(s string) bool

Next, let’s see how to replace strings using Regexp:

  1. func (re *Regexp) ReplaceAll(src, repl []byte) []byte
  2. func (re *Regexp) ReplaceAllFunc(src []byte, repl func([]byte) []byte) []byte
  3. func (re *Regexp) ReplaceAllLiteral(src, repl []byte) []byte
  4. func (re *Regexp) ReplaceAllLiteralString(src, repl string) string
  5. func (re *Regexp) ReplaceAllString(src, repl string) string
  6. func (re *Regexp) ReplaceAllStringFunc(src string, repl func(string) string) string

These are used in the crawling example, so we will not explain any further here.

Let’s take a look at the definition of Expand:

  1. func (re *Regexp) Expand(dst []byte, template []byte, src []byte, match []int) []byte
  2. func (re *Regexp) ExpandString(dst []byte, template string, src string, match []int) []byte

So how do we use Expand?

  1. func main() {
  2.     src := []byte(`
  3.         call hello alice
  4.         hello bob
  5.         call hello eve
  6.     `)
  7.     pat := regexp.MustCompile(`(?m)(call)\s+(?P<cmd>\w+)\s+(?P<arg>.+)\s*$`)
  8.     res := []byte{}
  9.     for _, s := range pat.FindAllSubmatchIndex(src, -1) {
  10.         res = pat.Expand(res, []byte("$cmd('$arg')\n"), src, s)
  11.     }
  12.     fmt.Println(string(res))
  13. }

At this point, you’ve learnt the whole regexp package in Go. I hope that you can understand more by studying examples of key methods, so that you can do something interesting on your own.