Description

If gaps is true, it splits the document with the given pattern. If gaps is false, it extract the tokens matching the pattern.

Parameters

Name Description Type Required？ Default Value
pattern If gaps is true, it’s used as a delimiter; If gaps is false, it’s used as a token String “\s+”
gaps If gaps is true, it splits the document with the given pattern. If gaps is false, it extract the tokens matching the pattern Boolean true
minTokenLength The minimum of token length. Integer 1
toLowerCase If true, transform all the words to lower case。 Boolean true
selectedCol Name of the selected column used for processing String ✓
outputCol Name of the output column String null
reservedCols Names of the columns to be retained in the output table String[] null

Script Example

Code

  1. import numpy as np
  2. import pandas as pd
  4. data = np.array([
  5.     [0, 'That is an English Book!'],
  6.     [1, 'Do you like math?'],
  7.     [2, 'Have a good day!']
  8. ])
  10. df = pd.DataFrame({"id": data[:, 0], "text": data[:, 1]})
  11. inOp1 = dataframeToOperator(df, schemaStr='id long, text string', op_type='batch')
  12. op = RegexTokenizer().setSelectedCol("text").setGaps(False).setToLowerCase(True).setOutputCol("token").setPattern("\\w+")
  13. op.transform(inOp1).print()

Results

  1. id    text    token
  2. 0    0    That is an English Book!    that is an english book
  3. 1    2    Have a good day!    have a good day
  4. 2    1    Do you like math?    do you like math