Description

Transform a document to a sparse vector based on the inverse document frequency(idf) statistics provided by DocHashCountVectorizerTrainBatchOp. It uses MurmurHash 3 to get the hash value of a word as the index.

Parameters

Name Description Type Required？ Default Value
selectedCol Name of the selected column used for processing String ✓
outputCol Name of the output column String null
reservedCols Names of the columns to be retained in the output table String[] null

Script Example

Code

  1. import numpy as np
  2. import pandas as pd
  3. data = np.array([
  4.     [0, u'二手旧书:医学电磁成像'],
  5.     [1, u'二手美国文学选读（ 下册 ）李宜燮南开大学出版社 9787310003969'],
  6.     [2, u'二手正版图解象棋入门/谢恩思主编/华龄出版社'],
  7.     [3, u'二手中国糖尿病文献索引'],
  8.     [4, u'二手郁达夫文集（ 国内版 ）全十二册馆藏书']])
  9. df = pd.DataFrame({"id": data[:, 0], "text": data[:, 1]})
  10. inOp1 = BatchOperator.fromDataframe(df, schemaStr='id int, text string')
  11. inOp2 = StreamOperator.fromDataframe(df, schemaStr='id int, text string')
  13. segment = SegmentBatchOp().setSelectedCol("text").linkFrom(inOp1)
  14. train = DocHashCountVectorizerTrainBatchOp().setSelectedCol("text").linkFrom(segment)
  15. predictBatch = DocHashCountVectorizerPredictBatchOp().setSelectedCol("text").linkFrom(train, segment)
  16. [model,predict] = collectToDataframes(train, predictBatch)
  17. print(model)
  18. print(predict)
  20. segment = SegmentStreamOp().setSelectedCol("text").linkFrom(inOp2)
  21. predictStream = DocHashCountVectorizerPredictStreamOp(train).setSelectedCol("text").linkFrom(segment)
  22. predictStream.print(refreshInterval=-1)
  23. StreamOperator.execute()

Results

Model
  1. rowID  model_id                                         model_info
  2. 0         0  {"numFeatures":"262144","minTF":"1.0","feature...
  3. 1   1048576  {"0":-0.6061358035703156,"37505":1.09861228866...
Output Data
  1.    id                                               text
  2. 0   0  $262144$10121:1.0 64444:1.0 119456:1.0 206232:...
  3. 1   1  $262144$0:6.0 37505:1.0 46743:1.0 93228:1.0 11...
  4. 2   2  $262144$40170:1.0 70777:1.0 96509:1.0 126159:1...
  5. 3   3  $262144$206232:1.0 214785:1.0 251090:1.0 25565...
  6. 4   4  $262144$0:4.0 87711:1.0 138080:1.0 162140:1.0 ...