Description

Transform data type from Kv to Vector.

Parameters

Name Description Type Required？ Default Value
handleInvalid Strategy to handle unseen token String “ERROR”
reservedCols Names of the columns to be retained in the output table String[] null
vectorCol Name of a vector column String ✓
vectorSize Size of the vector Long -1
kvCol Name of the KV column String ✓
kvColDelimiter Delimiter used between key-value pairs when data in the input table is in sparse format String “,”
kvValDelimiter Delimiter used between keys and values when data in the input table is in sparse format String “:”

Script Example

Code

  1. import numpy as np
  2. import pandas as pd
  5. data = np.array([['1', '{"f1":"1.0","f2":"2.0"}', '$3$1:1.0 2:2.0', '1:1.0,2:2.0', '1.0,2.0', 1.0, 2.0],
  6. ['2', '{"f2":"4.0","f4":"8.0"}', '$3$1:4.0 2:8.0', '1:4.0,2:8.0', '4.0,8.0', 4.0, 8.0]])
  8. df = pd.DataFrame({"row":data[:,0], "json":data[:,1], "vec":data[:,2], "kv":data[:,3], "csv":data[:,4], "f0":data[:,5], "f1":data[:,6]})
  9. data = dataframeToOperator(df, schemaStr="row string, json string, vec string, kv string, csv string, f0 double, f1 double",op_type="batch")
  12. op = KvToVectorBatchOp()\
  13.     .setKvCol("kv")\
  14.     .setReservedCols(["row"]).setVectorCol("vec").setVectorSize(5)\
  15.     .linkFrom(data)
  16. op.print()

Results

  1. |row|vec|
  2. |-|-----|
  3. |1|$5$1.0 2.0|
  4. |2|$5$4.0 8.0|