Description

Transform data type from Csv to Json.

Parameters

Name Description Type Required？ Default Value
handleInvalid Strategy to handle unseen token String “ERROR”
reservedCols Names of the columns to be retained in the output table String[] null
jsonCol Name of the CSV column String ✓
csvCol Name of the CSV column String ✓
schemaStr Formatted schema String ✓
csvFieldDelimiter Field delimiter String “,”
quoteChar quote char Character “\””

Script Example

Code

  1. import numpy as np
  2. import pandas as pd
  5. data = np.array([['1', '{"f1":"1.0","f2":"2.0"}', '$3$1:1.0 2:2.0', '1:1.0,2:2.0', '1.0,2.0', 1.0, 2.0],
  6. ['2', '{"f2":"4.0","f4":"8.0"}', '$3$1:4.0 2:8.0', '1:4.0,2:8.0', '4.0,8.0', 4.0, 8.0]])
  8. df = pd.DataFrame({"row":data[:,0], "json":data[:,1], "vec":data[:,2], "kv":data[:,3], "csv":data[:,4], "f0":data[:,5], "f1":data[:,6]})
  9. data = dataframeToOperator(df, schemaStr="row string, json string, vec string, kv string, csv string, f0 double, f1 double",op_type="stream")
  12. op = CsvToJsonStreamOp()\
  13.     .setCsvCol("csv").setSchemaStr("f0 double, f1 double")\
  14.     .setReservedCols(["row"]).setJsonCol("json")\
  15.     .linkFrom(data)
  16. op.print()
  17. StreamOperator.execute()

Results

  1. |row|json|
  2. |---|----|
  3. | 1 |{"f1":"1.0","f2":"2.0"}|
  4. | 2 |{"f2":"4.0","f4":"8.0"}|