Tabular

pipeline pipeline

The Tabular pipeline splits tabular data into rows and columns. The tabular pipeline is most useful in creating (id, text, tag) tuples to load into Embedding indexes.

Example

The following shows a simple example using this pipeline.

  1. from txtai.pipeline import Tabular
  3. # Create and run pipeline
  4. tabular = Tabular("id", ["text"])
  5. tabular("path to csv file")

See the link below for a more detailed example.

NotebookDescription
Transform tabular data with composable workflowsTransform, index and search tabular dataOpen In Colab

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

  1. # Create pipeline using lower case class name
  2. tabular:
  3.     idcolumn: id
  4.     textcolumns:
  5.       - text
  7. # Run pipeline with workflow
  8. workflow:
  9.   tabular:
  10.     tasks:
  11.       - action: tabular

Run with Workflows

  1. from txtai.app import Application
  3. # Create and run pipeline with workflow
  4. app = Application("config.yml")
  5. list(app.workflow("tabular", ["path to csv file"]))

Run with API

  1. CONFIG=config.yml uvicorn "txtai.api:app" &
  3. curl \
  4.   -X POST "http://localhost:8000/workflow" \
  5.   -H "Content-Type: application/json" \
  6.   -d '{"name":"tabular", "elements":["path to csv file"]}'

Methods

Python documentation for the pipeline.

Creates a new Tabular pipeline.

Parameters:

NameTypeDescriptionDefault
idcolumn

column name to use for row id

None
textcolumns

list of columns to combine as a text field

None
content

if True, a dict per row is generated with all fields. If content is a list, a subset of fields is included in the generated rows.

False

Source code in txtai/pipeline/data/tabular.py

  1. 23
  2. 24
  3. 25
  4. 26
  5. 27
  6. 28
  7. 29
  8. 30
  9. 31
  10. 32
  11. 33
  12. 34
  13. 35
  14. 36
  15. 37
  16. 38
  17. 39
  1. def init(self, idcolumn=None, textcolumns=None, content=False):
  2.     “””
  3.     Creates a new Tabular pipeline.
  5.     Args:
  6.         idcolumn: column name to use for row id
  7.         textcolumns: list of columns to combine as a text field
  8.         content: if True, a dict per row is generated with all fields. If content is a list, a subset of fields
  9.                  is included in the generated rows.
  10.     “””
  12.     if not PANDAS:
  13.         raise ImportError(‘Tabular pipeline is not available - install “pipeline” extra to enable’)
  15.     self.idcolumn = idcolumn
  16.     self.textcolumns = textcolumns
  17.     self.content = content

Splits data into rows and columns.

Parameters:

NameTypeDescriptionDefault
data

input data

required

Returns:

TypeDescription

list of (id, text, tag)

Source code in txtai/pipeline/data/tabular.py

  1. 41
  2. 42
  3. 43
  4. 44
  5. 45
  6. 46
  7. 47
  8. 48
  9. 49
  10. 50
  11. 51
  12. 52
  13. 53
  14. 54
  15. 55
  16. 56
  17. 57
  18. 58
  19. 59
  20. 60
  21. 61
  22. 62
  23. 63
  24. 64
  25. 65
  26. 66
  27. 67
  28. 68
  29. 69
  30. 70
  31. 71
  32. 72
  33. 73
  34. 74
  35. 75
  36. 76
  37. 77
  38. 78
  39. 79
  40. 80
  41. 81
  42. 82
  1. def call(self, data):
  2.     “””
  3.     Splits data into rows and columns.
  5.     Args:
  6.         data: input data
  8.     Returns:
  9.         list of (id, text, tag)
  10.     “””
  12.     items = [data] if not isinstance(data, list) else data
  14.     # Combine all rows into single return element
  15.     results = []
  16.     dicts = []
  18.     for item in items:
  19.         # File path
  20.         if isinstance(item, str):
  21.             _, extension = os.path.splitext(item)
  22.             extension = extension.replace(“.”, “”).lower()
  24.             if extension == “csv”:
  25.                 df = pd.read_csv(item)
  27.             results.append(self.process(df))
  29.         # Dict
  30.         if isinstance(item, dict):
  31.             dicts.append(item)
  33.         # List of dicts
  34.         elif isinstance(item, list):
  35.             df = pd.DataFrame(item)
  36.             results.append(self.process(df))
  38.     if dicts:
  39.         df = pd.DataFrame(dicts)
  40.         results.extend(self.process(df))
  42.     return results[0] if not isinstance(data, list) else results