Caption

The caption pipeline reads a list of images and returns a list of captions for those images.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import Caption
# Create and run pipeline
caption = Caption()
caption("path to image file")

See the link below for a more detailed example.

Notebook	Description
Generate image captions and detect objects	Captions and object detection for images

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

# Create pipeline using lower case class name
caption:
# Run pipeline with workflow
workflow:
  caption:
    tasks:
      - action: caption

Run with Workflows

from txtai.app import Application
# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("caption", ["path to image file"]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &
curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"caption", "elements":["path to image file"]}'

Methods

Python documentation for the pipeline.

Source code in txtai/pipeline/image/caption.py

def __init__(self, path=None, quantize=False, gpu=True, model=None, **kwargs):
    if not PIL:
        raise ImportError('Captions pipeline is not available - install "pipeline" extra to enable')
    # Call parent constructor
    super().__init__("image-to-text", path, quantize, gpu, model, **kwargs)

Builds captions for images.

This method supports a single image or a list of images. If the input is an image, the return type is a string. If text is a list, a list of strings is returned

Parameters:

Returns:

Source code in txtai/pipeline/image/caption.py

def __call__(self, images):
    """
    Builds captions for images.
    This method supports a single image or a list of images. If the input is an image, the return
    type is a string. If text is a list, a list of strings is returned
    Args:
        images: image|list
    Returns:
        list of captions
    """
    # Convert single element to list
    values = [images] if not isinstance(images, list) else images
    # Open images if file strings
    values = [Image.open(image) if isinstance(image, str) else image for image in values]
    # Get and clean captions
    captions = []
    for result in self.pipeline(values):
        text = " ".join([r["generated_text"] for r in result]).strip()
        captions.append(text)
    # Return single element if single element passed in
    return captions[0] if not isinstance(images, list) else captions