LLM

The LLM pipeline runs prompts through a large language model (LLM). This pipeline autodetects the LLM framework based on the model path.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import LLM
# Create and run LLM pipeline
llm = LLM()
llm(
  """
  Answer the following question using the provided context.
  Question:
  What are the applications of txtai?
  Context:
  txtai is an open-source platform for semantic search and
  workflows powered by language models.
  """
)

The LLM pipeline automatically detects the underlying LLM framework. This can also be manually set.

from txtai.pipeline import LLM
# Set method as litellm
llm = LLM("vllm/Open-Orca/Mistral-7B-OpenOrca", method="litellm")
# Set method as llama.cpp
llm = LLM("TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
           method="llama.cpp")

Models can be externally loaded and passed to pipelines. This is useful for models that are not yet supported by Transformers and/or need special initialization.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from txtai.pipeline import LLM
# Load Mistral-7B-OpenOrca
path = "Open-Orca/Mistral-7B-OpenOrca"
model = AutoModelForCausalLM.from_pretrained(
  path,
  torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(path)
llm = LLM((model, tokenizer))

See the links below for more detailed examples.

Notebook	Description
Prompt-driven search with LLMs	Embeddings-guided and Prompt-driven search with Large Language Models (LLMs)
Prompt templates and task chains	Build model prompts and connect tasks together with workflows
Build RAG pipelines with txtai	Guide on retrieval augmented generation including how to create citations
Integrate LLM frameworks	Integrate llama.cpp, LiteLLM and custom generation frameworks
Generate knowledge with Semantic Graphs and RAG	Knowledge exploration and discovery with Semantic Graphs and RAG

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

# Create pipeline using lower case class name
# Use `generator` or `sequences` to force model type
llm:
# Run pipeline with workflow
workflow:
  llm:
    tasks:
      - action: llm

Similar to the Python example above, the underlying Hugging Face pipeline parameters and model parameters can be set in pipeline configuration.

llm:
  path: Open-Orca/Mistral-7B-OpenOrca
  torch_dtype: torch.bfloat16

Run with Workflows

from txtai.app import Application
# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("llm", [
  """
  Answer the following question using the provided context.
  Question:
  What are the applications of txtai? 
  Context:
  txtai is an open-source platform for semantic search and
  workflows powered by language models.
  """
]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &
curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"sequences", "elements": ["Answer the following question..."]}'

Methods

Python documentation for the pipeline.

Creates a new LLM.

Parameters:

Source code in txtai/pipeline/llm/llm.py

20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

def init(self, path=None, method=None, kwargs):
    “””
    Creates a new LLM.
    Args:
        path: model path
        method: llm model framework, infers from path if not provided
        kwargs: model keyword arguments
    “””
    # Default LLM if not provided
    path = path if path else “google/flan-t5-base”
    # Generation instance
    self.generator = GenerationFactory.create(path, method, kwargs)

Generates text using input text

Parameters:

Returns:

Source code in txtai/pipeline/llm/llm.py

36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

def call(self, text, maxlength=512, kwargs):
    “””
    Generates text using input text
    Args:
        text: text|list
        maxlength: maximum sequence length
        kwargs: additional generation keyword arguments
    Returns:
        generated text
    “””
    # Run LLM generation
    return self.generator(text, maxlength, kwargs)