FAQ

Below is a list of frequently asked questions and common issues encountered.

Questions

Question

What models are recommended?

Answer

See the model guide.

Question

What is the best way to track the progress of an embeddings.index call?

Answer

Wrap the list or generator passed to the index call with tqdm. See #478 for more.

Question

What is the best way to analyze the content of a txtai index?

Answer

txtai has a console application that makes this easy. Read this article to learn more.

Question

How can models be externally loaded and passed to embeddings and pipelines?

Answer

Embeddings example.

from transformers import AutoModel, AutoTokenizer
from txtai import Embeddings
# Load model externally
model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
# Pass to embeddings instance
embeddings = Embeddings(path=model, tokenizer=tokenizer)

LLM pipeline example.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from txtai import LLM
# Load Mistral-7B-OpenOrca
path = "Open-Orca/Mistral-7B-OpenOrca"
model = AutoModelForCausalLM.from_pretrained(
  path,
  torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(path)
llm = LLM((model, tokenizer))

Common issues

Issue

Embeddings query errors like this:

SQLError: no such function: json_extract

Solution

Upgrade Python version as it doesn’t have SQLite support for json_extract

Issue

Segmentation faults and similar errors on macOS

Solution

Set the following environment parameters.

• Disable OpenMP threading via the environment variable export OMP_NUM_THREADS=1
• Disable PyTorch MPS device via export PYTORCH_MPS_DISABLE=1
• Disable llama.cpp metal via export LLAMA_NO_METAL=1

Issue

Error running SQLite ANN on macOS

AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension'

Solution

See this note for options on how to fix this.

Issue

ContextualVersionConflict and/or package METADATA exception while running one of the examples notebooks on Google Colab

Solution

Restart the kernel. See issue #409 for more on this issue.

Issue

Error installing optional/extra dependencies such as pipeline

Solution

The default MacOS shell (zsh) and Windows PowerShell require escaping square brackets

pip install 'txtai[pipeline]'