我的书签
添加书签
移除书签Index format
This section documents the txtai index format. Each component is designed to ensure open access to the underlying data in a programmatic and platform independent way
If an underlying library has an index format, that is used. Otherwise, txtai persists content with MessagePack serialization.
To learn more about how these components work together, read the Index Guide and Query Guide.
ANN
Approximate Nearest Neighbor (ANN) index configuration for storing vector embeddings.
| Component | Storage Format |
|---|---|
| Faiss | Local file format provided by library |
| Hnswlib | Local file format provided by library |
| Annoy | Local file format provided by library |
| NumPy | Local NumPy array files via np.save / np.load |
| Postgres via pgvector | Vector tables in a Postgres database |
Core
Core embeddings index files.
| Component | Storage Format |
|---|---|
| Configuration | Embeddings index configuration stored as JSON |
| Index Ids | Embeddings index ids serialized with MessagePack. Only enabled when when content storage (database) is disabled. |
Database
Databases store metadata, text and binary content.
| Component | Storage Format |
|---|---|
| SQLite | Local database files with SQLite |
| DuckDB | Local database files with DuckDB |
| Postgres | Postgres relational database via SQLAlchemy. Supports additional databases via this library. |
Graph
Graph nodes and edges for an embeddings index
| Component | Storage Format |
|---|---|
| NetworkX | Nodes and edges exported to local file serialized with MessagePack |
| Postgres | Nodes and edges stored in a Postgres database. Supports additional databases. |
Scoring
Sparse/keyword indexing
| Component | Storage Format |
|---|---|
| Local index | Metadata serialized with MessagePack. Terms stored in SQLite. |
| Postgres | Text indexed with Postgres Full Text Search (FTS) |
