Disk-based vector search

Introduced 2.17

For low-memory environments, OpenSearch provides disk-based vector search, which significantly reduces the operational costs for vector workloads. Disk-based vector search uses binary quantization, compressing vectors and thereby reducing the memory requirements. This memory optimization provides large memory savings at the cost of slightly increased search latency while still maintaining strong recall.

To use disk-based vector search, set the mode parameter to on_disk for your vector field type. This parameter will configure your index to use secondary storage.

To create an index for disk-based vector search, send the following request:

  1. PUT my-vector-index
  2. {
  3.   "settings" : {
  4.     "index": {
  5.       "knn": true
  6.     }
  7.   },
  8.   "mappings": {
  9.     "properties": {
  10.       "my_vector_field": {
  11.         "type": "knn_vector",
  12.         "dimension": 8,
  13.         "space_type": "innerproduct",
  14.         "data_type": "float",
  15.         "mode": "on_disk"
  16.       }
  17.     }
  18.   }
  19. }

copy

By default, the on_disk mode configures the index to use the faiss engine and hnsw method. The default compression_level of 32x reduces the amount of memory the vectors require by a factor of 32. To preserve the search recall, rescoring is enabled by default. A search on a disk-optimized index runs in two phases: The compressed index is searched first, and then the results are rescored using full-precision vectors loaded from disk.

To reduce the compression level, provide the compression_level parameter when creating the index mapping:

  1. PUT my-vector-index
  2. {
  3.   "settings" : {
  4.     "index": {
  5.       "knn": true
  6.     }
  7.   },
  8.   "mappings": {
  9.     "properties": {
  10.       "my_vector_field": {
  11.         "type": "knn_vector",
  12.         "dimension": 8,
  13.         "space_type": "innerproduct",
  14.         "data_type": "float",
  15.         "mode": "on_disk",
  16.         "compression_level": "16x"
  17.       }
  18.     }
  19.   }
  20. }

copy

For more information about the compression_level parameter, see Compression levels. Note that for 4x compression, the lucene engine will be used.

If you need more granular fine-tuning, you can override additional k-NN parameters in the method definition. For example, to improve recall, increase the ef_construction parameter value:

  1. PUT my-vector-index
  2. {
  3.   "settings" : {
  4.     "index": {
  5.       "knn": true
  6.     }
  7.   },
  8.   "mappings": {
  9.     "properties": {
  10.       "my_vector_field": {
  11.         "type": "knn_vector",
  12.         "dimension": 8,
  13.         "space_type": "innerproduct",
  14.         "data_type": "float",
  15.         "mode": "on_disk",
  16.         "method": {
  17.           "params": {
  18.             "ef_construction": 512
  19.           }
  20.         }
  21.       }
  22.     }
  23.   }
  24. }

copy

The on_disk mode only works with the float data type.

Ingestion

You can perform document ingestion for a disk-optimized vector index in the same way as for a regular vector index. To index several documents in bulk, send the following request:

  1. POST _bulk
  2. { "index": { "_index": "my-vector-index", "_id": "1" } }
  3. { "my_vector_field": [1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5], "price": 12.2 }
  4. { "index": { "_index": "my-vector-index", "_id": "2" } }
  5. { "my_vector_field": [2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5], "price": 7.1 }
  6. { "index": { "_index": "my-vector-index", "_id": "3" } }
  7. { "my_vector_field": [3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5, 3.5], "price": 12.9 }
  8. { "index": { "_index": "my-vector-index", "_id": "4" } }
  9. { "my_vector_field": [4.5, 4.5, 4.5, 4.5, 4.5, 4.5, 4.5, 4.5], "price": 1.2 }
  10. { "index": { "_index": "my-vector-index", "_id": "5" } }
  11. { "my_vector_field": [5.5, 5.5, 5.5, 5.5, 5.5, 5.5, 5.5, 5.5], "price": 3.7 }
  12. { "index": { "_index": "my-vector-index", "_id": "6" } }
  13. { "my_vector_field": [6.5, 6.5, 6.5, 6.5, 6.5, 6.5, 6.5, 6.5], "price": 10.3 }
  14. { "index": { "_index": "my-vector-index", "_id": "7" } }
  15. { "my_vector_field": [7.5, 7.5, 7.5, 7.5, 7.5, 7.5, 7.5, 7.5], "price": 5.5 }
  16. { "index": { "_index": "my-vector-index", "_id": "8" } }
  17. { "my_vector_field": [8.5, 8.5, 8.5, 8.5, 8.5, 8.5, 8.5, 8.5], "price": 4.4 }
  18. { "index": { "_index": "my-vector-index", "_id": "9" } }
  19. { "my_vector_field": [9.5, 9.5, 9.5, 9.5, 9.5, 9.5, 9.5, 9.5], "price": 8.9 }

copy

Search is also performed in the same way as in other index configurations. The key difference is that, by default, the oversample_factor of the rescore parameter is set to 3.0 (unless you override the compression_level). For more information, see Rescoring quantized results using full precision. To perform vector search on a disk-optimized index, provide the search vector:

  1. GET my-vector-index/_search
  2. {
  3.   "query": {
  4.     "knn": {
  5.       "my_vector_field": {
  6.         "vector": [1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5],
  7.         "k": 5
  8.       }
  9.     }
  10.   }
  11. }

copy

Similarly to other index configurations, you can override k-NN parameters in the search request:

  1. GET my-vector-index/_search
  2. {
  3.   "query": {
  4.     "knn": {
  5.       "my_vector_field": {
  6.         "vector": [1.5, 2.5, 3.5, 4.5, 5.5, 6.5, 7.5, 8.5],
  7.         "k": 5,
  8.         "method_parameters": {
  9.             "ef_search": 512
  10.         },
  11.         "rescore": {
  12.             "oversample_factor": 10.0
  13.         }
  14.       }
  15.     }
  16.   }
  17. }

copy

Radial search does not support disk-based vector search.

Model-based indexes

For model-based indexes, you can specify the on_disk parameter in the training request in the same way that you would specify it during index creation. By default, on_disk mode will use the Faiss IVF method and a compression level of 32x. To run the training API, send the following request:

  1. POST /_plugins/_knn/models/test-model/_train
  2. {
  3.     "training_index": "train-index-name",
  4.     "training_field": "train-field-name",
  5.     "dimension": 8,
  6.     "max_training_vector_count": 1200,
  7.     "search_size": 100,
  8.     "description": "My model",
  9.     "space_type": "innerproduct",
  10.     "mode": "on_disk"
  11. }

copy

This command assumes that training data has been ingested into the train-index-name index. For more information, see Building a k-NN index from a model.

You can override the compression_level for disk-optimized indexes in the same way as for regular k-NN indexes.

Next steps