Search with k-NN filters

Introduced 2.4

You can create custom filters using Query domain-specific language (DSL) search options to refine your k-NN searches. You define the filter criteria within the knn_vector field’s filter subsection in your query. You can use any of the OpenSearch query DSL query types as a filter. This includes the common query types: term, range, regexp, and wildcard, as well as custom query types. To include or exclude results, use Boolean query clauses. You can also specify a query point with the knn_vector type and search for nearest neighbors that match your filter criteria. To run k-NN queries with a filter, the Lucene search engine and Hierarchical Navigable Small World (HNSW) method are required.

To learn more about how to use query DSL Boolean query clauses, see Boolean queries. For more details about the knn_vector data type definition, see k-NN Index.

How does a k-NN filter work?

The OpenSearch k-NN plugin version 2.2 introduced support for the Lucene engine in order to process k-NN searches. The Lucene engine provides a search that is based on the HNSW algorithm in order to represent a multi-layered graph. The OpenSearch k-NN plugin version 2.4 can incorporate filters for searches based on Lucene 9.4.

After a filter is applied to a set of documents to be searched, the algorithm decides whether to perform pre-filtering for an exact k-NN search or modified post-filtering for an approximate search. The approximate search with filtering ensures the top number of closest vectors in the results.

Lucene also provides the capability to operate its KnnVectorQuery across a subset of documents. To learn more about this capability, see the Apache Lucene Documentation.

To learn more about all available k-NN search approaches, including approximate k-NN, exact k-NN with script score, and pre-filtering with painless extensions, see k-NN.

Filtered search performance

Filtering that is tightly integrated with the Lucene HNSW algorithm implementation allows you to apply k-NN searches more efficiently, both in terms of relevancy of search results and performance. Consider, for example, an exact search using post-filtering on a large dataset that returns results slowly and does not ensure the required number of results specified by k. With this new capability, you can create an approximate k-NN search, apply filters, and get the number of results that you need. To learn more about approximate searches, see Approximate k-NN search.

The HNSW algorithm decides which type of filtering to apply to a search based on the volume of documents and number of k points in the index that you search with a filter.

How the algorithm evaluates a doc set

VariableDescription
NThe number of documents in the index.
PThe number of documents in the search set after the filter is applied using the formula P <= N.
qThe search vector.
kThe maximum number of vectors to return in the response.

To learn more about k-NN performance tuning, see Performance tuning.

Filter approaches by use case

Depending on the dataset that you are searching, you might choose a different approach to minimize recall or latency. You can create filters that are:

  • Very restrictive: Returns the lowest number of documents (for example, 2.5%).
  • Somewhat restrictive: Returns some documents (for example, 38%).
  • Not very restrictive: Returns the highest number of documents (for example, 80%).

The restrictive percentage indicates the number of documents the filter returns for any given document set in an index.

Number of VectorsFilter Restrictive PercentagekRecallLatency
10M2.5100Scoring scriptScoring script
10M38100Lucene filterBoolean filter
10M80100Scoring scriptLucene filter
1M2.5100Lucene filterScoring script
1M38100Lucene filterlucene_filtering / Scoring script
1M80100Boolean filterlucene_filtering

In this context, Scoring script is essentially a brute force search, whereas a Boolean filter is an approximate k-NN search with post-filtering.

To learn more about the dynamic searches you can perform with the score script plugin, see Exact k-NN with scoring script.

In a Boolean query that uses post-filtering, you can join a k-NN query with a filter using a bool must query clause.

Example request

The following k-NN query uses a Boolean query clause to filter results:

  1. POST /hotels-index/_search
  2. {
  3.     "size": 3,
  4.     "query": {
  5.         "bool": {
  6.             "filter": {
  7.                 "bool": {
  8.                     "must": [
  9.                         {
  10.                             "range": {
  11.                                 "rating": {
  12.                                     "gte": 8,
  13.                                     "lte": 10
  14.                                 }
  15.                             }
  16.                         },
  17.                         {
  18.                             "term": {
  19.                                 "parking": "true"
  20.                             }
  21.                         }
  22.                     ]
  23.                 }
  24.             },
  25.             "must": [
  26.                 {
  27.                     "knn": {
  28.                         "location": {
  29.                             "vector": [
  30.                                 5.0,
  31.                                 4.0
  32.                             ],
  33.                             "k": 20
  34.                         }
  35.                     }
  36.                 }
  37.             ]
  38.         }
  39.     }
  40. }

Example response

The Boolean query filter returns the following results in the response:

  1. {
  2.   "took" : 95,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 5,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 0.72992706,
  16.     "hits" : [
  17.       {
  18.         "_index" : "hotels-index",
  19.         "_id" : "3",
  20.         "_score" : 0.72992706,
  21.         "_source" : {
  22.           "location" : [
  23.             4.9,
  24.             3.4
  25.           ],
  26.           "parking" : "true",
  27.           "rating" : 9
  28.         }
  29.       },
  30.       {
  31.         "_index" : "hotels-index",
  32.         "_id" : "6",
  33.         "_score" : 0.3012048,
  34.         "_source" : {
  35.           "location" : [
  36.             6.4,
  37.             3.4
  38.           ],
  39.           "parking" : "true",
  40.           "rating" : 9
  41.         }
  42.       },
  43.       {
  44.         "_index" : "hotels-index",
  45.         "_id" : "5",
  46.         "_score" : 0.24154587,
  47.         "_source" : {
  48.           "location" : [
  49.             3.3,
  50.             4.5
  51.           ],
  52.           "parking" : "true",
  53.           "rating" : 8
  54.         }
  55.       }
  56.     ]
  57.   }
  58. }

Use case 1: Very restrictive 2.5% filter

A very restrictive filter returns the lowest number of documents in your dataset. For example, the following filter criteria specifies hotels with feedback ratings less than or equal to 3. This 2.5% filter only returns 1 document:

  1.                 "filter": {
  2.                     "bool": {
  3.                         "must": [
  4.                             {
  5.                                 "range": {
  6.                                     "rating": {
  7.                                         "lte": 3
  8.                                     }
  9.                                 }
  10.                             }
  11.                         ]
  12.                     }
  13.                 }

Use case 2: Somewhat restrictive 38% filter

A somewhat restrictive filter returns 38% of the documents in the data set that you search. For example, the following filter criteria specifies hotels with parking and feedback ratings less than or equal to 8 and returns 5 documents:

  1.                "filter": {
  2.                     "bool": {
  3.                         "must": [
  4.                             {
  5.                                 "range": {
  6.                                     "rating": {
  7.                                         "lte": 8
  8.                                     }
  9.                                 }
  10.                             },
  11.                             {
  12.                                 "term": {
  13.                                     "parking": "true"
  14.                                 }
  15.                             }
  16.                         ]
  17.                     }
  18.                 }

Use case 3: Not very restrictive 80% filter

A filter that is not very restrictive will return 80% of the documents that you search. For example, the following filter criteria specifies hotels with feedback ratings greater than or equal to 5 and returns 10 documents:

  1.                 "filter": {
  2.                     "bool": {
  3.                         "must": [
  4.                             {
  5.                                 "range": {
  6.                                     "rating": {
  7.                                         "gte": 5
  8.                                     }
  9.                                 }
  10.                             }
  11.                         ]
  12.                     }
  13.                 }

You can search with a filter by following these three steps:

  1. Create an index and specify the requirements for the Lucene engine and HNSW requirements in the mapping.
  2. Add your data to the index.
  3. Search the index and specify these three items in your query:
    • One or more filters defined by query DSL
    • A vector reference point defined by the vector field
    • The number of matches you want returned with the k field

You can use a range query to specify hotel feedback ratings and a term query to require that parking is available. The criteria is processed with Boolean clauses to indicate whether or not the document contains the criteria.

Consider a dataset that contains 12 documents, a search reference point, and documents that meet two filter criteria.

Graph of documents with filter criteria

Step 1: Create a new index with a Lucene mapping

Before you can run a k-NN search with a filter, you need to create an index, specify the Lucene engine in a mapping, and add data to the index.

You need to add a location field to represent the location and specify it as the knn_vector type. The most basic vector can be two-dimensional. For example:

  1.   "type": "knn_vector",
  2.   "dimension": 2,

Requirement: Lucene engine with HNSW method

Make sure to specify “hnsw” method and “lucene” engine in the knn_vector field description, as follows:

  1. "my_field": {
  2.                 "type": "knn_vector",
  3.                 "dimension": 2,
  4.                 "method": {
  5.                     "name": "hnsw",
  6.                     "space_type": "l2",
  7.                     "engine": "lucene"
  8.                 }
  9.             }

Example request

The following request creates a new index called “hotels-index”:

  1. PUT /hotels-index
  2. {
  3.     "settings": {
  4.         "index": {
  5.             "knn": true,
  6.             "knn.algo_param.ef_search": 100,
  7.             "number_of_shards": 1,
  8.             "number_of_replicas": 0
  9.         }
  10.     },
  11.     "mappings": {
  12.         "properties": {
  13.             "location": {
  14.                 "type": "knn_vector",
  15.                 "dimension": 2,
  16.                 "method": {
  17.                     "name": "hnsw",
  18.                     "space_type": "l2",
  19.                     "engine": "lucene",
  20.                     "parameters": {
  21.                         "ef_construction": 100,
  22.                         "m": 16
  23.                     }
  24.                 }
  25.             }
  26.         }
  27.     }
  28. }

Example response

Upon success, you should receive a “200-OK” status with the following response:

  1. {
  2.   "acknowledged" : true,
  3.   "shards_acknowledged" : true,
  4.   "index" : "hotels-index"
  5. }

Step 2: Add data to your index

Next, add data to your index with a PUT HTTP request. Make sure that the search criteria is defined in the body of the request.

Example request

The following request adds 12 hotel documents that contain criteria such as feedback ratings and whether or not parking is available:

  1. POST /_bulk
  2. { "index": { "_index": "hotels-index", "_id": "1" } }
  3. { "location": [5.2, 4.4], "parking" : "true", "rating" : 5 }
  4. { "index": { "_index": "hotels-index", "_id": "2" } }
  5. { "location": [5.2, 3.9], "parking" : "false", "rating" : 4 }
  6. { "index": { "_index": "hotels-index", "_id": "3" } }
  7. { "location": [4.9, 3.4], "parking" : "true", "rating" : 9 }
  8. { "index": { "_index": "hotels-index", "_id": "4" } }
  9. { "location": [4.2, 4.6], "parking" : "false", "rating" : 6}
  10. { "index": { "_index": "hotels-index", "_id": "5" } }
  11. { "location": [3.3, 4.5], "parking" : "true", "rating" : 8 }
  12. { "index": { "_index": "hotels-index", "_id": "6" } }
  13. { "location": [6.4, 3.4], "parking" : "true", "rating" : 9 }
  14. { "index": { "_index": "hotels-index", "_id": "7" } }
  15. { "location": [4.2, 6.2], "parking" : "true", "rating" : 5 }
  16. { "index": { "_index": "hotels-index", "_id": "8" } }
  17. { "location": [2.4, 4.0], "parking" : "true", "rating" : 8 }
  18. { "index": { "_index": "hotels-index", "_id": "9" } }
  19. { "location": [1.4, 3.2], "parking" : "false", "rating" : 5 }
  20. { "index": { "_index": "hotels-index", "_id": "10" } }
  21. { "location": [7.0, 9.9], "parking" : "true", "rating" : 9 }
  22. { "index": { "_index": "hotels-index", "_id": "11" } }
  23. { "location": [3.0, 2.3], "parking" : "false", "rating" : 6 }
  24. { "index": { "_index": "hotels-index", "_id": "12" } }
  25. { "location": [5.0, 1.0], "parking" : "true", "rating" : 3 }

Example response

Upon success, you should receive a “200-OK” status with entries for each document ID added to the index. The following response is truncated to only show one document:

  1. {
  2.   "took" : 140,
  3.   "errors" : false,
  4.   "items" : [
  5.     {
  6.       "index" : {
  7.         "_index" : "hotels-index",
  8.         "_id" : "1",
  9.         "_version" : 2,
  10.         "result" : "updated",
  11.         "_shards" : {
  12.           "total" : 1,
  13.           "successful" : 1,
  14.           "failed" : 0
  15.         },
  16.         "_seq_no" : 12,
  17.         "_primary_term" : 3,
  18.         "status" : 200
  19.       }
  20.     }
  21.   ]
  22. }

Step 3: Search your data with a filter

Now you can create a k-NN search that specifies filters by using query DSL Boolean clauses. You need to include your reference point to search for nearest neighbors. Provide an x-y coordinate for the point within the vector field, such as "vector": [ 5.0, 4.0].

To learn more about how to specify ranges with query DSL, see Range query.

Example request

The following request creates a k-NN query that only returns the top hotels rated between 8 and 10 and that provide parking. The filter criteria to indicate the range for the feedback ratings uses a range query and a term query clause to indicate “parking”:

  1. POST /hotels-index/_search
  2. {
  3.     "size": 3,
  4.     "query": {
  5.         "knn": {
  6.             "location": {
  7.                 "vector": [
  8.                     5.0,
  9.                     4.0
  10.                 ],
  11.                 "k": 3,
  12.                 "filter": {
  13.                     "bool": {
  14.                         "must": [
  15.                             {
  16.                                 "range": {
  17.                                     "rating": {
  18.                                         "gte": 8,
  19.                                         "lte": 10
  20.                                     }
  21.                                 }
  22.                             },
  23.                             {
  24.                                 "term": {
  25.                                     "parking": "true"
  26.                                 }
  27.                             }
  28.                         ]
  29.                     }
  30.                 }
  31.             }
  32.         }
  33.     }
  34. }

Sample Response

The following response indicates that only three hotels met the filter criteria:

  1. {
  2.   "took" : 47,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 3,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 0.72992706,
  16.     "hits" : [
  17.       {
  18.         "_index" : "hotels-index",
  19.         "_id" : "3",
  20.         "_score" : 0.72992706,
  21.         "_source" : {
  22.           "location" : [
  23.             4.9,
  24.             3.4
  25.           ],
  26.           "parking" : "true",
  27.           "rating" : 9
  28.         }
  29.       },
  30.       {
  31.         "_index" : "hotels-index",
  32.         "_id" : "6",
  33.         "_score" : 0.3012048,
  34.         "_source" : {
  35.           "location" : [
  36.             6.4,
  37.             3.4
  38.           ],
  39.           "parking" : "true",
  40.           "rating" : 9
  41.         }
  42.       },
  43.       {
  44.         "_index" : "hotels-index",
  45.         "_id" : "5",
  46.         "_score" : 0.24154587,
  47.         "_source" : {
  48.           "location" : [
  49.             3.3,
  50.             4.5
  51.           ],
  52.           "parking" : "true",
  53.           "rating" : 8
  54.         }
  55.       }
  56.     ]
  57.   }
  58. }

Additional complex filter query

Depending on how restrictive you want your filter to be, you can add multiple query types to a single request, such as term, wildcard, regexp, or range. You can then filter out the search results with the Boolean clauses must, should, and must_not.

Example request

The following request returns hotels that provide parking. This request illustrates multiple alternative mechanisms to obtain the parking filter criteria. It uses a regular expression for the value true, a term query for the key-value pair "parking":"true", a wildcard for the characters that spell “true”, and the must_not clause to eliminate hotels with “parking” set to false:

  1. POST /hotels-index/_search
  2. {
  3.     "size": 3,
  4.     "query": {
  5.         "knn": {
  6.             "location": {
  7.                 "vector": [
  8.                     5.0,
  9.                     4.0
  10.                 ],
  11.                 "k": 3,
  12.                 "filter": {
  13.                     "bool": {
  14.                         "must": {
  15.                             "range": {
  16.                                 "rating": {
  17.                                     "gte": 1,
  18.                                     "lte": 6
  19.                                 }
  20.                             }
  21.                         },
  22.                         "should": [
  23.                             {
  24.                                 "term": {
  25.                                     "parking": "true"
  26.                                 }
  27.                             },
  28.                             {
  29.                                 "wildcard": {
  30.                                     "parking": {
  31.                                         "value": "t*e"
  32.                                     }
  33.                                 }
  34.                             },
  35.                             {
  36.                                 "regexp": {
  37.                                     "parking": "[a-zA-Z]rue"
  38.                                 }
  39.                             }
  40.                         ],
  41.                         "must_not": [
  42.                             {
  43.                                 "term": {
  44.                                     "parking": "false"
  45.                                 }
  46.                             }
  47.                         ],
  48.                         "minimum_should_match": 1
  49.                     }
  50.                 }
  51.             }
  52.         }
  53.     }
  54. }

Example response

The following response indicates a few results for the search with filters:

  1. {
  2.   "took" : 94,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 1,
  6.     "successful" : 1,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : {
  12.       "value" : 3,
  13.       "relation" : "eq"
  14.     },
  15.     "max_score" : 0.8333333,
  16.     "hits" : [
  17.       {
  18.         "_index" : "hotels-index",
  19.         "_id" : "1",
  20.         "_score" : 0.8333333,
  21.         "_source" : {
  22.           "location" : [
  23.             5.2,
  24.             4.4
  25.           ],
  26.           "parking" : "true",
  27.           "rating" : 5
  28.         }
  29.       },
  30.       {
  31.         "_index" : "hotels-index",
  32.         "_id" : "7",
  33.         "_score" : 0.154321,
  34.         "_source" : {
  35.           "location" : [
  36.             4.2,
  37.             6.2
  38.           ],
  39.           "parking" : "true",
  40.           "rating" : 5
  41.         }
  42.       },
  43.       {
  44.         "_index" : "hotels-index",
  45.         "_id" : "12",
  46.         "_score" : 0.1,
  47.         "_source" : {
  48.           "location" : [
  49.             5.0,
  50.             1.0
  51.           ],
  52.           "parking" : "true",
  53.           "rating" : 3
  54.         }
  55.       }
  56.     ]
  57.   }
  58. }