Doc Opeartion

http://router_server represents the router service, $db_name is the created library name, $space_name is the created space name, and $id is the unique id of the data record.

_id is the unique identifier of the record generated by the server, which can be specified by the user. This unique identifier needs to be used to modify and delete data.

$id is a unique identifier generated by the server using the specified value when inserting data. The $id value cannot use special characters such as URL paths. If the record with the unique identifier already exists in the library, it will be updated and overwritten. Single Insertion ——–

document/upsert

If primary_id is set, the specified primary key will be used. If not set, generated by Vearch.

If the _id specified when inserting already exists, the existing data is updated; otherwise, it is inserted.

When the documents in the inserted data contain multiple pieces of data, it is a batch insertion. It is generally recommended that the number of batch insertions does not exceed 100 pieces.

Insertion and update now support passing in the values of only some fields. When inserting and only passing in some fields, the vector field must be included. There is no such restriction when updating.

Do not specify unique identification id when inserting

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "documents": [{
  6.         "field_int": 90399,
  7.         "field_float": 90399,
  8.         "field_double": 90399,
  9.         "field_string": "111399",
  10.         "field_vector": {
  11.             "feature": [...]
  12.         }
  13.     }, {
  14.         "field_int": 45085,
  15.         "field_float": 45085,
  16.         "field_double": 45085,
  17.         "field_string": "106085",
  18.         "field_vector": {
  19.             "feature": [...]
  20.         }
  21.     }, {
  22.         "field_int": 52968,
  23.         "field_float": 52968,
  24.         "field_double": 52968,
  25.         "field_string": "113968",
  26.         "field_vector": {
  27.             "feature": [...]
  28.         }
  29.     }]
  30. }
  31. ' http://router_server/document/upsert

field_vector is feature field. All field names, value types, and table structures are consistent

Specify unique identifier when inserting

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "documents": [{
  6.         "_id": "1000000",
  7.         "field_int": 90399,
  8.         "field_float": 90399,
  9.         "field_double": 90399,
  10.         "field_string": "111399",
  11.         "field_vector": {
  12.             "feature": [...]
  13.         }
  14.     }, {
  15.         "_id": "1000001",
  16.         "field_int": 45085,
  17.         "field_float": 45085,
  18.         "field_double": 45085,
  19.         "field_string": "106085",
  20.         "field_vector": {
  21.             "feature": [...]
  22.         }
  23.     }, {
  24.         "_id": "1000002",
  25.         "field_int": 52968,
  26.         "field_float": 52968,
  27.         "field_double": 52968,
  28.         "field_string": "113968",
  29.         "field_vector": {
  30.             "feature": [...]
  31.         }
  32.     }]
  33. }
  34. ' http://router_server/document/upsert

The format of the return value of the upsert interface is as follows

  1. {
  2.     "code": 0,
  3.     "msg": "success",
  4.     "total": 3,
  5.     "document_ids": [{
  6.         "_id": "-526059949411103803",
  7.         "status": 200,
  8.         "error": "success"
  9.     }, {
  10.         "_id": "1287805132970120733",
  11.         "status": 200,
  12.         "error": "success"
  13.     }, {
  14.         "_id": "-1948185285365684656",
  15.         "status": 200,
  16.         "error": "success"
  17.     }]
  18. }

total identifies the number of successful insertions, and document_ids returns the generated _id and insertion result information.

document/query

The /document/query interface is used to accurately search for data that exactly matches the query conditions. The search does not include vector data.

Two methods are supported: one is to obtain documents directly through primary keys, and the other is to obtain corresponding documents based on filter conditions.

If partition_id is set, get the corresponding document on the specified data partition. At this time, the meaning of document_id is the document number on the partition. document_id can be [0, max_docid] of the specified partition, and max_docid and partition information can be obtained through the cluster/health interface. Complete data for the cluster can be obtained this way.

Find data based on unique id identifier

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "query": {
  6.         "document_ids": ["6560995651113580768", "-5621139761924822824", "-104688682735192253"]
  7.     },
  8.     "vector_value": true
  9. }
  10. ' http://router_server/document/query

Get the corresponding document on the specified data partition. At this time, document_id can be [0, max_docid] of the specified partition.

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "query": {
  6.         "document_ids": [
  7.         "10000",
  8.         "10001",
  9.         "10002"
  10.         ],
  11.         "partition_id": "1"
  12.     },
  13.     "vector_value": true
  14. }
  15. ' http://router_server/document/query

Find based on Filter expression of custom scalar field

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "query": {
  6.         "filter": [
  7.         {
  8.             "range": {
  9.             "field_int": {
  10.                 "gte": 1000,
  11.                 "lte": 100000
  12.             }
  13.             }
  14.         },
  15.         {
  16.             "term": {
  17.             "field_string": [
  18.                 "322"
  19.             ]
  20.             }
  21.         }
  22.         ]
  23.     },
  24.     "vector_value": false
  25. }
  26. ' http://router_server/document/query

Query interface return format

  1. {
  2.     "code": 0,
  3.     "msg": "success",
  4.     "total": 3,
  5.     "documents": [{
  6.         "_id": "6560995651113580768",
  7.         "_source": {
  8.             "field_double": 202558,
  9.             "field_float": 102558,
  10.             "field_int": 1558,
  11.             "field_string": "1558"
  12.         }
  13.     }, {
  14.         "_id": "-5621139761924822824",
  15.         "_source": {
  16.             "field_double": 210887,
  17.             "field_float": 110887,
  18.             "field_int": 89887,
  19.             "field_string": "89887"
  20.         }
  21.     }, {
  22.         "_id": "-104688682735192253",
  23.         "_source": {
  24.             "field_double": 207588,
  25.             "field_float": 107588,
  26.             "field_int": 46588,
  27.             "field_string": "46588"
  28.         }
  29.     }]
  30. }

Parameter Description:

field name

field type

must

remarks

document_ids

json array

false

filter or document_ids must have one

partition_id

json array

false

specify get document on which partition

filter

json array

false

query criteria filtering: numeric filtering + label filtering

fields

json array

false

Specify which fields to return. By default, only the unique id and score are returned.

vector_value

bool

false

default false

size

int

false

Specify the number of returned results, the default is 50

Supports similarity retrieval based on specified ID or vector value, and returns the specified Top K most similar Documents.

Supports similarity retrieval based on the primary key id (Document ID) or vector value, together with the Filter expression of a custom scalar field.

document_ids passes in the unique record id. The background processing first queries the characteristics of the record based on the unique id, and then uses the characteristics to perform similar queries and returns matching results.

Search based on document_ids

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "query": {
  4.         "document_ids": [
  5.             "3646866681750952826"
  6.         ],
  7.         "filter": [
  8.         {
  9.             "range": {
  10.                 "field_int": {
  11.                     "gte": 1000,
  12.                     "lte": 100000
  13.                 }
  14.             }
  15.         }
  16.         ]
  17.     },
  18.     "retrieval_param": {
  19.         "metric_type": "L2"
  20.     },
  21.     "size": 3,
  22.     "db_name": "ts_db",
  23.     "space_name": "ts_space"
  24. }
  25. ' http://router_server/document/search

Search based on vector Supports single or multiple queries. Multiple queries can splice the features of multiple queries into a feature array (such as defining 128-dimensional features and querying 10 in batches. Then 10 128-dimensional features are spliced into a 1280-dimensional feature array in order and assigned to the feature field), After receiving the request, the background splits it according to the characteristic field dimensions defined by the table structure, and returns the matching results in order.

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "query": {
  4.         "vector": [
  5.         {
  6.             "field": "field_vector",
  7.             "feature": [
  8.                 "..."
  9.             ]
  10.         }
  11.         ],
  12.         "filter": [
  13.         {
  14.             "range": {
  15.                 "field_int": {
  16.                     "gte": 1000,
  17.                     "lte": 100000
  18.                 }
  19.             }
  20.         }
  21.         ]
  22.     },
  23.     "retrieval_param": {
  24.         "metric_type": "L2"
  25.     },
  26.     "size": 3,
  27.     "db_name": "ts_db",
  28.     "space_name": "ts_space"
  29. }
  30. ' http://router_server/document/search

multi-vector search The table space supports multiple feature fields when defined, so the query can support the features of the corresponding data.

Take two vectors for each record as an example: define table structure fields

  1. {
  2.     "field1": {
  3.         "type": "vector",
  4.         "dimension": 128
  5.     },
  6.     "field2": {
  7.         "type": "vector",
  8.         "dimension": 256
  9.     }
  10. }

field1 and field2 are both vector fields. When querying, the search conditions can specify two vectors:

  1. {
  2.     "query": {
  3.         "vector": [{
  4.             "field": "filed1",
  5.             "feature": [0.1, 0.2, 0.3, 0.4, 0.5],
  6.             "min_score": 0.9
  7.         },
  8.         {
  9.             "field": "filed2",
  10.             "feature": [0.8, 0.9],
  11.             "min_score": 0.8
  12.         }]
  13.     }
  14. }

The intersection of field1 and field2 filtering results is obtained. Other parameters and request addresses are the same as ordinary queries.

search interface return format

  1. {
  2.     "code": 0,
  3.     "msg": "success",
  4.     "documents": [
  5.         [{
  6.             "_id": "6979025510302030694",
  7.             "_score": 16.55717658996582,
  8.             "_source": {
  9.                 "field_double": 207598,
  10.                 "field_float": 107598,
  11.                 "field_int": 6598,
  12.                 "field_string": "6598"
  13.             }
  14.         }, {
  15.             "_id": "-104688682735192253",
  16.             "_score": 17.663991928100586,
  17.             "_source": {
  18.                 "field_double": 207588,
  19.                 "field_float": 107588,
  20.                 "field_int": 46588,
  21.                 "field_string": "46588"
  22.             }
  23.         }, {
  24.             "_id": "8549822044854277588",
  25.             "_score": 17.88829803466797,
  26.             "_source": {
  27.                 "field_double": 220413,
  28.                 "field_float": 120413,
  29.                 "field_int": 99413,
  30.                 "field_string": "99413"
  31.             }
  32.         }]
  33.     ]
  34. }

The overall json structure of the query parameters is as follows:

  1. {
  2.     "query": {
  3.         "vector": [],
  4.         "filter": []
  5.     },
  6.     "retrieval_param": {"nprobe": 20},
  7.     "fields": ["field1", "field2"],
  8.     "is_brute_search": 0,
  9.     "online_log_level": "debug",
  10.     "quick": false,
  11.     "vector_value": false,
  12.     "load_balance": "leader",
  13.     "l2_sqrt": false,
  14.     "size": 10
  15. }

Parameter Description:

field name

field type

must

remarks

vector

json array

false

query feature, vector or document_ids must have one

document_ids

json array

false

query feature, vector or document_ids must have one

filter

json array

false

query criteria filtering: numeric filtering + label filtering

fields

json array

false

Specify which fields to return. By default, only the unique id and score are returned.

is_brute_search

int

false

default 0

online_log_level

string

false

The value is debug, which turns on printing debugging logs.

quick

bool

false

default false

vector_value

bool

false

default false

load_balance

string

false

Load balancing algorithm, random by default

l2_sqrt

bool

false

The default is false, and the root sign is used for the l2 distance calculation result.

sort

json array

false

Specify field sorting (only for matching results, not the whole)

size

int

false

Specify the number of returned results, the default is 50

The retrieval_param parameter specifies the parameters for model calculation. Different models support different parameters, as shown in the following example:

  • metric_type: calculation type, supports InnerProduct and L2, the default is L2.

  • nprobe: Search bucket number.

  • recall_num: The number of recalls, the default is equal to the value of size in the query parameter, set the number to search from the index, and then calculate the size closest values.

  • parallel_on_queries: Default 1, parallelism between searches; 0 represents parallelism between buckets.

  • efSearch: distance of graph traversal.

IVFPQ:

  1. "retrieval_param": {
  2.     "parallel_on_queries": 1,
  3.     "recall_num" : 100,
  4.     "nprobe": 80,
  5.     "metric_type": "L2"
  6. }

GPU:

  1. "retrieval_param": {
  2.     "recall_num" : 100,
  3.     "nprobe": 80,
  4.     "metric_type": "L2"
  5. }

HNSW:

  1. "retrieval_param": {
  2.     "efSearch": 64,
  3.     "metric_type": "L2"
  4. }

IVFFLAT:

  1. "retrieval_param": {
  2.     "parallel_on_queries": 1,
  3.     "nprobe": 80,
  4.     "metric_type": "L2"
  5. }

FLAT:

  1. "retrieval_param": {
  2.     "metric_type": "L2"
  3. }
  • vector json structure elucidation:
  1. "vector": [{
  2.           "field": "field_name",
  3.           "feature": [0.1, 0.2, 0.3, 0.4, 0.5],
  4.           "min_score": 0.9,
  5.           "boost": 0.5
  6.        }]

  1. vector: Support multiple (including multiple feature fields when defining table structure correspondingly).

  2. field: Specifies the name of the feature field when the table is created.

  3. feature: Transfer feature, dimension must be the same when defining table structure

  4. min_score: Specify the minimum score of the returned result, min_score can specify the minimum score of the returned result, and max_score can specify the maximum score. For example, set “min_score”: 0.8, “max_score”: 0.95 to filter the result of 0.8 <= score <= 0.95. At the same time, another way of score filtering is to use the combination of “symbol”: “>=”, “value”: 0.9. The value types supported by symbol include: >, >=, < and <= four kinds, and the values of value.

  5. boost: Specify the weight of similarity. For example, if the similarity score of two vectors is 0.7 and boost is set to 0.5, the returned result will multiply the score 0.7 * 0.5, which is 0.35.Does not take effect when using a single vector.

  • filter json structure elucidation:
  1. "filter": [
  2.     {
  3.         "range": {
  4.             "field_name": {
  5.                 "gte": 160,
  6.                 "lte": 180
  7.             }
  8.         }
  9.     },
  10.     {
  11.         "term": {
  12.             "field1": ["100", "200", "300"],
  13.             "operator": "or"
  14.         }
  15.     },
  16.     {
  17.         "term": {
  18.             "field2": ["a", "b", "c"],
  19.             "operator": "and"
  20.         }
  21.     },
  22.     {
  23.         "term": {
  24.             "field3": ["A1", "B2"],
  25.             "operator": "not"
  26.         }
  27.     }
  28. ]

  1. filter: Multiple conditions are supported. Multiple conditions are intersecting.

  2. range: Specify to use the numeric field integer / float filtering, the file name is the numeric field name, gte and lte specify the range, lte is less than or equal to, gte is greater than or equal to, if equivalent filtering is used, lte and gte settings are the same value. The above example shows that the query field_name field is greater than or equal to 160 but less than or equal to 180.

  3. term: With label filtering, field_name is a defined label field, which allows multiple value filtering. You can intersect “operator”: “or”, merge: “operator”: “and”. The above example indicates that the query field name segment value is “100”, “200” or “300”.

  • is_brute_search: Specify the query type. 0 means to use index if the feature has been created, and violent search if it has not been created; - 1 means to use index only for search, and 1 means not to use index only for violent search. The default value is 0.

  • quick: By default, the PQ recall vector is calculated and refined in the search results. In order to speed up the processing speed of the server to true, only recall can be specified, and no calculation and refined.

  • vector_value: In order to reduce the network overhead, the search results contain only scalar information fields without feature data by default, and set to true to specify that the returned results contain the original feature data.

  • online_log_level: Set “debug” to specify to print more detailed logs on the server, which is convenient for troubleshooting in the development and test phase.

  • size: Specifies the maximum number of results to return. use the size value specified in the URL first.

  • load_balance: leader，random，no_leader，least_connection，default random。

document/delete

Deletion supports two methods: specifying document_ids and filtering conditions.

Delete specified document_ids

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "query": {
  6.         "document_ids": ["4501743250723073467", "616335952940335471", "-2422965400649882823"]
  7.     }
  8. }
  9. ' http://router_server/document/delete

Delete documents that meet the filter conditions. size specifies the number of items to delete for each data fragment.

  1. curl -H "content-type: application/json" -XPOST -d'
  2. {
  3.     "db_name": "ts_db",
  4.     "space_name": "ts_space",
  5.     "query": {
  6.         "filter": [
  7.         {
  8.             "range": {
  9.             "field_int": {
  10.                 "gte": 1000,
  11.                 "lte": 100000
  12.             }
  13.             }
  14.         },
  15.         {
  16.             "term": {
  17.             "field_string": [
  18.                 "322"
  19.             ]
  20.             }
  21.         }
  22.         ]
  23.     },
  24.     "size": 3
  25. }
  26. ' http://router_server/document/delete

Delete interface return format

  1. {
  2.     "code": 0,
  3.     "msg": "success",
  4.     "total": 3,
  5.     "document_ids": ["4501743250723073467", "616335952940335471", "-2422965400649882823"]
  6. }