Doc Opeartion

Doc Opeartion

http://${VEARCH_URL} represents the vearch service, $db_name is the created library name, $space_name is the created space name, and $id is the unique id of the data record.

_id is the unique identifier of the record generated by the server, which can be specified by the user. This unique identifier needs to be used to modify and delete data.

$id is a unique identifier generated by the server using the specified value when inserting data. The $id value cannot use special characters such as URL paths. If the record with the unique identifier already exists in the library, it will be updated and overwritten.

document/upsert

If primary_id is set, the specified primary key will be used. If not set, generated by Vearch.

If the _id specified when inserting already exists, the existing data is updated; otherwise, it is inserted.

When the documents in the inserted data contain multiple pieces of data, it is a batch insertion. It is generally recommended that the number of batch insertions does not exceed 100 pieces.

Insertion and update now support passing in the values of only some fields. When inserting and only passing in some fields, the vector field must be included. There is no such restriction when updating.

Do not specify unique identification id when inserting

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "documents": [{
        "field_int": 90399,
        "field_float": 90399,
        "field_double": 90399,
        "field_string": "111399",
        "field_vector": [...]
    }, {
        "field_int": 45085,
        "field_float": 45085,
        "field_double": 45085,
        "field_string": "106085",
        "field_vector":  [...]
    }, {
        "field_int": 52968,
        "field_float": 52968,
        "field_double": 52968,
        "field_string": "113968",
        "field_vector":  [...]
    }]
}
' http://${VEARCH_URL}/document/upsert

field_vector is vector field. All field names, value types, and table structures are consistent

Specify unique identifier when inserting

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "documents": [{
        "_id": "1000000",
        "field_int": 90399,
        "field_float": 90399,
        "field_double": 90399,
        "field_string": "111399",
        "field_vector":  [...]
    }, {
        "_id": "1000001",
        "field_int": 45085,
        "field_float": 45085,
        "field_double": 45085,
        "field_string": "106085",
        "field_vector": [...]
    }, {
        "_id": "1000002",
        "field_int": 52968,
        "field_float": 52968,
        "field_double": 52968,
        "field_string": "113968",
        "field_vector": [...]
    }]
}
' http://${VEARCH_URL}/document/upsert

The format of the return value of the upsert interface is as follows

{
    "code": 0,
    "msg": "success",
    "data": {
        "total": 3,
        "document_ids": [
            {
                "_id": "-526059949411103803"
            },
            {
                "_id": "1287805132970120733"
            },
            {
                "_id": "-1948185285365684656"
            }
        ]
    }
}

total identifies the number of successful insertions, and document_ids returns the generated _id and insertion result information.

document/query

The /document/query interface is used to accurately search for data that exactly matches the query conditions. The search does not include vector data.

Two methods are supported: one is to obtain documents directly through primary keys, and the other is to obtain corresponding documents based on filter conditions.

If partition_id is set, get the corresponding document on the specified data partition. At this time, the meaning of document_id is the document number on the partition. document_id can be [0, max_docid] of the specified partition, and max_docid and partition information can be obtained through the http://master_server/dbs/$db_name/spaces/$space_name interface. Complete data for the cluster can be obtained this way.

query Parameter Description:

field name	field type	must	remarks
document_ids	string array	否	Query conditions, filter and document_ids must contain one item
partition_id	int	否	Specify which partition to obtain data from, used in combination with document_ids
filters	json	否	Query condition filtering: numerical filtering + label filtering, filter and document_ids must contain one item
fields	string array	否	Specify which fields to return. By default, all fields except vector fields are returned.
vector_value	bool	否	Defaults to false, whether to return a vector
limit	int	否	Specify the number of returned results, the default is 50

filters json format description:

"filters": [
    "operator": "AND",
    "conditions": [
        {
            "field": "field_int",
            "operator": ">=",
            "value": 1
        },
        {
            "field": "field_int",
            "operator": "<=",
            "value": 3
        },
        {
            "field": "field_string",
            "operator": "IN",
            "value": ["aaa", "bbb"]
        }
    ]
]

filters format description:

field name	field type	must	remarks
operator	string	true	only support AND now
conditions	json array	true

Filter conditions support multiple conditions, and there is an intersection relationship between multiple conditions, that is, the outermost operator currently supports AND.

conditions format description:

field name	field type	must	remarks
field	string	true
operator	string	true	support >, >=, <, <=, IN
value	json	true

(2) conditions specific filtering conditions, currently supports two types of field type filtering, numeric type and string type (including string array type) Numeric type operators: >, >=, <, <=; String operator type IN

Find data based on unique id identifier

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "document_ids": ["6560995651113580768", "-5621139761924822824", "-104688682735192253"]
    "vector_value": true
}
' http://${VEARCH_URL}/document/query

Get the corresponding document on the specified data partition. At this time, document_id can be [0, max_docid] of the specified partition.

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "document_ids": ["0", "1", "2"],
    "partition_id": 1,
    "vector_value": true
}
' http://${VEARCH_URL}/document/query

Find based on Filter expression of custom scalar field

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "filters": {
        "operator": "AND",
        "conditions": [
            {
                "field": "field_int",
                "operator": ">=",
                "value": 1
            },
            {
                "field": "field_int",
                "operator": "<=",
                "value": 3
            }
        ]
    }
}
' http://${VEARCH_URL}/document/query

Query interface return format

{
    "code": 0,
    "msg": "success",
    "data": {
        "total": 3,
        "documents": [{
            "_id": "6560995651113580768",
            "field_double": 202558,
            "field_float": 102558,
            "field_int": 1558,
            "field_string": "1558"
        }, {
            "_id": "-5621139761924822824",
            "field_double": 210887,
            "field_float": 110887,
            "field_int": 89887,
            "field_string": "89887"
        }, {
            "_id": "-104688682735192253",
            "field_double": 207588,
            "field_float": 107588,
            "field_int": 46588,
            "field_string": "46588"
        }]
    }
}

document/search

Supports similarity retrieval based on vector value, together with the Filter expression of a custom scalar field, and returns the specified Top K most similar Documents.

Parameter Description:

field name	field type	must	remarks
vectors	json array	true	embedding value
filters	json array	false	query criteria filtering: numeric filtering + label filtering
fields	json array	false	Specify which fields to return. By default, only the unique id and score are returned.
is_brute_search	int	false	default 0
vector_value	bool	false	default false
load_balance	string	false	Load balancing algorithm, random by default
limit	int	false	Specify the number of returned results, the default is 50
ranker	json	false	For further processing of multi-vector results, currently only WeightedRanker is supported, which specifies the weight of similarity.
index_params	json	false	Specify parameters for model calculation

The overall json structure of the query parameters is as follows:

{
    "vectors": [],
    "filters": [],
    "index_params": {"nprobe": 20},
    "fields": ["field1", "field2"],
    "is_brute_search": 0,
    "vector_value": false,
    "load_balance": "leader",
    "limit": 10,
    "ranker": {
        "type": "WeightedRanker",
        "params": [0.5, 0.5]
    }
}

The index_params parameter specifies the parameters for model calculation. Different models support different parameters, as shown in the following example:

metric_type: calculation type, supports InnerProduct and L2, the default is L2.
nprobe: Search bucket number.
recall_num: The number of recalls, the default is equal to the value of size in the query parameter, set the number to search from the index, and then calculate the size closest values.
parallel_on_queries: Default 1, parallelism between searches; 0 represents parallelism between buckets.
efSearch: distance of graph traversal.

IVFPQ:

"index_params": {
    "parallel_on_queries": 1,
    "recall_num" : 100,
    "nprobe": 80,
    "metric_type": "L2"
}
When recall_num is set, the original vector will be used for calculation rearrangement (fine sorting)

GPU:

"index_params": {
    "recall_num" : 100,
    "nprobe": 80,
    "metric_type": "L2"
}

HNSW:

"index_params": {
    "efSearch": 64,
    "metric_type": "L2"
}

IVFFLAT:

"index_params": {
    "parallel_on_queries": 1,
    "nprobe": 80,
    "metric_type": "L2"
}

FLAT:

"index_params": {
    "metric_type": "L2"
}

vectors json structure elucidation:

"vectors": [{
    "field": "field_name",
    "feature": [0.1, 0.2, 0.3, 0.4, 0.5],
    "min_score": 0.9
}]

vectors: Support multiple (including multiple feature fields when defining table structure correspondingly).
field: Specifies the name of the feature field when the table is created.
feature: Transfer feature, dimension must be the same when defining table structure
min_score: Specify the minimum score of the returned result, min_score can specify the minimum score of the returned result, and max_score can specify the maximum score. For example, set “min_score”: 0.8, “max_score”: 0.95 to filter the result of 0.8 <= score <= 0.95. At the same time, another way of score filtering is to use the combination of “symbol”: “>=”, “value”: 0.9. The value types supported by symbol include: >, >=, < and <= four kinds, and the values of value.

filter json structure elucidation:

Refer to the description of filter json in the query interface section.

is_brute_search: Specify the query type. 0 means to use index if the feature has been created, and violent search if it has not been created; - 1 means to use index only for search, and 1 means not to use index only for violent search. The default value is 0.
vector_value: In order to reduce the network overhead, the search results contain only scalar information fields without feature data by default, and set to true to specify that the returned results contain the original feature data.
limit: Specifies the maximum number of results to return. use the limit value specified in the URL first.
load_balance: leader, random, no_leader, least_connection, default random。

Search based on vector Supports single or multiple queries. Multiple queries can splice the features of multiple queries into a feature array (such as defining 128-dimensional features and querying 10 in batches. Then 10 128-dimensional features are spliced into a 1280-dimensional feature array in order and assigned to the feature field), After receiving the request, the background splits it according to the characteristic field dimensions defined by the table structure, and returns the matching results in order.

curl -H "content-type: application/json" -XPOST -d'
{
    "vectors": [
        {
            "field": "field_vector",
            "feature": [
                "..."
            ]
        }
    ],
    "filters": {
        "operator": "AND",
        "conditions": [
            {
                "field": "field_int",
                "operator": ">=",
                "value": 1
            },
            {
                "field": "field_int",
                "operator": "<=",
                "value": 3
            },
            {
                "field": "field_string",
                "operator": "IN",
                "value": [
                    "aaa",
                    "bbb"
                ]
            }
        ]
    },
    "index_params": {
        "metric_type": "L2"
    },
    "limit": 3,
    "db_name": "ts_db",
    "space_name": "ts_space"
}
' http://${VEARCH_URL}/document/search

multi-vector search The table space supports multiple feature fields when defined, so the query can support the features of the corresponding data.

Take two vectors for each record as an example: define table structure fields

{
    "field_vector1": {
        "type": "vector",
        "dimension": 128
    },
    "field_vector2": {
        "type": "vector",
        "dimension": 256
    }
}

field1 and field2 are both vector fields. When querying, the search conditions can specify two vectors:

{
    "vectors": [{
        "field": "field_vector1",
        "feature": [...]
    },
    {
        "field": "field_vector2",
        "feature": [...]
    }],
    "ranker": {
        "type": "WeightedRanker",
        "params": [0.5, 0.5]
    }
}

The intersection of field1 and field2 filtering results is obtained. Other parameters and request addresses are the same as ordinary queries.

search interface return format

{
    "code": 0,
    "msg": "success",
    "data": {
        "documents": [
            [{
                "_id": "6979025510302030694",
                "_score": 16.55717658996582,
                "field_double": 207598,
                "field_float": 107598,
                "field_int": 6598,
                "field_string": "6598"
            }, {
                "_id": "-104688682735192253",
                "_score": 17.663991928100586,
                "field_double": 207588,
                "field_float": 107588,
                "field_int": 46588,
                "field_string": "46588"
            }, {
                "_id": "8549822044854277588",
                "_score": 17.88829803466797,
                "field_double": 220413,
                "field_float": 120413,
                "field_int": 99413,
                "field_string": "99413"
            }]
        ]
    }
}

document/delete

Deletion supports two methods: specifying document_ids and filtering conditions.

Delete specified document_ids

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "document_ids": ["4501743250723073467", "616335952940335471", "-2422965400649882823"]
}
' http://${VEARCH_URL}/document/delete

Delete documents that meet the filter conditions. size specifies the number of items to delete for each data fragment.

curl -H "content-type: application/json" -XPOST -d'
{
    "db_name": "ts_db",
    "space_name": "ts_space",
    "filters": {
        "operator": "AND",
        "conditions": [
            {
                "field": "field_int",
                "operator": ">=",
                "value": 1
            },
            {
                "field": "field_int",
                "operator": "<=",
                "value": 3
            }
        ]
    },
    "limit": 3
}
' http://${VEARCH_URL}/document/delete

Delete interface return format

{
    "code": 0,
    "msg": "success",
    "data": {
        "total": 3,
        "document_ids": ["4501743250723073467", "616335952940335471", "-2422965400649882823"]
    }
}