1. Basic Concepts
1.1 Basic Concepts
1.1.1 Bitset
In Milvus, a bitset is an array of digits, which can be used to express certain data compactly and efficiently instead of using int, float or chars. By default the bitness is only set to .0 10``1 if certain requirements are met
1.1.2 Channel
There are two different Channels in Milvus, called PChannel and VChannel, where each PChannel corresponds to the topic of log storage, and each VChannel corresponds to the fragments in the collection.
PChannel
PChannel represents a physical channel. Each PChannel corresponds to a topic for log storage. By default, a set of 256 PChannels will be allocated to store logs that record data insertions, deletions, and updates when the Milvus cluster starts.
VChannel
VChannel represents a logical channel. Each VChannel represents a shard in the collection. Each collection will be assigned a set of VChannels for record data insertion, deletion and update. VChannels are logically separated but physically share resources
1.1.3 Collection
The collection in Milvus is equivalent to the table in the relational database management system (RDBMS). In Milvus, collections are used to store and manage entities.
1.1.4 Dependency
A dependency is a program that another program depends on to work. Dependencies of Milvus include etcd (to store metadata), MinIO or S3 (object storage) and Pulsar (to manage snapshot logs).
1.1.5 Entity
An entity (entity) consists of a set of fields that represent real-world objects. Each entity in Milvus is represented by a unique primary key.
You can customize the primary key. If not manually configured, Milvus will automatically assign primary keys to entities. If you choose to configure your own custom primary key, please note that Milvus does not currently support primary key deduplication. Therefore, there may be duplicate primary keys in the same collection.
1.1.6 Field
is the unit that constitutes an entity. Fields can be structured data (e.g. numbers, strings) or vectors.
Since Milvus 2.0, scalar filtering is available!
1.1.7 Partition
Partition (partition) is a part of collection (collection). Milvus supports splitting collection data into multiple parts in physical storage. This process is called partitioning, and each partition can contain multiple segments.
1.1.8 Schema
Mode (Schema) is the meta-information that defines data types and data attributes. Each collection (collection) has its own collection schema, which defines all the fields of the collection, automatic ID (primary key) assignment enablement and collection description. Also included in the collection schema is a field schema that defines the name, data type, and other properties of the field.
1.1.9 Segment
Segment is a data file automatically created by Milvus to save inserted data. A collection can have multiple segments, and a segment can have multiple entities. During a vector similarity search, Milvus scans each segment and returns the search results. A segment can be growing or closed. A growing segment continues to receive newly inserted data until it is sealed. The sealed segment will no longer receive any new data and will be flushed to object storage, leaving new data to be inserted into the newly created growing segment. A growing segment will be sealed either because the number of entities it holds reaches a predefined threshold, or because the extent of the "growth" state exceeds the specified limit.
1.1.10 Sharding
refers to assigning write operations to different nodes to make full use of the parallel computing potential of the Milvus cluster to write data. By default, a single collection contains two shards. Milvus adopts a sharding method based on the primary key hash. Milvus' development roadmap includes support for more flexible sharding methods, such as random sharding and custom sharding.
Partitioning reduces the read load by specifying a partition name, while sharding spreads the write load across multiple servers.
1.1.11 Vector index
The vector index is a reorganized data structure derived from the original data, which can greatly speed up the process of vector similarity search. Milvus supports multiple vector index types.
1.1.12 Embedding Vector
The embedding vector is the feature abstraction of unstructured data, such as emails, IoT sensor data, Instagram photos, protein structures, etc. Mathematically, an embedding vector is an array of floating point or binary numbers. Modern embedding techniques are used to convert unstructured data into embedding vectors.
1.1.13 Unstructured data
Unstructured data, including images, video, audio and natural language, is information that does not follow a predefined model or organization. This data type accounts for about 80% of the world's data and can be converted into vectors using various artificial intelligence (AI) and machine learning (ML) models.
1.1.14 Normalization
Normalization refers to the process of converting the embedding (vector) into a norm equal to 1. If the inner product (IP) is used to compute embedding similarity, all embeddings must be normalized. After normalization, the inner product is equal to the cosine similarity.
1.2 Vector index
1.2.1 In-memory Index
This topic lists the various types of in-memory indexes supported by Milvus, the most suitable scenarios for each index, and the parameters that users can configure to achieve better search performance.
Indexing is the process of organizing data efficiently, and it plays an important role in making similarity searches useful by dramatically speeding up time-consuming queries on large datasets.
To improve query performance, an index type can be specified for each vector field.
Currently, only one index type is supported for vector fields. Milvus automatically deletes the old index when switching the index type.
1.2.2 ANNS vector indexes
Most vector index types supported by Milvus use the Approximate Nearest Neighbor Search (ANNS) algorithm. Compared with precise retrieval, which is usually very time-consuming, the core idea of ANNS is no longer limited to returning the most accurate results, but only searches the neighbors of the target. ANNS improves retrieval efficiency by sacrificing accuracy within an acceptable range.
According to the implementation method, the ANNS vector index can be divided into four categories:
Tree-based index
Graph-based index
Hash-based index
Quantization-based index
1.2.3 On-disk Index
To improve query performance, you can specify an index type for each vector field.
Currently, only one index type is supported for vector fields. Milvus automatically deletes the old index when switching the index type.
prerequisite
Make sure you have set make disk_index=ON
Milvus is running on Ubuntu 18.04.6 or newer
The path ${MILVUS_ROOT_path}/MILVUS/data is mounted to NVMe SSD for full performance.
restrictive
Only floating-point vectors of at least 32 dimensions can be used in the data.
Only use Euclidean distance (L2) or inner product (IP) to measure the distance between vectors. Two ports used to
connect Milvus 2.1 Milvus uses two ports 19530 and 9091
Port 19530 is used for gRPC. When you use different Milvus SDK to connect to Milvus server, it is default port
9091 port for RESTful API. When you use the HTTP client to connect to the Milvus server, use
2.2 to create a connection Milvus
creates a connection code as follows
final MilvusServiceClient milvusClient = new MilvusServiceClient(
ConnectParam.newBuilder()
.withHost("localhost")
.withPort(19530)
.build()
);
// Release the connection
milvusClient.close()
The parameters are explained as follows:
Host: IP address of the Milvus server.
Port: Port of the Milvus server
Three collection management
3.1 Create
a collection A collection consists of one or more partitions. When creating a new collection, Milvus creates the default partition _default.
The following example builds a two-shard collection named book, where the primary key field is named book_id, a word_count type is INT64 scalar field (field), and a two-dimensional floating-point vector field is named book_intro.
The collection to be created must contain a primary key field and a vector field. INT64 and String are supported data types on primary key fields.
FieldType fieldType1 = FieldType.newBuilder()
.withName("book_id")
.withDataType(DataType.Int64)
.withPrimaryKey(true)
.withAutoID(false)
.build();
FieldType fieldType2 = FieldType.newBuilder()
.withName("word_count")
.withDataType(DataType.Int64)
.build();
FieldType fieldType3 = FieldType.newBuilder()
.withName("book_intro")
.withDataType(DataType.FloatVector)
.withDimension(2)
.build();
CreateCollectionParam createCollectionReq = CreateCollectionParam.newBuilder()
.withCollectionName("book")
.withDescription("Test book search")
.withShardsNum(2)
.addFieldType(fieldType1)
.addFieldType(fieldType2)
.addFieldType(fieldType3)
.build();
Parameter Description Option
Name Name of the field to create. N/A
Description Description of the field to create. N/A
DataType Data type of the field to create. For primary key field:entity.FieldTypeInt64 (numpy.int64)For scalar field:entity.FieldTypeBool (Boolean)entity.FieldTypeInt64 (numpy.int64)entity.FieldTypeFloat (numpy.float32)entity.FieldTypeDouble (numpy.double)For vector field:entity.FieldTypeBinaryVector (Binary vector)entity.FieldTypeFloatVector (Float vector)
PrimaryKey (Mandatory for primary key field) Switch to control if the field is primary key field. True or False
AutoID Switch to enable or disable Automatic ID (primary key) allocation. True or False
Dimension (Mandatory for vector field) Dimension of the vector. [1, 32768]
CollectionName Name of the collection to create. N/A
Description (Optional) Description of the collection to create. N/A
ShardsNum Number of the shards for the collection to create. [1,64]
3.2 Use constraints to create a collection
milvusClient.createCollection( createCollectionReq);
3.3 Check the collection information
3.3.1 Determine whether the collection exists
R<Boolean> respHasCollection = milvusClient.hasCollection(
HasCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
if (respHasCollection.getData() == Boolean.TRUE) {
System.out.println("Collection exists.");
}
3.3.2 Check Collection Details
R<DescribeCollectionResponse> respDescribeCollection = milvusClient.describeCollection(
// Return the name and schema of the collection.
DescribeCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
DescCollResponseWrapper wrapperDescribeCollection = new DescCollResponseWrapper(respDescribeCollection.getData());
System.out.println(wrapperDescribeCollection);
R<GetCollectionStatisticsResponse> respCollectionStatistics = milvusClient.getCollectionStatistics(
// Return the statistics information of the collection.
GetCollectionStatisticsParam.newBuilder()
.withCollectionName("book")
.build()
);
GetCollStatResponseWrapper wrapperCollectionStatistics = new GetCollStatResponseWrapper(respCollectionStatistics.getData());
System.out.println("Collection row count: " + wrapperCollectionStatistics.getRowCount());
3.3.3 Traversing all collections
R<ShowCollectionsResponse> respShowCollections = milvusClient.showCollections(
ShowCollectionsParam.newBuilder().build()
);
System.out.println(respShowCollections);
3.4 Deleting a collection
Deleting a collection will irreversibly delete all the data in it.
milvusClient.dropCollection(
DropCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
3.5 Collection aliases
Collection aliases are globally unique, so the same alias cannot be assigned to different collections. However, you can assign multiple aliases to a collection.
3.5.1 Set collection alias
milvusClient.createAlias(
CreateAliasParam.newBuilder()
.withCollectionName("book")
.withAlias("publication")
.build()
);
Parameter Description
CollectionName Name of the collection to create alias on.
Alias Collection alias to create.
3.5.2 Delete collection alias
milvusClient.dropAlias(
DropAliasParam.newBuilder()
.withAlias("publication")
.build()
);
3.5.3 Modify collection alias
milvusClient.alterAlias(
AlterAliasParam.newBuilder()
.withCollectionName(“book”)
.withAlias(“publication”)
.build()
);
Parameter Description
CollectionName Name of the collection to alter alias to.
Alias Collection alias to alter.
3.6 Loading Collections
This topic describes how collections are loaded into memory prior to searching or querying. All search and query operations in Milvus are performed in memory.
Milvus2.1 allows users to load collections as multiple copies to utilize CPU and memory resources of additional query nodes. This feature increases overall QPS and throughput without requiring additional hardware. In the current version, PyMilvus supports it.
Notice:
In the current version, the amount of data to be loaded must be below 90% of the total memory resources of all query nodes in order to reserve memory resources for the execution engine.
In the current version, all online query nodes will be divided into multiple replica groups according to the replica number specified by the user. All replica groups should have the minimum memory resources to load one replica of the provided collection. Otherwise, an error will be returned.
milvusClient.loadCollection(
LoadCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
There are restrictive conditions as follows:
An error will be returned when attempting to load a partition when the parent collection is already loaded. Future releases will support freeing partitions from loaded collections and then (if needed) loading additional partitions.
When attempting to load a collection that is already loaded, "Loading succeeded" will be returned.
An error will be returned when attempting to load the collection when the subpartition is already loaded. A future release will support loading collections when some partitions are already loaded.
It is not allowed to load different partitions in the same collection via separate RPCs.
3.7 Release collections
milvusClient.releaseCollection(
ReleaseCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
Four Partition Management
Milvus allows you to divide a large amount of vector data into a small number of partitions. Searches and other operations can then be restricted to a single partition to improve performance.
A collection consists of one or more partitions.
When creating a new collection, Milvus creates the default partition _default.
4.1 Create partitions
milvusClient.createPartition(
CreatePartitionParam.newBuilder()
.withCollectionName("book")
.withPartitionName("novel")
.build()
);
Parameter Description
CollectionName Name of the collection to create a partition in.
PartitionName Name of the partition to create.
4.2 Check the partition information
4.2.1 Determine whether the partition exists
R<Boolean> respHasPartition = milvusClient.hasPartition(
HasPartitionParam.newBuilder()
.withCollectionName("book")
.withPartitionName("novel")
.build()
);
if (respHasPartition.getData() == Boolean.TRUE) {
System.out.println("Partition exists.");
}
4.2.2 Traverse all partitions
R<ShowPartitionsResponse> respShowPartitions = milvusClient.showPartitions(
ShowPartitionsParam.newBuilder()
.withCollectionName("book")
.build()
);
System.out.println(respShowPartitions);
4.3 Deleting a partition
Deleting a partition will irreversibly delete all the data in it.
milvusClient.dropPartition(
DropPartitionParam.newBuilder()
.withCollectionName("book")
.withPartitionName("novel")
.build()
);
4.4 Loading Partitions
Loading partitions into memory instead of entire collections can significantly reduce memory usage. All search and query operations in Milvus are performed in memory.
Milvus2.1 allows users to load partitions as multiple copies to utilize CPU and memory resources of additional query nodes. This feature improves overall QPS and throughput with additional hardware. In the current version, PyMilvus supports it.
Note the following:
In the current version, the amount of data to be loaded must be below 90% of the total memory resources of all query nodes in order to reserve memory resources for the execution engine.
In the current version, all online query nodes will be divided into multiple replica groups according to the replica number specified by the user. All replica groups should have the minimum memory resources to load one replica of the provided collection. Otherwise, an error will be returned.
milvusClient.loadPartitions(
LoadPartitionsParam.newBuilder()
.withCollectionName(“book”)
.withPartitionNames([“novel”])
.build()
);
parameter Description
CollectionName Name of the collection to load partitions from.
PartitionNames List of names of the partitions to load.
The restrictive conditions are as follows:
An error will be returned when attempting to load a partition when the parent collection is already loaded. Future releases will support freeing partitions from loaded collections and then (if needed) loading additional partitions.
When attempting to load a collection that is already loaded, "Loading succeeded" will be returned.
An error will be returned when attempting to load the collection when the subpartition is already loaded. A future release will support loading collections when some partitions are already loaded.
It is not allowed to load different partitions in the same collection via separate RPCs.
4.5 Release Partition
milvusClient.releasePartitions(
ReleasePartitionsParam.newBuilder()
.withCollectionName("book")
.withPartitionNames(["novel"])
.build()
);
Parameter Description
CollectionName Name of the collection to release partition.
PartitionNames List of names of the partitions to release.
5. Data Management
You can also use MilvusDM to migrate data to Milvus, which is an open source tool specially designed for Milvus to import and export data.
Milvus 2.1 supports VARCHAR data type on scalar fields. When building an index for a VARCHAR type scalar field, the default index type is trie.
The following example inserts 2000 rows of randomly generated data as sample data (the Milvus CLI example uses a pre-built remote CSV file containing similar data). Real applications may use much higher dimensional vectors than the example. You can prepare your own data to replace the examples.
5.1 Insert entity
5.1.1 Prepare data
First, prepare the data to be inserted. The data type of the data to be inserted must match the schema of the collection, otherwise Milvus will throw an exception.
Random ran = new Random();
List<Long> book_id_array = new ArrayList<>();
List<Long> word_count_array = new ArrayList<>();
List<List<Float>> book_intro_array = new ArrayList<>();
for (long i = 0L; i < 2000; ++i) {
book_id_array.add(i);
word_count_array.add(i + 10000);
List<Float> vector = new ArrayList<>();
for (int k = 0; k < 2; ++k) {
vector.add(ran.nextFloat());
}
book_intro_array.add(vector);
}
5.1.2 Inserting data
When inserting data into a collection, by specifying partition_name, you can choose which partition to insert the data into.
List<InsertParam.Field> fields = new ArrayList<>();
fields.add(new InsertParam.Field("book_id", DataType.Int64, book_id_array));
fields.add(new InsertParam.Field("word_count", DataType.Int64, word_count_array));
fields.add(new InsertParam.Field("book_intro", DataType.FloatVector, book_intro_array));
InsertParam insertParam = InsertParam.newBuilder()
.withCollectionName("book")
.withPartitionName("novel")
.withFields(fields)
.build();
milvusClient.insert(insertParam);
Parameter Description
fieldName Name of the field to insert data into.
DataType Data type of the field to insert data into.
data Data to insert into each field.
CollectionName Name of the collection to insert data into.
PartitionName (optional) Name of the partition to insert data into.
NOTE: The maximum number of inserted vectors is 32,768
5.2 Insert entities through files
Milvus2.2 now supports inserting a batch of entities from files. Compared to the insert() method, this feature reduces network transfers between Milvus clients, brokers, Pulsar, and data nodes. Now you can import a batch of entities from a file into a collection with just one line of code.
This topic describes how to insert multiple entities in a batch from a JSON file.
5.2.1 Prepare the json file
Organize the data to be inserted into a row-based JSON file. You can name the file whatever you want, but the root key in the file must be "row".
In the file, each entity corresponds to a dictionary. The keys of the dictionary are the primary fields, and the values of the dictionary contain the remaining fields. Entities in the file must match the collection schema.
For binary vectors, use uint8 arrays. Each uint8 value represents 8 dimensions, and the value must be between [0, 255]. For example, [1, 0, 0, 2, 0, 1, 1] is a 16-dimensional binary vector and should be written as [128, 7] in the JSON file.
The file size should not be larger than 1 GB.
The content of the example file is as follows:
{ "rows":[ {"book_id": 101, "word_count": 13, "book_intro": [1.1, 1.2]}, {" book_id": 102, "word_count": 25, "book_intro": [2.1, 2.2]}, {"book_id": 103, "word_count": 7, "book_intro": [3.1, 3.2]}, {"book_id": 104, "word_count": 12, "book_intro": [4.1, 4.2] }, {"book_id": 105, "word_count": 34, "book_intro": [5.1, 5.2]} ] } 5.2.2 Insert entity from pymilvus import utility task_id = utility.do_bulk_insert( collection_name="book", partition_name= "2022", files=["test.json"] ) 5.3 Deleting Entities Milvus supports deleting entities by primary key filtered using Boolean expressions.
Note the following:
If the consistency level is set lower than Strong, deleted entities can still be retrieved immediately after deletion.
Entities deleted outside of the pre-specified time span cannot be retrieved again.
Frequent delete operations can affect system performance.
5.3.1 Prepare a Boolean expression
Prepare a Boolean expression that filters entities to delete.
Milvus only supports deleting entities with an explicitly specified primary key, which can be achieved simply by using the term expression in . Other operators can only be used for queries or scalar filtering in vector searches. See Boolean Expression Rules for details.
The following example filters data with primary key values 0 and 1.
private static final String DELETE_EXPR = “book_id in [0,1]”;
milvusClient.delete(
DeleteParam.newBuilder()
.withCollectionName("book")
.withExpr(DELETE_EXPR)
.build()
);
Deletes an entity using the boolean expression created. Milvus returns a list of IDs of deleted entities.
Parameter Description
CollectionName Name of the collection to delete entities from.
expr Boolean expression that specifies the entities to delete.
PartitionName (optional) Name of the partition to delete entities from.
5.4 Compressing data
By default, Milvus supports automatic data compression. You can configure Milvus to enable or disable compression and auto-compression.
If automatic compression is disabled, you can still compress data manually.
To ensure the accuracy of time travel search, Milvus keeps data operation logs within the range specified in common.retensionDuration. Therefore, data manipulated during this period will not be compressed.
5.4.1 Manually compress data
R response = milvusClient.manualCompaction(
ManualCompactionParam.newBuilder()
.withCollectionName(“book”)
.build()
);
long compactionID = response.getData().getCompactionID();
5.4.2 Check the compression status
milvusClient.getCompactionState(GetCompactionStateParam.newBuilder()
.withCompactionID(compactionID)
.build()
);
Six Index Management
Vector index is the organizational unit of metadata used to speed up vector similarity search. If there is no index built on the vector, Milvus will perform a brute force search by default.
Note the following:
Milvus 2.1 Supports Index on Scarlar Fields.
By Default, Milvus Does Not Index A Segment with Less than 1,024 ROWS. To CHANGE This Parameter, Configure rootcoor D. MinSegmentsizeToenableIndex in Milvus.yaml. 6.1 Prepare for the number of indexes for index (
if
you want to be a scalar field To build an index, no index construction parameters are required, and the default index type is a dictionary tree).
final IndexType INDEX_TYPE = IndexType.IVF_FLAT; // IndexType
final String INDEX_PARAM = "{\"nlist\":1024}"; // ExtraParam
Parameter Description Options
IndexType Type of index used to accelerate the vector search. For floating point vectors:FLAT (FLAT)IVF_FLAT (IVF_FLAT)IVF_SQ8 (IVF_SQ8)IVF_PQ (IVF_PQ)HNSW (HNSW)ANNOY (ANNOY)DISKANN* (DISK_ANN)For binary vectors:BIN_FLAT (BIN_FLAT)BIN_IVF_FLAT (BIN_IVF_FLAT)
ExtraParam Building parameter(s) specific to the index. See In-memory Index and On-disk Index for more information.
6.2 Create an index
milvusClient.createIndex(
CreateIndexParam.newBuilder()
.withCollectionName("book")
.withFieldName("book_intro")
.withIndexType(INDEX_TYPE)
.withMetricType(MetricType.L2)
.withExtraParam(INDEX_PARAM)
.withSyncMode(Boolean.FALSE)
.build()
);
6.3 Deleting an index
Deleting an index irreversibly deletes all corresponding index files.
milvusClient.dropIndex(
DropIndexParam.newBuilder()
.withCollectionName("book")
.withFieldName("book_intro")
.build()
);
Parameter Description
CollectionName Name of the collection to drop index on.
FieldName Name of the vector field to drop index on.
Seven search and query
7.1 Search
Carry out vector similarity search
The vector similarity search in Milvus calculates the distance between the query vector and the vectors in the collection using the specified similarity measure and returns the most similar results. By specifying Boolean expressions that filter scalar fields or primary key fields, it is possible to perform hybrid searches, and even use "time travel" to search.
The example below shows how to perform a vector similarity search on a 2000-row dataset of book ID (primary key), word count (scalar field), and book description (vector field), simulating a search for a specific book based on a vectorized description. Milvus will return the most similar results based on the query vector and search parameters you define.
7.1.1 Loading collections
All search and query operations in Milvus are performed in memory. Load the collection into memory before performing the vector similarity search.
milvusClient.loadCollection(
LoadCollectionParam.newBuilder()
.withCollectionName(“book”)
.build()
);
7.1.2 Prepare search parameters
Prepare parameters suitable for search scenarios. The following example defines that the search will compute distances using Euclidean distance and retrieve vectors from the ten closest clusters built from the IVF_FLAT index.
final Integer SEARCH_K = 2; // TopK
final String SEARCH_PARAM = “{“nprobe”:10, \”offset\”:5}”; // Params
Parameter Description Options
TopK Number of the most similar results to return. N/A
Params Search parameter(s) specific to the index. See Vector Index for more information.
Search parameters
Parameter Description Range
nprobe Number of units to query CPU: [1, nlist] GPU: [1, min(2048, nlist)]
By adjusting nprobe, an ideal balance between accuracy and speed can be found for a given scenario. The results of the IVF_FLAT performance test show that the query time increases dramatically as the number of target input vectors (nq) and the number of clusters to search (nprobe) increase.
7.1.3 Perform vector search
Use Milvus to search for vectors. To search for specific partitions, specify a list of partition names.
Milvus supports setting consistency levels specifically for searches. The examples in this topic set the consistency level to Strong. You can also set the consistency level to Bounded, Session or Eventually. For more information on Milvus' four consistency levels, see Consistency.
List search_output_fields = Arrays.asList(“book_id”);
List<List> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));
SearchParam searchParam = SearchParam.newBuilder()
.withCollectionName(“book”)
.withConsistencyLevel(ConsistencyLevelEnum.STRONG)
.withMetricType(MetricType.L2)
.withOutFields(search_output_fields)
.withTopK(SEARCH_K)
.withVectors(search_vectors)
.withVectorFieldName(“book_intro”)
.withParams(SEARCH_PARAM)
.build();
R respSearch = milvusClient.search(searchParam);
Parameter Description Options
CollectionName Name of the collection to load. N/A
MetricType Metric type used for search. This parameter must be set identical to the metric type used for index building.
OutFields Name of the field to return. Vector field is not supported in current release.
Vectors Vectors to search with. N/A
VectorFieldName Name of the field to search on. N/A
Expr Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
ConsistencyLevel The consistency level used in the query. STRONG, BOUNDED, andEVENTUALLY.
Check the primary key value of the most similar vector and its distance.
SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
System.out.println(wrapperSearch.getIDScore(0));
System.out.println(wrapperSearch.getFieldData(“book_id”, 0));
when When the search is complete, the loaded collection in Milvus is released to reduce memory consumption.
milvusClient.releaseCollection(
ReleaseCollectionParam.newBuilder()
.withCollectionName("book")
.build());
The restrictions are as follows:
Feature Maximum limit
Length of a collection name 255 characters
Number of partitions in a collection 4,096
Number of fields in a collection 256
Number of shards in a collection 256
Dimensions of a vector 32,768
Top K 16,384
Target input vectors 16,384
7.1.4 搜索API
7.1.4.1 SearchParam
Use the to construct a object.SearchParam.Builder``SearchParam
import io.milvus.param.SearchParam;
SearchParam.Builder builder = SearchParam.newBuilder();
Methods of :SearchParam.Builder
Method Description Parameters
withCollectionName(collectionName) Sets the collection name. Collection name cannot be empty or null. collectionName: The name of the collection to query.
withConsistencyLevel(ConsistencyLevelEnum consistencyLevel) Sets the consistency level used in the query. If no consistency level is specified, the default level is .ConsistencyLevelEnum.BOUNDED consistencyLevel: The consistency level used in the query.
withPartitionNames(List partitionNames) Set the list of partition names to specify the query scope (optional). partitionNames: The name list of the partitions to query.
addPartitionName(String partitionName) Add partitions to specify the query scope (optional). partitionName: The name of the partition to query.
withTravelTimestamp(Long ts) Specifies the absolute timestamp in the query to get results based on the data view at the specified time point (optional). The default is, which is used by the server to perform queries on the full data view.
withOutFields(List outFields) Specifies output scalar fields (optional). If output fields are specified, the returned query result will contain the values of these fields outFields: The name list of output fields.
addOutField(String fieldName) specifies an output scalar field (optional). fieldName: The name of an output field .
withExpr(String expr) Sets the expression to filter scalar fields before searching (optional) expr: The expression used to filter scalar fields. withMetricType
(MetricType metricType) Sets the metric type of ANN search .The default value is .MetricType.L2 metricType: The metric type.
withVectorFieldName(String vectorFieldName) Sets the target vector field by name. Field name cannot be empty or null. vectorFieldName: A vector field name.
withTopK(Integer topK) Set the topK value for ANN search. Available range is [116384]. topK: The topK value.
withVectors(List<?> vectors) Set target vectors. Up to 16384 vectors are allowed. vectors:If target field type is float vector, is required.List< List>If target field type is binary vector, is required.List
withRoundDecimal(Integer decimal) Specifies the number of decimal places for the returned distance. Available range is [-1, 6]. By default, the method returns all numbers -1 decimal: the number of digits to keep after the decimal point.
withParams(String params) specifies search parameters in JSON format. Valid keys are as follows: special parameters related to indexing, such as nprobe efsearch_k metric type, as key, or as value metric_type L2IP offset for pagination, keyword is, value is integer offset pagination limit with keyword and integer as value` Limit
build() Constructs a object.SearchParam N/A
The can throw the following exceptions: SearchParam.Builder.build()
ParamException: error if the parameter is invalid.
7.1.4.2 Returns
This method catches all exceptions and returns an object R
If the API fails on the server side, it will return an error code and message from the server.
If the API fails with an RPC exception, returns the exception's error message R.Status.Unknown
If the API succeeds, returns the valid value saved by the R template. You can get results using SearchResultsSearchResultsWrapperr
7.1.4.3 SearchResultsWrapper
A tool class to encapsulate the .SearchResults
import io.milvus.response.SearchResultsWrapper;
SearchResultsWrapper wrapper = new SearchResultsWrapper(searchResults);
Methods of :SearchResultsWrapper
Method Description Parameters Returns
getFieldData(String fieldName, int indexOfTarget) Get the data of the specified output field. If the field does not exist or is illegal, then throws SearchParam ParamExceptionindexOfTarget fieldName: A field name which is specified by the of (the order number of a target vector).withOutFields() SearchParam.indexOfTarget getIDScore(int indexOfTarget) 获取返回的ID分数对。如果非法，则抛出。如果返回的结果非法，则抛出.search()ParamException indexOfTargetIllegalResponseException indexOfTarget: Order number of a target vector. List
returned by getFieldData The meaning of the value is as follows:
Returns for float vector field.List<List>
Returns for binary vector field.List
Returns for int64 field.List
Returns for int32/int16/int8 field.List
Returns for boolean field.List
Returns for float field.List
Returns for double field.List
Returns for VARCHAR field.List
7.1.4.4 IDScore
A tool class to hold a pair of ID and distance.
Methods of :SearchResultsWrapper.IDScore
Method Description Returns
getLongID() Gets the integer ID if the primary key type is Int64. long
getStrID() Gets the string ID if the primary key type is VarChar. String
getScore() Gets the distance value. float
Whether is an accurate distance or not depands on the index type:getScore()
FLAT
The score is is accurate distance.
IVF_FLAT
The score is accurate distance.
IVF_SQ8/IVF_PQ
The score is not accurate.
HNSW/ANNOY
The score is accurate.
7.1.4.5 Example code
import io.milvus.param.*;
import io.milvus.response.SearchResultsWrapper;
import io.milvus.grpc.SearchResults;
SearchParam param = SearchParam.newBuilder()
.withCollectionName("collection1")
.withMetricType(MetricType.IP)
.withTopK(10)
.withVectors(targetVectors)
.withVectorFieldName("field1")
.withConsistencyLevel(ConsistencyLevelEnum.EVENTUALLY)
.withParams("{\"nprobe\":10,\"offset\":2, \"limit\":3}")
.build();
R<SearchResults> response = client.search(param)
if (response.getStatus() != R.Status.Success.getCode()) {
System.out.println(response.getMessage());
}
SearchResultsWrapper wrapper = new SearchResultsWrapper(response.getData());
System.out.println("Search results:");
for (int i = 0; i < targetVectors.size(); ++i) {
List<SearchResultsWrapper.IDScore> scores = results.getIDScore(i);
for (int j = 0; j < scores.size(); ++j) {
SearchResultsWrapper.IDScore score = scores.get(j);
System.out.println("Top " + j + " ID:" + score.getLongID() + " Distance:" + score.getScore());
}
}
7.1.5 Data Insertion API
MilvusClient interface. This method inserts the entity into the specified collection.
R insert(InsertParam requestParam);
7.1.5.1 InsertParam
import io.milvus.param.InsertParam;
InsertParam.Builder builder = InsertParam.newBuilder();
Method Description Parameters
withCollectionName(String collectionName) Set the target collection name. Collection name cannot be empty or null. collectionName: The name of the collection to insert data into.
withPartitionName(String partitionName) Set target partition name (optional). partitionName: The name of the partition to insert data into.
withFields(List<InsertParam.Field> fields) Set the data to be inserted. Field list cannot be empty. Note that the primary key field does not require entry if auto-id is enabled. fields: A list of 'fields' objects, each representing a field.
build() Constructs an InsertParam object. N/A
InsertParam.Builder.build() may throw the following exceptions
ParamException: An error occurred when the parameter was invalid.
7.1.5.2 Field
Method Description
Field(String name, List<?> values) This class only provides a Constructor to create a Field object.
The parameters are explained as follows:
name: The name of the data field.
values:
If data type is Bool, values is List of Boolean;
If data type is Int64, values is List of Long;
If data type is Float, values is List of Float;
If data type is Double, values is List of Double;
If data type is VARCHAR, values is List of String;
If data type is FloatVector, values is List of List Float;
If data type is BinaryVector, values is List of ByteBuffer.
7.1.5.3 示例代码
import io.milvus.param.*;
import io.milvus.grpc.MutationResult;
List<InsertParam.Field> fields = new ArrayList<>();
int rowCount = 10000;
List<Long> ids = new ArrayList<>();
for (long i = 0L; i < rowCount; ++i) {
ids.add(i);
}
List<List<Float>> vectors = generateFloatVectors(rowCount);
List<InsertParam.Field> fieldsInsert = new ArrayList<>();
fieldsInsert.add(new InsertParam.Field("id", ids));
fieldsInsert.add(new InsertParam.Field("vec", vectors));
InsertParam param = InsertParam.newBuilder()
.withCollectionName(collectionName)
.withFields(fieldsInsert)
.build();
R<MutationResult> response = client.insert(param);
if (response.getStatus() != R.Status.Success.getCode()) {
System.out.println(response.getMessage());
}
7.2 Hybrid Search
A hybrid search is essentially a vector search with attribute filtering. You can restrict the search with certain criteria by specifying a Boolean expression that filters on a scalar field or a primary key field.
The following example demonstrates how to perform a hybrid search on top of a regular vector search. Suppose you want to search for certain books based on vectorized descriptions, but you only want books within a certain word count. You can then specify Boolean expressions to filter fields in the search parameters. Milvus will only search for similar vectors among entities matching the expression. word_count
Scalar fields of entities can be filtered during vector searches by specifying a Boolean expression. The following example limits the search to vectors within a specified range of values.
word_count
milvusClient.loadCollection(
LoadCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
final Integer SEARCH_K = 2;
final String SEARCH_PARAM = "{\"nprobe\":10, \”offset\”:5}";
List<String> search_output_fields = Arrays.asList("book_id");
List<List<Float>> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));
SearchParam searchParam = SearchParam.newBuilder()
.withCollectionName("book")
.withMetricType(MetricType.L2)
.withOutFields(search_output_fields)
.withTopK(SEARCH_K)
.withVectors(search_vectors)
.withVectorFieldName("book_intro")
.withExpr("word_count <= 11000")
.withParams(SEARCH_PARAM)
.build();
R<SearchResults> respSearch = milvusClient.search(searchParam);
Parameter Description Options
CollectionName Name of the collection to load. N/A
MetricType Metric type used for search. This parameter must be set identical to the metric type used for index building.
OutFields Name of the field to return. Vector field is not supported in current release.
TopK Number of the most similar results to return. N/A
Vectors Vectors to search with. N/A
VectorFieldName Name of the field to search on. N/A
Expr Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
Params Search parameter(s) specific to the index. See Vector Index for more information.
检查返回的结果
SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
System.out.println(wrapperSearch.getIDScore(0));
System.out.println(wrapperSearch.getFieldData(“book_id”, 0));
7.3 Vector Queries
Unlike vector similarity searches, vector queries retrieve vectors through scalar filtering based on Boolean expressions. Milvus supports many data types and various Boolean expressions in scalar fields. Boolean expressions filter on scalar or primary key fields and retrieve all results that match the filter.
The following example shows how to perform a vector query on a 2000-row dataset of book ID (primary key), word count (scalar field), and book description (vector field), simulating the case of querying a specific book by book ID.
The following example filters a vector with some book_id value and returns the result's book_id field and book_intro.
milvusClient.loadCollection(
LoadCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
List<String> query_output_fields = Arrays.asList("book_id", "word_count");
QueryParam queryParam = QueryParam.newBuilder()
.withCollectionName("book")
.withConsistencyLevel(ConsistencyLevelEnum.STRONG)
.withExpr("book_id in [2,4,6,8]")
.withOutFields(query_output_fields)
.withOffset(0L)
.withLimit(10L)
.build();
R<QueryResults> respQuery = milvusClient.query(queryParam);
Parameter Description Options
CollectionName Name of the collection to load. N/A
OutFields Name of the field to return. Vector field is not supported in current release.
Expr Boolean expression used to filter attribute. See Boolean Expression Rules for more information.
ConsistencyLevel The consistency level used in the query. STRONG, BOUNDED, andEVENTUALLY.
检查执行结果
QueryResultsWrapper wrapperQuery = new QueryResultsWrapper(respQuery.getData());
System.out.println(wrapperQuery.getFieldWrapper(“book_id”).getFieldData());
System.out.println(wrapperQuery.getFieldWrapper(“word_count”).getFieldData());
八 完整实例代码
package com.whty.common.graph.milvus;
import io.milvus.Response.SearchResultsWrapper;
import io.milvus.client.MilvusServiceClient;
import io.milvus.grpc.DataType;
import io.milvus.grpc.SearchResults;
import io.milvus.param.ConnectParam;
import io.milvus.param.IndexType;
import io.milvus.param.MetricType;
import io.milvus.param.R;
import io.milvus.param.collection.CreateCollectionParam;
import io.milvus.param.collection.FieldType;
import io.milvus.param.collection.LoadCollectionParam;
import io.milvus.param.collection.ReleaseCollectionParam;
import io.milvus.param.dml.InsertParam;
import io.milvus.param.dml.SearchParam;
import io.milvus.param.index.CreateIndexParam;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
/**
* @author yishui
* @version 1.0.0
* @since 1.0.0
*/
public class DemoTest {
private static MilvusServiceClient connect() {
final MilvusServiceClient milvusClient = new MilvusServiceClient(
ConnectParam.newBuilder()
.withHost("192.168.171.132")
.withPort(19530)
.build()
);
return milvusClient;
}
private static CreateCollectionParam createCollectionReq() {
FieldType fieldType1 = FieldType.newBuilder()
.withName("book_id")
.withDataType(DataType.Int64)
.withPrimaryKey(true)
.withAutoID(false)
.build();
FieldType fieldType2 = FieldType.newBuilder()
.withName("word_count")
.withDataType(DataType.Int64)
.build();
FieldType fieldType3 = FieldType.newBuilder()
.withName("book_intro")
.withDataType(DataType.FloatVector)
.withDimension(2)
.build();
CreateCollectionParam createCollectionReq = CreateCollectionParam.newBuilder()
.withCollectionName("book")
.withDescription("Test book search")
.withShardsNum(2)
.addFieldType(fieldType1)
.addFieldType(fieldType2)
.addFieldType(fieldType3)
.build();
return createCollectionReq;
}
private static void createCollection(MilvusServiceClient milvusClient, CreateCollectionParam createCollectionReq) {
milvusClient.createCollection(createCollectionReq);
}
private static void insertData(MilvusServiceClient milvusClient) {
Random ran = new Random();
List<Long> book_id_array = new ArrayList<>();
List<Long> word_count_array = new ArrayList<>();
List<List<Float>> book_intro_array = new ArrayList<>();
for (long i = 0L; i < 2000; ++i) {
book_id_array.add(i);
word_count_array.add(i + 10000);
List<Float> vector = new ArrayList<>();
for (int k = 0; k < 2; ++k) {
vector.add(ran.nextFloat());
}
book_intro_array.add(vector);
}
List<InsertParam.Field> fields = new ArrayList<>();
fields.add(new InsertParam.Field("book_id", DataType.Int64, book_id_array));
fields.add(new InsertParam.Field("word_count", DataType.Int64, word_count_array));
fields.add(new InsertParam.Field("book_intro", DataType.FloatVector, book_intro_array));
InsertParam insertParam = InsertParam.newBuilder()
.withCollectionName("book")
.withPartitionName("novel")
.withFields(fields)
.build();
milvusClient.insert(insertParam);
}
static final IndexType INDEX_TYPE = IndexType.IVF_FLAT; // IndexType
static final String INDEX_PARAM = "{\"nlist\":1024}"; // ExtraParam
private static void buildIndex(MilvusServiceClient milvusClient) {
milvusClient.createIndex(
CreateIndexParam.newBuilder()
.withCollectionName("book")
.withFieldName("book_intro")
.withIndexType(INDEX_TYPE)
.withMetricType(MetricType.L2)
.withExtraParam(INDEX_PARAM)
.withSyncMode(Boolean.FALSE)
.build()
);
}
public static void main(String[] args) {
MilvusServiceClient milvusClient = connect();
CreateCollectionParam createCollectionReq = createCollectionReq();
createCollection(milvusClient, createCollectionReq);
insertData(milvusClient);
buildIndex(milvusClient);
milvusClient.loadCollection(
LoadCollectionParam.newBuilder()
.withCollectionName("book")
.build()
);
final Integer SEARCH_K = 2; // TopK
final String SEARCH_PARAM = "{\"nprobe\":10}"; // Params
List<String> search_output_fields = Arrays.asList("book_id");
List<List<Float>> search_vectors = Arrays.asList(Arrays.asList(0.1f, 0.2f));
SearchParam searchParam = SearchParam.newBuilder()
.withCollectionName("book")
.withMetricType(MetricType.L2)
.withOutFields(search_output_fields)
.withTopK(SEARCH_K)
.withVectors(search_vectors)
.withVectorFieldName("book_intro")
.withParams(SEARCH_PARAM)
.build();
R<SearchResults> respSearch = milvusClient.search(searchParam);
SearchResultsWrapper wrapperSearch = new SearchResultsWrapper(respSearch.getData().getResults());
System.out.println("wrapperSearch.getIDScore" + wrapperSearch.getIDScore(0));
System.out.println("wrapperSearch.getFieldData" + wrapperSearch.getFieldData("book_id", 0));
milvusClient.releaseCollection(
ReleaseCollectionParam.newBuilder()
.withCollectionName("book")
.build());
}
}
The dependencies are as follows:
<dependency>
<groupId>io.milvus</groupId>
<artifactId>milvus-sdk-java</artifactId>
<version>2.2.1</version>
</dependency>