"Vector Database Guide" - AI native vector database Milvus Cloud 2.3 new features ScaNN index and Iterator

• ScaNN index

Milvus currently supports the FastScan algorithm in Faiss, which performs well in various benchmarks. Compared with HNSW, it is about 20% improved, about 7 times that of IVFFlat, and the index building speed is faster. ScaNN is similar to IVFPQ in terms of algorithm. It clusters into buckets, and then uses PQ to quantize the vectors in the buckets. The difference is that ScaNN is more aggressive in quantification and has higher computational efficiency when paired with SIMD, but the accuracy loss will be greater and requires the original The process of vector refinement.

The following table shows the performance of ScaNN, HNSW and IVFFLAT on the Cohere1M (768-dimensional) data set. The data comes from VectorDBBench.

• Iterator

Pymilvus provides the iterator interface, which can pull data through iterators. In Query and Range Search scenarios, data exceeding the 16384 data limit can be obtained through iterators. Iterator is similar to the scroll interface of ES and the cursor in the relational database, and is more suitable for background batches.