[Tencent Cloud Ecosystem Conference] Tencent Cloud Vector Database
- Personal homepage : ζ小菜鸡
- Hello everyone, I am a newbie, let us experience Tencent Cloud Vector Database together
- If the article is helpful to you, welcome to follow, like, and collect (three consecutive clicks)
1. Introduction to Tencent Cloud Vector Database
1. What is Tencent Cloud Vector Database?
Tencent Cloud Vector Database (Tencent Cloud VectorDB) is a fully managed, self-developed enterprise-level distributed database service dedicated to storing, retrieving, and analyzing multi-dimensional vector data. The database supports multiple index types and similarity calculation methods. A single index supports a vector scale of 1 billion, and can support millions of QPS and millisecond-level query latency. Tencent Cloud Vector Database can not only provide an external knowledge base for large models and improve the accuracy of large model answers, but can also be widely used in AI fields such as recommendation systems and natural language processing.
2. Product features and advantages of Tencent Cloud Vector Database
(1) High-performance
vector database Tencent Cloud VectorDB supports 1 billion-level vector data scale in a single index, and can support millions of QPS and millisecond-level query latency.
(2) Low cost
: Just follow the instructions of the console to quickly create a vector database instance and host the whole process platform without any installation, deployment, operation and maintenance operations, reducing machine costs, operation and maintenance costs, and labor costs. .
(3) High
availability The vector database Tencent Cloud VectorDB provides multi-copy high availability features to improve disaster recovery capabilities and ensure that the database can still run normally when facing challenges such as node failures and load changes.
(4) Simple and easy to use.
It supports rich vector retrieval capabilities. Users can quickly operate the database through the API, and the development efficiency is high. At the same time, the console provides complete data management and monitoring capabilities, and is simple and convenient to operate.
(5) The large-scale
vector database architecture supports horizontal expansion, and a single instance can support tens of billions of QPS, easily meeting the vector storage and retrieval needs in AI scenarios.
(6) Stable and reliable
vector database Tencent Cloud VectorDB is derived from Tencent Group’s self-developed vector search engine OLAMA. Nearly 40 business lines are running stably, with an average of 100 billion search requests processed daily, and service continuity and stability are guaranteed. .
2. Product specifications of vector database
(1) Product Specifications
Tencent Cloud VectorDB adopts a distributed deployment architecture, with each node communicating and coordinating with each other to achieve data storage and retrieval. Client requests are distributed to each node through Load balance.
(2) Node type
Tencent Cloud vector database is divided into two types: storage type and computing type based on the different allocation ratios of storage node CPU and memory resources.
a. Storage type: Mainly used to store and manage large-scale vector data. Its main advantages are: it provides low query latency and can efficiently store and manage vector data. It is especially suitable for large amounts of data, rapid data growth, and relatively query QPS. Lower-level scenarios, such as face recognition, image search, etc.
b. Computational type: Mainly used to quickly find and retrieve vector data, supporting highly concurrent query requests. Its main advantage is: providing higher query QPS and lower query latency, suitable for scenarios with heavy traffic and delay sensitivity. For example: real-time recommendations, advertising, etc.
(3) Number of nodes
Tencent Cloud Vector Database adopts a distributed architecture to support multi-node communication and coordination. Currently, it supports 3 to 10 nodes.
(4) Node specifications
Different node types have different corresponding product specifications. For details, please see the table below.
3. Key concepts of vector database
Vector
A vector can be understood as an ordered collection of values, usually used to represent the attributes or characteristics of an object or thing. These values can have different dimensions, each dimension representing an attribute or characteristic. In the field of machine learning and artificial intelligence, vectors are often used to represent data such as images, text, and audio. By calculating the distance or similarity between vectors, tasks such as classification, clustering, and retrieval can be achieved.
OLAMA
OLAMA is a vector engine developed by Tencent. It has the characteristics of high performance, high availability, simplicity and ease of use. It supports a single index of 1 billion vector scale, is suitable for AI computing and retrieval scenarios, and has stably served nearly 40 online businesses.
Instance
An instance is an independently running database environment in Tencent Cloud. It is the basic unit for users to purchase vector database services and exists as a separate process. A database instance can contain multiple databases created by users. You can create, modify, and delete instances in the console. Instances are independent of each other and resources are isolated. There is no preemption problem between CPU, memory, persistent memory, IO, etc.
Database
A database is a warehouse that organizes, stores, and manages data according to a data structure. One instance can create multiple Databases.
Collection
In a vector database, a collection refers to a group of documents, similar to a table in a relational database, which can contain multiple pieces of document data. Collections have no fixed structure and can be inserted into different formats and types of data. Vector database supports multi-sharding and multi-copy features in the collection dimension. You can specify the number of shards and copies as needed when creating a collection.
Document
In a vector database, a collection can be viewed as a table, and a Document can be viewed as a row of data in the table. Each Document represents a complete document object, including multiple Fields, and each Field represents a property or field in the document. The document of a vector database is a set of key-value pairs (key:value), and each document has a unique primary key (id) and a vector field (vector). When inserting a document, the vector database does not need to set the same fields, and fields can be added or deleted when inserting data.
Field
Each Field is a key:value pair, representing an attribute or field in the document. Each Field has its own type and value range, which can be different types of data such as strings and numbers.
Node
From the resource perspective of a vector database cluster, nodes are resource units used to store data. A running vector database instance usually contains many nodes, and multiple copies and shards of the collection are distributed on several nodes. Node is one of the basic units that make up a vector database cluster.
Shard
In order to support larger-scale data, the collection is generally divided into multiple parts according to a certain dimension, and each part is a shard distributed on several nodes (Node). In order to ensure reliability and availability, multiple shards of the same collection will be distributed on different nodes (Node).
Replica
For the backup data of the same shard (Shard), there will be at least 2 copies of a shard. Replica sharding serves as a redundant backup to protect data from loss in the event of hardware failure, and provides services for read operations such as vector retrieval and document query, ensuring that the database can still run normally in the face of challenges such as node failure and load changes.
Index
An index is a special data structure used to quickly find and access data, stored in memory. The index itself does not store data, but rather stores pointers or key-value pairs pointing to the location where the data is stored. Tencent Cloud VectorDB supports common vector indexes such as FLAT and HNSW. For an introduction to indexing, see Vector Retrieval.
KNN (K-Nearest Neighbor Search)
KNN refers to K-Nearest Neighbor Search, which is a method based on brute force search. Its principle is: calculate the distance between the vector to be queried and all vectors in the database, and then sort them according to the distance from small to large. , select the K nearest vectors as query results. The advantage of the KNN algorithm is that it can guarantee accurate results, but for large-scale vector data, the calculation amount will be very large and the efficiency is low.
ANN (Approximate Nearest Neighbor Search)
ANN stands for Approximate Nearest Neighbor Search, which is a method for quickly finding nearest neighbors in high-dimensional data space. Compared with exact nearest neighbor search, ANN sacrifices certain accuracy in exchange for higher search speed, and therefore has higher efficiency when processing large-scale data sets. ANN methods often preprocess the data to reduce the number of distance calculations at query time. The advantages of the ANN algorithm are fast speed and high efficiency, but compared to the KNN algorithm, its results may not be accurate enough.
HNSW (Hierarchical Navigable Small World)
HNSW is a graph-based high-dimensional vector similarity search algorithm, its full name is: Hierarchical Navigable Small World. It constructs a graph to represent the similarity relationship between vectors and uses some optimization strategies to speed up the search process.
4. Use of vector database to create a new database instance
【1】Use a Tencent Cloud account to log in to the Vector Database console. (Xiao Caiji has registered a Tencent Cloud account and completed real-name authentication. If you do not have a Tencent Cloud account, you need to register and log in to Tencent Cloud first)
【2】Click New to enter the new vector database instance page. (Diamond chose the computer model, and the vector size is relatively small)
【3】Please refer to the table below, configure the following parameters, and purchase an instance.
【4】Click Apply Now to automatically return to the instance list page. The current instance status is Creating. Wait for the instance status to be updated to Running before you can use it.
5. First experience with vector database
[1] Users can purchase cloud server CVM instances through the Tencent Cloud console (CVM and vector database must be in the same private network VPC) to initiate performance testing.
(Here, the rookie has created a new instance of the performance test, and then logs in in the operation column on the right)
(The interface after logging in is as shown in the figure. There is an editor option in it to facilitate user operations. The rookie uses the editor)
The editor page is as shown in the figure
[2] Test tool preparation
1. Download the test tool: ann-benchmarks-dev.zip, and then upload the test tool and data set to the test client CVM
2. Install test tool dependencies
Software dependencies: If the python version is greater than 3.6.8, use python3 on the recommended operating system version. (The newbie’s choice is python3)
a. First install the operating system dependency package in the test client CVM terminal, as shown in the following figure:
b. Enter the test tool directory in the terminal and install the python running dependencies as shown below: (Be sure to enter the test tool directory for installation here)
(There are not many modules to install, and the novice will have to wait for about 10 seconds before the installation is completed)
c. The result of successful installation of python dependent environment is shown in the figure below:
[3] Connect vector database
1: When starting the test, you can copy the default configuration file 'ann_benchmarks/algorithms/vector_db/config.yml' from the project directory
to your own working directory under the project to make modifications, and then specify it through the command line parameter –definitions own configuration file to run tests.
2: Select the test data set and find ann-benchamrk's existing data set named sift-128-euclidean, which has exactly 128 dimensions. You can use this data set for testing. The data set command ends with euclidean, which means using the L2 similarity algorithm
3: CVM connection test vector database
a. Ensure that the database instance has been created and the status is running. As shown below:
(In the instance list, we click the instance ID, or click Manage in its operation column to enter the instance details page. You can see the specific information of the vector database)
b. Obtain the intranet IP address and network port of the vector database instance (the specific information is in the previous picture of Xiaocaiji)
c. Obtain the API access key of the vector database (you can directly copy the secret key. You can click the display of the API key column to view the API key information and copy it.)
d. To configure outbound rules in the Tencent Cloud CVM security group, you need to add the IP and port of the Tencent Cloud vector database to the outbound rules. Configure inbound rules in the Tencent Cloud vector database security group, and add the CVM IP and port to the inbound rules
(1) Tencent Cloud Security Group
Add inbound rules later
(2) Vector database security group, configure security group information. As shown below:
[4] Test run example
a: Copy the default configuration file to your own configuration file path (it is not possible for novices to operate directly on the console, and must be operated in the test file) as shown below:
b. Set your own configuration parameters in mytest.yml as needed.
c. Run the test through the following command. The tool will automatically download the ann-benchmark official data set from the external network and write it to the Tencent Cloud vector database. (The rookie did not pay attention to the format when inputting the command and kept reporting an error. You need to pay attention to the command format here.) As shown in the figure below:
d. After the test is completed, the test results are saved in the results folder. You need to use another command line tool to convert the results into a readable csv file. As shown below:
6. Retrieval performance stress test using vector database
[1] When exploring the specified recall rate, the query ef conditions need to be set (Note: This mode needs to add the only_query parameter to avoid re-insertion of data) a. Find the exploration mode,
modify the KNNSeekMode in the configuration file to true, and test this mode The tool will repeatedly run the query with different ef values until the recall of the best match is obtained. As shown below:
b. Configure the exploration parameter
KNNSeekStartEF: Specify which ef reference value to start querying
KNNSeekStep: Specify how many units the ef value changes each time in the exploration mode; such as 1:
KNNSeekStartEF+=1 or subtract. (Step size for each increment or decrement)
KNNSeekExpect: The recall rate expected to be found, such as 0.95
(this is still set in the configuration file) as shown below:
c. Run the tool and enter the command in the console. As shown in the picture below: (The novice here still needs to remind everyone to pay attention to the format of the command)
d. The results are shown as follows (KNNSeekExpect=0.95 in the picture):
【2】Search retrieval performance test
a. Find the EF of the desired test or set EF, and explore the query conditions that need to be set to obtain the specified recall rate through '[1]' to find the EF of the desired recall rate; or set yourself as needed The ef value
b. Stress test query configuration, in the configuration file, use the 8-core stress test query ef=111 as follows. Note that KNNSeekMode needs to be set to false. As shown below:
c. Stress test command. During stress test, you need to turn on the –batch parameter and set –runs to a larger value for long-term operation. As shown below:
【3】View the stress test results
a. Log in to the Vector Database console using a Tencent Cloud account. In the instance list, find the instance for which monitoring data needs to be viewed. As shown below:
b. Click the target instance ID, or click Manage in its operation column to enter the instance details page and find instance monitoring. as the picture shows:
c: Tencent Cloud Vector Database Console provides monitoring of key performance indicators such as CPU, memory, QPS, and latency of instances. Available from a convenient console. As shown below:
[4] When the query cannot fully charge the CPU, the solution
is limited by the threading model of the python language. Sometimes the testing tool cannot fully load the test pressure, causing the instance performance to fail to reach the limit. In this case, multiple processes or multiple CVMs can be used Perform pressure testing at the same time. as the picture shows:
7. Summary
Tencent Cloud Vector Database is provided to users as a service dedicated to storing and retrieving vector data. It has significant advantages in high performance, high availability, large scale, low cost, simplicity and ease of use, stability and reliability, and intelligent operation and maintenance. It is a fully managed self-developed enterprise-level distributed database service dedicated to storing, retrieving, and analyzing multi-dimensional vector data. Not only can it provide an external knowledge base for large models and improve the accuracy of large model answers, it can also be widely used in AI fields such as recommendation systems, NLP services, computer vision, and intelligent customer service, with a wide range of applications.