September 7-8, Shenzhen International Convention and Exhibition Center Hall 18
Here it comes, here it comes, Tencent’s annual technology event with the highest standards, the largest scale, and the widest coverage in the field of industrial Internet — Tencent Global Digital Ecosystem Conference .
On September 7 , we will focus on new trends in the future development of the industry , release key progress on core digital tools such as cloud computing, big data, artificial intelligence, security, and SaaS , and work with ecological partners to launch the latest industry scenario solutions. At the same time, we join hands with the world's authoritative business magazine lists to discuss the development trend of digital-physical integration with China's best CEOs , and analyze the best practices for industrial renewal with China's top 500 companies. We will talk about application examples such as localization, enterprises going global, and large industry models , and explore key issues such as how to build enterprise security immunity and how to create new business growth through SaaS product organization collaboration.
Since the amount of information in this conference is too large, Tomato will focus on " vector database " and bring you a detailed introduction and evaluation.
1. What is a vector database?
Vector database is a database system specially used to store and process vector data. It provides efficient vector data storage, similarity search, clustering and classification functions by optimizing the storage structure and query algorithm. Vector databases play an important role in applications in image, audio, text and other fields.
Vector databases usually provide rich query interfaces and functions, such as range query, k -nearest neighbor query, similarity matching, etc. At the same time, the vector database also supports high concurrency and distributed deployment to cope with the needs of large-scale data and high concurrent access.
The vector database workflow includes the following steps:
- 1. Storage of vector data : Vector data is usually high-dimensional numerical data, such as image feature vectors, text word vectors, etc.; vector databases use vector-based storage structures to facilitate quick query and processing;
- 2. Vector indexing : Vector databases use algorithms such as PQ, LSH or HNSW to index vectors and map vectors to data structures for faster searches;
- 3. Vector query : The vector database compares the query vector with the vectors in the database to find the nearest neighbor vector;
- 4. Return of query results : The vector database returns query results, which usually include a list of vectors most similar to a given vector, similarity scores between vectors and other information; this link can use different similarity measures to reorder the nearest neighbors.
2. Tencent Cloud Vector Database - Test Preparation
Overview of preparation work: a vector database + a client to execute the test code + the official test data set and method of ann-benchamrk.
2.1 Environment preparation
- 1. Log in to Tencent Cloud: https://console.cloud.tencent.com
- 2. Create a vector database: Create a vector database instance with one click from the Tencent Cloud console.
ps. Tencent Cloud Vector Database provides users with a variety of instance specifications, which can be selected as needed.
- Operating system selection: The recommended version is 'TencentOS Server 3.1 (TK4)'. During the actual testing of this version, the installation of python3-related dependencies went smoothly;
- Disk size selection: It is recommended to reserve enough disk space. 200G should be able to meet the actual testing needs. You can also refer to the size of the actual data set to create;
- Memory size selection: Because during the actual test process, each process needs to load the complete set of tested data into memory, you can refer to 130% of the data set size * the number of test tool processes to select the machine memory;
Wait a few minutes and it will be built. The effect is as follows: 
2.2 Data preparation
2.1 Upload test tools and data sets to the test client
Test data source: ann-benchamrk official data set test tool can be automatically downloaded from the official external website. Official website address: https://ann-benchmarks.com/
2.2 Install test tool dependencies
Software dependencies : python version is greater than 3.6.8, just use python3 on the recommended operating system version
2.2.1 Install operating system dependency packages
yum install python3-pillow-devel.x86_64
2.2.2 Solve the test tool and install python running dependencies
cd ann-benchmarks
pip3 install -r requirements.txt
2.2.3 Required dependency packages:
ansicolors==1.1.8
docker
h5py
matplotlib
numpy
pyyaml
psutil
scikit-learn
jinja2
pytest
dataclasses-json==0.5.7
dacite
urllib3
enum34
typing
tqdm
threadpool
3. Performance test
3.1 Test the single-core query performance of 128- dimensional data under HNSW index
From the test data set description, find ann-benchamrk 's existing data set named sift-128-euclidean, which has exactly 128 dimensions. You can use this data set for testing. The data set command ends with euclidean , which means the L2 similarity algorithm is used.
After executing the test, take a look at the resource performance of the database.
- CPU usage is close to 100%
- Low memory and disk usage
Requests:
- Request quantity QPS (Count/s) Maximum value: 1092.92 Minimum value: 0.00 Average value: 63.90
- Request success rate (%) Maximum value: 100.00 Minimum value: 0.00 Average value: 6.39
- Request timeout rate (%) Maximum value: 0.00 Minimum value: 0.00 Average value: 0.00
- Request exception rate (%) Maximum value: 0.00 Minimum value: 0.00 Average value: 0.00
- Failed request quantity QPS (Count/s) Maximum value: 0.00 Minimum value: 0.00 Average value: 0.00
- Average request time (ms) Maximum value: 0.70 Minimum value: 0.00 Average value: 0.04
Network : When Tomato downloaded the test results, it caused a peak.
The single-core performance of vector databases is impressive. It exhibits excellent processing power and efficient computing speed. Whether it's data storage, index building, or query processing, vector databases enable fast parallel computing and achieve superior performance on a single core.
4. Summary
Tencent Cloud Vector Database focuses on processing large-scale vector data and uses efficient indexing and query algorithms to quickly perform similarity searches and high-dimensional vector calculations. Enables users to perform complex data analysis and machine learning tasks with ease.
Tencent Cloud Vector Database also has excellent scalability and stability. It supports automatic horizontal expansion and can elastically expand according to data scale and user needs, ensuring high availability and performance stability of the database. At the same time, Tencent Cloud Vector Database provides a friendly management interface and flexible API interface, allowing users to easily operate and manage data.
In addition, Tencent Cloud Vector Database also focuses on data security and privacy protection. It uses multi-level security measures, including data encryption, access control and firewalls, to ensure the security and privacy protection of user data.
In short, Tencent Cloud Vector Database is highly regarded for its high performance, scalability and data security. It provides users with powerful data processing and analysis capabilities and is an impressive cloud database solution.
For more exciting events, please pay attention to Tencent Global Digital Ecosystem Conference .