Vector database technology panorama
After long-term internal exploration and optimization, the vector database product structure adopted by Douyin is as shown in the figure below: Based on cloud infrastructure, it provides various engines that have been deeply polished and optimized, providing everything from multi-modal data writing, to vector generation, and then A complete set of full-link solutions to online retrieval, as well as flexible scheduling and monitoring after going online.
Scenario implementation practice of Volcano Engine vector database
After technical practice within the Douyin Group, the vector database currently covers more than 50 business lines, basically supporting all internal vector retrieval scenarios, such as Douyin, Toutiao, Dianchedi, Tuchong, Volcano Engine Oncall Intelligent Question and Answer and Cutting The main business scenarios include intelligent search, AIGC cross-modal retrieval, recommendation and deduplication, intelligent question and answer, related sorting, cluster analysis and data mining, etc., and the scale of multiple scenario libraries reaches tens of billions.
The following uses Tu Chong and Volcano Engine Oncall intelligent question answering as examples to demonstrate the application practice of vector databases.
● Intelligent search scenarios—Tuchong’s image search
Tuchong provides the ability to search pictures by pictures, and is committed to providing users with genuine material content and digital asset management solutions. Currently, Tuchong Creative has 460 million pictures and over 20 million high-definition videos in its library. A large number of users search and query pictures and videos every day. Billion-level massive data puts forward higher requirements for vector retrieval service capabilities. How can businesses flexibly set up sharding? When the amount of data increases significantly, how can they avoid redeploying clusters, speed up index construction, and save money?