Take action and pay attention
Don’t get lost with useful information
Since the beginning of the development of the Internet, search technology has bloomed with amazing social and economic value. With the rapid development of the information society, data is growing explosively. Search technology meets the needs of information sharing and rapid retrieval through data collection and processing.
The cloud search service ESCloud is a fully managed online distributed search service provided by Volcano Engine . It is compatible with Elasticsearch, Kibana and other software and commonly used open source plug-ins. It can provide multi-condition retrieval, statistics, and reports for structured and unstructured text, helping to achieve one-click deployment, elastic scaling, simplified operation and maintenance, and quickly build practical services such as log analysis and information retrieval analysis.
With the rise of Serverless and the general trend, the Volcano Engine cloud search service has been upgraded to a new cloud-native architecture .
Cloud search service cloud native version
k-NN, native vector search and database in the era of large models
With the emergence of applications in emerging fields such as recommendation and audio and video and the demand for large model scenarios, it is imperative to introduce multi-modal search to meet more complex search needs. We add vector search capabilities based on full-text retrieval to achieve analysis and retrieval of unstructured data .
In the scenario of vector search, a machine learning model is used to generate vectors to represent data objects (text, images, audio and video, etc.); vector distance represents the similarity between objects. Commonly used vector libraries use the ANN algorithm to complete retrieval of massive vectors in a very short time.
k-NN can be used as a vector database. By introducing an advanced vector algorithm library to build a vector index, it will also persist the constructed vector index to disk, making the index more stable. Combined with the inverted index of ESCloud products, the capabilities of vector retrieval and full-text retrieval can be integrated to achieve more powerful hybrid search capabilities. Based on ESCloud's cluster, the k-NN vector database can provide large-scale distributed capabilities and bring users scalable vector searches.
Scenario case
Business scenarios based on k-NN mainly fall into the following six categories, which are currently used in complex business scenarios within ByteDance:
Multi-modal search: including image search, semantic search, audio and video similarity retrieval, etc.;
Intelligent recommendation: video recommendation, advertising recommendation, relationship recommendation, product recommendation, etc.;
Intelligent Q&A: FAQ based on Transformer, domain knowledge Q&A based on LLM, and generative QA based on LangChain collection;
Data deduplication: review and deduplication of videos, audios, and pictures, and copyright detection of various materials;
Security risk control: fraud detection, anti-crime detection, risk assessment, anomaly detection;
Other applications: data mining, data analysis, search reordering, text search and image search.
Take the copywriting similarity recognition scheme as an example.
In the scenario where users push copywriting, in order to ensure the user experience, it is necessary to ensure that the pushed copywriting does not contain duplicate content. Therefore, similarity identification and deduplication of each pushed content will be performed. Each copy is generated Embedding through the BERT model and retrieved once in cloud search. If the similarity is lower than the threshold, it will be judged as new copywriting, which will be written into the k-NN vector database and gradually improved into a copywriting library; if the similarity is higher than the threshold, it will be judged as duplicate copywriting and the amount of push will be reduced.
The cloud search service ESCloud is compatible with Elasticsearch, Kibana and other software and commonly used open source plug-ins. It provides multi-condition retrieval, statistics, and reports of structured and unstructured text. It can achieve one-click deployment, elastic scaling, simplified operation and maintenance, and quickly build log analysis. , information retrieval analysis and other business capabilities.
Scan the QR code to learn more product details
Click "Read the original text" to learn more product details