In the past year, the rise of large language models represented by ChatGPT and LLaMA has pushed the development of vector databases to new heights.
Vector databases are a new type of database that is increasingly popular in the fields of machine learning and artificial intelligence and can help support new search engines based on neural networks rather than keywords. Vector databases differ from traditional relational databases, such as PostgreSQL, in that they were originally designed to store tabular data in the form of rows and columns. It is also significantly different from newer NoSQL databases, such as MongoDB, which primarily store data in JSON documents.
Vector databases are designed for storing and retrieving a specific type of data: vector embeddings. They are essentially filters that are run on new data during the inference part of the machine learning process.
In large model deployments, vector databases can be used to store vector embeddings resulting from large model training. By storing potentially billions of vector embeddings representing extensive training of large models, the vector database performs the all-important similarity search, finding the best match between the user prompt (the question he or she asked) and a specific vector embedding.
After the popularity of large models, more companies began to invest heavily in vector databases to improve algorithm accuracy and efficiency. According to relevant statistics, the AI investment field in April 2023 showed a growth trend, especially in the vector database field. Investment activities are quite active. Vector database startups such as Pinecone, Chroma and Weviate have all received financing this month.
The current vector database plays an important role in the era of large model gold mining. It is like a good shovel, helping to dig out more and more valuable resources.
But what cannot be ignored is that