This is actually a rather tortuous and rare story. Since I joined the company in the middle, I have participated in the construction of Milvus since version 2.0, and our company has a history of about 5 years in the field of vector databases.
Initially, we saw the opportunity and realized that there were engines such as Faiss that were very mature in handling vector retrieval and that many large vendors were using them. Therefore, our first version design is very simple, only adding some persistent logs and some basic RPC to Faiss. At that time, many people thought that Milvus was just a layer of encapsulation on Faiss. This was the earliest view of Milvus. In fact, this conclusion can now be applied to most of the vector databases being started. Their current architecture is the same as our earliest 1.0 architecture. There is almost no difference.
In this process, we made many attempts from version 0.1 to 1.0, mainly influenced by the earliest important customer. Although I won't mention the customer's name, they initially used an early version of Milvus, version 0.1, to process the 10 million data set, and it worked very well and they were pleased with it.
However, they found that as the amount of data increased, Milvus could not scale to meet their needs. They hope to process larger data sets, reaching 1 billion or more, but this requires facing many challenges. To this end, we have adopted many techniques, including special optimization methods and larger models, to solve the problem of 1 billion data.
Since then, we have recognized that performance and scalability of vector retrieval (in late 2020 to early 2021) are very critical issues. Therefore, we began to build the core capabilities of version 2.0.