TiDB Hackathon 2023 has just ended, and I carefully reviewed all the projects. Without emphasizing that the project must use artificial intelligence (AI)-related technologies, eye-catching projects almost uniformly use AI to build their applications. The advent of large-scale language models (LLM) allows individual developers to add reasoning capabilities to programs in just 5 minutes, something that in the past could only be accomplished by very large teams. From the perspective of application developers, the AI era has also arrived.
In these AI applications, vector databases are ubiquitous. Although most of these projects still use relational databases, they no longer seem to play an obvious role. Are relational databases still worthy of the attention of application developers?
In order to answer this question clearly, we need to understand how vector databases differ from traditional relational databases.
What is a vector database?
In order to figure this out, I spent some time studying vector databases. Next, I will explain in the simplest language what a vector database is.
Most things in this world are multi-characteristic. For example, you can describe a person using many different types of dimensions such as height, weight, personality, gender, dressing style, hobbies, etc. Usually if you are willing, you can infinitely expand this dimension or feature to describe an object. The more dimensions or features, the more accurate the description of an object or event.
Now, if we start to use one dimension to express Emoji expressions, 0 represents happiness and 1 represents sadness. The number size from 0 to 1 can express the sadness and joy of the corresponding expression, as shown on the x-axis below:
But you will find that if there is only one dimension to describe emotional Emoji, it is general and inaccurate. For example, there are many types of Emoji that can express happiness. So at this time we usually add new dimensions to describe it better. For example, we add the Y axis here, with 0 representing yellow and 1 representing white. After adding, the points expressing each expression on the coordinate axis become tuples of (x, y).
If you are smart, you must have discovered that even if we add the new description dimension of the Y axis, there are still Emojis that we cannot distinguish. for example
So what to do? The solution is still simple, just add another dimension. In the coordinate system, the z axis is added. We simply set the new dimension to whether or not we are wearing a hat (note that the value of each dimension here is as simple as possible for the sake of explanation, and does not mean that the real world is also so simple). Use 0 to represent not wearing it and 1 to represent wearing it. So now we have a three-dimensional coordinate point of (x, y, z) to describe an Emoji.
Of course, in the real world, a thing does not have so few properties, so we need to add many dimensions to describe it, so descriptions like high-dimensional arrays appear (0.123, 0.295, 0.358, 0.222...) . At this point we are very close to the "vectors" in the vector database. In fact, the vector database stores such arrays to represent various data, including pictures, videos, text, etc. These things are transformed into high-dimensional arrays through our above-mentioned conversion method, and then saved.
Maybe at this point you still don’t understand the role of a vector database: Why do we change things into this form?
Simply put, this is because after turning it into a vector, we have a way to quantify the correlation and similarity between any two things in the world. Through our demonstration just now, the closer things are in each dimension, the closer they will be in space. By calculating the distance between two points, you can determine the similarity between them.
So if we have an Emoji that has never appeared before, we can use the above method to turn this Emoji into a vector (0.01, 1, 0).
By calculating the vectors already stored in the library, you can find out the closest Emoji is
The next closest thing is
As evidence, you can look at PineCone Query Data ( https://docs.pinecone.io/docs/query-data#sending-a-query ) for an example of obtaining data (Score can simply be thought of as similarity):
index.query(
vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3],
top_k=3,
include_values=True
)
# Returns:
# {'matches': [{'id': 'C',
# 'score': -1.76717265e-07,
# 'values': [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]},
# {'id': 'B',
# 'score': 0.080000028,
# 'values': [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2]},
# {'id': 'D',
# 'score': 0.0800001323,
# 'values': [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]}],
# 'namespace': ''}Values is the retrieved vector (which can be considered the corresponding Emoji in our example). This means that we can vectorize all the query criteria to find what is closest to what we want. If we replace Emoji with text, we can implement "semantic" search. If you replace Emoji with pictures or videos, you can implement picture or video similarity recommendations.
Why do AI applications often rely on vector databases?
To explain it in one sentence: "Large models" can only remember limited things.
This is very similar to our brain. In the process of communication, it is impossible for us to give all our knowledge to the other party in the conversation. Usually we can only make certain "inferences" through limited context. So in current AI applications, reasoning capabilities are provided by LLM, and the most relevant context that needs to be expressed is found from your brain. So by analogy, the vector database is similar to the memory or knowledge base of LLM. Therefore, to complete an AI-related function, without the help of a vector database, usually the functions and accuracy that large AI models can complete are very limited.
Looking along this line of thinking, in addition to some less precise fuzzy matching, there are actually many searches/indexes that require very precise and deterministic search/indexing in real life. This is similar to how we usually record some important information in a notebook, and then retrieve it accurately through the index when needed.
Therefore, the biggest difference between vector databases and relational databases is the data storage method and index query method. And it is precisely because of the precise indexes that exist in relational databases that it can obtain the corresponding information at the millisecond level. Corresponding to business systems that require high-speed access, such as accounts, products, order information, etc., it still needs to be completed by it.
Let’s take the Hackathon award-winning application Heuristic AI ( https://devpost.com/software/cx-8lh7ps ) as an example to show you how to use these two types of databases in a real project. .
In our daily life, when the electronic products we use malfunction, we usually need to read complicated user manuals to get the relevant solutions, and it takes a lot of time to learn. This project accomplished the following:
- Import all product manuals into the vector database
- Use natural language to describe the problems encountered and find the most relevant context in the vector database through semantic search.
- Pack the context into Prompt and send it to OpenAI to generate the corresponding solution.
The rough technical implementation is as follows:
If this software ended here, it would basically be a toy. Usually it is also necessary to add a user authentication and management system to the system. In addition, it is usually necessary to add an analysis system for business data in the background, such as how many online users have used the product, how often it is used, and other dimensions. These functions need to be implemented using traditional databases:
Of course, as a Hackathon project, this software is actually relatively complete. But if it is to be further commercialized, the following aspects need to be considered:
○ The amount of user data has increased dramatically, and the scalability and stability of the system have improved
○ Data backup and recovery in multiple data centers and disaster situations
None of this is cool, and it's even painful, but it's still an area we need to take cautiously and seriously. Fortunately, from this Hackathon, another trend can be observed with the naked eye: Serverless, which is helping developers continue to reduce the technical difficulty of productizing an application.
Efficiency improvements brought about by serverless basic software
What can be observed: Independent developers play an increasingly prominent role in project development. Independent developers play an increasingly prominent role in project development. Compared with the past, there is no longer a need for a large team of 3-4 people to work together. Today's excellent projects are often completed by 1-2 developers, or even individuals alone.
Behind this trend, the wave of serverless serves as an important driving force. With Serverless, developers can focus on business logic without having to worry about the details of the underlying infrastructure. This time I don’t see any developers using local deployment to implement their own applications. Vercel is used for front-end and business code deployment, and Qrdrant ( https://qdrant.tech/ ) or Pinecone ( ) for back-end components, such as Vector database . https://www.pinecone.io/ ), and the relational database uses TiDB Cloud Serverless ( https://bit.ly/3PsYJle ). Using this set, basically an engineer can complete a Demo-level application.
In this era, the field of AI is not the only one that excels. Other traditional technologies are actually providing developers with an increasingly convenient experience and are constantly iterating with the tide.
As long as the focus returns to the developers themselves, everyone will have a bright future.