We are in the midst of an artificial intelligence revolution. It disrupted any industry it touched, promising great innovations – but it also brought new challenges. Efficient data processing has become more important than ever for applications involving large language models, generative AI, and semantic search.
All of these new applications rely on vector embeddings, which are representations of data that contain semantic information that are critical for AIs to gain understanding and maintain long-term memories that they can draw on when performing complex tasks.
Embeddings are generated by AI models, such as large language models, with a large number of attributes or features, which makes their representation difficult to manage. In the context of AI and machine learning, these features represent different dimensions of data, which are critical for understanding patterns, relationships, and underlying structures.
That's why we need a dedicated database specifically designed to handle this type of data. Vector databases like Pinecone meet this requirement by providing optimized storage and query capabilities for embeddings. Vector databases have features of traditional databases that stand-alone vector indexes do not, and specialization in handling vector embeddings that traditional scalar-based databases lack.
The challenge with using vector embeddings is that traditional scalar-based databases cannot keep up with the complexity and scale of such data, making it difficult to extract insights and perform real-time analysis. This is where vector databases come into play - they are designed to handle this type of data and provide the performance, scalability and flexibility needed to get the most out of it.
With the help of vector databases, we can add advanced features to AI such as semantic information retrieval, long-term memory, etc. The diagram below gives us a better idea of the role of a vector database in such an application:
Let's break it down:
1. First, we use an embedding model to create vector embeddings for the content to be indexed.
2. The vector embedding is inserted into the vector database and references some of the original content that created the embedding.
3. When the application issues a query, we use the same embedding model to create embeddings for the query, and use these embeddings to query the database to find similar vector embeddings. As previously mentioned, these similar embeds are associated with the original content used to create them.
What is the difference between a vector index and a vector database?
Independent vector indexes like FAISS (Facebook AI Similarity Search) can significantly improve the search and retrieval of vector embeddings, but they lack features that are present in any database. On the other hand, vector databases are purpose-built for managing vector embeddings, which offer several advantages over using a separate vector index:
**Data Management:** Vector databases provide well-known and easy-to-use functions for data storage, such as inserting, deleting, and updating data. This makes managing and maintaining vector data easier than using standalone vector indexes such as FAISS, which require additional work to integrate with storage solutions.
**Metadata storage and filtering:** The vector database can store metadata associated with each vector entry. Users can then query the database with additional metadata filters for more fine-grained queries.
**Scalability:** Vector databases are designed to scale with growing data volumes and user needs, providing better support for distributed and parallel processing. Standalone vector indexes may require custom solutions to achieve similar levels of scalability (such as deploying and managing them on a Kubernetes cluster or other similar systems).
**Real-time updates:** Vector databases typically support real-time data updates, allowing dynamic changes to the data, whereas standalone vector indexes may require a full re-indexing process to incorporate new data, which can be time-consuming and computationally expensive.
**Backups and Collections:** Vector Database handles the routine of backing up all data stored in the database. Pinecone also allows users to selectively select specific indexes that can be backed up in "collections" where data is stored for later use.
**Ecosystem integration:** Vector databases can be more easily integrated with other components of the data processing ecosystem, such as ETL pipelines (such as Spark), analytics tools (such as Tableau and Segment), and visualization platforms (such as Grafana), thus simplifying Data management workflow. It can also easily integrate with other AI-related tools, such as plugins for LangChain, LlamaIndex, and ChatGPT.
**Data Security and Access Control:** Vector databases often provide built-in data security features and access control mechanisms to protect sensitive information, which may not be available in standalone vector indexing solutions.
In short, vector databases provide a superior solution for processing vector embeddings by addressing the limitations of standalone vector indexes such as scalability challenges, cumbersome integration process, and lack of real-time updates and built-in security measures, thus ensuring more efficient and a simplified data management experience.
How do vector databases work?
We all know how traditional databases work (more or less) - they store strings, numbers and other types of scalar data in rows and columns. A vector database, on the other hand, operates on vectors, so it's optimized and queried quite differently.
In traditional databases, we typically query the database for rows whose values usually exactly match our query. In a vector database, we apply a similarity metric to find the vectors most similar to our query.
Vector databases use a combination of different algorithms that all involve Approximate Nearest Neighbor (ANN) search. These algorithms optimize searches through hashing, quantization, or graph-based searches.
These algorithms are assembled into a pipeline that provides fast and accurate retrieval of the neighbors of the query vector. Since vector databases provide approximate results, the main tradeoff we consider is between accuracy and speed. The more accurate the result, the slower the query. However, a good system can provide ultra-fast searches with near-perfect precision.
The following is a common pipeline for vector databases:
**1. Indexing: **Vector databases use algorithms such as PQ, LSH or HNSW to index vectors (see below for more). This step maps the vector to a data structure for faster searching.
**2. Query:** The vector database compares the index query vector with the index vector in the dataset to find the nearest neighbor (applies the similarity metric used by the index)
**3. Post-processing:** In some cases, the vector database retrieves the final nearest neighbor from the dataset and post-processes it to return the final result. This step can include reordering the nearest neighbors using a different similarity measure.
In the following sections, we discuss each of these algorithms in more detail and explain how they contribute to the overall performance of vector databases.
algorithm
Several algorithms facilitate the creation of vector indices. Their common goal is to enable fast queries by creating data structures that can be traversed quickly. They usually convert the representation of the original vector into a compressed form to optimize the query process.
However, as a Pinecone user, you don't need to worry about the complexities and choices of these different algorithms. Pinecone is designed to handle all the complexities and algorithmic decisions behind the scenes, ensuring you get the best performance and results with ease. By leveraging Pinecone's expertise, you can focus on what really matters - extracting valuable insights and delivering powerful AI solutions.
The following sections explore several algorithms and their unique ways of handling vector embeddings. This knowledge will enable you to make informed decisions and appreciate the seamless performance Pinecone provides as you unleash your app's full potential.
random projection
The basic idea behind random projection is to project a high-dimensional vector into a low-dimensional space using a random projection matrix. We create a matrix of random numbers. The size of the matrix will be the target low-dimensional value we want. We then compute the dot product of the input vectors and matrices, resulting in a projection matrix with fewer dimensions than the original vectors but still retaining their similarity.
When we query, we use the same projection matrix to project the query vector onto the low-dimensional space. We then compare the projected query vector with the projected vectors in the database to find the nearest neighbors. Due to the reduced dimensionality of the data, the search process is significantly faster than searching the entire high-dimensional space.
Remember that random projection is an approximation and the quality of the projection depends on the properties of the projection matrix. In general, the more random the projection matrix, the better the quality of the projection. However, generating a truly random projection matrix can be computationally expensive, especially for large datasets. Learn more about random projections.
Product Quantification
Another approach to index building is Product Quantization (PQ), a lossy compression technique for high-dimensional vectors such as vector embeddings. It takes the original vector, breaks it down into smaller chunks, simplifies the representation of each chunk by creating a representative "code" for each chunk, and then puts all the chunks back together - without losing the pairing of similarities Action-critical information. The process of PQ can be divided into four steps: splitting, training, encoding and querying.
1. Split – The vector is split into segments.
2. Training – We build a “codebook” for each segment. Simply put - the algorithm generates a pool of potential "codes" that can be assigned to vectors. In practice - this "codebook" consists of the center points of the clusters created by performing k-means clustering on each segment of the vector. We have the same number of values in the segment codebook as we use for k-means clustering.
3. Coding – The algorithm assigns a specific code to each segment. In practice, we find the closest value to each vector segment in the codebook after training is complete. The PQ code of our segment will be the identifier of the corresponding value in the codebook. We can use as many PQ codes as we want, which means we can choose multiple values from the codebook to represent each segment.
4. Query – When we query, the algorithm decomposes the vector into subvectors and quantizes them using the same codebook. It then uses the index code to find the closest vector to the query vector.
The number of representative vectors in the codebook is a trade-off between the accuracy of the representation and the computational cost of searching the codebook. The more representative the vectors in the codebook are, the more accurate the representation of the vectors in the subspace is, but the computational cost of searching the codebook is higher. In contrast, the fewer representative vectors in the codebook, the lower the precision of the representation, but the lower the computational cost. Learn more about PQ.
position-sensitive hashing
Locality Sensitive Hashing (LSH) is a technique for indexing in the context of an approximate nearest neighbor search. It is optimized for speed while still providing approximate, non-exhaustive results. LSH maps similar vectors into "buckets" using a set of hash functions, as follows:
To find the nearest neighbor for a given query vector, we use the same hashing Greek function. The query vector is hashed to a specific table and then compared with other vectors in the same table to find the closest match. This method is much faster than searching the entire dataset because there are far fewer vectors in each hash table than in the entire space.
It's important to remember that LSH is an approximation method, and the quality of the approximation depends on the properties of the hash function. In general, the more hash functions used, the better the quality of the approximation. However, using a large number of hash functions can be computationally expensive and may not be feasible for large datasets.
Hierarchical Navigation Small World (HNSW)
HNSW creates a hierarchical tree structure where each node of the tree represents a set of vectors. Edges between nodes represent similarities between vectors. The algorithm starts by creating a set of nodes, each with a small number of vectors. This can be done randomly, or by clustering the vector using an algorithm like k-means, where each cluster becomes a node.
The algorithm then examines each node's vector and draws an edge between that node and the node most similar to the vector it has.
When we query the HNSW index, it uses this graph to navigate the tree, visiting the nodes most likely to contain the vector closest to the query. Learn more about HNSW.
similarity measure
Based on the algorithms discussed earlier, we need to understand the role of similarity measures in vector databases. These metrics are the basis for how vector databases compare and identify the most relevant results for a given query.
A similarity measure is a mathematical method for determining how similar two vectors are in a vector space. Similarity measures are used in vector databases to compare vectors stored in the database and find the most similar vector to a given query vector.
Several similarity measures can be used, including:
**Cosine Similarity:** Measures the cosine of the angle between two vectors in a vector space. It ranges from -1 to 1, where 1 means identical vectors, 0 means orthogonal vectors, and -1 means diametrically opposite vectors.
**Euclidean Distance:** Measures the straight-line distance between two vectors in a vector space. It ranges from 0 to infinity, where 0 represents the same vector and larger values represent increasingly different vectors.
**Dot Product:** Measures the product of the magnitude of two vectors and the cosine of the angle between them. It ranges from -∞ to ∞, where positive values indicate vectors pointing in the same direction, 0 indicates an orthogonal vector, and negative values indicate vectors pointing in the opposite direction.
The choice of similarity measure will have an impact on the results obtained from the vector database. It is also important to note that each similarity measure has its own advantages and disadvantages, and it is important to choose an appropriate measure based on the use case and requirements. Learn more about similarity measures.
filtering
Each vector stored in the database also includes metadata. In addition to the ability to query for similar vectors, vector databases can filter results based on metadata queries. To this end, vector databases typically maintain two indexes: a vector index and a metadata index. It then performs metadata filtering before or after the vector search itself, but in either case there are difficulties that slow down the query process.
The filtering process can be performed before or after the vector search itself, but each approach has its own challenges that can affect query performance:
**Pre-filtering:** In this approach, metadata filtering is done before the vector search. While this helps reduce the search space, it can also cause relevant results that don't match the metadata filter to be ignored. Additionally, extensive metadata filtering may slow down the query process due to increased computational overhead.
**POST FILTERING:** In this approach, metadata filtering is done after the vector search. This helps ensure that all relevant results are considered, but can also introduce additional overhead and slow down the query process as irrelevant results need to be filtered out after the search is complete.
To optimize the filtering process, vector databases use various techniques such as advanced indexing methods utilizing metadata or using parallel processing to speed up filtering tasks. Balancing the trade-off between search performance and filtering accuracy is critical to provide efficient and relevant query results in vector databases. Learn more about vector search filtering.
database operation
Unlike vector indexes, vector databases are equipped with a set of features that make them more qualified for use in large-scale production environments. Let's look at a general overview of the components involved in operating a database.
performance and fault tolerance
Performance and fault tolerance are closely related. The more data we have, the more nodes we need – the more potential for errors and failures. As with other types of databases, we want to ensure that queries are executed as quickly as possible even if some underlying nodes fail. This could be due to hardware failure, network failure, or other types of technical errors. Such failures can cause downtime or even incorrect query results.
To ensure high performance and fault tolerance, Vector Database uses sharding and replication to apply the following:
Sharding – Partitioning data across multiple nodes. There are different ways to partition data – for example, different data clusters can be partitioned by their similarity so that similar vectors are stored in the same partition. When a query is made, it is sent to all shards, and the results are retrieved and combined. This is called the "scatter-gather" pattern.
Replication – Creates multiple copies of data across different nodes. This ensures that even if a particular node fails, other nodes will be able to replace it. There are two main consistency models: eventual consistency and strong consistency. Eventual consistency allows temporary inconsistencies between different copies of data, which increases availability and reduces latency, but can lead to conflicts and even data loss. Strong consistency, on the other hand, requires that all copies of the data be updated before a write operation is considered complete. This method provides greater consistency, but may result in higher latency.
monitor
To effectively manage and maintain a vector database, we need a robust monitoring system to track important aspects of database performance, health, and overall status. Monitoring is critical to detecting potential issues, optimizing performance and ensuring smooth production operations. Some aspects of monitoring vector databases include:
Resource Usage – Monitoring resource usage such as CPU, memory, disk space, and network activity can identify potential issues or resource constraints that may affect database performance.
Query Performance – Query latency, throughput, and error rates may indicate potential systemic issues that need to be addressed.
System Health – Overall system health monitoring includes status of individual nodes, replication process, and other key components.
Access control
Access control is the process of managing and regulating user access to data and resources. It is an important part of data security, ensuring that only authorized users can view, modify or interact with sensitive data stored in a vector database.
Access control is important for the following reasons:
**Data Protection:** Since AI applications often deal with sensitive and confidential information, implementing strict access control mechanisms helps protect data from unauthorized access and potential damage.
**Compliance:** Many industries, such as healthcare and finance, are subject to strict data privacy regulations. Implementing proper access controls helps organizations comply with these regulations, protecting them from legal and financial repercussions.
**Accountability and Auditing:**Access control mechanisms enable organizations to maintain records of user activity in vector databases. This information is critical for auditing purposes, and when a security breach occurs, it helps trace any unauthorized access or modification.
**Scalability and Flexibility:** As an organization grows and evolves, its access control needs may change. A robust access control system allows seamless modification and extension of user privileges, ensuring data security remains constant throughout the growth of the organization.
backup and collection
When all else fails, vector databases provide the ability to rely on regularly created backups. These backups can be stored on external storage systems or cloud-based storage services, ensuring data security and recoverability. In the event of data loss or corruption, these backups can be used to restore the database to a previous state, minimizing downtime and impact on the overall system. With Pinecone, users can also choose to backup specific indexes and save them as "collections" that can later be used to populate new indexes.
Interface and SDK
This is where the rubber meets the road: developers who interact with databases want to use an easy-to-use API, using a familiar and comfortable toolset to do so. The vector database API layer simplifies the development of high-performance vector search applications by providing a user-friendly interface.
In addition to the API, vector databases usually provide a programming language-specific SDK to wrap the API. The SDK enables developers to more easily interact with databases in their applications. This enables developers to focus on their specific use cases, such as semantic text search, generative question answering, hybrid search, image similarity search, or product recommendations, without having to worry about the complexities of the underlying infrastructure.
Summarize
The exponential growth of vector embeddings in NLP, computer vision, and other AI applications has led to the emergence of vector databases as computational engines that allow us to efficiently interact with vector embeddings in applications.
Vector databases are purpose-built databases specifically designed to solve the problems that arise when managing vector embeddings in production scenarios. Therefore, they have significant advantages over traditional scalar-based databases and independent vector indexes.
In this post, we review key aspects of a vector database, including how it works, the algorithms it uses, and additional features that make it operationally ready in production scenarios. We hope this helps you understand the inner workings of vector databases. Fortunately, this is not something you have to know to use Pine Cone. Pinecone handles all of these (and then some) considerations, letting you focus on the rest of your application.