How to implement a vector database for AI

Recommendation: Use NSDT scene editor to help you quickly build 3D application scenes

However, AI models are a bit like gourmet chefs. They can work wonders, but they require quality ingredients. AI models do a good job on most inputs, but they really shine if they receive input in the most optimized format. That's the whole point of a vector database.

Over the course of this article, we'll dig into what vector databases are, why they're becoming more and more important in the world of artificial intelligence, and then we'll look at a step-by-step guide to implementing a vector database.

Jump ahead:

What is a vector database?
Why do you need a vector database?
Implementing a Vector Database: A Step-by-Step Guide
prerequisites
Setting up the Weaviate project
Create our node.js project
Set up our vector database
set client
migrate data
add document
delete document
Add a query function to the database
Combining Vector Embeddings and AI
AI Model Settings
query our data
Test our query

What is a vector database?

Before we start exploring vector databases, it's important to understand what a vector is in the context of programming and machine learning.

In programming, a vector is essentially a one-dimensional array of numbers. If you've ever written code involving 3D graphics or machine learning algorithms, chances are you've worked with vectors.

const vector4_example = [0.5, 1.5, 6.0, 3.4]

They're just arrays of numbers, usually floats, that we refer to by their dimensions. For example, a is a three-element array of floating-point numbers, and a is a four-element array of floating-point numbers. It's that simple!vector3vector4

But vectors are more than just arrays of numbers. In the context of machine learning, vectors play a key role in representing and manipulating data in high-dimensional spaces. This allows us to perform the complex operations and calculations that drive AI models.

Now that we have vectors in hand, let's turn our attention to vector databases.

At first glance, you might think, "Hey, since vectors are just arrays of numbers, can't we use a regular database to store them?" Well, technically, you can. But this is where things get interesting.

A vector database is a special type of database optimized for storing and performing operations on large amounts of vector data. So while your regular database can indeed store arrays, vector databases go a step further and provide specialized tools and operations to work with vectors.

In the next section, we discuss why vector databases are necessary, and the advantages they bring. So hang in there because things are about to get more interesting!

Why do you need a vector database?

Now that we have a solid understanding of what vector databases are, let's dig into why they are so necessary in the field of artificial intelligence and machine learning.

The key word here is performance. Vector databases typically process hundreds of millions of vectors per query, a performance that is much faster than traditional databases can achieve when processing vectors.

So, what makes a vector database so fast and efficient? Let's take a look at some of the key features that set them apart.

complex math

Vector databases are designed to perform complex mathematical operations on vectors, such as filtering and locating "nearby" vectors. These operations are crucial in machine learning contexts, where models often need to find vectors that are close to each other in a high-dimensional space.

For example, a common data analysis technique, cosine similarity, is often used to measure how similar two vectors are. Vector databases are good at these types of calculations.

dedicated vector index

Like well-organized libraries, databases require a good indexing system to quickly retrieve requested data. Vector databases offer specialized vector indexes that allow faster and more deterministic retrieval of data than traditional databases (as opposed to random databases).

With these indexes, the vector database can quickly locate the vectors needed by the AI model and generate the results quickly.

compact storage

In the world of big data, storage space is a precious commodity. Vector databases also shine here to store vectors in a more compact way. Techniques such as compression and quantized vectors are used to keep as much data as possible in memory, further reducing load and query latency.

Fragmentation

When dealing with large amounts of data, it may be beneficial to distribute the data across multiple machines, a process known as sharding. Many databases can do this, but SQL databases in particular require more effort to scale out. On the other hand, vector databases often have sharding built into their architecture, allowing them to easily handle large amounts of data.

In short, while traditional databases can store and perform operations on vectors, they are not optimized for the task. Vector databases, on the other hand, are built for exactly this purpose. They provide the speed, efficiency, and specialized tools needed to process large amounts of vector data, making them essential tools in the field of artificial intelligence and machine learning.

In the next section, we compare vector databases to other types of databases and explain how they fit into the larger database ecosystem. We're just getting started!

Implementing a Vector Database: A Step-by-Step Guide

For the purpose of this guide, we'll use Weaviate, a popular vector database service, to implement a simple vector database that you can build upon for any use case.

You can clone the starter template here and run to get set.npm install

prerequisites

Previous JS knowledge will help : all code written in this tutorial will be in JavaScript, and we will also be using the Weaviate JavaScript SDK.
Node and npm : We'll be working in a JavaScript environment on the server.
OpenAI API key : We will use their embedding model to convert our data into embeddings to store in our database
Weaviate Account : We will be using their managed database service; you can get a free account here

Setting up the Weaviate project

After creating your account, you need to set up your project through the Weaviate dashboard. Go to the WCS console and click Create Cluster :

Create a Weaviate cluster

Select the "Free Sandbox" tier and provide a cluster name . Select Yes when it asks you to enable authentication :

Enable authentication in the cluster

Click Create . After a few minutes, you should see a check mark Done.

Click on Details to view the cluster details as we will use them in the next section. These include:

a woven url
Authentication details (Weaviate API key; click the key icon to display)

Create our node.js project

With the prerequisites in place, we can create the vector database and query it. To proceed, you'll need a new Node project; you can clone the template on GitHub here, which includes everything you need to get started.

Alternatively, you can create one by running:

mkdir weaviate-vector-database && cd weaviate-vector-database
npm init -y && npm install dotenv openai weaviate-ts-client
mkdir src

Edit the file and add the script as follows:package.jsonstart

// ...rest of package.json
"scripts": {
"start": "node src/index.js"
},
// ...rest of package.json

Create a file to store sensitive information such as API keys. Write the command and open the newly created file in a code editor, then paste the following content making sure to replace the placeholders with actual values:.envtouch .env.env

// .env
OPENAI_KEY="<OPENAI_API_KEY>"
WEAVIATE_API_KEY="<WEAVIATE_API_KEY>"
WEAVIATE_URL="<WEAVIATE_URL>"
DATA_CLASSNAME="Document"

Set up our vector database

Once the project is setup, we can add some code to setup and use our vector database. Let's quickly summarize what we're going to achieve:

Helper function, which:
connect to our database
Batch vectorize and upload documents
Find most similar items
A main function that uses the helper functions above to upload documents and query the database in one go

set client

Having said that, let's create our first file to store the database connection and helper functions. Create a new file by running, and let's start filling it out:touch src/database.js

// src/database.js
import weaviate, { ApiKey } from "weaviate-ts-client";
import { config } from "dotenv";

config();

async function setupClient() {
let client;

try {
client = weaviate.client({
scheme: "https",
host: process.env.WEAVIATE_URL,
apiKey: new ApiKey(process.env.WEAVIATE_API_KEY),
headers: { "X-OpenAI-Api-Key": process.env.OPENAI_API_KEY },
});
} catch (err) {
console.error("error >>>", err.message);
}

return client;
}
// ... code continues below

Let's break down what's going on here. First, we import the necessary packages, Weaviate client and dotenv configuration. dotenv is a load of environment variables from a file into .Weaviate and OpenAI keys and URLs are often stored in environment variables to keep them confidential and away from the codebase..envprocess.env

Here's what happens in the function:setupClient()

We initialize a variableclient
We have a block that sets up the connection to the Weaviate server. If any error occurs during this process, we will print the error message to the consoletry…catch

Inside the block, we use the method to create a new Weaviate client. , and parameters are taken from the environment variables we settryweaviate.client()schemehostapiKey

Finally, we pass in OpenAI's headers, since we'll be using OpenAI's Ada model to vectorize our data.

migrate data

With the client set up, let's run the migration with some collections of dummy data, fictional creatures, places, and events. Later, we will query GPT-3 against this data.

If you didn't clone the starter template, follow these steps:

Create a new file by runningtouch src/data.js
Copy the content of the file from here and paste it in

Spend some time browsing the data in. Then, add a new import at the top of the file:src/data.jssrc/database.js

// ...other imports
import { FAKE_XORDIA_HISTORY } from "./data";

Below Functions, add a new function as follows:setupClient

async function migrate(shouldDeleteAllDocuments = false) {
try {
const classObj = {
class: process.env.DATA_CLASSNAME,
vectorizer: "text2vec-openai",
moduleConfig: {
"text2vec-openai": {
model: "ada",
modelVersion: "002",
type: "text",
},
},
};

const client = await setupClient();

try {
  const schema = await client.schema
    .classCreator()
    .withClass(classObj)
    .do();
  console.info("created schema >>>", schema);
} catch (err) {
  console.error("schema already exists");
}

if (!FAKE_XORDIA_HISTORY.length) {
  console.error(`Data is empty`);
  process.exit(1);
}

if (shouldDeleteAllDocuments) {
  console.info(`Deleting all documents`);
  await deleteAllDocuments();
}

console.info(`Inserting documents`);
await addDocuments(FAKE_XORDIA_HISTORY);

} catch (err) {
console.error("error >>>", err.message);
}
}

Again, let's break down what's going on here.

The function accepts a single parameter that determines whether to clear the database when migrating data.migrateshouldDeleteAllDocuments

In our block, we create an object called . This object represents the schema of the class in Weaviate (make sure to add a to the file) that uses the vectorizer. This determines how text documents are configured and represented in the database, and tells Weaviate to vectorize our data using OpenAI's "ada" model.try…catchclassObjCLASS_NAME.envtext2vec-openai

Then, we create the schema using method chaining. This sends a request to the Weaviate server to create the document class defined in . After successfully creating the schema, we log the schema object to the console, and display a message. Errors are now handled with simple messages logged to the console.client.schema.classCreator().withClass(classObj).do()classObjcreated schema >>>

We can check the length of the dummy data to be migrated. If empty, the code ends here. We can use a function (will be added later) to clear the database if the .deleteAllDocumentsshouldDeleteAllDocumentstrue

Finally, using a function (which we'll add next), we upload all the items to be vectorized and stored in Weaviate.addDocuments

add document

We can go ahead and vectorize and upload our text document. This is actually a two-step process where:

Raw text strings converted to vectors using the OpenAI Ada model
Converted vectors will be uploaded to our Weaviate database

Thankfully, these are automatically handled by the Weaviate SDK we use. Let's go ahead and create functions to do this. Open the same file and paste the following:src/database.js

// code continues from above
const addDocuments = async (data = []) => {
const client = await setupClient();
let batcher = client.batch.objectsBatcher();
let counter = 0;
const batchSize = 100;

for (const document of data) {
const obj = {
class: process.env.DATA_CLASSNAME,
properties: { ...document },
};

batcher = batcher.withObject(obj);
if (counter++ == batchSize) {
  await batcher.do();
  counter = 0;
  batcher = client.batch.objectsBatcher();
}

}

const res = await batcher.do();
return res;
};
// ... code continues below

As before, let's break down what's going on here.

First, we call the function defined earlier to set and get the Weaviate client instancesetupClient()
We initialize a batch processor usingclient.batch.objectsBatcher()
We also define a counter variable and a variable and set it to 100. A counter keeps track of how many documents have been added to the current batch and defines how many documents should be included in each batchbatchSizebatchSize
Then, we iterate over each document in the data array:

For each document, we create an object that represents the document in the format Weaviate expects so that it can be expanded into the object's properties
Then, we usebatcher.withObject(obj)
If the counter is equal to the batch size (meaning the batch is full), we upload the batch to Weaviate, reset the counter to , and create a new batch processor for the next batch of documentsbatcher.do()0

After all documents have been processed and added to the batch, if there is a remaining batch that has not been uploaded (because it did not arrive), you can use Upload Remaining Batch.batchSizebatcher.do()

The last step here happens when the function returns the response from the last call. This response will contain details about the upload, such as whether the upload was successful and any errors that occurred.batcher.do()

Essentially, this function helps us efficiently upload large numbers of documents to our Weaviate instance by grouping them into manageable batches.addDocuments()

delete document

Let's add the code used in the function. Below the function, add the following code:deleteAllDocumentsmigrateaddDocuments

// code continues from above
async function deleteAllDocuments() {
const client = await setupClient();
const documents = await client.graphql
.get()
.withClassName(process.env.DATA_CLASSNAME)
.withFields("_additional { id }")
.do();

for (const document of documents.data.Get[process.env.DATA_CLASSNAME]) {
await client.data
.deleter()
.withClassName(process.env.DATA_CLASSNAME)
.withId(document._additional.id)
.do();
}
}
// ... code continues below

This function is relatively simple.

We use the class namesetupClientidDocument
Then using a loop we delete each document usingfor...ofid

This approach works because we have a small amount of data. For larger databases, a technique is needed to delete all documents, since the limit per request is only 200 entries at a time.batching

Add a query function to the database

Now that we have a method for uploading data to the database, let's add a function to query the database. In this example, we'll perform a "nearest neighbor search" to find documents similar to our query.

In the same file, add the following:src/database.js

// code continues from above
async function nearTextQuery({
concepts = [""],
fields = "text category",
limit = 1,
}) {
const client = await setupClient();
const res = await client.graphql
.get()
.withClassName("Document")
.withFields(fields)
.withNearText({ concepts })
.withLimit(limit)
.do();

return res.data.Get[process.env.DATA_CLASSNAME];
}

export { migrate, addDocuments, deleteAllDocuments, nearTextQuery };

Again, let's break down what's going on here:

nearTextQuery()is an asynchronous function that accepts an object as a parameter. This object can contain three properties:

概念: array of strings representing the terms we are searching for
field: A string representing what we want returned in search results 字段. In this example, we request from and fieldtextcategory
限制: the maximum number of results we want returned from the search query

We call the function to get the Weaviate client instancesetupClient()
We build GraphQL queries using a series of methods:

client.graphql.get(): Initialize the GraphQL query
.withClassName("Document"): we specify that we want to search in the "Documents" object
.withFields（fields）：We specify which fields to return in the result
.withNearText（{ concepts }）: This is where the magic happens! We specify the concept Weaviate will use to search for semantically similar documents
.withLimit(limit): we specify the maximum number of results to return
Finally, execute the query.do()

The response from the query is stored in a variable and then returned on the next lineres
Finally, we export all functions defined here for use elsewhere

In a nutshell, this function helps us to search for semantically similar documents in a Weaviate instance based on the terms provided.nearTextQuery()

Let's migrate the data so we can query it in the next section. Open a terminal and run.npm run start"migrate"

Combining Vector Embeddings and AI

Large language models like GPT-3 and ChatGPT are designed to process input and generate useful output, a task that requires understanding complex meanings and relationships between words and phrases.

They do this by representing words, sentences, or even entire documents as high-dimensional vectors. By analyzing the similarities and differences between these vectors, AI models can understand context, semantics and even nuances in our language.

So, where do vector databases come from? Let's think of vector databases as librarians for AI models. In a huge book library (or, in our case, a vector), an AI model needs to quickly find the most relevant books for a particular query. Vector databases do this by efficiently storing these "books" and providing fast and precise retrieval when needed.

This is critical for many AI applications. For example, in a chatbot application, an AI model needs to find the most relevant responses to user questions. It does this by converting the user's questions and potential responses into vectors, and then uses a database of vectors to find the response most similar to the user's question.

With this in mind, we will use the database above to feed an AI model, GPT-3.5, with the context of our own data. This will allow the model to answer questions about the data it was not trained on.

AI Model Settings

Create a new file by running and pasting the following:touch src/data.js

import { Configuration, OpenAIApi } from "openai";

const configuration = new Configuration({
apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

async function getChatCompletion({ prompt, context }) {
const chatCompletion = await openai.createChatCompletion({
model: "gpt-3.5-turbo",
messages: [
{
role: "system",
content: You are a knowledgebase oracle. You are given a question and a context. You answer the question based on the context. Analyse the information from the context and draw fundamental insights to accurately answer the question to the best of your ability. Context: ${context} ,
},
{ role: "user", content: prompt },
],
});

return chatCompletion.data.choices[0].message;
}

export { getChatCompletion };

As usual, let's break down the file:

We import some required modules from the package and initialize an instance ofopenaiopenai
We define a function that takes a hint, some context, and configures the GPT-3.5 model to respond as a knowledge base oraclegetChatCompletion
Finally, we return the response and export the function

query our data

By setting up our vector database and AI model, we can finally query our data by combining these two systems. Leveraging the powerful effects of embedding and the impressive natural language capabilities of GPT-3.5, we will be able to interact with our data in more expressive and customizable ways.

First create a new file and run. Then paste the following:touch src/index.js

import { config } from "dotenv";
import { nearTextQuery } from "./database.js";
import { getChatCompletion } from "./model.js";

config();

const queryDatabase = async (prompt) => {
console.info(Querying database);
const questionContext = await nearTextQuery({
concepts: [prompt],
fields: "title text date",
limit: 50,
});

const context = questionContext
.map((context, index) => {
const { title, text, date } = context;
return Document ${index + 1} Date: ${date} Title: ${title} ${text} ;
})
.join("\n\n");

const aiResponse = await getChatCompletion({ prompt, context });
return aiResponse.content;
};

const main = async () => {
const command = process.argv[2];
const params = process.argv[3];

switch (command) {
case "migrate":
return await migrate(params === "--delete-all");
case "query":
return console.log(await queryDatabase(params));
default:
// do nothing
break;
}
};

main();

In this file, we've pulled together all the work we've done so far to allow us to query data via the command line. As usual, let's explore what's going on here:

First, we import the necessary modules and set our environment variables with the packagedotenv
Next, we create a function that accepts a text hint, which we use to perform a "near-text" query against the vector database. We limited the results to 50, and we specifically asked for the Title, Text, and Date fields to match the conceptqueryDatabase
This basically returns documents that are semantically similar to any significant term in our search query (embeddings are powerful!
We then map the received context, format it, and pass it to the AI model to generate completions. Using context, GPT-3.5's natural language processing (NLP) capabilities shine as it is able to generate more accurate and meaningful responses based on our data
Finally, we reach the function. Here, we use command line arguments to perform various tasks. If we pass, we can migrate our data (with optional flags, just in case we want to clean up our slate and start fresh), and with that, we can test our query functionsmainmigrate--delete-allquery

Test our query

congratulate. If you got this far, you deserve a pat on the back - you can finally test your code.

Open a terminal and run the following command:

npm run start "query" "what are the 3 most impressive achievements of humanity in the story?"

The query is sent to your Weaviate vector database where it is compared to other similar vectors and returns the 50 most similar vectors based on their text. This contextual data is then formatted and sent along with your query to OpenAI's GPT-3.5 model, where it is processed and a response is generated.

If all goes well, you should get a response similar to the following:

Responses from query tests

Feel free to explore this fictional world, make more queries, or better yet, bring your own data and witness the power of vectors and embeddings for yourself.

If you encounter any errors at this point, compare your code with the final version here, and make sure the file is created and populated..env

Conclusions and next steps

In this tutorial, we've explored a little bit the power of vectors and vector databases. Using tools like Weaviate and GPT-3, we have seen first-hand the potential of these technologies to shape AI applications, from improving personalized chatbots to enhancing machine learning algorithms. Be sure to check out our GitHub too!

However, this is just the beginning. If you want to learn more about working with vector databases, consider:

Gain insight into advanced concepts such as using vector metadata, sharding, compression for more flexible and efficient data storage and retrieval
Try more sophisticated approaches to integrating vector embeddings into AI applications for richer, more nuanced results

Thanks for sticking it out to the end, hope it was a good use of your time.

Are you adding new JS libraries to improve performance or build new functionality? What if they did the opposite?

There is no doubt that the front end is becoming more and more complex. As you add new JavaScript libraries and other dependencies to your app, you'll need greater visibility to ensure your users don't run into unknown issues.

LogRocket is a front-end application monitoring solution that lets you replay JavaScript errors as if they occurred in your own browser, so you can react to them more efficiently.

LogRocket works flawlessly with any application, regardless of framework, and has plugins for logging from Redux, Vuex, and other contexts from @ngrx/store. Instead of guessing why a problem occurred, you can summarize and report on the state of your application when the problem occurred. LogRocket also monitors application performance, reporting metrics such as client CPU load, client memory usage, and more.

Original link: How to implement AI's vector database (mvrlink.com)