With a vector database, do we still need a SQL database?

"Do I need a regular SQL database in addition to a vector database?"

This is a question we are often asked. If users have other scalar data information in addition to vector data, their business may need to filter the data based on certain conditions before performing semantic similarity searches , for example:

In the legal field, it may only be necessary to search for relevant legal terms from a specific database;
In retail, you may need to search for men's shoes in a certain size;
When searching for images, you may want to search for posters for movies released between 2010-2016 that have IMDB movie ratings higher than 7.0.

To this, our answer is – no. With the vector database Milvus or the fully managed Milvus service - Zilliz Cloud, there is no need to maintain an additional SQL database to store scalars. With just one system, users can send hybrid queries that implement "vector search + scalar filtering" to obtain more accurate search results.

Among them, Milvus allows users to perform conditional filtering based on scalar data when performing vector searches. The data attributes can be any field except vectors. Milvus will create a vector index for the vector field and conduct a vector similarity search. At the same time, it can also perform metadata filtering on the search results through expressions. Just enter a filter expression when searching and Milvus will do both automatically for you.

This tutorial uses Zilliz Cloud Pipelines - Zilliz Cloud's built-in functionality for encoding unstructured data into Embedding vectors, while supporting direct search of vectors with text and filter expressions. We will demonstrate how to use scalar filtering to recall only document fragments that match certain criteria, such as a specific source URL, or a specific file name. You can also use similar ideas to recall documents with specific tags, such as publication year, version number, etc.

01. Create Collection and Pipelines

This tutorial requires the free version of Zilliz Cloud (overseas version). Zilliz Cloud is a fully managed Milvus service that deploys the user's database on a Serverless cloud server, but we can still use the Zilliz Cloud vector database locally by calling the PyMiluvs API interface. The following text used for testing is from the PyMilvus documentation

Open https://cloud.zilliz.com/ and create a "Starter" version cluster.

Add the Collection name and click "Create Collection and Cluster".

By default, 1 Collection is created when creating a Zilliz Cloud cluster, which will not be used in this tutorial. When we create Zilliz Cloud Pipelines later, another Collection will be automatically created. Note that the two Collections are not the same.

Click Pipelines in the left navigation bar, follow the interface prompts to create Pipelines and upload data:

a. Please choose to create an "Ingestion Pipeline" first.

b. Select the Serverless cluster you just created, enter the Collection and Pipeline names respectively, and click "Add function".

c. Select INDEX_DOC function, enter the function name, leave other parameter values as default, and click "Add". This function will slice the document into vectors.

d. (Optional) Click "Add function" again.

e. (Optional) Select the PRESERVE function, name it, and click "Add". This function is used to save the tag information of the document.

Click "Create Ingestion Pipeline". Now, we have finished creating the Ingestion Pipeline and Collection.

Click "Create Deletion and Search Pipeline".
Enter the Pipelines list page and click the button "▶️" to run the Ingestion Pipeline.

Ingestion Pipeline supports uploading your files on object storage (such as AWS S3 and Google Cloud Storage). In this example we upload the data to AWS S3. Once the upload is complete, click Share via Pre-signed-URL. Copy the sharing link (Pre-signed URL). If there is no object storage, you can use the test file link https://publicdataset.zillizcloud.com/milvus_doc.md provided by us as the Pre-signed URL.

Paste the Pre-signed URL in the code and click Run. This step will fragment the file to extract vectors and import them into the vector database Collection.

Enter the collection page and check whether the Collection and Schema are correct. At this point the vectors for the document fragments should already be displayed in the Data Preview.

Afterwards, the data can be queried on the Playground interface or by calling the API.

02. Use scalar filtering to recall vectors matching specific labels

Find "Search Pipeline" in the Pipeline list and click the button "▶️" on the right to run the Search Pipeline.
In the request, enter a question and click Run.

Edit "Filter". Please use Boolean expressions. After clicking Run, you can see that Zilliz Cloud has filtered the search results based on the conditions you entered.

Metadata filtering with Zilliz Cloud Pipelines is that easy! You can conditionally filter on all scalar fields except vector fields using Boolean expressions .

03. Search through API interface

Similarly, we can also search by calling the API interface. When using the API, the user needs to provide the following two points:

Zilliz API Token
Pipeline ID

We can obtain the API Token through the cluster details page.

To obtain the Pipeline ID, first find the Search Pipeline on the Pipelines list page, and then copy the Pipeline ID in the Pipeline ID column. Paste the Pipeline ID into the URL when calling the API interface.


import requests, json
url = "https://controller.api.gcp-us-west1.zillizcloud.com/v1/pipelines/pipe-xxxx/run"
headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {TOKEN}",
}
data = {
    "data": {
        "query_text": SAMPLE_QUESTION
    },
    "params": {
        "limit": TOP_K,
        "offset": 0,
        # Any of these fields can be used in filter expression.
        "outputFields": ["chunk_text", "chunk_id", "doc_name", "source"],
        "filter": "doc_name == 'param.html'"
    }
}

# Send the POST request
response = requests.post(url, headers=headers, json=data)

Metadata filtering searches using the API are that easy! If you are interested in learning more about how to use Zilliz Cloud Pipelines, you can refer to the Notebook to use Pipelines to build a RAG Q&A robot with tag filtering function. You are welcome to try it.