Elasticsearch: Semantic search using Elasticsearch

In the digital age, search engines play an important role in retrieving data by browsing the vast amount of information available on the internet. This method involves users entering specific terms or phrases into the search bar, expecting the search engine to return results that match those exact keywords.

While keyword searching is valuable for streamlining information retrieval, it has its limitations. One of the main drawbacks is its reliance on lexical matching. Keyword searches treat each word in the query as a separate entity, often resulting in results that may not exactly match the user's intent. Additionally, ambiguous queries may yield different interpretations, leading to mixed or inaccurate results.

Another key limitation arises when dealing with languages in which context strongly affects meaning. The meaning of words depends largely on the context. Using keywords alone may not capture these queries correctly, which may lead to misunderstandings.

As our digital environment continues to evolve, so do our expectations for more refined and intuitive search experiences. This paved the way for the emergence of semantic search, an approach that aims to transcend the limitations of traditional keyword-based approaches. By focusing on the intent and contextual meaning of search queries, semantic search offers a promising solution to the challenges posed by keyword searches.

As shown in the picture above, if we search by keyword, we want to search for apple fruit, but we may end up getting relevant information about apple (Apple) company. Actually it's not what we want.

What is semantic search?

Semantic search is an advanced way of searching for content on the internet. It's not just about matching words, it's about understanding what you're really looking for. It finds the meaning behind your words and how they relate to each other.

This technology uses technologies such as artificial intelligence and understanding human language. Almost like it says something about humans! It looks at the big picture, checking for words with similar meanings and other ideas related to the question you're asking.

Basically, semantic search helps you get exactly what you need from the vast amount of content on the internet. It's like talking to a super-smart search engine that not only gets the words you say, but also what you're actually looking for. This makes it perfect for doing research, finding information, and even getting recommendations that match your interests.

Benefits of Semantic Search

Precision and relevance : Semantic search delivers highly relevant results by understanding user intent and context.
Natural Language Understanding : It understands complex queries and makes natural language interactions more effective.
Disambiguation : It resolves ambiguous queries to provide accurate results based on user behavior and context.
Personalization : Semantic search learns from user behavior to get customized results that increase relevance over time.

Semantic search in Elastic Search

Elastic Search offers semantic search that focuses on the meaning and context of search queries rather than just matching keywords. It uses natural language processing (NLP) and vector search to achieve this goal. Elastic has its own pre-trained representation model called Elastic Learned Sparse EncodeR (ELSER) .

Before getting into ELSER, let's learn more about NLP and vector search.

Natural Language Processing (NLP)

Natural language processing is a branch of artificial intelligence that focuses on enabling computers to understand, interpret, and generate human language in valuable and useful ways.

NLP involves a set of techniques and algorithms that allow computers to process and analyze large amounts of natural language data. This includes the following tasks:

Text understanding : NLP helps computers understand the content of an article. It can find important content in a text, such as names, relationships, and feelings.
Text processing : This involves tasks such as breaking sentences into words or phrases, reducing words to their basic forms, and identifying different parts of sentences.
Named Entity Recognition (NER) : NLP can identify special things in text, such as the names of people, places, or organizations. This helps to understand what is being discussed.

vector search

Vector search is a technique that involves representing data points or information as vectors in a multidimensional space. Each dimension of the space represents a different characteristic or attribute of the document or data point.

In this vector space, similar documents or data points are closer to each other. This allows efficient similarity-based searches. For example, if you are searching for documents that are similar to a given document, you can calculate the similarity between the vectors representing the documents to find the closest match.

Vector search is used in a wide variety of applications, including:

Recommendation System : It helps in recommending similar items to users based on their preferences.
Information retrieval : It allows finding similar documents in large corpora.
Anomaly Detection : It helps in identifying anomalies or abnormal data points.

How NLP and vector search work

1) Vector embedding:
In this step, NLP involves converting text data into numerical vectors. Convert each word in the text into a high-dimensional vector using techniques like word embeddings

2. Similarity score:

The engine compares the vectorized query to the vectorized document to determine their similarity.

3) Artificial neural network algorithm:

The approximate nearest neighbor (ANN) algorithm can effectively find approximate nearest neighbors in high-dimensional space.

4) Query processing:

A user's query undergoes similar processing to a document to produce a vector representation.

5) Distance calculation:

The engine calculates the distance (similarity score) between the vectorized query and the document.

6) Nearest neighbor search:

The engine looks for documents whose embeddings are closest to the query embeddings.

7) Ranking results:

Results are ranked based on similarity scores.

ELSER

ELSER is a pre-trained model specifically designed to excel at understanding context and intent without the need for complex fine-tuning. Currently only available for English, ELSER's out-of-the-box adaptability makes it an invaluable tool for a variety of natural language processing tasks. Its utilization of sparse vector representation improves the efficiency of processing text data. ELSER's vocabulary contains approximately 30,000 terms and optimizes queries by replacing terms with their contextual counterparts, ensuring precise and comprehensive search results.

Let's take a deeper look at how to leverage the potential of ELSER to enhance search capabilities in Elasticsearch. You can refer to the article " Elasticsearch: Deploying ELSER - Elastic Learned Sparse EncoderR " to configure your own ELSER.

Step 1: Create an index with the required mappings

In Elasticsearch, an "index" is a collection of documents that share common characteristics or belong to similar categories. It is similar to a table in a relational database or a type in some other NoSQL database. Each document in the index is assigned a unique identifier and contains structured data in JSON format.
Define a map of indexes that will contain the tokens generated by the model based on your input. The index must have a field of type rank_features to index the ELSER output.

PUT <index-name>
{
  "mappings": {
    "properties": {
      "ml.tokens": { 
        "type": "rank_features" 
      },
      "name": { 
        "type": "text" 
      }
    }
  }
}

Step 2: Create an ingestion pipeline using an inference processor

The ingestion pipeline in Elasticsearch enables you to apply various transformations to your data before indexing. These transformations include tasks such as field removal, text value extraction, and data enrichment.
A pipeline consists of a customizable set of tasks called processors. These processors operate in a sequential manner, implementing specific modifications to the incoming document. Use an inference processor to create an ingestion pipeline to use ELSER to perform inference on the data being ingested.

PUT _ingest/pipeline/<pipeline-name>
{
  "processors": [
    {
      "inference": {
        "model_id": ".elser_model_1",
        "target_field": "ml",
        "field_map": { 
          "text": "text_field"
        },
        "inference_config": {
          "text_expansion": { 
            "results_field": "tokens"
          }
        }
      }
    }
  ]
}

Step 3: Add data to the index

The index mapping and ingest pipeline are set up and now we can start adding data to the index.
The ingestion pipeline acts on the incoming data and adds relevant markup to the document

curl -X POST 'https://<url>/<index-name>/_doc?pipeline=<ingest-pipeline-name' 
  -H 'Content-Type: application/json' 
  -H 'Authorization: ApiKey <Replace_with_created_API_key>' 
  -d '{
  "name" : "How to Adapt Crucial Conversations to Global Audiences"
}'

The ingestion pipeline acts on the incoming data and adds the relevant tokens to the document:

{
 "name" : "How to Adapt Crucial Conversations to Global Audiences",
 "ml":{
    "tokens": {
        "voice": 0.057680283,
        "education": 0.18481751,
        "questions": 0.4389099,
        "adaptation": 0.6029656,
        "language": 0.4136539,
        "quest": 0.082507774,
        "presentation": 0.035054933,
        "context": 0.2709603,
        "talk": 0.17953876,
        "communication": 1.0619682,
        "international": 0.38651025,
        "different": 0.25769454,
        "conversation": 1.03593,
        "train": 0.021380302,
        "audience": 0.97641367,
        "development": 0.33928272,
        "adapt": 0.90020984,
        "certification": 0.45675382,
        "cultural": 0.63132435,
        "distraction": 0.38943478,
        "success": 0.09179027,
        "cultures": 0.82463825,
        "essay": 0.2730616,
        "institute": 0.21582486,
       },
       "model_id":".elser_model_1"
    }
}

Step 4: Perform a semantic search

Use text expansion queries to perform semantic searches. Provide query text and ELSER model ID.
Text expansion queries use a natural language processing model to convert the query text into a list of token weight pairs, which are then used in queries against the rank_features field.

GET <index-name>/_search
{
   "query":{
      "text_expansion":{
         "ml.tokens":{
            "model_id":".elser_model_1",
            "model_text":<query_text>
         }
      }
   }
}

Step 5: Combine semantic search with other queries

We can also combine text_expansion with other queries in a compound query to get more granular results.

GET my-index/_search
{
  "query": {
    "bool": { 
      "should": [
        {
          "text_expansion": {
            "ml.tokens": {
              "model_text": <query_text>,
              "model_id": ".elser_model_1",
            }
          }
        },
        {
          "query_string": {
            "query": <query_text>,
          }
        }
      ]
    }
  }
}

We can also combine text_expansion with other queries in a compound query to get more granular results.
Text_expansion queries generally produce higher scores compared to other queries in Elasticsearch. We can adjust the relevance score using the boost parameter.