Elasticsearch: Understanding TLS Log Errors Using ESRE and Generative AI

By DAVID HOPE

This blog presents a novel application of the Elasticsearch Relation Engine (ESRE​​) and its Elastic Learned Sparse Encoder capabilities, particularly in log analysis.

The recently released Elasticsearch Relevance Engine™ (ESRE™) includes a set of key features that enhance search capabilities and allow Elasticsearch® to be queried using the same natural language that is used to ask generative AI questions.

This makes me wonder how we can use this for logging. It's fairly easy to get started with Elastic Learned Sparse Encoder models , an important ESRE feature that supports semantic search - we'll show how to use it for logging in this blog. This feature is very different from what Elasticsearch® currently does because it understands the meaning of words and searches extensively for words that have a similar meaning or are synonymous with the word I searched for.

The Elastic Learned Sparse Encoder is also an efficient search ranking solution, producing better results than traditional BM25 and vector-based KNN searches in out-of-domain spaces (i.e., when the model is not trained strictly on user data). Late interaction models improve search accuracy by encoding raw text to provide more contextual information when queried.

How do Elastic Learned sparse encoders work?

When an Elastic Learned Sparse Encoder is applied to raw text, such as a log message, it produces a data structure. The keys of this structure represent the terms and their synonyms found in the original text. In a process called term expansion (text expansion), the model adds and removes terms and synonyms from the data from a static vocabulary of 30K fixed terms based on their relevance to the original text . This is similar to vector embeddings, creating an auxiliary data structure (or structures) and storing it in a field, which can then be used for instant semantic matching in queries.

Each term also has an associated score, which captures its contextual importance in the original text. We refer to this data structure as the (scored) bag of words, or BOW.

BOW example

Note how the words of the input text are captured, stemmed, synonyms are added, and finally scored. Note also that the "Scored Bag of Words" structure is just a map of [word => Score]. You'll see later how this is generated for our syslog.

At query time, the input query string goes through the same reasoning process behind the scenes that we do on the raw log text at ingest time to generate the BOW. This BOW is then matched against the BOW of the original log text in the index, similar to a k-nearest neighbor search with embedding vectors . The final ranking score for search results is the result of a formula that takes into account the scores of matching words in the query BOW and the original log text.

Let's take an example of how useful this is, if I'm searching for "Are there any TLS warnings?". In my syslog, the following log line " 2023/05/24 17:10:33: Error getting config status, workload certificates may not be configured: HTTP 404 " matches, although there is no exact match in that particular log line. ESRE understands TLS in relation to certificates.

A word of caution here: this search is a "fuzzy" search and may not get you the results you're looking for. You can even use a better model for this purpose, such as one trained on your own data. It's important to remember that this is just another tool to help point us in the right direction when we need to resolve ambiguity. Being able to talk about the problem in general during the search without having to be very specific about the errors we see helps us with root cause analysis.

You can get it up and running, and start using it to search logs with just a few clicks.

This can change the way you approach problem detection and repair, as ESRE may return logs that you may not have thought to analyze before but are relevant to the problem you are trying to solve. This is a win for reducing the MTTx indicator. You can then use the documents returned from Elastic and the same search terms to ask questions to a generative AI engine like ChatGPT, which we explore at the end and in other use cases on this blog .

At the end of this blog, you'll have a high-level flow that looks like this:

setup steps

In the next few steps, I'll walk through:

• Get an account on Elastic Cloud (must be version 8.8 and above) and install Elastic Agent
• Set up ESRE in this environment and connect logs to ESRE
• Search logs through ESRE using Kibana®
• Use ChatGPT to quickly create applications that work with Elasticsearch and ESRE

Step 0: Create an account on Elastic Cloud

A few notes before proceeding:

1) Make sure to click " Advanced Options " (shown below) when creating your Elastic Cloud instance and add a machine learning node with at least 4GB RAM for this blog.

2) When you follow the instructions below, be sure to follow the "Add System" integration steps.

Keeping these considerations in mind, follow the instructions to get started with Elastic Cloud.

Keeping these considerations in mind, follow the instructions to get started with Elastic Cloud .

Step 1: Install the Elastic Learned Sparse Encoder model

These steps will deploy the Elastic Learned Sparse Encoder for use with syslog. The documentation page is here. Chinese guides are available here.

Elastic Learned Sparse Encoder is a retrieval model trained by Elastic that enables you to perform semantic search to retrieve more relevant search results. This search type gives you search results based on contextual meaning and user intent, rather than exact keyword matches.

Go to Analytics -> Machine Learning:

Click on Model Management -> Trained Models:

Select Elastic Learned Sparse EncodeR v1 and click the download button on the far right.

Once downloaded, we need to launch it.

For the purposes of this blog, the default settings are fine.

Step 2: Modify the index template

Next, we need to add a few fields to the incoming document for the Elastic Learned Sparse Encoder to use for tokens and searches. We can do this by updating the index template for the log type we want to use for the model.

In this section, we'll add a token field and a text_field — both of which are required for use with the Elastic Learned Sparse Encoder model. When running inference, the model will look for log messages in the text_field field. The inference step then converts the log message text into a list of token-weight pairs, which is stored in the tokens field - this is why we need these mappings.

Index templates allow us to define the format and mapping of documents in an index, in this case a syslog index.

Go to "Stack Management".

Next, go to Index Management -> Index Templates. We will modify the "syslog" index as follows:

Click on the index name, choose Manage in the lower right corner, and then choose Edit.

Skip to Step 4: "Mappings".

And add the following mappings: ml.tokens and text_field.

Click Next and save.

Step 3: Add a custom ingest pipeline

Next, we'll add a custom ingestion pipeline so that the Elastic Learned Sparse Encoder can add tokens to documents sent to Elastic. This is required for semantic search -- please dig into this . The cool thing is that we can see the token in the documentation, which gives you an intuition of how it works.

The pipeline step first copies the log message into the text_field field, since this is where the model looks for the data it needs. Through an inference step, it converts the log message text into a list of token-weight pairs, which is stored in the ml.tokens field.

Note that if your log ingestion on this index is very busy, inference will add a little bit of processing time - this may not be a good idea if you have a very heavy load, but for our small use case here, this should work good.

First, go to the Ingest pipeline:

Create a new ingest pipeline named "logs-system.syslog@custom". This is a special name called from the main syslog ingestion pipeline and exists in all "Managed" pipelines so you can add custom ingestion steps .

Now let's add two processors to this ingestion pipeline:

1) The first processor will set the text_field with the log message field containing the log data we want to search using the Set processor and Copy from operation as shown below.

2) The second processor added is the "inference" processor, which will generate the tokens required for semantic search. Set it up as follows.

Field mapping:

{
 "text": "text_field"
}

Inference configuration:

{
 "text_expansion": {
   "results_field": "tokens"
 }
}

Once done, click " Create pipeline ".

Step 4: Scroll Index

To force an index rollover and use the new index template we created, go to Dev Tools and do the following POST:

POST /logs-system.syslog-default/_rollover/

Step 5: Welcome to Semantic Search with Logs

Now, go to discovery and create a new data view that only shows syslog:

You might have to wait a bit for some data to populate here. Once there is data, you should see the ML Tokens field populated as follows:

This is the magic of Elastic Sparse Learned Encoder. The token you can see here enhances how it can understand the semantics of a log line and enables you to find logs with a similar meaning to the search parameter.

Now we can run semantic search. Change the language box below from KQL to Lucene.

Enter the following in the search box. Here, text augmentation queries use a natural language processing model to convert the query text into a list of token weight pairs, which are then used in queries against the rank features field we defined above.

{"text_expansion":{"ml.tokens":{"model_id":".elser_model_1","model_text":"Are there any TLS warnings?"}}}

Congratulations! You just did semantic search with Elastic.

Now here's one more thing to do: we need to sort the results so we see the top search results. Add the _score field as shown below.

There is also a "message" field.

Now we can sort them from highest to lowest, which tells us what is a strong match (similar to how search results are ranked on Google):

Now let's see if we can connect it to ChatGPT!

Step 6: Connect it to ChatGPT

Now let ChatGPT write some code for us. We asked ChatGPT:

“Can you write some code to query Elasticsearch and then send the documents that are returned to ChatGPT?”

It writes the following code:

from elasticsearch import Elasticsearch
from openai import OpenAI, ChatCompletion

# Initialize Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])

# Search for documents in Elasticsearch
body = {
    "query": {
        "match_all": {}
    }
}

res = es.search(index='your_index_name', body=body)

# Initialize OpenAI
openai = OpenAI(api_key='your_openai_api_key')

# Send documents to ChatGPT
for hit in res['hits']['hits']:
    message = {
        "role": "system",
        "content": f"Document : {hit['_source']}"
    }

    # Generate response from ChatGPT
    response = openai.ChatCompletion.create(
      model="gpt-4.0-turbo",
      messages=[message]
    )

    # Print the model's response
    print(response['choices'][0]['message']['content'])

Now this is easily modified so we can use the new Elasticsearch "Semantic query" as shown below.

Change the following lines:

# Search for documents in Elasticsearch
body = {
    "query": {
        "match_all": {}
    }
}

That's all for now. Note that this will ensure that we only get the top three results and thus the most relevant results:

body = {
  "query": {
    "text_expansion": {
      "ml.tokens": {
        "model_id": ".elser_model_1",
        "model_text": "Are there any TLS warnings?"
      }
    }
  },
  "size": 3,
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ]
}

and change this line:

"content": f"Document : {hit['_source']}"

becomes like this:

 "content": f”Are there any TLS warnings? Log line : {hit['_source']}"

This allows us to include Elasticsearch results in ChatGPT queries, making hallucinations much less likely and more likely to give us a relevant diagnosis. Check out the final result here.

ChatGPT responds:

The error message you're seeing seems to suggest that there might be an issue related to Transport Layer Security (TLS) certificates. The HTTP 404 error indicates that the requested resource cannot be found.

• Incorrect or incomplete configuration: The error suggests that workload certificates may not be configured. If your service requires TLS certificates to communicate securely, but these have not been properly set up, this could cause the issue. You should review your configuration to make sure TLS is correctly set up
• Wrong path: The system might not be able to find the certificates because they're not located at the specified path in your configuration. You need to ensure that the path to the certificate files is correct.
• Permission issues: There could be issues with file or directory permissions that are preventing your system from accessing the certificate files. You should ensure that the relevant user or service has the necessary permissions to access these files.
• Server or service is not available: The HTTP 404 error could indicate that the server or service you're trying to reach is not currently available or doesn't exist. You need to check if the server is running and that the endpoint is correct. Now if you didn’t want to do this yourself, I have created an application that does all of this over here, in this GitHub repository. When you run it, you get this nice graphical user interface:

So, with everything wired up, the architecture of our solution ends up looking like this:

Now that you understand how to set up an Elastic Learned Sparse Encoder model, I challenge readers to figure out what other uses they can use the model for, and what logging use cases they can come up with. You can also extend this example to cache and store ChatGPT responses in Elastic, like this blog , enabling you to start a knowledge base. You can augment this with your own knowledge over time.

Superior out-of-the-box relevance for SRE

In this blog, we guide you through the process of setting up an Elastic Learned Sparse Encoder for log analysis and challenge our readers to explore new uses for the model. A simple example might involve probing for any TLS-related issues, as shown in this blog.

With the launch of the Elasticsearch relevance engine, we're bringing the unique Elastic retrieval model into your hands. This ready-to-use model can be easily integrated with our full range of data ingestion methods, including the ingestion pipeline shown in this blog. Out of the box, anyone can use it in its searchable corpus, and it's compact enough to fit within the confines of a laptop's memory.

Elastic's learned sparse encoders provide cross-domain semantic search for search use cases (including logs), as demonstrated in this blog, delivering highly relevant search results without custom tuning or training.

Field tests consistently show that hybrid ranking techniques produce the most relevant sets of search results. However, one key factor - RRF - has been absent. We are now integrating RRF for your application search needs, enabling you to blend vector and text search capabilities.

Learn more about Elasticsearch and the possibilities of generative AI .

In this blog post, we may have used third-party generative artificial intelligence tools that are owned and operated by their respective owners. Elastic has no control over third-party tools, and we are not responsible for their content, operation, or use, nor shall we be liable for any loss or damage that may arise from your use of such tools. Exercise caution when using artificial intelligence tools with personal, sensitive or confidential information. Any data you submit may be used for artificial intelligence training or other purposes. There can be no guarantee that information you provide will be secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative artificial intelligence tool before using it.

Elastic, Elasticsearch and related marks are trademarks, logos or registered trademarks of Elasticsearch NV. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.