Let's talk about the online problem of changing Elasticsearch space for time...

1. Online practical problems

I would like to ask my friends, does the slop of ngram affect the search results?

1. Preconditions:

The SPUCodeText of product A is: OWBB050C99JER0021001
The SPUCodeText of commodity B is: VSA00293ABBLACKFW2022
The SPUCodeText of commodity C is: 2WHGG0VNT03HHFC99FW2022

2. Current situation: search for the SPUCodeText code of product A: OWBB050, the product cannot be queried if the slop is set to 49-54; the product A can only be queried if the slop is set to 55 or above;

3. Pursue the goal: search SPUCodeText for any combination of 4 numbers and above, and you can find out the product?

For reasons of space, DSL definitions and query statements are omitted.

——The source of the topic: Dead Elasticsearch Knowledge Planet https://t.zsxq.com/08rmVBnhA

2. Interpretation of the problem

Major premise: The storage of the product code is similar to the storage of the mobile phone number we mentioned in the previous video, and the traditional word segmentation (default standard, Chinese ik_max_word, etc.) cannot handle it.

It needs to be implemented with the help of Ngram custom word segmentation.

Then the problem comes: the data after Ngram word segmentation, there is a problem with match_phrase + slop retrieval, and the slop must be set to a large value to get it done!

What causes it? Is there a more concise way?

3. Elasticsearch space for time

What is space for time? Take the example of the current World Cup to understand.

As the commentator said: "15 people can win". 15 people are far more than the normal 11 people by 4 people, which means that there is more space in exchange for time or results. Of course, the facts of the game are far from what the commentator said.

The essence of Ngram word segmentation in Elasticsearch is the way of exchanging space for time. Segmenting documents at a very small granularity will cause a surge in space storage and affect the writing speed, but in exchange for improved retrieval efficiency!

4. Realization after streamlining the problem

PUT /products-001
{
  "settings": {
     "max_ngram_diff": 40,
      "analysis": {
        "analyzer": {
          "ruishan_ngram_analyzer": {
            "filter": [
              "lowercase"
            ],
            "type": "custom",
            "tokenizer": "ruishan_ngram_tokenizer"
          }
        },
        "tokenizer": {
          "ruishan_ngram_tokenizer": {
            "token_chars": [
              "letter",
              "digit"
            ],
            "min_gram": 3,
            "type": "ngram",
            "max_gram": 40
          }
        }
      }
    },
  "mappings": {
     "properties" : {
        "id" : {
          "type" : "keyword"
        },
        "sPUCodeText" : {
          "type" : "text",
          "analyzer" : "ruishan_ngram_analyzer"
        }
      }
  }
}

PUT products-001/_bulk
{"index":{"_id":1}}
{"id":1,"sPUCodeText":"OWBB050C99JER0021001"}
{"index":{"_id":2}}
{"id":2,"sPUCodeText":"VSA00293ABBLACKFW2022"}
{"index":{"_id":3}}
{"id":3,"sPUCodeText":"2WHGG0VNT03HHFC99FW2022"}

GET products-001/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "sPUCodeText": {
              "query": "OWBB050"
            }
          }
        }
      ]
    }
  }
}

See the following results, a match can be done!

Next, can match_phrase be used?

What about match_phrase plus a larger slop? Can it be done? !

After repeated tests, it is necessary to set the slop at least 52 to get it done, as shown below.

why? Why is it 52 ?

5. What is the essence of match_phrase phrase matching retrieval?

In layman's terms: the word segmentation results of the query part to be retrieved (such as the beginning: OWBB050) must be exactly the same as the order and position of the word segmentation results in the document (such as: OWBB050C99JER0021001)!

You can view word segmentation results through the analyzer api, as follows:

POST products-001/_analyze
{
  "field": "sPUCodeText",
  "text": ["OWBB050C99JER0021001"]
}

The term unit after word segmentation, "OWBB050" is shown on the left side of the figure below, and "OWBB050C99JER0021001" is shown on the right side of the figure below.

The two are not consistent , which is the reason why they cannot be matched, that is, there is a deviation!

6. The essence of parameter slop under match_phrase phrase retrieval

A picture completely understands!

The same color represents: the word to be retrieved is the same as the word segmentation result in the source document.

The calculation method of the maximum difference, for example: the term "050" after word segmentation ranks 15 in the term to be searched, and ranks 67 in the source document "OWBB050C99JER0021001".

Bad: 67-15=52.

Therefore, slop fills up the maximum difference of 52, and the retrieval and data recall can be realized!

If the slop is set to 51, it will not work! At least 52 or more can recall data.

7. Summary

After word segmentation similar to Ngram, we have worked hard on the spatial level! There is no need to work hard at the time level and retrieval level!

A direct match search will surely retrieve the result!

The above-mentioned filter can be cached, and it is recommended to use it.

So, is there a faster way to write it?

Careful students will find that only when "OWBB05" is changed to lowercase "owbb05" can the data be recalled, but direct capitalization and direct term retrieval cannot recall the data!

why? Leave a message for everyone to think about!