1. Background
Since 2020, the search for content annotation results has been one of the core high-frequency usage scenarios of the back-end business in the community. In order to support complex back-end search, we have stored an additional copy of the key information of the community content in Elasticsearch for use as a secondary index. With the subdivision, iteration, and passage of time in the annotation business, the number of documents indexed and the RT of the search began to gradually increase. The following is the current monitoring status of this index.
This article introduces how the community uses IndexSorting to optimize the search performance of billion-level documents from the initial 2000ms to 50ms . If you encounter similar problems and scenarios, I believe that after reading it, you will be able to earn tons of income from one line of code.
2. Exploration process
2.1 Initial optimization
The initial requirement is very simple, just fetch the latest dynamic paging display . At this time, the implementation is also simple and rude, as long as the function is satisfied. The query statement is as follows:
{
"track_total_hits": true,
"sort": [
{
"publish_time": {
"order": "desc"
}
}
],
"size": 10
}
Since no filtering conditions were added when the home page was loaded, it became to find the latest 10 published content from the billion-level content library .
For this query, it is easy to find that the problem occurs in the sorting of large result sets. To solve the problem, two paths are naturally thought of:
- remove sort
- Narrow the result set
After weighing user demands and development costs, it was decided to "hold it first, then optimize": when the user opens the homepage, the filter condition of "published within the last week" is added by default. At this time, the statement becomes:
{
"track_total_hits": true,
"query": {
"bool": {
"filter": [
{
"range": {
"publish_time": {
"gte": 1678550400,
"lt": 1679155200
}
}
}
]
}
},
"sort": [
{
"publish_time": {
"order": "desc"
}
}
],
"size": 10
}
After this change was launched, the effect can be said to be immediate. The loading speed of the homepage immediately dropped to less than 200ms, with an average RT60ms. This change also reduced the pressure from the business for us, and bought a lot of research time for subsequent optimization.
Although the loading speed of the search home page is obviously faster, it does not actually solve the fundamental problem - the sorting of specified fields in ES large result sets is still very slow . For business, the experience of some boundary functions on the result page is still not satisfactory, such as export, full dynamic search, etc. This point can also be clearly seen from the monitoring: slow queries still exist, and are accompanied by a small number of interface timeouts.
To be honest, our understanding of ES in this period is still relatively basic. We can only say that we can use it, know about sharding, inverted index, and correlation scoring, and then it is gone. In short, we have a direction and start to catch up.
2.2 Fine grinding
2.2.1 Knowledge accumulation
With the problems left over from before, we started to start again and learn ES from scratch. To optimize search performance, the first thing we need to know is how search works. Let's take the simplest search as an example to disassemble the entire search request process.
(1) Search request
{
"track_total_hits":false,
"query": {
"bool": {
"filter": [
{
"term": {
"category_id.keyword": "xxxxxxxx"
}
}
]
}
},
"size": 10
}
Accurately query the documents whose category_id is "xxxxxxxx", fetch 10 pieces of data, no sorting or total number is required
The overall process is divided into 3 steps:
- The client initiates a request to Node1
- As the coordinating node, Node1 forwards the request to each primary or secondary shard of the index, and each shard executes the query locally.
- Each node returns its own data, and the coordinating node summarizes and returns to the client
This process can be roughly depicted in the figure:
We know that ES relies on the capabilities provided by Lucene. The real search occurs in Lucene, and we need to continue to understand the search process in Lucene.
（2）Lucene
Lucene contains four basic data types, namely:
- Index: Index, composed of many Documents.
- Document: Composed of many Fields, it is the smallest unit of Index and Search.
- Field: It consists of many terms, including Field Name and Field Value.
- Term: Consists of many bytes. Generally, each smallest unit after the word segmentation of the Field Value of the Text type is called Term.
Before introducing the search process of the Lucene index, let me talk about the smallest data storage unit that makes up the Lucene index——Segment.
The Lucene index is composed of many segments, and each segment contains the term dictionary of the document, the inverted list of the term dictionary, the columnar storage DocValues of the document, and the forward index. It can independently and directly provide search function externally, which is almost a shrunken version of Lucene index.
(3) Term dictionary and posting list
The picture above shows the general appearance of the Term dictionary and its inverted list. Of course, there are some important data structures here, such as:
- FST: term index, built in memory. Can quickly realize single term, term range, term prefix and wildcard query.
- BKD-Tree: for fast lookup of numeric types (including spatial points).
- SkipList: the data structure of the inverted list
There are many details here, and those who are interested can learn about it separately. This does not affect our overall search process, but I will repeat it here. With the Term dictionary and inverted list, we can directly get the result set that matches the search criteria. Next, we only need to retrieve the entire doc from the forward index through the docID and return it. This is because the basic disk of ES will not be slow in theory. We guess that the slow query occurs in sorting. So what happens when you add a sort to the request? for example:
{
"track_total_hits":false,
"query": {
"bool": {
"filter": [
{
"term": {
"category_id.keyword": "xxxxxxxx"
}
}
]
}
},
"sort": [
{
"publish_time": {
"order": "desc"
}
}
],
"size": 10
}
The docIds obtained through the inverted list are out of order. Now that the sorting field is specified, the easiest and direct way is to extract all of them, and then sort the top 10. This can certainly achieve the effect, but the efficiency is conceivable. So how does Lucene solve it?
（4）DocValues
Inverted index can solve the fast mapping from word to document, but it needs fast mapping from document number to value when aggregation operations such as classification, sorting and mathematical calculation of retrieval results are required. And the positive index is too bloated, what should I do?
At this time, you guys may directly think of columnar storage. There is nothing wrong with it. Lucene introduced a columnar storage structure based on docId——DocValues
| Document number | column value | column value mapping |
|---|---|---|
| 0 | 2023-01-13 | 2 |
| 1 | 2023-01-12 | 1 |
| 2 | 2023-03-13 | 3 |
For example, DocValues=[2023-01-13, 2023-01-12,2023-03-13] in the above table
If the column value is a string, Lucene will sort the original string value according to the dictionary to generate a digital ID. Such preprocessing can further speed up the sorting speed. So we got DocValues=[2, 1, 3]
The columnar storage form of Docvalues can speed up our traversal. At this point, a regular search request to fetch the first N records can be regarded as the real dismantling completion. The analysis of word frequency, relevance scoring, aggregation and other functions is not discussed here, so this article greatly simplifies the whole process and data structure. If you are interested in this part, welcome to discuss together.
At this time, the problem of slow sorting has gradually surfaced: Although Docvalues is columnar storage and preprocesses complex values into simple values to avoid complex comparisons during queries, it still cannot hold back the large dataset we need to sort. .
It seems that ES is trying its best, and it seems that it is really not good at solving the slow query problem in our scene.
However, spiritual readers must have thought that if the inverted list can be stored in the order we pre-specified, the entire sorting time can be saved .
2.2.2 Index Sorting
Soon the ES official document "How to tune for search speed" mentioned a search optimization method - Index Sorting (Index Sorting) appeared in our field of vision.
From the description in the document, we can know that index sorting improves search performance mainly in two aspects:
- For multi-condition parallel queries (a and b and ...), index sorting can help us store unqualified documents together and skip a large number of unmatched documents. But this trick only works for low-cardinality fields that are often used for filtering.
- Early break: When the order specified by the search sort and the index sort is the same, only the first N documents of each segment need to be compared, and the other documents only need to be used for the total calculation. For example: there is a timestamp in our document, and we often need to search and sort according to the timestamp. At this time, if the specified index sort is consistent with the search sort, the efficiency of search and sort can usually be greatly improved.
Early interruption! ! ! It was simply what was lacking, so we started researching around this.
(1) Enable index sorting
{
"settings": {
"index": {
"sort.field": "publish_time", // 可指定多个字段
"sort.order": "desc"
}
},
"mappings": {
"properties": {
"content_id": {
"type": "long"
},
"publish_time": {
"type": "long"
},
...
}
}
}
As in the example above, documents are sorted in descending order of the publish_time field when they are written to disk.
In the previous paragraphs we repeatedly mentioned docID and positive index. Here we briefly introduce their relationship by the way. First, each document in the Segment will be assigned a docID. The docID starts from 0 and is assigned sequentially. When there is no IndexSorting, docID is allocated according to the order in which the documents are written. After IndexSorting is set, the order of docID is consistent with the order of IndexSorting.
The following figure describes the relationship between docID and forward index:
So look back at our original query again:
{
"track_total_hits":true,
"sort": [
{
"publish_time": {
"order": "desc"
}
}
],
"size": 10
}
When querying in Lucene, it is found that the order of the inverted list of the result set is just in descending order of publish_time, so the query can return after the first 10 pieces of data are queried, which achieves early interruption and saves sorting overhead. So what is the price?
(2) Consideration
IndexSorting is different from query-time sorting. The essence is to preprocess data when writing. So sort fields can only be specified at creation time and cannot be changed. And because the data needs to be sorted when writing, it will also have a certain negative impact on the writing performance.
We mentioned before that Lucene itself has various optimizations for sorting, so if the search result set itself does not have so much data, even if this function is not enabled, it can still have a good RT.
In addition, since the total number still needs to be calculated most of the time, after the index sorting is enabled, the sorting process can only be interrupted in advance, and the total number of the result set must still be counted. If you can not check the total number, or get the total number in another way, then you can make better use of this feature.
summary:
- For the scenario where the top N items are sorted for a large result set, index sorting can significantly improve search performance .
- Index sorting can only be specified when creating the index and cannot be changed . If you have more than one specified field sorting scenario, you may need to choose the sorting field carefully.
- Not getting the total number can make better use of index sorting .
- Enabling index sorting will reduce write performance to a certain extent. Here is a screenshot of ElaticsearchBenchmarks data for your reference.
见：Elasticsearch Benchmarks
2.3 Effect
Since our business is far from reaching the ES write bottleneck, and there are few scenarios where the sorting fields are frequently changed. After a brief trade-off, it was determined that index sorting was exactly what we needed, so we began to use online real data to conduct a simple performance test on the effect of index sorting.
(1) Performance test: home page
(2) Performance test: other
Here, after the index sorting is turned on, the search combination test of several general conditions and time windows is randomized
It can be seen that the effect is very obvious, there is no spike like before, and RT is also very stable, so we decided to officially launch this function.
(3) Online effect
slow query
!
Overall before and after comparison
Basically as we expected, the search RT has been greatly reduced, and the slow query has completely disappeared.
2.4 Subsequent optimization
In the process of exploration, we actually found some other optimization methods. In view of the development costs and benefits, some of them have not been fully applied to the production environment. Here are some of them, hoping to give you some inspiration.
- Do not get the total number: In most scenarios, not querying the total number can reduce overhead and improve performance. The search interface after ES 7.x does not return the total number by default, which is evident.
- Custom routing rules: From the query process above, we can see that ES will poll all fragments to obtain the desired data. If we can control the location of data fragments, we can also save a lot of overhead. For example: If we have a large number of scenarios in the future to check the dynamics of a certain user, we can control the sharding by user, which avoids shard polling and improves search efficiency.
- keyword: Not all numbers should be stored as numeric fields. If your numeric values are rarely used for range queries, but are often used for term queries and are sensitive to search rt. Then keyword is the most suitable storage method.
- Data preprocessing: Just like IndexSoting, if we can preprocess the data when writing, it can also save the overhead of searching. This combination
_ingest/pipelinemay have unexpected effects.
3. Write at the end
I believe that everyone who has seen this can see that our optimization does not involve very advanced technical difficulties. We are only in the process of solving problems, and we have gradually changed from a beginner to a beginner. Coming to a big cow may directly bypass our detours from the very beginning, but a journey of thousands of miles begins with a single step. Finally, here is a summary of experience and feelings to share with everyone, hoping to give some references to beginners like us.
ES does not perform well in scenarios where large result sets specify field sorting, and we should try to avoid such scenarios when using it. If it cannot be avoided, proper IndexSorting settings can greatly improve sorting performance .
Optimization is endless. We should weigh costs and benefits, and concentrate resources to solve the most priority and important problems.
Text: Kelp
This article belongs to Dewu technology original, source: Dewu technology official website
It is strictly forbidden to reprint without the permission of Dewu Technology, otherwise legal responsibility will be investigated according to law! Copyright belongs to the author. For commercial reprint, please contact the author for authorization, for non-commercial reprint, please indicate the source.