ElasticSearch教程——Search相关、deep paging问题及解决方案

其他 2018-10-07 20:09:50 阅读次数: 0
版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/gwd1154978352/article/details/82943037

ElasticSearch汇总请查看：ElasticSearch教程——汇总篇

 

搜索所有索引

GET /_search

返回结果

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 16,
    "successful": 16,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 1,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "config:6.4.0",
        "_score": 1,
        "_source": {
          "type": "config",
          "updated_at": "2018-09-18T09:30:18.949Z",
          "config": {
            "buildNum": 17929,
            "telemetry:optIn": true
          }
        }
      },
      {
        "_index": "blog",
        "_type": "article",
        "_id": "eTmX5mUBtZGWutGW0TNs",
        "_score": 1,
        "_source": {
          "title": "New version of Elasticsearch released!",
          "content": "Version 1.0 released today!",
          "priority": 10,
          "tags": [
            "announce",
            "elasticsearch",
            "release"
          ]
        }
      },
      {
        "_index": "ecommerce",
        "_type": "product",
        "_id": "2",
        "_score": 1,
        "_source": {
          "name": "jiajieshi yagao",
          "desc": "youxiao fangzhu",
          "price": 25,
          "producer": "jiajieshi producer",
          "tags": [
            "fangzhu"
          ]
        }
      },
      {
        "_index": "ecommerce",
        "_type": "product",
        "_id": "J3fLFWYBBoLynJN1-kOG",
        "_score": 1,
        "_source": {
          "name": "test yagao",
          "desc": "youxiao fangzhu"
        }
      },
      {
        "_index": "blog",
        "_type": "article",
        "_id": "1",
        "_score": 1,
        "_source": {
          "id": "1",
          "title": "New version of Elasticsearch released!",
          "content": "Version 1.0 released today!",
          "priority": 10,
          "tags": [
            "announce",
            "elasticsearch",
            "release"
          ]
        }
      },
      {
        "_index": "ecommerce",
        "_type": "product",
        "_id": "KXfSFWYBBoLynJN1TUPo",
        "_score": 1,
        "_source": {
          "name": "test yagao2",
          "desc": "youxiao fangzhu2"
        }
      },
      {
        "_index": "index",
        "_type": "fulltext",
        "_id": "1",
        "_score": 1,
        "_source": {
          "content": "中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"
        }
      },
      {
        "_index": "ecommerce",
        "_type": "product",
        "_id": "3",
        "_score": 1,
        "_source": {
          "name": "zhonghua yagao",
          "desc": "caoben zhiwu",
          "price": 40,
          "producer": "zhonghua producer",
          "tags": [
            "qingxin"
          ]
        }
      }
    ]
  }
}

返回参数含义

took：整个搜索请求花费了多少毫秒

hits.total：本次搜索，返回了几条结果
hits.max_score：本次搜索的所有结果中，最大的相关度分数是多少，每一条document对于search的相关度，越相关，_score分数越大，排位越靠前
hits.hits：默认查询前10条数据，完整数据，_score降序排序

shards：shards fail的条件（primary和replica全部挂掉），不影响其他shard。默认情况下来说，一个搜索请求，会打到一个index的所有primary shard上去，当然了，每个primary shard都可能会有一个或多个replic shard，所以请求也可以到primary shard的其中一个replica shard上去。

timeout：默认无timeout，当搜索得特别深，需要花费很长时间的时候我们可以设置timeout，当时间达到这个timeout的时候就返回当前的搜索结果（不继续搜索下去了）

timeout=10ms，timeout=1s，timeout=1m
GET /_search?timeout=10m

查询指定index

GET /blog/_search

查询多个指定index

GET /.kibana,blog/_search

按照通配符去匹配多个索引

GET /*log/_search

返回结果

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "blog",
        "_type": "article",
        "_id": "eTmX5mUBtZGWutGW0TNs",
        "_score": 1,
        "_source": {
          "title": "New version of Elasticsearch released!",
          "content": "Version 1.0 released today!",
          "priority": 10,
          "tags": [
            "announce",
            "elasticsearch",
            "release"
          ]
        }
      },
      {
        "_index": "blog",
        "_type": "article",
        "_id": "1",
        "_score": 1,
        "_source": {
          "id": "1",
          "title": "New version of Elasticsearch released!",
          "content": "Version 1.0 released today!",
          "priority": 10,
          "tags": [
            "announce",
            "elasticsearch",
            "release"
          ]
        }
      }
    ]
  }
}

搜索一个index下指定的type的数据

GET /index1/type1/_search

搜索一个index下多个type的数据

由于在6.0之后每个index下最多只能有一个type，故在该版本及其以后无意义

GET /index1/type1,type2/_search

搜索多个index下的多个type的数据

由于在6.0之后每个index下最多只能有一个type，故在该版本及其以后无意义

GET /index1,index2/type1,type2/_search

搜索所有index下的指定type的数据

由于在6.0之后每个index下最多只能有一个type，故在该版本及其以后无意义

GET /_all/type1,type2/_search

分页搜索

添加"?from=0&size=2"

注：当数据量达到50000条以上时，用下面的scroll滚动的方式进行代替

GET /_search?from=0&size=2

返回结果

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 16,
    "successful": 16,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 1,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "config:6.4.0",
        "_score": 1,
        "_source": {
          "type": "config",
          "updated_at": "2018-09-18T09:30:18.949Z",
          "config": {
            "buildNum": 17929,
            "telemetry:optIn": true
          }
        }
      },
      {
        "_index": "blog",
        "_type": "article",
        "_id": "eTmX5mUBtZGWutGW0TNs",
        "_score": 1,
        "_source": {
          "title": "New version of Elasticsearch released!",
          "content": "Version 1.0 released today!",
          "priority": 10,
          "tags": [
            "announce",
            "elasticsearch",
            "release"
          ]
        }
      }
    ]
  }
}

deep paging问题

deep paging简单来说，就是搜索的特别深，比如总共有60000条数据，现在有3个primary shard，每个shard上分20000条，每页是10条数据，这个时候你要搜索到第1000页，实际上要拿到的是10001-10010，该怎么拿呢？
请求首先可能是打到一个不包含这个index的shard的node上，这个node就是一个coordinate node，这个coordinate node就会将搜索请求转发到index的三个shard所在的node上去。

要搜索60000条数据中的第1000页，实际上每个shard都要将内部的20000条数据中的第1-10010条数据拿出来，不是10条，是10010条数据，3个shard每个shard都返回10010条数据给coordinate node，coordinate node会收到总共30030条数据，然后排序取到所需的那10条数据，其实就是我们要的最后的第1000页的10条数据。

举个例子，现在有60个带编号的球（从1到60），我现在随机给他们放到三个篮子里面（他们在篮子里面已经排好序了），现在我要取出第10-12个球，那我是不是应该先把各个篮子里面前12个球取出来放到一起（篮子里面的球是随机放的，无规律），共计36个球，然后汇总进行排序后，在这个结果中取出第10-12个球！！！

缺点

搜索过深的时候就需要在coordinate node上保存大量的数据，还要进行大量数据的排序，排序之后再取出对应的那一页，所以这个过程，既消耗网络宽带，耗费内存，还消耗cpu。这就是deep paging的性能问题，我们应该尽量避免出现这种deep paging操作。

解决方案

为了解决上面的问题，elasticsearch提出了一个scroll滚动的方式，这个滚动的方式原理就是通过每次查询后，返回一个scroll_id。根据这个scroll_id 进行下一页的查询。可以把这个scroll_id理解为通常关系型数据库中的游标。但是，这种scroll方式的缺点是不能够进行反复查询，也就是说，只能进行下一页，不能进行上一页。

经过分析，如果数据达到了50000条以上，那么用户基本上是不会考虑每条都去看的，用户需要的是最后对数据分析处理后的结果。而如果小于50000条的时候我们可以使用from size的方式进行分页的查询。那么这种方式存在是为了什么情景呢。应该是为了分批次的检索所有数据。

实现步骤

1.首先取出前2条，并且得到scroll_id（这里的3s代表的是持续滚动时间，如果过了3秒钟，还没有查询下一页，那么这个scroll_id就会失效）。

GET /_search?scroll=3s&size=2

返回结果

{
  "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoEAAAAAAAAAPIFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAADyhZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAAA8sWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAPJFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAADzRZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAAA8wWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAPOFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAADzxZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAAA9cWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAPQFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAD0RZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAAA9UWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAPWFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAD0hZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAAA9MWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAPUFnhEVi1HVGViVFJxYzdlczBoRFI0clE=",
  "took": 10,
  "timed_out": false,
  "_shards": {
    "total": 16,
    "successful": 16,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 1,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "config:6.4.0",
        "_score": 1,
        "_source": {
          "type": "config",
          "updated_at": "2018-09-18T09:30:18.949Z",
          "config": {
            "buildNum": 17929,
            "telemetry:optIn": true
          }
        }
      },
      {
        "_index": "blog",
        "_type": "article",
        "_id": "eTmX5mUBtZGWutGW0TNs",
        "_score": 1,
        "_source": {
          "title": "New version of Elasticsearch released!",
          "content": "Version 1.0 released today!",
          "priority": 10,
          "tags": [
            "announce",
            "elasticsearch",
            "release"
          ]
        }
      }
    ]
  }
}

2.再次查询下一页，注意，这里查询时不需要指定index，只需要指定scroll_id和本次的持续滚动时间。

说白了，想要第几页，循环请求几次就行了，在设置的时间内scroll_id是不会变的

GET /_search/scroll?scroll=3s&scroll_id=DnF1ZXJ5VGhlbkZldGNoEAAAAAAAAAWtFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFuxZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABa4WeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAWwFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFrxZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABbIWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAWxFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFvBZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABbMWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAW0FnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFtRZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABbYWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAW3FnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFuBZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABbkWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAW6FnhEVi1HVGViVFJxYzdlczBoRFI0clE=

或者

POST /_search/scroll
{
 "scroll" : "3s",
 "scroll_id":"DnF1ZXJ5VGhlbkZldGNoEAAAAAAAAAXIFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAF1RZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABccWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXWFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFyRZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABcoWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXLFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFzBZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABc0WeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXOFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFzxZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABdAWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXSFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAF0RZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABdQWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXTFnhEVi1HVGViVFJxYzdlczBoRFI0clE="
}

删除对应scroll_id

当我们搜索完毕或者说已经滚动到最后的时候，我们可以选择删除scroll_id

DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoEAAAAAAAAAXIFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAF1RZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABccWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXWFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFyRZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABcoWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXLFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFzBZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABc0WeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXOFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAFzxZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABdAWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXSFnhEVi1HVGViVFJxYzdlczBoRFI0clEAAAAAAAAF0RZ4RFYtR1RlYlRScWM3ZXMwaERSNHJRAAAAAAAABdQWeERWLUdUZWJUUnFjN2VzMGhEUjRyUQAAAAAAAAXTFnhEVi1HVGViVFJxYzdlczBoRFI0clE=

删除所有scroll_id

DELETE /_search/scroll/_all

猜你喜欢

转载自blog.csdn.net/gwd1154978352/article/details/82943037
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
周排行
循环神经网络（rnn）讲解
Tigao教程四：单独的关节运动
金蝶K3WISE15.0-注册套打教程
如何在Mac上配置Kubernetes
Android应用结束自身进程的方法
SpringMVC学习 十三 拦截器栈
中国驻洛杉矶总领馆举行新春招待会
HttpClient get post 发送
11 - three.js 笔记 - 绘制三维字体模型
Mysql递归获取某个父节点下面的所有子节点和子节点上的所有父节点
每日归档
更多
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022