ES7.x的查询语句及java查询语句

ES 简介

Elasticsearch 7.x 是一个基于 Lucene 的分布式搜索引擎，它提供了一个分布式全文搜索引擎，可以快速、准确地搜索、分析和存储海量数据。Elasticsearch 7.x 的主要特点包括：

• 分布式架构：Elasticsearch 7.x 是一个分布式系统，它可以运行在多个节点上，每个节点都是相互独立的，并且数据会自动分片和复制到多个节点上，提高了系统的可用性和可扩展性。

• 实时搜索：Elasticsearch 7.x 可以快速地索引和搜索海量数据，支持实时搜索，即当数据发生变化时，可以立即搜索到最新的结果。

• 多数据类型支持：Elasticsearch 7.x 支持多种数据类型的索引和搜索，包括文本、数字、日期、地理位置等，可以满足各种不同类型数据的搜索需求。

• 分词和分析：Elasticsearch 7.x 支持文本分词和分析，它可以自动将文本分解成单词，然后进行索引和搜索，提高了搜索的准确性和效率。

• RESTful API：Elasticsearch 7.x 提供了基于 RESTful API 的操作方式，可以通过 HTTP 请求与 Elasticsearch 进行交互，方便易用。

• 多语言支持：Elasticsearch 7.x 支持多种编程语言的客户端，包括 Java、Python、Ruby、PHP 等，可以方便地通过这些客户端进行索引和搜索操作。

• 实时监控和诊断：Elasticsearch 7.x 提供了实时监控和诊断工具，可以方便地监控集群状态、索引性能、节点状态等，提高了系统的可维护性和可靠性。
  总之，Elasticsearch 7.x 是一个功能强大、性能优越、易于使用的分布式搜索引擎，可以满足各种不同规模的数据搜索和分析需求。

基本信息查询

查看所有索引

# 查看所有索引
GET _cat/indices

green open indexName1                                 785t4eK4SiarjIj6d_qDtA 1 0       0  0    283b    283b
green open indexName2                              7WB8hk3cRMKuTUJhKWni9g 1 0       0  0    283b    283b
green open indexName3                                PDJWT_VoS9iWSfEzUe5LEQ 1 0       0  0    283b    283b
green open indexName4                  0qc3VwyvS9SIQ97GbILvJw 1 0       0  0    283b    283b

public Set<String> getAllIndex() {
    
    
    GetAliasesRequest request = new GetAliasesRequest();
    GetAliasesResponse response = null;
    try {
    
    
        response = restHighLevelClient.indices().getAlias(request, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }

    Map<String, Set<AliasMetadata>> aliases = response.getAliases();
    Set<String> strings = aliases.keySet();
    return strings;
}

查看索引 mapping

# 查看mapping
GET indexName/_mapping

{
    
    
  "indexName" : {
    
    
    "mappings" : {
    
    
      "properties" : {
    
    
        "field1" : {
    
    
          "type" : "integer"
        },
        "field2" : {
    
    
          "type" : "keyword"
        },
        "field3" : {
    
    
          "type" : "integer"
        },
        "field4" : {
    
    
          "type" : "long"
        },
        "field5" : {
    
    
          "type" : "keyword"
        },
        "field6" : {
    
    
          "type" : "integer"
        }
     }
   }
}

public Map<String, String> getMapping(String indexName){
    
    
    GetMappingsRequest request = new GetMappingsRequest().indices(indexName);
    GetMappingsResponse response = null;
    try {
    
    
        response = restHighLevelClient.indices().getMapping(request, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }
    Map<String, Object> mappings = response.mappings().get(indexName).getSourceAsMap();
    Map<String, Map<String, Object>> properties = (Map<String, Map<String, Object>>) mappings.get("properties");

    Map<String, String> result = new HashMap<>();
    properties.keySet().forEach(key -> {
    
    
        String value = properties.get(key).get("type").toString();
        System.out.println(key);
        result.put(key, value);
    } );

    return result;
}

查索引中所有值

GET indexName/_search
{
    
    
 "query":{
    
    
   "match_all": {
    
    }
 }
}

{
    
    
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    
    
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    
    
    "total" : {
    
    
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
    
    
        "_index" : "indexName",
        "_type" : "_doc",
        "_id" : "ngGsfYcB7ZKupvf2r47w",
        "_score" : 1.0,
        "_source" : {
    
    
        	"field1" : "value1",
        	"field2" : "value2"
        }
      },
      
      {
    
    
        "_index" : "indexName",
        "_type" : "_doc",
        "_id" : "ngGsfYcB7ZKupvf2r47w",
        "_score" : 1.0,
        "_source" : {
    
    
        	"field1" : "value1",
        	"field2" : "value2"
        }
      }
   	 ]
   }
}

public List<Map<String, Object>> getAll(String indexName) {
    
    
    SearchRequest searchRequest = new SearchRequest(indexName);
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
            .query(QueryBuilders.matchAllQuery())
            .size(10000);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = null;
    try {
    
    
        search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }
    SearchHit[] hits = search.getHits().getHits();
    List<Map<String, Object>> result = new ArrayList<>();
    Arrays.stream(hits).iterator().forEachRemaining(oneDoc -> {
    
    
        Map<String, Object> sourceAsMap = (Map<String, Object>) oneDoc.getSourceAsMap();
    
        result.add(sourceAsMap);
    });
    return result;
}

过滤

ES7.x中的bool查询是一种复合查询，可以将多个查询组合起来进行复杂的逻辑运算。bool查询中常用的子查询有以下几种：

1. must查询：所有的子查询都必须匹配才算匹配成功。
2. must_not查询：所有的子查询都必须不匹配才算匹配成功。
3. should查询：至少有一个子查询匹配成功就算匹配成功，可以指定多个should查询，可以通过minimum_should_match参数控制至少需要匹配的子查询数量。
4. filter查询：和must查询相似，但是不会计算得分，只用于过滤文档。 可以将这些子查询组合起来进行复杂的逻辑运算，例如可以使用must查询和should查询组合来实现"必须匹配A和B，或者匹配C"的查询条件。

每种子查询下面又有各种过滤条件：

1. 相等：QueryBuilders.termQuery(fieldName, value)
2. 取值范围：QueryBuilders.rangeQuery(fieldName).lt(leftValue).gt(rightValue)
   • 除此之外还有 lte、gte、equals 等。

以下查询语句表示查询索引名为 indexName 中 fileTypeEnum 字段值必须是 TEMP、timestamp 字段值必须在 1680769009720-1680769909720之间，count 字段必须不是 19 的值。

GET /indexName/_search
{
    
    
  "query": {
    
     
    "bool": {
    
    
      "must": [
        {
    
    
          "term": {
    
    
            "fileTypeEnum": {
    
    
              "value": "TEMP"
            }
          }
        },
        {
    
    
          "range": {
    
    
            "timestamp": {
    
    
              "gte": 1680769009720,
              "lte": 1680769909720
            }
          }
        }
      ],
      "must_not": [
        {
    
    
          "term": {
    
    
            "count": {
    
    
              "value": "19"
            }
          }
        }
      ]

    }
  },
  "size": 10000
}

public List<Map<String, Object>> search(String indexName, SearchSourceBuilder searchSourceBuilder) {
    
    
    SearchRequest searchRequest = new SearchRequest(indexName);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = null;
    try {
    
    
        search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }
    SearchHit[] hits = search.getHits().getHits();
    List<Map<String, Object>> result = new ArrayList<>();
    Arrays.stream(hits).iterator().forEachRemaining(oneDoc -> {
    
    
        Map<String, Object> sourceAsMap = (Map<String, Object>) oneDoc.getSourceAsMap();
        result.add(sourceAsMap);
    });
    return result;
}

定义了 search 函数，只需要传入index名称和一个SearchSourceBuilder对象即可，其中在SearchSourceBuilder对象中可以加入各种过滤条件。

SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
    .query(QueryBuilders.boolQuery().must(QueryBuilders.termQuery("name", "张三")))
    .query(QueryBuilders.boolQuery().mustNot(QueryBuilders.rangeQuery("timestape")
                                             				.lte(1001)
                                             				.gte(1101)));
List<Map<String, Object>> result = search(indexName, searchSourceBuilder);

模糊查询

模糊查询：QueryBuilders.wildcardQuery(fieldName, queryString)

• * 表示匹配零个或多个任意字符；
• ? 表示匹配一个任意字符。

GET /indexName/_search
{
    
    
  "query": {
    
    
    "wildcard": {
    
    
      "location": "*123*"
    }
  }
}

{
    
    
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    
    
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    
    
    "total" : {
    
    
      "value" : 270,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
    
    
        "_index" : "indexName",
        "_type" : "_doc",
        "_id" : "ngGsfYcB7ZKupvf2r47w",
        "_score" : 1.0,
        "_source" : {
    
    
        	"field1" : "value1",
        	"field2" : "value2"
        }
      },
      
      {
    
    
        "_index" : "indexName",
        "_type" : "_doc",
        "_id" : "ngGsfYcB7ZKupvf2r47w",
        "_score" : 1.0,
        "_source" : {
    
    
        	"field1" : "value1",
        	"field2" : "value2"
        }
      }
   	 ]
   }
}

List<String> contextLoads(String indexName) throws IOException {
    
    
     // 模糊搜索
     // *表示匹配任意多个字符（包括零个字符）
     // ?表示匹配任意单个字符
     WildcardQueryBuilder queryWildcard = QueryBuilders.wildcardQuery("location", "*123*");

     SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
             .query(queryWildcard)
             .from(0) // 设置分页
             .size(500) // 默认只显示10个
             .fetchSource(new String[]{
    
    "location"}, null) // 只返回location字段
             ;

    SearchRequest searchRequest = new SearchRequest(indexName);
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = null;
    try {
    
    
        search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }
    SearchHit[] hits = search.getHits().getHits();
    List<String> result = new ArrayList<>();
    Arrays.stream(hits).iterator().forEachRemaining(oneDoc -> {
    
    
        Map<String, Object> sourceAsMap = (Map<String, Object>) oneDoc.getSourceAsMap();
        result.add(sourceAsMap.get("location").toString());
    });
    return result
}

去重

对 indexName 中的 location 字段去重。

POST /indexName/_search
{
    
    
  "size": 0,
  "aggs": {
    
    
    "distinct": {
    
    
      "terms": {
    
    
        "field": "location"
      }
    }
  }
}

{
    
    
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    
    
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    
    
    "total" : {
    
    
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    
    
    "distinct" : {
    
    
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 688,
      "buckets" : [
        {
    
    
          "key" : "30-200-1680592680000",
          "doc_count" : 172
        },
        {
    
    
          "key" : "30-200-1680592801000",
          "doc_count" : 172
        },
        {
    
    
          "key" : "30-200-1680593042000",
          "doc_count" : 172
        },
        {
    
    
          "key" : "30-200-1680593161000",
          "doc_count" : 172
        }
      ]
    }
  }
}

通过聚合去重

public List<String>  getDistance() {
    
    
    SearchRequest searchRequest = new SearchRequest(indexName);

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
            .aggregation(AggregationBuilders.terms("distinct_location").field("location").size(10));

    searchRequest.source(searchSourceBuilder);
    SearchResponse searchResponse = null;
    try {
    
    
        searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }
    Terms distinctValues = searchResponse.getAggregations().get("distinct_location");

    List<String> collect = distinctValues.getBuckets().stream().map(Terms.Bucket::getKeyAsString).collect(Collectors.toList());
    return collect;
}

通过查询去重

List<String> getDistance2() throws IOException {
    
    
    SearchRequest searchRequest = new SearchRequest(indexName);
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
            .collapse(new CollapseBuilder("location")) // 根据字段去重
            .from(0) // 设置分页
            .size(10) // 默认只显示10个
            .fetchSource(new String[]{
    
    "location"}, null) // 只返回以上字段
            ;
    searchRequest.source(searchSourceBuilder);
    SearchResponse search = null;
    try {
    
    
        search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        throw new RuntimeException(e);
    }
    SearchHit[] hits = search.getHits().getHits();
    List<String> result = new ArrayList<>();
    Arrays.stream(hits).iterator().forEachRemaining(oneDoc -> {
    
    
        Map<String, Object> sourceAsMap = (Map<String, Object>) oneDoc.getSourceAsMap();
        result.add(sourceAsMap.get("location").toString());
    });
    return result;
}

过滤后去重

GET /indexName/_search
{
    
    
  "query": {
    
    
    "bool": {
    
    
      "must": [
        {
    
    
          "match": {
    
    
            "fileTypeEnum": {
    
    
              "query": "TEMP"
            }
          }
        },
        {
    
    
          "wildcard": {
    
    
            "location": "*13*"
          }
        }
      ]
    }
  },
  "aggs": {
    
    
    "distinct": {
    
    
      "terms": {
    
    
        "field": "location"
      }
    }
  }
}

    String name = "location"; // 名称实际上是 location 字段
    // 根据数据类型选择索引名称
    SearchRequest searchRequest = new SearchRequest().indices(indexName);

    /**
     * 封装查询条件
     */
    BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
    // fieldName1
    if (StringUtils.isNotEmpty(DTO.getSubsystem())) {
    
    
        boolQueryBuilder.must(QueryBuilders.termQuery(fieldName1, rawDataAssetsDTO.getSubsystem()));
    }
    // fieldName1
    if (StringUtils.isNotEmpty(DTO.getSubject())) {
    
    
        boolQueryBuilder.must(QueryBuilders.termQuery(fieldName1, DTO.getSubject()));
    }
    // 模糊查询 location
    if (StringUtils.isNotEmpty(DTO.getLocation())) {
    
    
        boolQueryBuilder.must(QueryBuilders.wildcardQuery(name, "*" + DTO.getLocation() + "*"));
    }

    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
            .query(boolQueryBuilder)
            .aggregation(AggregationBuilders.terms("result").field(name).size(100))
            ;

    searchRequest.source(searchSourceBuilder);
    SearchResponse searchResponse = null;
    try {
    
    
        searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
    } catch (IOException e) {
    
    
        logger.error("查询ES数据异常：" + e.getMessage());
    }

    Terms distinctValues = searchResponse.getAggregations().get("result");

    List<String> result = distinctValues.getBuckets().stream().map(Terms.Bucket::getKeyAsString).collect(Collectors.toList());
    return result;