ES client (RestHighLevelClient, SpringDataElasticsearch framework) usage guide

RestHighLevelClient

Client introduction

Clients in various languages ​​are provided on the elasticsearch official website :

Choose Java REST Client

Select the Java High Level Rest Client version, here are the APIs used

es dependency

    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>6.2.4</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>elasticsearch-rest-client</artifactId>
        <version>6.2.4</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>elasticsearch-rest-high-level-client</artifactId>
        <version>6.2.4</version>
    </dependency>

    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>fastjson</artifactId>
        <version>1.2.70</version>
    </dependency>

Index library and mapping

Create the type and its mapping relationship while creating the index library, but it is not recommended to use the java client to complete these operations for the following reasons:

• The index library and mapping are often completed during initialization and do not require frequent operations. It is better to configure them in advance

• The official index creation library and mapping API are very cumbersome, and the json structure needs to be concatenated through strings:

  request.mapping(
          "{\n" +
          "  \"properties\": {\n" +
          "    \"message\": {\n" +
          "      \"type\": \"text\"\n" +
          "    }\n" +
          "  }\n" +
          "}", 
          XContentType.JSON);
  

  
  

Therefore, it is recommended to use the Rest style API to implement these operations.

Take such a product data as an example to create an index library:

New entity class:

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@NoArgsConstructor
@AllArgsConstructor
public class Item {
    
    
    private Long id;
    private String title; //标题
    private String category;// 分类
    private String brand; // 品牌
    private Double price; // 价格
    private String images; // 图片地址
}

• id: It can be considered as the primary key, and it can be used to judge whether the data is repeated in the future, regardless of word segmentation, and the keyword type can be used
• title: search field, word segmentation is required, you can use text type
• category: Product classification, this is the whole, regardless of words, you can use the keyword type
• brand: brand, similar to classification, regardless of word, you can use keyword type
• price: price, this is double type
• images: pictures, fields used for display, no search, index is false, no word segmentation, keyword type can be used

Mapping configuration:

PUT /item
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "id": {
          "type": "keyword"
        },
        "title": {
          "type": "text",
          "analyzer": "ik_max_word"
        },
        "category": {
          "type": "keyword"
        },
        "brand": {
          "type": "keyword"
        },
        "images": {
          "type": "keyword",
          "index": false
        },
        "price": {
          "type": "double"
        }
      }
    }
  }
}

view add result

document manipulation

Initialize the http client: RestHighLevelClient

To complete any operation, you need to pass the RestHighLevelClient client

Getting started example

@RunWith(SpringRunner.class)
@SpringBootTest
class EsDemoApplicationTests {
    
    

    RestHighLevelClient client;

    /**
     * 初始化连接
     */
    @Before
    void init() {
    
    
        //初始化：高级客户端
        client = new RestHighLevelClient(RestClient.builder(
            new HttpHost("192.168.85.135", 9201, "http"),
            new HttpHost("192.168.85.135", 9202, "http"),
            new HttpHost("192.168.85.135", 9203, "http")
        ));
    }
    
    @After
    void close() throws IOException {
    
    
        client.close();
    }
}

Application level case

application.yml configuration file

# es集群名称
elasticsearch.clusterName=single-node-cluster
# es用户名
elasticsearch.userName=elastic
# es密码
elasticsearch.password=elastic
# es 是否启用用户密码
elasticsearch.passwdEnabled=true
# es host ip 地址(集群)：本次使用的是单机模式
elasticsearch.hosts=43.142.243.124:9200
# es 请求方式
elasticsearch.scheme=http
# es 连接超时时间
elasticsearch.connectTimeOut=1000
# es socket 连接超时时间
elasticsearch.socketTimeOut=30000
# es 请求超时时间
elasticsearch.connectionRequestTimeOut=500
# es 连接保持活跃时间(ms)
elasticsearch.keepAliveStrategy=180000
# es 最大连接数
elasticsearch.maxConnectNum=100
# es 每个路由的最大连接数
elasticsearch.maxConnectNumPerRoute=100

es connection configuration class

import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.ArrayList;
import java.util.List;

/**
 * restHighLevelClient 客户端配置类
 */
@Slf4j
@Data
@Configuration
@ConfigurationProperties(prefix = "elasticsearch")
public class ElasticsearchConfig {
    
    

    // es host ip 地址（集群）
    private String hosts;
    // es用户名
    private String userName;
    // es密码
    private String password;
    // es 是否启用用户密码
    private boolean passwdEnabled;
    // es 请求方式
    private String scheme;
    // es集群名称
    private String clusterName;
    // es 连接超时时间
    private int connectTimeOut;
    // es socket 连接超时时间
    private int socketTimeOut;
    // es 请求超时时间
    private int connectionRequestTimeOut;
    // es 连接保持活跃时间
    private int keepAliveStrategy;
    // es 最大连接数
    private int maxConnectNum;
    // es 每个路由的最大连接数
    private int maxConnectNumPerRoute;

    @Bean(name = "restHighLevelClient")
    public RestHighLevelClient restHighLevelClient() {
    
    
        // 拆分地址。单节点配一个地址即可
        List<HttpHost> hostLists = new ArrayList<>();
        hosts = hosts.replace("http://", "");
        String[] hostList = hosts.split(",");
        for (String addr : hostList) {
    
    
            String host = addr.split(":")[0];
            String port = addr.split(":")[1] == null ? "9200": addr.split(":")[1];
            hostLists.add(new HttpHost(host, Integer.parseInt(port), scheme));
        }
        // 转换成 HttpHost 数组
        HttpHost[] httpHost = hostLists.toArray(new HttpHost[]{
    
    });

        // 构建连接对象
        RestClientBuilder builder = RestClient.builder(httpHost);

        // 连接延时配置
        builder.setRequestConfigCallback(requestConfigBuilder -> {
    
    
            requestConfigBuilder
                .setConnectTimeout(connectTimeOut)
                .setSocketTimeout(socketTimeOut)
                .setConnectionRequestTimeout(connectionRequestTimeOut);
            return requestConfigBuilder;
        });


        builder.setHttpClientConfigCallback(httpClientBuilder -> {
    
    
            httpClientBuilder
                // 连接数配置
                .setMaxConnTotal(maxConnectNum)
                .setMaxConnPerRoute(maxConnectNumPerRoute)
                // 连接保持活跃时间配置
                .setKeepAliveStrategy((HttpRequest, HttpResponse) -> keepAliveStrategy);
            // 设置用户名、密码
            if (passwdEnabled) {
    
    
                CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, password));
                httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
            return httpClientBuilder;
        });

        return new RestHighLevelClient(builder);
    }
}

Note:

• KeepAliveStrategy ：

  The HTTP specification does not determine how long a persistent connection may or should be kept alive.

  Some HTTP servers use the non-standard Keep-Alive header to tell clients how many period seconds they want to keep the connection alive on the server side.

  If this information is available, HttpClient will make use of it.

  If the Keep-Alive header is not present in the response, HttpClient assumes the connection is kept alive indefinitely.

  However many real HTTP servers are configured to drop persistent connections after a certain period of inactivity to conserve system resources, often without notifying clients.

Create indexes and maps

Index mappings

mapping_test.json

{
    
    
    "properties": {
    
    
        "brandName": {
    
    
            "type": "keyword"
        }, 
        "categoryName": {
    
    
            "type": "keyword"
        }, 
        "createTime": {
    
    
            "type": "date", 
            "format": "yyyy-MM-dd HH:mm:ss"
        }, 
        "id": {
    
    
            "type": "long"
        }, 
        "price": {
    
    
            "type": "double"
        }, 
        "saleNum": {
    
    
            "type": "integer"
        }, 
        "status": {
    
    
            "type": "integer"
        }, 
        "stock": {
    
    
            "type": "integer"
        }, 
        "spec": {
    
    
            "type": "text", 
            "analyzer": "ik_max_word", 
            "search_analyzer": "ik_smart"
        }, 
        "title": {
    
    
            "type": "text", 
            "analyzer": "ik_max_word", 
            "search_analyzer": "ik_smart"
        }
    }
}

import com.example.test.service.es.IndexTestService;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.client.IndicesClient;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.client.indices.CreateIndexResponse;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.client.indices.GetIndexResponse;
import org.elasticsearch.cluster.metadata.MappingMetaData;
import org.elasticsearch.common.xcontent.XContentType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.Map;

/**
 * 索引服务类
 */
@Service
public class IndexTestServiceImpl implements IndexTestService {
    
    
    
    @value("classpath:json/mapping_test.json")
    private resoure mappingTest;

    @Autowired
    RestHighLevelClient restHighLevelClient;
    
    // 分片数的配置名
    private String shardNumName = "number_of_shards";
    
    // 副本数的配置名
    private String replicaNumName = "number_of_replicas";
    
    // 索引名
    private String index = "goods"

    @Override
    public boolean indexCreate() throws Exception {
    
    
        // 1、创建 创建索引request 参数：索引名
        CreateIndexRequest indexRequest = new CreateIndexRequest(index);
        // 2、设置索引的settings
        indexRequest.settings(Settings.builder().put(shardNumName, 3).put(replicaNumName, 1));
        // 3、设置索引的mappings(表结构)
        String mappingJson = IOUtils.toString(mappingTest.getInputStream(), Charset.forName("UTF-8"));
        indexRequest.mapping(mappingJson, XContentType.JSON);
        // 4、 设置索引的别名
        // 5、 发送请求
        // 5.1 同步方式发送请求
        // 请求服务器
        IndicesClient indicesClient = restHighLevelClient.indices();
        CreateIndexResponse response = indicesClient.create(indexRequest, RequestOptions.DEFAULT);

        return response.isAcknowledged();
    }


    /**
     * 获取表结构
     * GET goods/_mapping
     */
    @Override
    public Map<String, Object> getMapping(String indexName) throws Exception {
    
    
        IndicesClient indicesClient = restHighLevelClient.indices();

        // 创建get请求
        GetIndexRequest request = new GetIndexRequest(indexName);
        // 发送get请求
        GetIndexResponse response = indicesClient.get(request, RequestOptions.DEFAULT);
        // 获取表结构
        Map<String, MappingMetaData> mappings = response.getMappings();
        Map<String, Object> sourceAsMap = mappings.get(indexName).getSourceAsMap();
        return sourceAsMap;
    }

    /**
     * 删除索引库
     */
    @Override
    public boolean indexDelete(String indexName) throws Exception {
    
    
        IndicesClient indicesClient = restHighLevelClient.indices();
        // 创建delete请求方式
        DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest(indexName);
        // 发送delete请求
        AcknowledgedResponse response = indicesClient.delete(deleteIndexRequest, RequestOptions.DEFAULT);

        return response.isAcknowledged();
    }

    /**
     * 判断索引库是否存在
     */
    @Override
    public boolean indexExists(String indexName) throws Exception {
    
    
        IndicesClient indicesClient = restHighLevelClient.indices();
        // 创建get请求
        GetIndexRequest request = new GetIndexRequest(indexName);
        // 判断索引库是否存在
        boolean result = indicesClient.exists(request, RequestOptions.DEFAULT);

        return result;
    }
}

New document: index

Documentation: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/6.8/java-rest-high-document-index.html

Example:

// 新增文档
@Test
void add() throws IOException {
    
    
    // 准备文档
    Item item = new Item(1L, "小米手机9", "手机",
            "小米", 3499.00, "http://image.leyou.com/13123.jpg");
    // 将对象转换为Json
    String json = JSON.toJSONString(item);
    // 创建索引请求 参数为： 索引库名   类型名  文档ID
    IndexRequest request = new IndexRequest("item", "_doc", item.getId().toString());
    // 将Json格式的数据放入到请求中
    request.source(json, XContentType.JSON);
    // 发送请求
    IndexResponse response = client.index(request);
    // 打印结果
    System.out.println("结果为：" + response);
}

response:

response = IndexResponse[index=item,type=docs,id=1,version=1,result=created,seqNo=0,primaryTerm=1,shards={"total":2,"successful":2,"failed":0}]

View documentation: get

Example:

// 根据ID获取文档
@Test
void get() throws IOException {
    
    
    // 创建get请求对象 参数为： 索引库名   类型名  文档ID
    GetRequest request = new GetRequest("item","_doc","1");
    // 发送请求
    GetResponse response = client.get(request);
    // 解析结果 结果为Json
    String source = response.getSourceAsString();
    // 将Json数据转换为对象 参数为： Json字符串  类的字节码
    Item item = JSON.parseObject(source, Item.class);
    // 打印结果
    System.out.println(item);
}

Update documentation: update

Example:

// 根据ID更新文档
@Test
void update() throws IOException{
    
    
    // 准备文档
    Item item = new Item(1L, "小米手机9", "手机",
            "小米", 3699.00, "http://image.leyou.com/13123.jpg");
    // 将对象转换为Json
    String json = JSON.toJSONString(item);
    // 创建Update请求对象 参数为： 索引库名   类型名  文档ID
    UpdateRequest request = new UpdateRequest("item","_doc","1");
    // 将Json格式的数据放入到请求中
    request.doc(json,XContentType.JSON);
    // 发送请求
    UpdateResponse response = client.update(request);
    // 打印结果
    System.out.println("结果为：" + response);
}

Delete a document: delete

Example:

// 根据ID删除文档
@Test
void delete() throws IOException {
    
    
    // 创建Delete请求对象 参数为： 索引库名   类型名  文档ID
    DeleteRequest request = new DeleteRequest("item","_doc","1");
    // 发送请求
    DeleteResponse response = client.delete(request);
    // 打印结果
    System.out.println("结果为：" + response);
}

Add in batches: bulk

Example:

    // 批量插入
    @Test
    void bulkInsert() throws IOException {
    
    
        // 准备文档数据：
        List<Item> list = new ArrayList<>();
        list.add(new Item(1L, "小米手机7", "手机", "小米", 3299.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(2L, "坚果手机R1", "手机", "锤子", 3699.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(3L, "华为META10", "手机", "华为", 4499.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(4L, "小米Mix2S", "手机", "小米", 4299.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(5L, "荣耀V10", "手机", "华为", 2799.00, "http://image.leyou.com/13123.jpg"));
        // 创建批量新增请求
        BulkRequest request = new BulkRequest();
        // 遍历集合
        for (Item item : list) {
    
    
            // 将索引请求添加到批量请求对象中
            request.add(new IndexRequest("item", "_doc", item.getId().toString())
                    .source(JSON.toJSONString(item), XContentType.JSON));
        }
        // 发送请求
        BulkResponse response = client.bulk(request);
        // 打印结果
        System.out.println("结果为：" + response);
    }

Document search: search

Query all: matchAllQuery

Example:

    /**
     * 查询文档
     */
    @Test
    public void searchDoc() throws IOException {
    
    
        // 创建查询请求对象 指定查询的索引名称
        SearchRequest request = new SearchRequest("item");
        // 指定查询的源
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        // 构建查询条件
        QueryBuilder query = QueryBuilders.matchAllQuery(); // 查询的是所有的文档

        // 添加查询条件
        sourceBuilder.query(query);

        // 添加查询源
        request.source(sourceBuilder);
        // 发送请求
        SearchResponse response = client.search(request);
        // 分析响应结果
        // 返回命中的数据对象
        SearchHits searchHits = response.getHits();
        // 获取命中的文档个数
        long totalHits = searchHits.totalHits;
        System.out.println("命中的文档个数为: " + totalHits);
        // 获取命中的数据
        SearchHit[] hits = searchHits.getHits();
        // 遍历数组
        for (SearchHit hit : hits) {
    
    
            String sourceAsString = hit.getSourceAsString();
            // 转换成对象
            Item item = JSON.parseObject(sourceAsString, Item.class);
            System.out.println(item);
        }
    }

In the above code, the search condition is added through sourceBuilder.query(QueryBuilders.matchAllQuery()). The parameters accepted by the query() method are: QueryBuilder interface type.

The QueryBuilder interface provides many implementation classes, corresponding to different types of queries, such as: term query, match query, range query, boolean query, etc. To use various queries, just pass different parameters to the sourceBuilder.query() method. And these implementation classes do not need to be new, the official QueryBuilders factory is provided to build various implementation classes

Keyword search: matchQuery

In fact, the change of the search type is just that the query objects constructed by QueryBuilders are different, and the other codes are basically the same:

// 使用match查询,参数为 1 查询的字段 2 查询的关键字
QueryBuilder query = QueryBuilders.matchQuery("title","小米");

Keyword exact match: termQuery

// 使用term查询,参数为 1 查询的字段 2 查询的关键字
QueryBuilder query = QueryBuilders.termQuery("title","小米手机");

Range query: rangeQuery

The following range keywords are supported:

method	illustrate
gt(Object from)	more than the
gte(Object from)	greater or equal to
lt(Object from)	less than
lte(Object from)	less than or equal to

Example:

// 使用rangeQuery查询,参数为 查询的字段
QueryBuilder query = QueryBuilders.rangeQuery("price").gte(2000).lt(4000);  // 参数为：查询的字段 后面是链式的调用

response:

item = Item(id=2, title=坚果手机R1, category=手机, brand=锤子, price=3699.0, 
            images=http://image.leyou.com/13123.jpg)
item = Item(id=5, title=荣耀V10, category=手机, brand=华为, price=2799.0, 
            images=http://image.leyou.com/13123.jpg)
item = Item(id=1, title=小米手机7, category=手机, brand=小米, price=3299.0, 
            images=http://image.leyou.com/13123.jpg)

filter: fetchSource

By default, all data in the index library will be returned. If you want to return only some fields, you can control it through fetchSource.

Example:

    /**
     * 查询文档
     */
    @Test
    public void searchDoc() throws IOException {
    
    
        // 创建查询请求对象 指定查询的索引名称
        SearchRequest request = new SearchRequest("item");
        // 指定查询的源
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        // 构建查询条件
        // QueryBuilder query = QueryBuilders.matchAllQuery(); // 查询的是所有的文档

        // 使用match查询,参数为 1 查询的字段 2 查询的关键字
        // QueryBuilder query = QueryBuilders.matchQuery("title","小米");

        // 使用term查询,参数为 1 查询的字段 2 查询的关键字
        // QueryBuilder query = QueryBuilders.termQuery("title","小米手机");

        // 使用rangeQuery查询,参数为 1 查询的字段
        QueryBuilder query = QueryBuilders.rangeQuery("price").gte(3000).lte(4000);

        // 添加查询条件
        sourceBuilder.query(query);

        // 添加过滤
        String[] includes = {
    
    "id", "title", "price"};
        String[] excludes = {
    
    };
        sourceBuilder.fetchSource(includes, excludes);

        // 添加查询源
        request.source(sourceBuilder);
        // 发送请求
        SearchResponse response = client.search(request);
        // 分析响应结果
        // 返回命中的数据对象
        SearchHits searchHits = response.getHits();
        // 获取命中的文档个数
        long totalHits = searchHits.totalHits;
        System.out.println("命中的文档个数为: " + totalHits);
        // 获取命中的数据
        SearchHit[] hits = searchHits.getHits();
        // 遍历数组
        for (SearchHit hit : hits) {
    
    
            String sourceAsString = hit.getSourceAsString();
            // 转换成对象
            Item item = JSON.parseObject(sourceAsString, Item.class);
            System.out.println(item);
        }
    }

Sort: sort

Example:

// 排序
sourceBuilder.sort("price", SortOrder.DESC);

Pagination: from, size

Example:

// 分页
int current = 1;
int size = 2;
int start = (current - 1) * size;
sourceBuilder.from(start);
sourceBuilder.size(size);
// 搜索
SearchResponse response = client.search(request);

Highlight: HighlightBuilder

Example:

	@Test
    public void testHighlight() throws IOException{
    
    
        // 创建搜索对象
        SearchRequest request = new SearchRequest();
        // 指定查询的源
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        // 添加查询条件，通过QueryBuilders获取各种查询
        sourceBuilder.query(QueryBuilders.matchQuery("title", "小米手机"));

        // 高亮
        HighlightBuilder highlightBuilder = new HighlightBuilder()	//创建高亮构建器对象
                .field("title") // 指定高亮字段
                .preTags("<em style='color:red'>")  // 添加高亮前缀
                .postTags("</em>"); // 添加高亮后缀
        sourceBuilder.highlighter(highlightBuilder);
        request.source(sourceBuilder);

        // 获取结果
        SearchResponse response = client.search(request);
        SearchHits hits = response.getHits();
        SearchHit[] hitList = hits.getHits();
        for (SearchHit hit : hitList) {
    
    
            // 获取高亮结果
            Map<String, HighlightField> fields = hit.getHighlightFields();
            // 取出标题
            HighlightField titleField = fields.get("title");
            // 拼接为字符串
            Text[] fragments = titleField.fragments();
            String title = fragments[0].string();
            // 获取其它字段,并转换成对象
            Item item = JSON.parseObject(hit.getSourceAsString(), Item.class);
            // 覆盖title
            item.setTitle(title);
            System.out.println(item);
        }
    }

key code:

• Add a highlighted field to the query condition:
  • new HighlightBuilder() : Create a highlight builder
  • .field("title") : specify the highlighted field
  • .preTags("") and .postTags("") : specify the highlighted pre and post tags
• Parsing highlighted results:
  • hit.getHighlightFields(); Get highlight results

Aggregation: aggregation

Let's try aggregation again, using the brand field to aggregate to see which brands are there and how many of each brand there are.

The key to aggregation is to figure out these points:

• What is the aggregated field
• What is the type of aggregation
• give the aggregation a name

Similar to queries, aggregation conditions are set through the sourceBuilder.aggregation() method, and the parameter is an interface:

• AggregationBuilder, this interface also has a large number of implementation classes, representing different types of aggregation.

Also, there is no need to go to new by yourself. The official provides a factory to help create instances:

Example:

    /**
     * 聚合
     */
    @Test
    public void testAgg() throws IOException{
    
    
        // 创建搜索对象
        SearchRequest request = new SearchRequest();
        // 指定查询的源
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        // 添加查询条件，通过QueryBuilders获取各种查询
        sourceBuilder.query(QueryBuilders.matchAllQuery());
        // 添加排序
        sourceBuilder.sort("price", SortOrder.ASC);
        // 配置size为0，因为不需要数据，只要聚合结果
        sourceBuilder.size(0);
        // 添加聚合
        sourceBuilder.aggregation(AggregationBuilders.terms("brandAgg").field("brand"));
        request.source(sourceBuilder);
        // 获取结果
        SearchResponse response = client.search(request);
        // 获取聚合结果
        Aggregations aggregations = response.getAggregations();
        // 获取某个聚合
        Terms terms = aggregations.get("brandAgg");
        // 获取桶
        for (Terms.Bucket bucket : terms.getBuckets()) {
    
    
            // 获取key，这里是品牌名称
            System.out.println("品牌 : " + bucket.getKeyAsString());
            // 获取docCount，就是数量
            System.out.println("count: " + bucket.getDocCount());
        }
    }

response:

品牌 : 华为
count: 2
品牌 : 小米
count: 2
品牌 : 锤子
count: 1

It is also possible to add sub-aggregations within an aggregate

Example:

	/**
     * 聚合
     */
    @Test
    public void testAgg() throws IOException{
    
    
        // 创建搜索对象
        SearchRequest request = new SearchRequest();
        // 指定查询的源
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        // 添加查询条件，通过QueryBuilders获取各种查询
        sourceBuilder.query(QueryBuilders.matchAllQuery());
        // 添加排序
        sourceBuilder.sort("price", SortOrder.ASC);
        // 配置size为0，因为不需要数据，只要聚合结果
        sourceBuilder.size(0);

        // 添加聚合
        TermsAggregationBuilder termsAggregationBuilder = AggregationBuilders.terms("brandAgg").field("brand");

        // 添加子聚合
        termsAggregationBuilder.subAggregation(AggregationBuilders.avg("avgPrice").field("price"));

        sourceBuilder.aggregation(termsAggregationBuilder);

        request.source(sourceBuilder);
        // 获取结果
        SearchResponse response = client.search(request);
        // 获取聚合结果
        Aggregations aggregations = response.getAggregations();
        // 获取某个聚合
        Terms terms = aggregations.get("brandAgg");
        // 获取桶
        for (Terms.Bucket bucket : terms.getBuckets()) {
    
    
            // 获取key，这里是品牌名称
            System.out.println("品牌 : " + bucket.getKeyAsString());
            // 获取docCount，就是数量
            System.out.println("count: " + bucket.getDocCount());

            // 获取子聚合
            Avg avgPrice = bucket.getAggregations().get("avgPrice");
            System.out.println("均价：" + avgPrice.getValue());
        }
    }

response:

品牌 : 华为
count: 2
均价：3649.0
品牌 : 小米
count: 2
均价：3799.0
品牌 : 锤子
count: 1
均价：3699.0

Spring Data Elasticsearch

Spring Data Elasticsearch is the elasticsearch component provided by Spring

Introduction to Spring Data Elasticsearch

Spring Data Elasticsearch (hereinafter referred to as SDE) is a submodule under the Spring Data project.

Check Spring Data's official website: https://spring.io/projects/spring-data

The mission of Spring Data is to provide a unified programming interface for various data access, whether it is a relational database (such as MySQL), a
non-relational database (such as Redis), or an index database like Elasticsearch. Thereby simplifying the developer's code and improving
development efficiency.
Contains modules for many different data manipulations:

Spring Data Elasticsearch page: https://spring.io/projects/spring-data-elasticsearch

feature:

• Support Spring's @Configuration-based Java configuration method, or XML configuration method
• Provides a convenient tool class ElasticsearchTemplate for operating ES. Including automatic intelligent mapping between documents and POJOs.
• Feature-rich object mapping using Spring's data transformation services
• Annotation-based metadata mapping, and can be extended to support more different data formats
• The corresponding implementation method is automatically generated according to the persistence layer interface, and there is no need to manually write the basic operation code (similar to mybatis, which is automatically implemented according to the interface). Of course, manual custom queries are also supported

Version relationship:

Note : If you use Spring Boot 2.3.X version, Spring Data Elasticsearch uses 4.0.X. After version 4, many APIs are not very compatible with ElasticSearch 6.X, so the version of Spring Boot should be controlled at 2.1.X-2.2.X.

Deploying Spring Data Elasticsearch

Official documentation: https://docs.spring.io/spring-data/elasticsearch/docs/3.0.1.RELEASE/reference/html/#reference

Integration steps:

1. rely

   <parent>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-parent</artifactId>
       <version>2.1.3.RELEASE</version>
       <relativePath/> <!-- lookup parent from repository -->
   </parent>
   <properties>
       <java.version>1.8</java.version>
   </properties>
   <dependencies>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
       </dependency>
   
       <dependency>
           <groupId>org.projectlombok</groupId>
           <artifactId>lombok</artifactId>
           <optional>true</optional>
       </dependency>
       <dependency>
           <groupId>org.springframework.boot</groupId>
           <artifactId>spring-boot-starter-test</artifactId>
           <scope>test</scope>
       </dependency>
       <dependency>
           <groupId>com.alibaba</groupId>
           <artifactId>fastjson</artifactId>
           <version>1.2.70</version>
       </dependency>
   </dependencies>	
   

2. Add ES address to configuration file

   Spring Data Elasticsearch has configured various SDE configurations and registered an ElasticsearchTemplate for use

   The bottom layer of ElasticsearchTemplate does not use RestHighLevelClient provided by Elasticsearch, but TransportClient, which does not use Http protocol communication, but accesses the tcp port opened by elasticsearch to the outside world, so the port set here is: 9300 instead of 9200

   spring:
     data:
       elasticsearch:
         # ES 集群名称
         cluster-name: elasticsearch
         # 这里使用的是TransportClient 连接的是TCP端口
         cluster-nodes: localhost:9300,localhost:9301,localhost:9302
   

   
   

Index library operations (ElasticsearchTemplate)

Create an index library

Add a test class, here you need to pay attention to the SpringBoot 2.1.X test class that needs to be added @RunWith(SpringRunner.class)and injectedElasticsearchTemplate

@RunWith(SpringRunner.class)
@SpringBootTest
public class EsDemoApplicationTests {
    
    
    @Autowired
	private ElasticsearchTemplate template;
}    

Prepare an entity class

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Document(indexName = "goods",type = "_doc",shards = 3, replicas = 1)
public class Goods {
    
    
    @Id
    private Long id;
    @Field(type = FieldType.Text, analyzer = "ik_max_word")
    private String title; //标题
    @Field(type = FieldType.Keyword)
    private String category;// 分类
    @Field(type = FieldType.Keyword)
    private String brand; // 品牌
    @Field(type = FieldType.Double)
    private Double price; // 价格
    @Field(type = FieldType.Keyword, index = false)
    private String images; // 图片地址
}

Description of annotations used:

• @Document: declare index library configuration
  • indexName: index library name
  • shards: number of shards, default 5
  • replicas: number of replicas, default 1
• @Id: Declare the id of the entity class
• @Field: Declare field attributes
  • type: the data type of the field
  • analyzer: specify the tokenizer type
  • index: Whether to create an index

Create an index library

    /**
     * 创建索引
     */
    @Test
    void testCreateIndex(){
    
    
        boolean b = template.createIndex(Goods.class);
        System.out.println("结果为："+b);
    }

If you do not add the @Document annotation, you will get an error if you run it directly, as shown below:

Add @Document annotation to run the test again, you can successfully create the index, look at the index information

create mapping

The @Id and @Filed annotations are used to configure the mapping relationship, and the mapping can be created using the putMapping() method:

/**
 * 创建类型映射
 */
@Test
public void testCreateMapping() {
    
    
    boolean b = template.putMapping(Goods.class);
    System.out.println("结果为：" + b);
}

View index information

new document

    @Autowired
	private ElasticsearchTemplate template;

    @Test
    public void add(){
    
    
        Goods goods = new Goods(1L,"小米手机10Pro","手机","小米",5999.00,"/images/123.jpg");
        IndexQuery query = new IndexQuery();
        query.setObject(goods);
        String index = template.index(query);
        System.out.println(index);
    }

ElasticsearchRepository interface

ElasticsearchRepositoryIt encapsulates the basic CRUD methods and can be ElasticsearchRepositoryused by inheritance:

public interface GoodsRepository extends ElasticsearchRepository<Goods,Long> {
    
    
}

ElasticsearchRepository<T, ID>T in corresponds to the type, and ID corresponds to the primary key type.

Added and updated documentation: save

Add a single document

// 保存文档
@Test
public void testSave() {
    
    
    Goods item = new Goods(6L, "小米手机10Pro", " 手机", "小米", 4699.00, "http://image.leyou.com/13123.jpg");
    goodsRepository.save(item);
}

update document

// 更新文档
@Test
public void testUpdate() {
    
    
    Goods item = new Goods(6L, "小米手机10Pro", " 手机", "小米", 4699.00, "http://image.leyou.com/13123.jpg");
    goodsRepository.save(item);
}

Add in batches

@Test
// 批量保存文档
public void addDocuments() {
    
    
    // 准备文档数据：
    List<Goods> list = new ArrayList<>();
    list.add(new Goods(1L, "小米手机7", "手机", "小米", 3299.00, "/13123.jpg"));
    list.add(new Goods(2L, "坚果手机R1", "手机", "锤子", 3699.00, "/13123.jpg"));
    list.add(new Goods(3L, "华为META10", "手机", "华为", 4499.00, "/13123.jpg"));
    list.add(new Goods(4L, "小米Mix2S", "手机", "小米", 4299.00, "/13123.jpg"));
    list.add(new Goods(5L, "荣耀V10", "手机", "华为", 2799.00, "/13123.jpg"));
    // 添加索引数据
    goodsRepository.saveAll(list);
}

View documentation: findById

query by id

// 根据id查询
@Test
public void testQueryById(){
    
    
    Optional<Goods> goodsOptional = goodsRepository.findById(3L);
    System.out.println(goodsOptional.orElse(null));
}

Delete document: deleteById

delete by id

// 删除文档
@Test
public void testDelete(){
    
    
    goodsRepository.deleteById(6L);
}

Query all: findAll

// 查询所有
@Test
public void testQueryAll(){
    
    
    Iterable<Goods> list = goodsRepository.findAll();
    list.forEach(System.out::println);
}

custom query

The query methods provided by GoodsRepository are limited, but it provides a very powerful custom query function: as long as you follow the syntax provided by SpringData, you can define method declarations arbitrarily:

public interface GoodsRepository extends ElasticsearchRepository<Goods, Long> {
    
    
    /**
     * 根据价格区间查询
     * @param from 开始价格
     * @param to 结束价格
     * @return 符合条件的goods
     */
    List<Goods> findByPriceBetween(double from, double to);
    List<Goods> findByTitle(String title);
    List<Goods> findByBrand(String brand);
}

Example of use:

// 范围查询
@Test
public void testConditionSearch(){
    
    
    List<Goods> list = goodsRepository.findByPriceBetween(3000, 4000);
    list.forEach(System.out::println);
}
@Test
public void testTitle(){
    
    
    // List<Goods> goods = goodsRepository.findByBrand("米");
    List<Goods> goods = goodsRepository.findByBrand("小米");
    goods.forEach(System.out::println);
}

Some examples of supported syntax:

Keyword	Sample	Elasticsearch Query String
And	findByNameAndPrice	{"bool" : {"must" : [ {"field" : {"name" : "?"}}, {"field" : {"price" : "?"}} ]}}
Or	findByNameOrPrice	{"bool" : {"should" : [ {"field" : {"name" : "?"}}, {"field" : {"price" : "?"}} ]}}
Is	findByName	{"bool" : {"must" : {"field" : {"name" : "?"}}}}
Not	findByNameNot	{"bool" : {"must_not" : {"field" : {"name" : "?"}}}}
Between	findByPriceBetween	{"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : ?,"include_lower" : true,"include_upper" : true}}}}}
LessThanEqual	findByPriceLessThan	{"bool" : {"must" : {"range" : {"price" : {"from" : null,"to" : ?,"include_lower" : true,"include_upper" : true}}}}}
GreaterThanEqual	findByPriceGreaterThan	{"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : null,"include_lower" : true,"include_upper" : true}}}}}
Before	findByPriceBefore	{"bool" : {"must" : {"range" : {"price" : {"from" : null,"to" : ?,"include_lower" : true,"include_upper" : true}}}}}
After	findByPriceAfter	{"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : null,"include_lower" : true,"include_upper" : true}}}}}
Like	findByNameLike	{"bool" : {"must" : {"field" : {"name" : {"query" : "?*","analyze_wildcard" : true}}}}}
StartingWith	findByNameStartingWith	{"bool" : {"must" : {"field" : {"name" : {"query" : "?*","analyze_wildcard" : true}}}}}
EndingWith	findByNameEndingWith	{"bool" : {"must" : {"field" : {"name" : {"query" : "*?","analyze_wildcard" : true}}}}}
Contains/Containing	findByNameContaining	{"bool" : {"must" : {"field" : {"name" : {"query" : "**?**","analyze_wildcard" : true}}}}}
In	findByNameIn(Collectionnames)	{"bool" : {"must" : {"bool" : {"should" : [ {"field" : {"name" : "?"}}, {"field" : {"name" : "?"}} ]}}}}
NotIn	findByNameNotIn(Collectionnames)	{"bool" : {"must_not" : {"bool" : {"should" : {"field" : {"name" : "?"}}}}}}
Near	findByStoreNear	Not Supported Yet !
True	findByAvailableTrue	{"bool" : {"must" : {"field" : {"available" : true}}}}
False	findByAvailableFalse	{"bool" : {"must" : {"field" : {"available" : false}}}}
OrderBy	findByAvailableTrueOrderByNameDesc	{"sort" : [{ "name" : {"order" : "desc"} }],"bool" : {"must" : {"field" : {"available" : true}}}}

Condition query: matchQuery

@Test
public void testSearch(){
    
    
    // QueryBuilder query = QueryBuilders.matchAllQuery();
    QueryBuilder query = QueryBuilders.matchQuery("title","小米");
    Iterable<Goods> goods = goodsRepository.search(query);
    goods.forEach(System.out::println);
}

Paging query: search

@Test
public void testPage(){
    
    
    QueryBuilder query = QueryBuilders.matchAllQuery();
    // 设置分页 page是从0开始
    PageRequest pageable = PageRequest.of(0, 2, Sort.by(Sort.Direction.DESC, "price"));
    Page<Goods> goodsPage = goodsRepository.search(query, pageable);
    System.out.println("总数：" + goodsPage.getTotalElements());
    List<Goods> goods = goodsPage.getContent();
    goods.forEach(System.out::println);
}

ElasticsearchTemplate interface

SDE also supports the use of ElasticsearchTemplate for native queries, and the construction of query conditions is done through a class called NativeSearchQueryBuilder, but the bottom layer of this class is still using tools such as QueryBuilders, AggregationBuilders, HighlightBuilders in the native API.

highlight

To support highlighting, you must customize the result processor to implement

import com.itheima.es.entity.Goods;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.aggregations.Aggregations;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.core.SearchResultMapper;
import org.springframework.data.elasticsearch.core.aggregation.AggregatedPage;
import org.springframework.data.elasticsearch.core.aggregation.impl.AggregatedPageImpl;
import java.util.ArrayList;
import java.util.List;

public class GoodsSearchResultMapper implements SearchResultMapper {
    
    
    @Override
    public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
    
    
        SearchHits searchHits = response.getHits();
        long total = searchHits.getTotalHits();
        float maxScore = searchHits.getMaxScore();
        // 定义content
        List<T> content = new ArrayList<>();
        SearchHit[] hits = searchHits.getHits();
        // 遍历文档
        for (SearchHit hit : hits) {
    
    
            // 获取json格式数据
            String sourceAsString = hit.getSourceAsString();
            // 转换称为对象
            Goods goods = JSON.parseObject(sourceAsString, Goods.class);
            // 解析高亮字段
            String title = hit.getHighlightFields().get("title").fragments()[0].string();
            // 替换原有的title
            goods.setTitle(title);
            content.add((T) goods);
        }
        Aggregations aggregations = response.getAggregations();
        String scrollId = response.getScrollId();

        return new AggregatedPageImpl(content,pageable,total,aggregations,scrollId,maxScore);
    }
}

When querying, you need to pass in a custom result processor

	/**
     * 查询结果高亮处理
     */
    @Test
    public void testHighlight() {
    
    
        // 构建查询条件
        QueryBuilder queryBuilder = QueryBuilders.matchQuery("title", "小米");
        // 定义高亮条件
        HighlightBuilder.Field field = new HighlightBuilder.Field("title")
                .preTags("<em style='color:red'>")
                .postTags("</em>");
        // 构建查询条件并设置高亮
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(queryBuilder)
                .withHighlightFields(field)
                .build();
        AggregatedPage<Goods> aggregatedPage = template.queryForPage(query, Goods.class, new GoodsSearchResultMapper());
        List<Goods> goods = aggregatedPage.getContent();
        goods.forEach(System.out::println);
    }

View Results:

polymerization

Example:

    /**
     * 聚合
     */
    @Test
    void testAgg() {
    
    
        // 针对品牌字段做分组
        AbstractAggregationBuilder agg = AggregationBuilders.terms("brandAgg").field("brand");
        // 添加子聚合来实现平均值的计算
        agg.subAggregation(AggregationBuilders.avg("priceAvg").field("price"));
        SearchQuery query = new NativeSearchQueryBuilder()
                .withQuery(QueryBuilders.matchAllQuery())
                .addAggregation(agg)
                .build();
        AggregatedPage<Goods> aggregatedPage = template.queryForPage(query, Goods.class);
        // 获取到品牌的聚合
        Terms brandAgg = (Terms) aggregatedPage.getAggregation("brandAgg");
        for (Terms.Bucket bucket : brandAgg.getBuckets()) {
    
    
            System.out.println("品牌：" + bucket.getKeyAsString());
            System.out.println("数量：" + bucket.getDocCount());
            Avg priceAvg = bucket.getAggregations().get("priceAvg");
            System.out.println("均价：" + priceAvg.getValue());
        }
    }

result: