Author: Robod
link: https: //zhuanlan.zhihu.com/p/261054645
Source: know almost
copyrighted by the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.
Preliminary preparation
Today's task is to use ElasticSearcher to realize the function of product search. For the installation and basic use of Elasticsearch, IK tokenizer, and Kibana, please see my other article Elasticsearch Getting Started Guide .
Construction of API project for search microservices
Create a Module called changgou-service-search-api under changgou-service-api . The functions we want to implement later are all based on Spring Data ElasticSearch , so related dependencies cannot be less:
<dependencies>
<!--goods API依赖-->
<dependency>
<groupId>com.robod</groupId>
<artifactId>changgou-service-goods-api</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<!--SpringDataES依赖-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
</dependencies>
Search microservice construction
In a new changgou-service-search project under changgou-service as a micro-search service. The JavaBean and Feign interfaces of the API project need to be used in the search microservice, so search-api and goods-api are added as dependencies .
<dependencies>
<!--依赖search api-->
<dependency>
<groupId>com.robod</groupId>
<artifactId>changgou-service-search-api</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.robod</groupId>
<artifactId>changgou-service-goods-api</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>
Startup classes and configuration files are naturally indispensable
@SpringBootApplication(exclude={DataSourceAutoConfiguration.class})
@EnableEurekaClient
@EnableFeignClients(basePackages = "com.robod.goods.feign")
@EnableElasticsearchRepositories(basePackages = "com.robod.mapper")
public class SearchApplication {
public static void main(String[] args) {
//解决SpringBoot的netty和elasticsearch的netty相关jar冲突
System.setProperty("es.set.netty.runtime.available.processors", "false");
SpringApplication.run(SearchApplication.class,args);
}
}
server:
port: 18085
spring:
application:
name: search
data:
elasticsearch:
cluster-name: my-application # 集群节点的名称，就是在es的配置文件中配置的
cluster-nodes: 192.168.31.200:9300 # 这里用的是TCP端口所以是9300
eureka:
client:
service-url:
defaultZone: http://127.0.0.1:7001/eureka
instance:
prefer-ip-address: true
feign:
hystrix:
enabled: true
#超时配置
ribbon:
ReadTimeout: 500000 # Feign请求读取数据超时时间
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 50000 # feign连接超时时间
Import data into ES
Importing data from MySQL to ES is roughly divided into the following steps:
First, we need to create a JavaBean to define the relevant mapping configuration, Index, Type, Field. Create a JavaBean called SkuInfo under the com.robod.entity package of changgou-service-search-api :
@Data
@Document(indexName = "sku_info", type = "docs")
public class SkuInfo implements Serializable {
@Id
private Long id;//商品id，同时也是商品编号
/**
* SKU名称
* FieldType.Text支持分词
* analyzer 创建索引的分词器
* searchAnalyzer 搜索时使用的分词器
*/
@Field(type = FieldType.Text, analyzer = "ik_smart",searchAnalyzer = "ik_smart")
private String name;
@Field(type = FieldType.Double)
private Long price;//商品价格，单位为：元
private Integer num;//库存数量
private String image;//商品图片
private String status;//商品状态，1-正常，2-下架，3-删除
private LocalDateTime createTime;//创建时间
private LocalDateTime updateTime;//更新时间
private String isDefault; //是否默认
private Long spuId;//SPU_ID
private Long categoryId;//类目ID
@Field(type = FieldType.Keyword)
private String categoryName;//类目名称,不分词
@Field(type = FieldType.Keyword)
private String brandName;//品牌名称，不分词
private String spec;//规格
private Map<String, Object> specMap;//规格参数
}
In SkuInfo, Index is set to "sku_info", Tpye is set to "docs", and word segmentation is set for several fields. Then create a Feign interface SkuFeign under the com.robod.goods.feign package of changgou-service-goods-api :
@FeignClient(name = "goods")
@RequestMapping("/sku")
public interface SkuFeign {
/**
* 查询所有的sku数据
* @return
*/
@GetMapping
Result<List<Sku>> findAll();
}
We will use this Feign to call the findAll method in the Goods microservice to get all Sku data in the database. Finally, in the changgou-service-search microservice, write the relevant code of the Controller, Service, and Dao layers to realize the function of data import.
//SkuEsController
@GetMapping("/import")
public Result importData(){
skuEsService.importData();
return new Result(true, StatusCode.OK,"数据导入成功");
}
-----------------------------------------------------------
//SkuEsServiceImpl
@Override
public void importData() {
List<Sku> skuList = skuFeign.findAll().getData();
List<SkuInfo> skuInfos = JSON.parseArray(JSON.toJSONString(skuList), SkuInfo.class);
//将spec字符串转化成map，map的key会自动生成Field
for (SkuInfo skuInfo : skuInfos) {
Map<String,Object> map = JSON.parseObject(skuInfo.getSpec(),Map.class);
skuInfo.setSpecMap(map);
}
skuEsMapper.saveAll(skuInfos);
}
-------------------------------------------------------------
//继承自ElasticsearchRepository，泛型为SkuInfo，主键类型为Long
public interface SkuEsMapper extends ElasticsearchRepository<SkuInfo,Long> {
}
Now run the program and visit http://localhost:18085/search/import to start importing.
After a long wait, more than 90,000 pieces of data were successfully imported into ES. It takes a long time, about fifteen minutes, it may be related to the configuration of the virtual machine.
When I finished this, I submitted it to Github, and then I changed it and went back to the previously submitted version. Then I started the project and reported an error saying that the Bean injection failed. I was puzzled. This is the normal version I submitted before. Why did it go wrong. Then carefully flipped through the log and found a line
This seems to be a problem with the index, delete the index, start the project, no problem, re-import the data to ES, and it's done!
Function realization
Search by keyword
Before starting to implement this function, you must first define the format of the front-end and back-end parameters. Map is used in the video, but I think Map is not good and its readability is too poor. A better approach is to encapsulate an entity class, so I added a SearchEntity as the format of the front-end and back-end parameters in the search-api project:
@Data
public class SearchEntity {
private long total; //搜索结果的总记录数
private int totalPages; //查询结果的总页数
private List<SkuInfo> rows; //搜索结果的集合
public SearchEntity() {
}
public SearchEntity(List<SkuInfo> rows, long total, int totalPages) {
this.rows = rows;
this.total = total;
this.totalPages = totalPages;
}
}
Then write the corresponding code in the search microservice
@GetMapping
public Result<SearchEntity> searchByKeywords(@RequestParam(required = false)String keywords) {
SearchEntity searchEntity = skuEsService.searchByKeywords(keywords);
return new Result<>(true,StatusCode.OK,"根据关键词搜索成功",searchEntity);
}
---------------------------------------------------------------------------------------------------
@Override
public SearchEntity searchByKeywords(String keywords) {
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
if (!StringUtils.isEmpty(keywords)) {
nativeSearchQueryBuilder.withQuery(QueryBuilders.queryStringQuery(keywords).field("name"));
}
AggregatedPage<SkuInfo> skuInfos = elasticsearchTemplate
.queryForPage(nativeSearchQueryBuilder.build(), SkuInfo.class);
List<SkuInfo> content = skuInfos.getContent();
return new SearchEntity(content,skuInfos.getTotalElements(),skuInfos.getTotalPages());
}
Then I started the project and visited it http://localhost:18085/search?keywords=小米. The result was an error and a failed to map was reported. Then I found the following in the error message:
It probably means that there is a problem with LocalDateTime, because the Date class is not very good, so I changed it to LocaDateTime. I looked at the content in Kibana and found
It turns out that ES automatically divides LocalDateTime into multiple files, but I don't want it to be divided into multiple files, and I don't want to use Date, what should I do? I found a method on the Internet and successfully solved my problem, which is to add @JsonSerialize and @JsonDeserialize annotations , so I added a few annotations to the createTime and updateTime of SkuInfo:
/**
* 只用后两个注解就可以实现LocalDateTime不分成多个Field，但是格式不对。
* 所以还需要添加前面两个注解去指定格式与时区
**/
@Field(type = FieldType.Date, format = DateFormat.custom, pattern = "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd")
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
@JsonSerialize(using = LocalDateTimeSerializer.class)
@JsonDeserialize(using = LocalDateTimeDeserializer.class)
private LocalDateTime createTime;//创建时间
@Field(type = FieldType.Date, format = DateFormat.custom, pattern = "yyyy-MM-dd HH:mm:ss || yyyy-MM-dd")
@JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss", timezone = "GMT+8")
@JsonSerialize(using = LocalDateTimeSerializer.class)
@JsonDeserialize(using = LocalDateTimeDeserializer.class)
private LocalDateTime updateTime;//更新时间
Now import again
There is no problem with the format now, let’s test it now
OK！
Classification statistics
When we search for a product on Xiaomi Mall, the following categories will be displayed to help users further filter products. In the table design of Changbu Mall, there is also a field called categoryName. The next step is to realize the classification and statistics of the data we searched out.
What we want to achieve is the effect in the figure, but in Elasticsearch instead of MySQL.
Modify SearchEntity, add a categoryList field:
private List<String> categoryList; //分类集合
Modify the searchByKeywords method in SkuEsServiceImpl and add the code for grouping statistics:
public SearchEntity searchByKeywords(String keywords) {
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
if (!StringUtils.isEmpty(keywords)) {
nativeSearchQueryBuilder.withQuery(QueryBuilders.queryStringQuery(keywords).field("name"));
//terms: Create a new aggregation with the given name.
nativeSearchQueryBuilder.addAggregation(AggregationBuilders.terms("categories_grouping")
.field("categoryName"));
}
NativeSearchQuery nativeSearchQuery = nativeSearchQueryBuilder.build();
AggregatedPage<SkuInfo> skuInfos = elasticsearchTemplate.queryForPage(nativeSearchQuery, SkuInfo.class);
StringTerms stringTerms = skuInfos.getAggregations().get("categories_grouping");
List<String> categoryList = new ArrayList<>();
for (StringTerms.Bucket bucket : stringTerms.getBuckets()) {
categoryList.add(bucket.getKeyAsString());
}
return new SearchEntity(skuInfos.getTotalElements(),skuInfos.getTotalPages(),
categoryList,skuInfos.getContent());
}
Now let's test it again:
OK! The function of grouping statistics has been realized.
summary
This article mainly writes about building the Elasticsearch environment, and then importing the data into ES. Finally, the functions of keyword search and classification statistics are realized.