文章目录
SpringBoot整合ElasticSearch
创建项目gulimall-search，选择依赖web
导入依赖
这里的版本要和所安装的ES版本匹配。
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.12.1</version>
</dependency>
在spring-boot-dependencies中所依赖的ES版本位7.15.2，要改掉
<properties>
<java.version>11</java.version>
<elasticsearch.version>7.12.1</elasticsearch.version>
</properties>
请求测试项，比如es添加了安全访问规则，访问es需要添加一个安全头，就可以通过requestOptions设置
官方建议把requestOptions创建成单实例
@Configuration
public class GuliESConfig {
public static final RequestOptions COMMON_OPTIONS;
static {
RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
COMMON_OPTIONS = builder.build();
}
// 注入
@Bean
public RestHighLevelClient esRestClient() {
RestClientBuilder builder = null;
// 可以指定多个es
builder = RestClient.builder(new HttpHost("localhost", 9200, "http"));
RestHighLevelClient client = new RestHighLevelClient(builder);
return client;
}
}
测试
@RunWith(SpringRunner.class) // 解决@Autowired注入为null的问题
@SpringBootTest
class GulimallSearchApplicationTests {
@Autowired
..........
@Test
void contextLoads() {
}
}
保存数据
保存方式分为同步和异步，异步方式多了个listener回调
@Test
public void indexData() throws IOException {
// 设置索引
IndexRequest indexRequest = new IndexRequest ("users");
indexRequest.id("1");
User user = new User();
user.setUserName("张三");
user.setAge(20);
user.setGender("男");
String jsonString = JSON.toJSONString(user);
//设置要保存的内容，指定数据和类型
indexRequest.source(jsonString, XContentType.JSON);
//执行创建索引和保存数据
IndexResponse index = client.index(indexRequest, GulimallElasticSearchConfig.COMMON_OPTIONS);
System.out.println(index);
}
获取数据
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-search.html
@Test
public void find() throws IOException {
// 1 创建检索请求
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("bank");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
// 构造检索条件
// sourceBuilder.query();
// sourceBuilder.from();
// sourceBuilder.size();
// sourceBuilder.aggregation();
sourceBuilder.query(QueryBuilders.matchQuery("address","mill"));
System.out.println(sourceBuilder.toString());
searchRequest.source(sourceBuilder);
// 2 执行检索
SearchResponse response = client.search(searchRequest, GuliESConfig.COMMON_OPTIONS);
// 3 分析响应结果
System.out.println(response.toString());
}
{
"took":198,
"timed_out":false,
"_shards": {
"total":1,"successful":1,"skipped":0,"failed":0},
"hits":{
"total":{
"value":4,"relation":"eq"},
"max_score":5.4032025,
"hits":[
{
"_index":"bank",
"_type":"account",
"_id":"970",
"_score":5.4032025,
"_source":{
"account_number":970,"balance":19648,
"firstname":"Forbes","lastname":"Wallace","age":28,
"gender":"M","address":"990 Mill Road","employer":"Pheast",
"email":"[email protected]","city":"Lopezo","state":"AK"}
},
{
"_index":"bank","_type":"account","_id":"136",
"_score":5.4032025,
"_source":{
"account_number":136,"balance":45801,"firstname":"Winnie",
"lastname":"Holland","age":38,"gender":"M","address":"198 Mill Lane",
"employer":"Neteria","email":"[email protected]","city":"Urie","state":"IL"
}
},
{
"_index":"bank","_type":"account","_id":"345",
"_score":5.4032025,
"_source":{
"account_number":345,"balance":9812,"firstname":"Parker",
"lastname":"Hines","age":38,"gender":"M",
"address":"715 Mill Avenue","employer":"Baluba","email":"[email protected]",
"city":"Blackgum","state":"KY"
}
},
{
"_index":"bank",
"_type":"account","_id":"472",
"_score":5.4032025,
"_source":{
"account_number":472,"balance":25571,"firstname":"Lee","lastname":"Long",
"age":32,"gender":"F","address":"288 Mill Street","employer":"Comverges",
"email":"[email protected]","city":"Movico","state":"MT"
}
}
]
}
}
@Test
public void find() throws IOException {
// 1 创建检索请求
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("bank");
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
// 构造检索条件
// sourceBuilder.query();
// sourceBuilder.from();
// sourceBuilder.size();
// sourceBuilder.aggregation();
sourceBuilder.query(QueryBuilders.matchQuery("address","mill"));
//AggregationBuilders工具类构建AggregationBuilder
// 构建第一个聚合条件:按照年龄的值分布
TermsAggregationBuilder agg1 = AggregationBuilders.terms("agg1").field("age").size(10);// 聚合名称
// 参数为AggregationBuilder
sourceBuilder.aggregation(agg1);
// 构建第二个聚合条件:平均薪资
AvgAggregationBuilder agg2 = AggregationBuilders.avg("agg2").field("balance");
sourceBuilder.aggregation(agg2);
System.out.println("检索条件"+sourceBuilder.toString());
searchRequest.source(sourceBuilder);
// 2 执行检索
SearchResponse response = client.search(searchRequest, GuliESConfig.COMMON_OPTIONS);
// 3 分析响应结果
System.out.println(response.toString());
}
转换bean
//获取java bean
SearchHits hits = response.getHits();
SearchHit[] hits1 = hits.getHits();
for (SearchHit hit : hits1) {
hit.getId();
hit.getIndex();
String sourceAsString = hit.getSourceAsString();
Account account = JSON.parseObject(sourceAsString, Account.class);
System.out.println(account);
}
Account(accountNumber=970, balance=19648, firstname=Forbes, lastname=Wallace, age=28, gender=M, address=990 Mill Road, employer=Pheast, email=forbeswallace@pheast.com, city=Lopezo, state=AK)
Account(accountNumber=136, balance=45801, firstname=Winnie, lastname=Holland, age=38, gender=M, address=198 Mill Lane, employer=Neteria, email=winnieholland@neteria.com, city=Urie, state=IL)
Account(accountNumber=345, balance=9812, firstname=Parker, lastname=Hines, age=38, gender=M, address=715 Mill Avenue, employer=Baluba, email=parkerhines@baluba.com, city=Blackgum, state=KY)
Account(accountNumber=472, balance=25571, firstname=Lee, lastname=Long, age=32, gender=F, address=288 Mill Street, employer=Comverges, email=leelong@comverges.com, city=Movico, state=MT)
Buckets分析信息
//获取检索到的分析信息
Aggregations aggregations = response.getAggregations();
Terms agg21 = aggregations.get("agg2");
for (Terms.Bucket bucket : agg21.getBuckets()) {
String keyAsString = bucket.getKeyAsString();
System.out.println(keyAsString);
}
搜索address中包含mill的所有人的年龄分布以及平均年龄，平均薪资
扫描二维码关注公众号，回复：
14580504 查看本文章
GET bank/_search
{
"query": {
"match": {
"address": "Mill"
}
},
"aggs": {
"ageAgg": {
"terms": {
"field": "age",
"size": 10
}
},
"ageAvg": {
"avg": {
"field": "age"
}
},
"balanceAvg": {
"avg": {
"field": "balance"
}
}
}
}
product-es
ES在内存中，所以在检索中优于mysql。ES也支持集群，数据分片存储。
需求：
- 上架的商品才可以在网站展示。
- 上架的商品需要可以被检索。
分析sku在es中如何存储
商品mapping
分析：商品上架在es中是存sku还是spu？
- 1、检索的时候输入名字，是需要按照sku的title进行全文检索的
- 2、检素使用商品规格，规格是spu的公共属性，每个spu是一样的
- 3、按照分类id进去的都是直接列出spu的，还可以切换。
- 4、我们如果将sku的全量信息保存到es中（包括spu属性〕就太多字段了
方案1：
{
skuId:1
spuId:11
skyTitile:华为xx
price:999
saleCount:99
attr:[
{
尺寸:5},
{
CPU:高通945},
{
分辨率:全高清}
]
缺点：如果每个sku都存储规格参数(如尺寸)，会有冗余存储，因为每个spu对应的sku的规格参数都一样
优点：方便检索
方案2：
sku索引
{
skuId:1
spuId:11
}
attr索引
{
spuId:11
attr:[
{
尺寸:5},
{
CPU:高通945},
{
分辨率:全高清}
]
}
先找到4000个符合要求的spu，再根据4000个spu查询对应的属性，封装了4000个id，long 8B*4000=32000B=32KB
1K个人检索，就是32MB
结论：如果将规格参数单独建立索引，会出现检索时出现大量数据传输的问题，会引起网络卡顿
因此选用方案1，以空间换时间
建立product索引
最终选用的数据模型：也就是es中存的信息
- { “type”: “keyword” }, ：保持数据精度问题，可以检索，但不分词
- “analyzer”: “ik_smart” ：中文分词器
- “index”: false, ：不可被检索，不生成index
- “doc_values”: false ：不可被聚合，es就不会维护一些聚合的信息， 默认为true
PUT product
{
"mappings":{
"properties": {
"skuId":{
"type": "long" },
"spuId":{
"type": "keyword" }, # 不可分词
"skuTitle": {
"type": "text",
"analyzer": "ik_smart" # 中文分词器
},
"skuPrice": {
"type": "keyword" }, # 保证精度问题
"skuImg" : {
"type": "keyword" }, # 视频中有false
"saleCount":{
"type":"long" },
"hasStock": {
"type": "boolean" },
"hotScore": {
"type": "long" },
"brandId": {
"type": "long" },
"catalogId": {
"type": "long" },
"brandName": {
"type": "keyword"}, # 视频中有false
"brandImg":{
"type": "keyword",
"index": false, # 不可被检索，不生成index，只用做页面使用
"doc_values": false # 不可被聚合，默认为true
},
"catalogName": {
"type": "keyword" }, # 视频里有false
"attrs": {
"type": "nested",
"properties": {
"attrId": {
"type": "long" },
"attrName": {
"type": "keyword",
"index": false,
"doc_values": false
},
"attrValue": {
"type": "keyword" }
}
}
}
}
}
冗余存储的字段：不用来检索，也不用来分析，节省空间
nested嵌入式对象
属性是"type": “nested”,因为是内部的属性进行检索
数组类型的对象会被扁平化处理（对象的每个属性会分别存储到一起）
user.name=["aaa","bbb"]
user.addr=["ccc","ddd"]
这种存储方式，可能会发生如下错误：
错误检索到{
aaa,ddd}，这个组合是不存在的
数组的扁平化处理会使检索能检索到本身不存在的，为了解决这个问题，就采用了嵌入式属性，数组里是对象时用嵌入式属性（不是对象无需用嵌入式属性）
商品上架
- 1、后台上架商品，传入spuId给服务，通过spuId查出当前spuid对应的所有sku信息，品牌的名字，还要查询当前sku的所有可以被用来检索的规格属性
- 2、通过远程调用库存服务，传入skuIdList，查询出当前skuIdList是否有库存，然后一一对应赋值给SkuEsModel的hasStock属性，然后再封装好es需要的vo对象SkuEsModel
- 3、通过远程调用es服务，传入
List<SkuEsModel>保存到es中，保存完成修改spuId的状态
根据spuId封装上架数据
@PostMapping("/{spuId}/up")
public R spuUp(@PathVar iable("spuId") Long spuId){
spuInfoService.up(spuId);
return R.ok();
}
public void up(Long spuId) {
// 1 查出当前spuid对应的所有sku信息，品牌的名字
List<SkuInfoEntity> skus = skuInfoService.getSkusBySpuId(spuId);
// 快速回去skus集合中的属性id的集合skuIdList
List<Long> skuIdList = skus.stream().map(SkuInfoEntity::getSkuId).collect(Collectors.toList());
// TODO 4 查询当前sku的所有可以被用来检索的规格属性
List<ProductAttrValueEntity> baseAttrs = attrValueService.baseAttrlistforspu(spuId);
List<Long> attrIds = baseAttrs.stream().map(attr -> {
return attr.getAttrId();
}).collect(Collectors.toList());
// 在指定的所有属性里面，挑出检索属性
List<Long> searchAttrIds = attrService.selectSearchAttrIds(attrIds);
Set<Long> idSet = new HashSet<>(searchAttrIds);
List<SkuEsModel.Attrs> attrsList = baseAttrs.stream().filter(item -> {
return idSet.contains(item.getAttrId());
}).map(item -> {
SkuEsModel.Attrs attrs1 = new SkuEsModel.Attrs();
BeanUtils.copyProperties(item, attrs1);
return attrs1;
}).collect(Collectors.toList());
// TODO 1 发送远程调用，库存系统查询是否有库存
Map<Long,Boolean> hasStockMap = null;
try {
R r = wareFeignService.getSkuHasStock(skuIdList);
hasStockMap = r.getData(new TypeReference<List<SkuHasStockTo>>(){
}).stream().collect(Collectors.toMap(SkuHasStockTo::getSkuId, item -> item.getHasStock()));
// Collectors.toMap(SkuHasStockTo::getSkuId, item -> item.getHasStock())
// 将SkuHasStockTo的id作为map的key，item的hasStock作为value封装成map
} catch (Exception e) {
log.error("库存服务查询异常原因: {}",e);
}
// 2 封装每个sku的信息
Map<Long, Boolean> finalStockMap = hasStockMap;
List<SkuEsModel> upProducts = skus.stream().map(sku -> {
// 组装需要的数据
SkuEsModel esModel = new SkuEsModel();
BeanUtils.copyProperties(sku,esModel);
esModel.setSkuPrice(sku.getPrice());
esModel.setSkuImg(sku.getSkuDefaultImg());
if (finalStockMap == null) {
esModel.setHasStock(true);
} else {
esModel.setHasStock(finalStockMap.get(sku.getSkuId()));
}
// TODO 热度评分
esModel.setHotScore(0L);
// TODO 3 查询品牌和查询分类的名字信息
BrandEntity brand = brandService.getById(esModel.getBrandId());
esModel.setBrandName(brand.getName());
esModel.setBrandImg(brand.getLogo());
CategoryEntity category = categoryService.getById(esModel.getCatalogId());
esModel.setCatalogName(category.getName());
// 设置检索属性
esModel.setAttrs(attrsList);
return esModel;
}).collect(Collectors.toList());
// TODO 5 将数据发送到es进行保存：gulimall-search
R r = searchFeignService.productStatusUp(upProducts);
if (r.getCode() == 0) {
// 远程调用成功
// TODO 6 修改当前的spu的状态，并且加上修改时间。update_time = now();
baseMapper.updateSpuState(spuId, ProductConstant.StatusEnum.SPU_UP.getCode());
} else {
// 远程调用失败
// 重复调用？接口幂等性,chong
}
}
product里组装好，search里上架
上架实体类
商品上架需要在es中保存spu信息并更新spu的状态信息，由于SpuInfoEntity与索引的数据模型并不对应，所以我们要建立专门的vo进行数据传输
@Data
public class SkuEsModel {
//common中
private Long skuId;
private Long spuId;
private String skuTitle;
private BigDecimal skuPrice;
private String skuImg;
private Long saleCount;
private boolean hasStock; // 是否有库存
private Long hotScore; // 热度评分
private Long brandId;
private Long catalogId;
private String brandName;
private String brandImg;
private String catalogName;
private List<Attr> attrs;
@Data
public static class Attr{
private Long attrId;
private String attrName;
private String attrValue;
}
}
库存量查询
上架要确保还有库存，也就是远程调用的时候查询库存服务
1、在ware微服务里添加"查询sku是否有库存"的controller
// sku的规格参数相同，因此我们要将查询规格参数提前，只查询一次
/**
* 查询sku是否有库存
* 返回skuId 和 stock库存量
*/
@PostMapping("/hasStock")
public R getSkuHasStock(@RequestBody List<Long> SkuIds){
List<SkuHasStockVo> vos = wareSkuService.getSkuHasStock(SkuIds);
return R.ok().setData(vos);
}
然后用feign调用
2、设置R的时候最后设置成泛型的
3、收集成map的时候，toMap()参数为两个方法，如SkyHasStockVo::getSkyId, item->item.getHasStock()
gulimall-search
- 将封装好的SkuInfoEntity，调用search的feign，保存到es中
/*** 上架商品*/
@PostMapping("/product") // ElasticSaveController
public R productStatusUp(@RequestBody List<SkuEsModel> skuEsModels){
boolean status;
try {
status = productSaveService.productStatusUp(skuEsModels);
} catch (IOException e) {
log.error("ElasticSaveController商品上架错误: {}", e);
return R.error(BizCodeEnum.PRODUCT_UP_EXCEPTION.getCode(), BizCodeEnum.PRODUCT_UP_EXCEPTION.getMsg());
}
if(!status){
return R.ok();
}
// 商品上架异常
return R.error(BizCodeEnum.PRODUCT_UP_EXCEPTION.getCode(), BizCodeEnum.PRODUCT_UP_EXCEPTION.getMsg());
}
public class ProductSaveServiceImpl implements ProductSaveService {
@Resource
private RestHighLevelClient client;
/**
* 将数据保存到ES
* 用bulk代替index，进行批量保存
* BulkRequest bulkRequest, RequestOptions options
*/
@Override // ProductSaveServiceImpl
public boolean productStatusUp(List<SkuEsModel> skuEsModels) throws IOException {
// 1.给ES建立一个索引 product
BulkRequest bulkRequest = new BulkRequest();
// 2.构造保存请求
for (SkuEsModel esModel : skuEsModels) {
// 设置es索引，EsConstant.PRODUCT_INDEX
IndexRequest indexRequest = new IndexRequest(EsConstant.PRODUCT_INDEX);
// 设置索引id
indexRequest.id(esModel.getSkuId().toString());
// json格式，数据
String jsonString = JSON.toJSONString(esModel);
indexRequest.source(jsonString, XContentType.JSON);
// 添加到文档
bulkRequest.add(indexRequest);
}
// bulk批量保存
BulkResponse bulk = client.bulk(bulkRequest, GuliESConfig.COMMON_OPTIONS);
// TODO 是否拥有错误
boolean hasFailures = bulk.hasFailures();
if(hasFailures){
List<String> collect = Arrays.stream(bulk.getItems()).map(item -> item.getId()).collect(Collectors.toList());
log.error("商品上架错误：{}",collect);
}
return hasFailures;
}
}
5、上架失败返回R.error(错误码,消息)
此时再定义一个错误码枚举。在接收端获取他返回的状态码
6、上架后再让数据库中变为上架状态
gulimall-search
依赖：thymeleaf
修改源文档index.html中的路径，加上/static前缀，交由nginx响应
修改hosts search.gulimall.com
修改nginx的配置文件 *.gulimall.com; 要注意这种配置方式不包含gulimall.com
server_name gulimall.com *.gulimall.com;
修改index.html成list.html。添加对应controller