A, ES cluster management
1.1, ES is how to address the high concurrency
ES is a distributed full-text retrieval framework hides the complexity of handling mechanism, the core of fragmentation mechanism, the cluster was found, fragmentation load balancing request routing.
1.2, ES basic concept nouns
（1）Cluster
On behalf of a cluster.
There are multiple nodes in the cluster, in which a master node (selected by election) This is an internal cluster for the. ES is to the center of the (non-central node), which is outside of the cluster is, since the external view ES cluster as a whole, the entire communication and ES cluster communication with any node are equivalent.
(2) Shards-- main fragments
Representative index fragmentation.
ES can be a complete index into a plurality of fragments, a big advantage is the index into multiple, distributed to different nodes, distributed search configuration. (Can not be changed after the number of fragments can only be created before the specified index, created)
(3) replicas-- sub-fragment
Representative index copy.
ES can be said that a plurality of copies of the index, a copy of the action: the system is to improve the fault tolerance, can be restored from the copy of a node when a fragment is corrupted or lost; effect 2: improve the query efficiency of ES, ES will automatic search request load balancing.
Note: call the shots fragmentation and vice fragmentation, in order to do fault tolerance
Note: number of fragments, is set to a server (node) is preferably a square
（4）Recovery
On behalf of Data Recovery (data restore).
ES will be performed when a node joins and leaves the machine according to the load index slice reallocation, data recovery will hang when the node restarts.
1.3, ES Why cluster
ES cluster index may be composed of a plurality of slices, and each slice can have multiple copies. By a single index into a plurality of slices, it can not handle large index running on a single server above. Since each tile has one or more copies, the copy distributed to multiple servers, may improve the load capacity of the query .
1.4, ES cluster analysis principle
(1) is divided into a plurality of indexes each fragmented shards stored (default 5 slices), each slice will be deployed on the distributed deployment of a plurality of different nodes.
Note: after a good index of the main primary shards defined fragment, behind not modified.
(2) a main fragment with a corresponding backup piece replics shards, replic shards respectively carry fault tolerance, load balancing requests.
Note: fragments corresponding to the primary backup slice can not be placed on the same server, the master slice can be primary shards and other sub-fragments replics shards can not be placed in a node on the same node (server).
1.5, some frequently asked questions
(1) why the Lord after a good definition of fragmentation can not be modified (Vice fragmentation can be changed)?
(1) Why can not the primary slice and secondary slice on the same server?
Second, build a clustered environment
(1) prepare three server cluster
| name of server |
IP addresses |
| node-1 |
192.168.212.182 |
| node-2 |
192.168.212.183 |
| node-3 |
192.168.212.184 |
(2) Service cluster configuration
One server configuration:
| vi elasticsearch.yml cluster.name: myes ### to ensure that the same three server nodes in the cluster name node.name: node-1 #### of each node name is not the same as the other two node-1, node-2 network.host: 192.168.212.180 #### actual server ip address discovery.zen.ping.unicast.hosts: [ "192.168.212.184", "192.168.212.185", "192.168.212.186"] more than ## service cluster ip discovery.zen.minimum_master_nodes: 1 |
Turn off the firewall systemctl stop firewalld.service
Default underlying open cluster 9300
The other two cloned directly on the line.
(3) Verifying the cluster effect
http://192.168.212.185:9200/_cat/nodes?pretty
Note clone data file does not cause data synchronization
The reported error solution
failed to send join request to master
Because the data files can also lead to cloning clone directly to clear each server data files.
Three, SpringBoot integrate ES
Case: SpringBoot ES achieve integration network disk search engine
3.1, a data structure analysis
OST /clouddisk/_mapping/disk
{
"disk": {
"properties": {
"baiduaddres": {
"type": "keyword"
},
"browsetimes": {
"type": "long"
},
"collectiontime": {
"type": "date"
},
"describe": {
"type": "text",
"analyzer": "ik_smart"
},
"filesize": {
"type": "float"
},
"name": {
"type": "text",
"analyzer": "ik_smart"
},
"sharpeople": {
"type": "keyword"
},
"shartime": {
"type": "date"
},
"source": {
"type": "keyword"
}
}
}
}
3.2, Maven relies
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.0.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>com.google.collections</groupId>
<artifactId>google-collections</artifactId>
<version>1.0-rc2</version>
</dependency>
<!-- springboot整合freemarker -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-freemarker</artifactId>
</dependency>
</dependencies>
3.3, application information
spring:
data:
elasticsearch:
####集群名称
cluster-name: myes
####地址
cluster-nodes: 192.168.212.180:9300
freemarker:
# 设置模板后缀名
suffix: .ftl
# 设置文档类型
content-type: text/html
# 设置页面编码格式
charset: UTF-8
# 设置页面缓存
cache: false
# 设置ftl文件路径
template-loader-path:
- classpath:/templates
# 设置静态文件路径，js,css等
mvc:
static-path-pattern: /static/**
Three-tier structure:
3.4, layer entity class
@Data
@Document(indexName = "clouddisk", type = "disk")
public class CloudDiskEntity {
@Id
private String id;
// 名称
private String name;
// 来源
private String source;
// 描述
private String describe;
// 分享时间
private Date shartime;
// 浏览次数
private Long browsetimes;
// 文件大小
private Double filesize;
// 分享人
private String sharpeople;
// 收录时间
private String collectiontime;
// 地址
private String baiduaddres;
3.5, repository (dao layer)
ElasticsearchRepository is to integrate good mine construction, you can use the direct successor
public interface CloudDiskDao extends ElasticsearchRepository<CloudDiskEntity, String> {
}
3.6, the control layer
@RestController
public class CloudDiskController {
@Autowired
private CloudDiskDao cloudDiskDao;
// springboot 整合 es 查询
// 根据id查询文档信息
@RequestMapping("/findById/{id}")
public Optional<CloudDiskEntity> findById(@PathVariable String id) {
return cloudDiskDao.findById(id);
}
// 实现分页查询
@RequestMapping("/search")
public List<CloudDiskEntity> search(String name, String describe,
@PageableDefault(page = 0, value = 2) Pageable pageable) {
// 1.创建查询对象
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
if (!StringUtils.isEmpty(name)) {
MatchQueryBuilder matchQuery = QueryBuilders.matchQuery("name", name);
boolQuery.must(matchQuery);
}
if (!StringUtils.isEmpty(describe)) {
MatchQueryBuilder matchQuery = QueryBuilders.matchQuery("describe", describe);
boolQuery.must(matchQuery);
}
// 2.调用查询接口
Iterable<CloudDiskEntity> search = cloudDiskDao.search(boolQuery, pageable);
// 3.将迭代器转换为集合
return Lists.newArrayList(search);
}
}
Conditions inquiry:
@Controller
public class SearchController {
@Autowired
private CloudDiskDao cloudDiskDao;
@RequestMapping("/search")
public String search(String keyword, String describe, @PageableDefault(page = 0, value = 5) Pageable pageable,
HttpServletRequest req) {
Long startTime = System.currentTimeMillis();
// 1.创建查询对象
BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
if (!StringUtils.isEmpty(keyword)) {
//模糊查询
MatchQueryBuilder matchQuery = QueryBuilders.matchQuery("name", keyword);
boolQuery.must(matchQuery);
}
if (!StringUtils.isEmpty(describe)) {
//精确查询
MatchQueryBuilder matchQuery = QueryBuilders.matchQuery("describe", describe);
boolQuery.must(matchQuery);
}
// 2.调用查询接口
Page<CloudDiskEntity> page = cloudDiskDao.search(boolQuery, pageable);
req.setAttribute("page", page);
// 记录总数
req.setAttribute("total", page.getTotalElements());
req.setAttribute("keyword", keyword);
// 计算分页总数
int totalPage = (int) ((page.getTotalElements() - 1) / pageable.getPageSize() + 1);
// 分页总数
req.setAttribute("totalPage", totalPage);
Long emdTime = System.currentTimeMillis();
req.setAttribute("time", emdTime - startTime);
return "search";
}
}
