introduction
ELasticsearch as a distributed search engine, can well support the cluster model, the level of dynamic expansion, failover distributed system characteristics, which is an important reason for their choice as the full-text search engine.
This article describes the configuration from scratch and expansion process of the cluster, so that you have a preliminary understanding how Elasticsearch cluster.
First, when a plurality Elasticsearch cluster nodes, nodes within the same network cluster by designating the same added cluster.name, all nodes share the load and pressure data. When a node joins a cluster or remove a node from the cluster, the cluster will be re-distributed evenly all the data.
Elasticsearch node divided into the main node and data node, the master node is responsible for management and coordination Elasticsearch clusters, including the increase of the index, delete, add nodes, removal, etc., but the master node is not responsible for data storage and search, which makes the master node does not have too much pressure, but remains lightweight state. Data node is responsible for data storage and search.
Whether the node as a master node can configure settings node.master, true represents as a master node, false not expressed as a master node.
A data node whether the node can configure settings node.data, true data node as represented, false indicates not serve as data nodes.
By default node.master and node.data are true, special attention node.master set to true only that this node qualify as a primary node, but does not necessarily mean becoming a master node, the master node cluster is generated by the election, the specific re-election strategy follow-up article to explain.
A cluster configuration
1.1 empty cluster
When we start a Elasticsearch node, the default is to create a Elasticsearch cluster, but this is only one cluster node. At this point if there is no index is created, the cluster are in a state of "empty".
View cluster status at this time by _cluster API
$ curl -X GET "localhost:9200/_cluster/health?pretty"
As a result, you can see the status is green, the number of nodes number_of_nodes to 1, and the number of nodes is also number_of_data_nodes 1, because the index is not created, the number of slices is 0.
status field indicates whether the current cluster is working properly in general. Its three colors have the following meanings:
- green all the main fragmentation and copy fragments are operating normally
- yellow all the main fragments are operating normally, but not all copies of fragments are operating normally.
- red main fragment could not run properly.
Because there is no copy of the master slice slices not normal operation, the status is green does not currently exist.
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
1.2 single-node cluster
Adding an index for the current cluster, the specified number of sheets of the main distributor 3, the number of copies of a slice.
$ curl -X PUT "localhost:9200/cumstomer?pretty" -H 'Content-Type: application/json' -d'
{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}
'
At this point of view the cluster state again
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 3,
"active_shards" : 3,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 3,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
At this time, the master can see the number of active slice 3, slice number unfragmented 3 is also not assigned three main fragments of the main fragment corresponding replica fragmentation, since the main and sub-copy sheet piece can not exist in the same node, the copy of the slice can not be allocated, then the state of the cluster is yellow.
1.2 two-node cluster
As noted above, because a copy of the fragments can not be assigned with the main fragment in the same node, then to achieve the so-called copy fragments can be properly allocated, you need more than one node. To add a new node cluster, you only need to specify the new cluster node name cluster.name same as the first node can be added to the cluster, the cluster will be re-slice after slice automatically added, due to the presence of three copies of the minutes sheet is not allocated, the three copies of the fragment will be assigned to the new node, as shown in FIG.
At this point of view the cluster state can be found, all fragments are normal distribution, the status of this group becomes green.
{
"cluster_name": "elasticsearch",
"status": "green",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 3,
"active_shards": 6,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}
Second, the level of expansion
2.1 Expansion of the main fragments
After adding two nodes after the above we can see three main fragments are in the same node, this node shared CPU, IO, and memory resources like this when the system traffic increased performance bottlenecks easily appear to be consider adding nodes to share a portion of the master slice to improve performance. For example, adding a node, the cluster consisting of three nodes, the cluster distribution sheet as shown carve
Node may be increased up to 6 nodes, each node is assigned an average of a slice can be maximized to improve the performance of each tile. It is noted here, continue to increase node has been unable to extend the main fragments, so the number of primary fragments determines the capacity of the entire cluster, during the design of the index when the need for special attention, careful planning.
2.2 copy of the fragmentation of expansion
Although the primary fragmentation can not continue expansion, but it is a copy of fragmentation can be dynamic expansion, increase the dynamic fragmentation hand makes high availability more, on the other hand a copy of the slices can provide access to search functions, multiple copies of fragments performance of the search can be extended, for example to modify the number of copies is 2 fragments.
$ curl -X PUT "localhost:9200/customer/_settings?pretty" -H 'Content-Type: application/json' -d'
{
"number_of_replicas" : 2
}
'
Modified slice allocated as shown below, a total of nine slices, slice two copies, respectively, on two nodes, two nodes can provide search service searches, the performance has been promoted to great. Of course, we can also continue to increase node and a copy of the growing fragmentation search performance.
Third, failover
We have repeatedly mentioned above, multiple copies be fragmented in order to ensure high availability of Elasticsearch. Now how we ensure this availability in the simulation of the following Elastic search fails.
Chief stopped Node 1 node, Node1 node is a master node, including P1, P2, R0 three fragmentation, necessary due to the operation of the cluster Elasticsearch have a master node, it will be a master re-election in the remaining two nodes node, and due to P1, P2 master slice lost, it will choose to upgrade two fragments from two copies of R1 and R2 is mainly sliced continue to provide services, the state following the final cluster, because this time P1, P2 are only one copy of the fragment, and we specify that two copies of the fragment, there is a copy of the slices are not working, so the cluster status is yellow.
Fourth, distributed indexing and search
Cluster contains more than one node, an index that contains multiple slices, then how the data is indexed it? And how to search it?
In fact, we can access any node in the cluster to obtain a complete data set, depending on Elasticsearch is distributed indexing and search method.
When we index a document, elasticsearch will first hash calculated from the ID of the document, by selecting a hash value of a master slice, and then sends the document to the slice, the slice is not necessarily in this node accessed node can be any node in the other, but of course the accessed automatically forwards the request to complete all copies of the document after the fragmentation points felled primary storage, the document will be sent to the primary slice go to achieve and the main sub-sliced data.
When we search for a document, also sends a request to any node, regardless of the data node or the master node, even node is neither the data nor the primary node, but are in this cluster, with a search function. After return to the client node receives a request to use the round-robin polling mechanism selection available primary copy fragment or fragments, the request is forwarded in the past, then all of the response data returned from respective nodes polymerization.
From this we can see that increasing the number of horizontal extension node is a good solution to improve search performance Elasticsearch, even if an ordinary node is not for data storage, calculation can be performed in response to the request and forwards and the polymerization results, the main reduction pressure nodes and data nodes to improve the overall performance of the system.
more content
- Blog: http://blog.liujintao.tech
- No public: Programming road