Big Data learning technology, has become a required course for many programmers, but there have been many technical forum for our discussion today so good programmers to share big data technology Zookeeper cluster management and elections, we can learn together!
1. Cluster machine monitoring
This is typically the kind used in the cluster state machine, the machine has a scene online rate higher requirements, can quickly respond to changes in the cluster machine. Such a scenario, tend to have a monitoring system, whether real-time detection machine cluster survival. Past practice is usually: monitoring system by some means (such as ping) to detect the timing of each machine or each machine its own regular reports to the monitoring system, "I'm alive." This approach works, but there are two obvious problems:
Cluster machines are subject to change when more implicated modify things.
A certain delay.
Use ZooKeeper has two properties, you can in real time to another cluster machine liveness monitoring system:
On the client node x registered a Watcher, then if x? Child of change, and will notify the client.
Creating EPHEMERAL types of nodes, once the session client and server end or expired, then the node will disappear.
For example, the monitoring system on / clusterServers a Watcher node registration, dynamic added after every machine, create a EPHEMERAL aboard the node type / clusterServers: / clusterServers / {hostname} Thus, the monitoring system will be able to know in real time by the machine. Save the case, as for the subsequent processing operations that the monitoring system.
2.Master election
In a distributed environment, the same business applications distributed on different machines, some business logic (e.g., a number of time-consuming calculation, network I / O process), often only one of the machines to make the entire cluster for execution, the remaining machines can share the results, which can greatly reduce duplication and improve performance, so the master election is the main problem encountered in this scenario.
ZooKeeper use strong consistency, to ensure the nodes created under high concurrency distributed global uniqueness, namely: there are multiple client requests to create / currentMaster node, after all, only a certain client requests to create success. Using this feature, it can easily be selected clusters in a distributed environment.
In addition, look at the evolution of this scenario, it is the dynamic Master election. This use to? EPHEMERAL_SEQUENTIAL characteristics of the type of node.
Mentioned above, all client requests to create, ultimately only one able to create success. In this slight variation, is to allow all requests can be created, but have to have the order of creation, then all requests to create the final result on a possible ZK is the case:
/ CurrentMaster / {sessionId} -1,? / CurrentMaster / {sessionId} -2,? / CurrentMaster / {sessionId} -3 ... .. each machine to select the smallest serial number as the Master, if linked to the machine, because the he created node will immediately hour, then after that the machine is the smallest of the Master.
3. Search System
In search system, if each machine in the cluster to generate a full index amount, time consuming and does not guarantee consistency between the index data with each other. So let the cluster Master to generate full amount of the index, and then sync to other machines in the cluster. In addition, disaster recovery measures Master election that can be manually specify the master at any time, that when used in zk can not obtain master information, you can obtain master to a place such as by way of http.
In Hbase, the election is to achieve dynamic HMaster use ZooKeeper. In Hbase implementation, ZK will be stored on the table and some addresses ROOT HMaster address, registered to the Zookeeper HRegionServer will put themselves in a temporary node (Ephemeral) a way that HMaster can perceive survival status of each HRegionServer at any time, At the same time, once HMaster problems, will be re-elected to a HMaster run, thus avoiding the problem of single point HMaster
学习大数据开发，可以参考好程序员提供的大数据学习路线，该学习路线提供完整的大数据开发知识体系，内容包含Linux&&Hadoop生态体系、大数据计算框架体系、云计算体系、机器学习&&深度学习。根据好程序员提供的大数据学习路线图可以让你对学习大数据需要掌握的知识有个清晰的了解，并快速入门大数据开发。