In a distributed cluster deployment mode, in order to maintain data consistency, usually we need to elect a leader to coordinate, from the cluster and the election of a new leader after leader hang. Election leader of the program there are many, there is agreement on Paxos and Raft had to know the students should have some understanding of the leader elections are generally in accordance with the principle of majority rule to implement, but because the distributed environment can not avoid network instability the data is not synchronized, time deviation and other issues, in order to do a good job leader election is not a particularly easy thing to do. This article will provide a simple solution to make use of Consul leader election.
principle
Consul's leader election is only two steps:
1, Create Session: Applications to participate in the elections separately to create Session, Session related to the survival of the state of health checks.
2, Acquire KV: multiple applications with the created Session to lock the same KV, only one application lock housing, lock successful application is the leader.
As shown above, by App2 is assumed here that the live lock Session KV, in fact, in order Session property KV Session2.
When will trigger the re-election of it?
- Session failure: Session is deleted, Session associated health check fails, Session TTL expired and so on.
- KV is deleted: This is no need of special note.
That application how they perceive these situations it?
Application after the election, should be kept to a KV blocking query, the query will return results on time-out or KV changed, this time to determine whether the application can initiate a new election returns based on the results.
Examples
Here is a Java example: This is a console program, the program will create a Session, and then try to use the Session Lock key is "program / leader" of Consul KV, KV will also try to set the value of the current node Id "007." Regardless of the success or failure of capture, and then will start a program for blocking "program / leader" of the query, the query returns while blocking KV will determine whether there is or there is bound to the Session, if there is any that does not exist, initiating elections, otherwise, continue blocking queries. This "blocking query -> elections" operation is an infinite loop operation.
package cn.bossma; import com.ecwid.consul.v1.ConsulClient; import com.ecwid.consul.v1.QueryParams; import com.ecwid.consul.v1.kv.model.GetValue; import com.ecwid.consul.v1.kv.model.PutParams; import com.ecwid.consul.v1.session.model.NewSession; import com.ecwid.consul.v1.session.model.Session; import org.apache.commons.lang3.StringUtils; /** * consul leader 选举演示程序 * * @author: bossma.cn */ public class Main { private static ConsulClient client = new ConsulClient(); private static String sesssionId = ""; private static String nodeId = "007"; private static String electName = "program/leader"; /** * @param args */ public static void main(String[] args) { System.out.println("starting"); watch(); } /** * 监控选举 * * @param: * @return: * @author: bossma.cn */ private static void watch() { System.out.println("start first leader election"); // 上来就先选举一次，看看结果 ElectResponse electResponse = elect(); System.out.printf("elect result: %s, current manager: %s" + System.getProperty("line.separator"), electResponse.getElectResult(), electResponse.getLeaderId()); long waitIndex = electResponse.modifyIndex++; int waitTime = 30; do { try { System.out.println("start leader watch query"); // 阻塞查询 GetValue kv = getKVValue(electName, waitTime, waitIndex); // kv被删除或者kv绑定的session不存在 if (null == kv || StringUtils.isEmpty(kv.getSession())) { System.out.println("leader missing, start election right away"); electResponse = elect(); waitIndex = electResponse.modifyIndex++; System.out.printf("elect result: %s, current manager: %s" + System.getProperty("line.separator"), electResponse.getElectResult(), electResponse.getLeaderId()); } else { long kvModifyIndex = kv.getModifyIndex(); waitIndex = kvModifyIndex++; } } catch (Exception ex) { System.out.print("leader watch异常：" + ex.getMessage()); try { Thread.sleep(3000); } catch (Exception ex2) { System.out.printf(ex2.getMessage()); } } } while (to true ); } / ** * 执行选举 * * @param : * @return : * @author : bossma.cn * / Private static ElectResponse Elect () { ElectResponse Response = new new ElectResponse (); Boolean electResult = to false ; // create an association the current node to the Session IF (StringUtils.isNotEmpty (sesssionId)) { the Session S = the getSession (sesssionId); IF ( null == S) { // here session associated health check only bind a health check consul node // actual use is recommended to check the current health of the application is also binding on, or if you just turn off the program, nor session fail sesssionId = the createSession (10 ); } } the else { sesssionId = the createSession (10 ); } // Get election Consul kV objects to be locked the GetValue kV = getKVValue (electName); IF ( null == kV) { kV = new new GetValue (); } // whoever captured KV, who is the Leader //Note: If the program is shut down soon after startup, session associated health checks may not fail, so the session will not fail // this time can be saved by sessionId program creates, first try sessionId last, after the restart electResult = acquireKV (electName, nodeId, sesssionId); // whether the election success, to get the current Leader kv = getKVValue (electName); response.setElectResult (electResult); response.setLeaderId (kv.getDecodedValue ()); response.setModifyIndex (kv.getModifyIndex ()); return the Response; } / ** * Creating the session * * @param : lockDealy session after the release of kv, kv bind again delay the session * @return : * @author: bossma.cn */ private static String createSession(int lockDelay) { NewSession session = new NewSession(); session.setLockDelay(lockDelay); return client.sessionCreate(session, QueryParams.DEFAULT).getValue(); } /** * 获取指定的session信息 * * @param: sessionId * @return: Session对象 * @author: bossma.cn */ private static Session getSession(String sessionId) { return client.getSessionInfo(sessionId, QueryParams.DEFAULT).getValue(); } /** * 使用Session捕获KV * * @param key * @param value * @param sessionId * @return * @author: bossma.cn */ public static Boolean acquireKV(String key, String value, String sessionId) { PutParams putParams = new PutParams(); putParams.setAcquireSession(sessionId); return client.setKVValue(key, value, putParams).getValue(); } / ** * Gets a value corresponding to the specified key * * @param : key * @return : * @author : bossma.cn * / Private static the GetValue getKVValue (String key) { return client.getKVValue (key) .getValue (); } / ** * Block acquires a value corresponding to the specified key * * @param : key, the waitTime, waitIndex * @return : * @author : bossma.cn * / Private static the GetValue getKVValue (String key, int waitTime, long waitIndex) { QueryParams paras = QueryParams.Builder.builder() .setWaitTime(waitTime) .setIndex(waitIndex) .build(); return client.getKVValue(key, paras).getValue(); } /** * leader选举结果 * * @author: bossma.cn */ private static class ElectResponse { private Boolean electResult = false; private long modifyIndex = 0; private String leaderId; public String getLeaderId() { return leaderId; } public void setLeaderId(String leaderId) { this.leaderId = leaderId; } public Boolean getElectResult() { return electResult; } public void setElectResult(Boolean electResult) { this.electResult = electResult; } public long getModifyIndex() { return modifyIndex; } public void setModifyIndex(long modifyIndex) { this.modifyIndex = modifyIndex; } } }
pit
1, Consul KV for elections must be locked session update, if the update by other means, will not be affected KV-bound Session, that is to say original program KV or be locked, but was modified other programs , which does not comply with the rules of the leader.
2, Session associated health check health check by default only the current node, if the application is stopped, Session and will not fail, it is proposed that the Session associated health checks include health screening applications; but if only the application of the health check, the server stops , application health check may still be healthy, so Session health check should check the health of applications and Consul nodes are incorporated.
3, if the program shut down soon after startup, session associated health checks may not fail, so the session will not fail, after the program starts If you create a new Session to lock KV, can not succeed locked KV, this time proposed to SessionId persistent storage, Session if there is, still use this Session to lock.
4, lockdelay: This is not a pit, a protection mechanism, but do not need to be considered useful. It can not allow a short session after the failure of other session to lock KV, this is because the application may still be normal, session associated health check of false positives, the application may still processing business, take some time to end deal with. You can also use this mechanism to prohibit the value 0 off.
5, may be more things to load leader, leader switching is too much trouble, considering the session associated health check false alarm problem, hope leader election program priority KV last lock, which can improve efficiency. At this point in the election process may increase in some logic: if the election when the leader was found last current program, then immediately the election; if we find the last leader is not the current program, wait two fixed period of time and then submit election.
Overall, Leader election program provided Consul is quite simple, either leader election cluster deployment, or standby deploy traditional, can be applied. The key is Session, we must combine their businesses thoughtful. Welcome to the last 800 234 939 415 Consul exchange group, to discuss various scenarios and issues Consul of use.