Redhat cluster 的心跳机制

Redhat cluster软件的架构分为两层：

• 底层的cluster messaging layer（集群信息层）负责在所有节点之间传递集群信息包括心跳信息
  RHEL6及之前是CMAN，从RHEL7开始是Corosync。
• 上层的cluster resource manager（集群资源管理器）管理划归集群所属的资源，包括IP地址、逻辑卷、文件系统等
  RHEL6及之前是RGManager，从RHEL7开始是Pacemaker。

心跳机制使用的是Totem协议，Totem是一个巨复杂的协议，心跳只是其中一个很小的部分，可以这么理解：Totem协议负责处理集群内部所有节点之间的通信问题，包含四个主要组件：Total Ordering Protocol, Membership Protocol, Recovery Protocol, Flow Control Mechanism。这里只讲与心跳有关的部分。

Totem的通信是两种方式互相配合进行的：multicast（多播）和token（令牌）。[注意multicast与broadcast(广播)不同，它相当于限定范围的广播，只有分组内的成员能收到。] 一个节点只有在持有token的时候才能发言，通过multicast给集群内所有节点发送信息，每一条multicast信息都附有顺序号，发送完成之后就把顺序号记录到token中，然后把token传给下一个节点，token的传递方式是一对一的，就像首尾相连的环形接力，从IP地址最小的节点依次传给下一个IP地址更大的节点，每个节点收到token之后，根据token里记录的顺序号检查自己是否漏收了multicast信息，如果漏收了，就在token里添加重传申请，然后把token传给下一个节点，下一个节点除了检查自己有没有漏收multicast之外，还会根据token里记录的重传申请把相应的multicast信息重新发送一次（当然前提是该节点已经收到了这条multicast信息）。

集群通过两种机制检测节点的健康状态：

• Token超时

如果token中断的时间超过了指定的期限，节点就会触发membership protocol，重组cluster，这个期限通过以下参数设置：<totem token=”XXX”/>
注：XXX以毫秒为单位。

在上述期限内，token会尝试重传，重传的次数是以下参数指定的，缺省值是4：
<totem token_retransmits_before_loss_const=”X”/>

• Multicast retransmit(重传)超过阈值

如果某个节点能收到token，但收不到multicast，那么经过若干次token循环之后，也会触发membership protocol，重组cluster，这个次数可以通过参数 fail_recv_const 设置，缺省值是2500次。

Totem协议有4种状态，代表集群运行的不同阶段，不同的状态下运行的子协议也不同：

• GATHER
• COMMIT
  Gather和Commit状态出现在集群组建的阶段，可以比喻为集合、报数的过程。运行的是membership protocol。
• RECOVERY
  Recovery状态出现在集群发生问题、进行恢复的阶段。运行的是recovery protocol。
• OPERATIONAL
  Operational状态表示集群正常工作的阶段。运行的是Totem Ordering Protocol。

有了这些基本概念，我们就可以大致看懂集群的日志了：

// 本来正常的集群突然发生token丢失（心跳信号丢失）： Apr 26 01:47:26 node1 openais[4252]: [TOTEM] The token was lost in the OPERATIONAL state. Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes). Apr 26 01:47:26 node1 openais[4252]: [TOTEM] Transmit multicast socket send buffer size (320000 bytes). // 集群开始重组，进入Gather状态，召集各节点： Apr 26 01:47:26 node1 openais[4252]: [TOTEM] entering GATHER state from 2. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering GATHER state from 0. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Creating commit token because I am the rep. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Storing new sequence id for ring 798 // 进入Commit状态： Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering COMMIT state. // 进入Recovery状态： Apr 26 01:47:28 node1 openais[4252]: [TOTEM] entering RECOVERY state. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] position [0] member 10.23.201.30: Apr 26 01:47:28 node1 openais[4252]: [TOTEM] previous ring seq 1940 rep 10.23.201.30 Apr 26 01:47:28 node1 openais[4252]: [TOTEM] aru 2b high delivered 2b received flag 1 Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Did not need to originate any messages in recovery. Apr 26 01:47:28 node1 openais[4252]: [TOTEM] Sending initial ORF token Apr 26 01:47:28 node1 openais[4252]: [CLM ] CLM CONFIGURATION CHANGE Apr 26 01:47:28 node1 openais[4252]: [CLM ] New Configuration: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.30) Apr 26 01:47:28 node1 kernel: dlm: closing connection to node 2 Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Left: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.31) Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Joined: Apr 26 01:47:28 node1 openais[4252]: [CLM ] CLM CONFIGURATION CHANGE Apr 26 01:47:28 node1 openais[4252]: [CLM ] New Configuration: Apr 26 01:47:28 node1 openais[4252]: [CLM ] r(0) ip(10.23.201.30) Apr 26 01:47:28 node1 openais[4252]: [CLM ] Members Left: Apr 26 01:47:29 node1 openais[4252]: [CLM ] Members Joined: Apr 26 01:47:29 node1 openais[4252]: [SYNC ] This node is within the primary component and will provide service. // 集群重组完成，进入Operational状态，只剩下一个节点： Apr 26 01:47:29 node1 openais[4252]: [TOTEM] entering OPERATIONAL state. Apr 26 01:47:29 node1 openais[4252]: [CLM ] got nodejoin message 10.23.201.30 Apr 26 01:47:29 node1 openais[4252]: [CPG ] got joinlist message from node 1 // 把不正常的节点fence掉： Apr 26 01:47:58 node1 fenced[4302]: node2 not a cluster member after 30 sec post_fail_delay Apr 26 01:47:58 node1 fenced[4302]: fencing node "node2" Apr 26 01:48:10 node1 fenced[4302]: fence "node2" success

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

// 本来正常的集群突然发生token丢失（心跳信号丢失）： Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] The token was lost in the OPERATIONAL state . Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] Receive multicast socket recv buffer size ( 320000 bytes ) . Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] Transmit multicast socket send buffer size ( 320000 bytes ) . // 集群开始重组，进入Gather状态，召集各节点： Apr 26 01 : 47 : 26 node1 openais [ 4252 ] : [ TOTEM ] entering GATHER state from 2. Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] entering GATHER state from 0. Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Creating commit token because I am the rep . Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Storing new sequence id for ring 798 // 进入Commit状态： Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] entering COMMIT state . // 进入Recovery状态： Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] entering RECOVERY state . Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] position [ 0 ] member 10.23.201.30 : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] previous ring seq 1940 rep 10.23.201.30 Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] aru 2b high delivered 2b received flag 1 Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Did not need to originate any messages in recovery . Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ TOTEM ] Sending initial ORF token Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] CLM CONFIGURATION CHANGE Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] New Configuration : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ]    r ( 0 ) ip ( 10.23.201.30 )    Apr 26 01 : 47 : 28 node1 kernel : dlm : closing connection to node 2 Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] Members Left : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ]    r ( 0 ) ip ( 10.23.201.31 )    Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] Members Joined : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] CLM CONFIGURATION CHANGE Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] New Configuration : Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ]    r ( 0 ) ip ( 10.23.201.30 )    Apr 26 01 : 47 : 28 node1 openais [ 4252 ] : [ CLM    ] Members Left : Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ CLM    ] Members Joined : Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ SYNC ] This node is within the primary component and will provide service . // 集群重组完成，进入Operational状态，只剩下一个节点： Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ TOTEM ] entering OPERATIONAL state . Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ CLM    ] got nodejoin message 10.23.201.30 Apr 26 01 : 47 : 29 node1 openais [ 4252 ] : [ CPG    ] got joinlist message from node 1 // 把不正常的节点fence掉： Apr 26 01 : 47 : 58 node1 fenced [ 4302 ] : node2 not a cluster member after 30 sec post_fail_delay Apr 26 01 : 47 : 58 node1 fenced [ 4302 ] : fencing node "node2" Apr 26 01 : 48 : 10 node1 fenced [ 4302 ] : fence "node2" success

来源 http://linuxperf.com/?p=120