CAP theory
For an introduction to CAP theory, you can directly read this article.
What are CAPs?
Consistency (
Consistency includes strong consistency, weak consistency, and eventual consistency.
Consistency actually refers to the consistency of data. Why is the data inconsistent?
As shown in the picture above, our service is provided externally in a cluster, and the client does not care about which node I requested this time. If my first request went to node A to update the data, but my second request was forwarded to node B. If the data of service A and service B have not been synchronized at this time, the data will be inconsistent at this time. In fact, it is difficult for us to transfer the data of service A to B immediately for synchronization when the data of service A changes, especially when the request is special. Frequently. Therefore, it is necessary to judge whether strong consistency must be ensured based on the actual scenario. For example, service clusters involving money and inventory must ensure strong consistency. Another example is the number of likes, views, or visits. These do not need to be consistent in real time. They only need to be consistent in the end.
Availability (Availability)
actually means that regardless of whether the data is correct or not, as long as the user requests it, the data will be returned to the user.
When we talk about consistency, if we want to ensure strong consistency, the client will not be able to provide external services if the request reaches B at this time, because once the service is provided, the data will still be old data, resulting in data inconsistency. But if no service is provided, wouldn't this service node be unusable? That means availability is not guaranteed. So we can also see that a certain compromise needs to be made between consistency and availability.
Partition tolerance
1 Partition
What is a partition? A distributed system is distributed in multiple subnetworks. Service nodes in a distributed system may not be able to communicate with each other due to network reasons or other factors. This situation is called partitioning.
2 What is zoning tolerance?
Because partitioning is generally unavoidable, we cannot fully guarantee that the two service nodes can fully communicate with each other without any exceptions.
总结下来就是在一个分布式系统中，这个P分区是一定存在的。我们需要根据业务场景来做C和A的一些取舍。
Common component guarantee patterns
- Nacos guarantees AP + CP
- Zookeeper guarantees CP
- Eureka guarantees AP
Distro Consistency Protocol (Temporary Node Protocol)
There is relatively little information on the distro protocol linked to the original text
, because it is Alibaba’s “self-created protocol.” Let’s summarize the key points of the distro protocol through the source code: The
distro protocol is a protocol created for the registration center;
there are two aspects: client and server. An important interaction, service registration and heartbeat sending;
the client registers with the server with the service as the dimension, and sends a heartbeat to the server every once in a while after registration. The heartbeat package needs to carry all the information of the registered service, which can be viewed on the client. Come, the server nodes are equal, so the requested node is random;
if the client request fails, another node will resend the request;
the server nodes store all data, but each node is only responsible for part of the service. After receiving the client After the client's "write" (registration, heartbeat, offline, etc.) request, the server node determines whether the requested service is its own responsibility. If so, process it, otherwise it will be handed over to the responsible node for processing;
each server node actively sends health information Other nodes are detected, and the responding node is regarded as a healthy node by that node;
after the server receives the client's service heartbeat, if the service does not exist, it will process the heartbeat request as a registration request;
if the server does not After receiving the client heartbeat, the service will be offline;
the responsible node will write the data and return after receiving write requests such as service registration and service heartbeat, and the background will asynchronously synchronize the data to other nodes; the
node will After the request, it is retrieved directly from the local machine and returned, regardless of whether the data is the latest.
New node synchronization mechanism:
DistroProtocol#startDistroTask
private void startDistroTask() {
if (EnvUtil.getStandaloneMode()) {
isInitialized = true;
return;
}
// 健康检查
startVerifyTask();
// 开始加载
startLoadTask();
}
Equality mechanism:
Each node in Nacos is equal and can handle write requests.
Asynchronous replication mechanism:
DistroProtocol#sync
public void sync(DistroKey distroKey, DataOperation action) {
sync(distroKey, action, DistroConfig.getInstance().getSyncDelayMillis());
}
Health check mechanism:
DistroProtocol#startDistroTask
private void startVerifyTask() {
GlobalExecutor.schedulePartitionDataTimedSync(new DistroVerifyTimedTask(memberManager, distroComponentHolder,
distroTaskEngineHolder.getExecuteWorkersManager()),
DistroConfig.getInstance().getVerifyIntervalMillis());
}
Local read mechanism (each node has a full amount of data):
However, write processing only has a part of the data, that is, there is a data splitting and divide-and-conquer mechanism.
InstanceController#list
/**
* Get all instance of input service.
*
* @param request http request
* @return list of instance
* @throws Exception any error during list
*/
@GetMapping("/list")
@Secured(action = ActionTypes.READ)
public Object list(HttpServletRequest request) throws Exception {
}
Routing forwarding mechanism:
DistroFilter#doFilter
if (distroMapper.responsible(distroTag)) {
filterChain.doFilter(req, resp);
return;
}
Raft consensus protocol (persistent node protocol)
This protocol is mainly about initiating elections and how to synchronize data after the election. After a node gets up, a vote will be initiated. If more than half of the data in the cluster approves the node, the node will become the leader. After becoming the leader, it will establish contact with other nodes.
V1 version election and synchronization
try {
if (stopWork) {
return;
}
if (!peers.isReady()) {
return;
}
RaftPeer local = peers.local();
local.leaderDueMs -= GlobalExecutor.TICK_PERIOD_MS;
if (local.leaderDueMs > 0) {
return;
}
// 来到这里以后已经和Master失联了
// reset timeout
local.resetLeaderDue();
local.resetHeartbeatDue();
// 发起投票，里面会向其它节点发起异步请求
sendVote();
} catch (Exception e) {
Loggers.RAFT.warn("[RAFT] error while master election {}", e);
}
// 过半
if (maxApproveCount >= majorityCount()) {
RaftPeer peer = peers.get(maxApprovePeer);
peer.state = RaftPeer.State.LEADER;
if (!Objects.equals(leader, peer)) {
leader = peer;
ApplicationUtils.publishEvent(new LeaderElectFinishedEvent(this, leader, local()));
Loggers.RAFT.info("{} has become the LEADER", leader.ip);
}
}
// 处理心跳
try {
if (stopWork) {
return;
}
if (!peers.isReady()) {
return;
}
RaftPeer local = peers.local();
local.heartbeatDueMs -= GlobalExecutor.TICK_PERIOD_MS;
if (local.heartbeatDueMs > 0) {
return;
}
local.resetHeartbeatDue();
sendBeat();
} catch (Exception e) {
Loggers.RAFT.warn("[RAFT] error while sending beat {}", e);
}
V2版本的选举与同步
涉及源码
com.alibaba.nacos.naming.consistency.DelegateConsistencyServiceImpl
com.alibaba.nacos.naming.consistency.persistent.PersistentConsistencyServiceDelegateImpl#PersistentConsistencyServiceDelegateImpl
com.alibaba.nacos.naming.consistency.persistent.impl.PersistentServiceProcessor#afterConstruct
com.alibaba.nacos.core.distributed.raft.JRaftProtocol#addRequestProcessors
com.alibaba.nacos.core.distributed.raft.JRaftServer#createMultiRaftGroup
com.alipay.sofa.jraft.RaftGroupService#start(boolean)
com.alipay.sofa.jraft.RaftServiceFactory#createAndInitRaftNode
com.alipay.sofa.jraft.core.NodeImpl#init
com.alipay.sofa.jraft.core.NodeImpl#electSelf
com.alipay.sofa.jraft.core.NodeImpl#becomeLeader
同步
com.alipay.sofa.jraft.ReplicatorGroup#addReplicator(com.alipay.sofa.jraft.entity.PeerId)
部分源码：
private BasePersistentServiceProcessor createNewPersistentServiceProcessor(ProtocolManager protocolManager,
ClusterVersionJudgement versionJudgement) throws Exception {
final BasePersistentServiceProcessor processor =
EnvUtil.getStandaloneMode() ? new StandalonePersistentServiceProcessor(versionJudgement)
: new PersistentServiceProcessor(protocolManager, versionJudgement);
processor.afterConstruct();
return processor;
}
PersistentServiceProcessor#afterConstruct
@Override
public void afterConstruct() {
super.afterConstruct();
String raftGroup = Constants.NAMING_PERSISTENT_SERVICE_GROUP;
this.protocol.protocolMetaData().subscribe(raftGroup, MetadataKey.LEADER_META_DATA, o -> {
if (!(o instanceof ProtocolMetaData.ValueItem)) {
return;
}
Object leader = ((ProtocolMetaData.ValueItem) o).getData();
hasLeader = StringUtils.isNotBlank(String.valueOf(leader));
Loggers.RAFT.info("Raft group {} has leader {}", raftGroup, leader);
});
this.protocol.addRequestProcessors(Collections.singletonList(this));
// If you choose to use the new RAFT protocol directly, there will be no compatible logical execution
if (EnvUtil.getProperty(Constants.NACOS_NAMING_USE_NEW_RAFT_FIRST, Boolean.class, false)) {
NotifyCenter.registerSubscriber(notifier);
waitLeader();
startNotify = true;
}
}
JRaft
public void init(RaftConfig config) {
if (initialized.compareAndSet(false, true)) {
this.raftConfig = config;
NotifyCenter.registerToSharePublisher(RaftEvent.class);
this.raftServer.init(this.raftConfig);
this.raftServer.start();
// There is only one consumer to ensure that the internal consumption
// is sequential and there is no concurrent competition
NotifyCenter.registerSubscriber(new Subscriber<RaftEvent>() {
@Override
public void onEvent(RaftEvent event) {
Loggers.RAFT.info("This Raft event changes : {}", event);
final String groupId = event.getGroupId();
Map<String, Map<String, Object>> value = new HashMap<>();
Map<String, Object> properties = new HashMap<>();
final String leader = event.getLeader();
final Long term = event.getTerm();
final List<String> raftClusterInfo = event.getRaftClusterInfo();
final String errMsg = event.getErrMsg();
// Leader information needs to be selectively updated. If it is valid data,
// the information in the protocol metadata is updated.
MapUtil.putIfValNoEmpty(properties, MetadataKey.LEADER_META_DATA, leader);
MapUtil.putIfValNoNull(properties, MetadataKey.TERM_META_DATA, term);
MapUtil.putIfValNoEmpty(properties, MetadataKey.RAFT_GROUP_MEMBER, raftClusterInfo);
MapUtil.putIfValNoEmpty(properties, MetadataKey.ERR_MSG, errMsg);
value.put(groupId, properties);
metaData.load(value);
// The metadata information is injected into the metadata information of the node
injectProtocolMetaData(metaData);
}
@Override
public Class<? extends Event> subscribeType() {
return RaftEvent.class;
}
});
}
}