Analysis of SOFA registry data synchronization

This article mainly analyzes the data synchronization module of SOFARegistry, and does not elaborate too much on the concept of registry and the infrastructure of SOFARegistry. For related introduction, please refer to the article " Registry Center under Massive Data - Introduction to SOFARegistry Architecture ".

The main writing ideas of this paper are roughly divided into the following two parts: the first part uses the role classification in SOFARegistry to explain which roles will perform data synchronization, and the second part analyzes the specific implementation of data synchronization.

SOFARegistry role classification

As shown above, SOFARegistry contains 4 roles:

Role	illustrate
Client	Provides the basic API capability for applications to access the service registry. The application system invokes the service subscription and service publishing capabilities of the service registry programmatically by relying on the client JAR package.
SessionServer	The session server is responsible for accepting the client's service publishing and service subscription requests, and as an intermediate layer, it forwards write operations to the DataServer layer. The SessionServer layer can be expanded with the growth of the number of business machines.
DataServer	The data server is responsible for storing specific service data. The data is stored in consistent Hash shards according to dataInfoId, and supports multiple copy backups to ensure high data availability. This layer can be expanded as the scale of the service data volume grows.
MetaServer	The metadata server is responsible for maintaining a consistent list of SessionServer and DataServer in the cluster. As an address discovery service within the SOFARegistry cluster, it can notify the entire cluster when the SessionServer or DataServer node changes.

Among these four roles, MetaServer, as the metadata server itself, does not handle actual business data, but is only responsible for maintaining a consistent list of SessionServer and DataServer in the cluster, and does not involve data synchronization; the core actions between Client and SessionServer are subscription and publishing. In a broad sense, the data synchronization between the user-side client and the SOFARegistry cluster can be found at: github.com/sofastack/s… , so it is not within the scope of this article.

As a session service, SessionServer mainly solves the problem of massive client connections, followed by caching all pub data published by clients; session itself does not persist service data, but transfers data to DataServer. The data of the DataServer storage service is stored in consistent Hash shards according to dataInfoId, and supports multi-copy backup to ensure high data availability.

From the functional analysis of SessionServer and DataServer, it can be concluded that:

• The service data cached by the SessionServer needs to be consistent with the service data stored by the DataServer

• DataServer supports multiple copies to ensure high availability, so data server needs to keep service data consistent between multiple copies of DataServer.

The above two data consistency guarantees in SOFARegistry are achieved through the data synchronization mechanism.

The concrete realization of data synchronization

The following mainly introduces the implementation details of data synchronization, mainly including data synchronization between SessionServer and DataServer and data synchronization between multiple copies of DataServer.

Data synchronization between SessionServer and DataServer

Data synchronization between SessionServer and DataServer is based on a push-pull combination mechanism

• Push: When the data changes, the DataServer will actively notify the SessionServer, and the SessionServer checks and confirms that it needs to be updated (compared with the version) and then actively obtains the data from the DataServer.
• 拉：除了上述的 DataServer 主动推以外，SessionServer 每隔一定的时间间隔，会主动向 DataServer 查询所有 dataInfoId 的 version 信息，然后再与 SessionServer 内存的 version 作比较，若发现 version 有变化，则主动向 DataServer 获取数据。这个“拉”的逻辑，主要是对“推”的一个补充，若在“推”的过程有错漏的情况可以在这个时候及时弥补。

关于推和拉两种模式检查的 version 有一些差异，可以详见下面 推模式下的数据同步 和 **拉模式下的数据同步 **中的具体介绍

推模式下的数据同步流程

推模式是通过 SyncingWatchDog 这个守护线程不断 loop 执行来实现数据变更检查和通知发起的。

// 这里遍历所有的 slot
for (SlotState slotState : slotTableStates.slotStates.values()) {
    try {
        sync(slotState, syncSessionIntervalMs, syncLeaderIntervalMs, slotTableEpoch);
    } catch (Throwable e) {
        SYNC_ERROR_LOGGER.error(
                "[syncCommit]failed to do sync slot {}, migrated={}",
                slotState.slot,
                slotState.migrated,
                e);
    }
}

按 slot 分组汇总数据版本。data 与每个 session 的连接都对应一个 SyncSessionTask，SyncSessionTask 负责执行同步数据的任务，核心同步逻辑在 com.alipay.sofa.registry.server.data.slot.SlotDiffSyncer#sync方法中完成，大致流程如下面时序图所示：

这上图圈红部分的逻辑第四步，根据 dataInfoId diff 更新 data 内存数据，这里仅处理了被移除的 dataInfoId，对于新增和更新的没有做任务处理，而是通过后面的第 5 -7 步来完成；这么做的主要原因在于避免产生空推送导致一些危险情况发生。

第 5 步中，比较的是所有变更 dataInfoId 的 pub version，具体比较逻辑参考后面 diffPublisher 小节中的介绍。

数据变更的事件通知处理

数据变更事件会被收集在 DataChangeEventCenter 的 dataCenter2Changes 缓存中，然后由一个守护线程 ChangeMerger 负责从 dataCenter2Changes 缓存中不断的读取，这些被捞到的事件源会被组装成 ChangeNotifier 任务，提交给一个单独的线程池(notifyExecutor)处理，整个过程全部是异步的。

拉模式下的数据同步流程

拉模式下，由 SessionServer 负责发起，com.alipay.sofa.registry.server.session.registry.SessionRegistry.VersionWatchDog默认情况下每 5 秒扫描一次版本数据，如果版本有发生变更，则主动进行拉取一次，流程大致如下：

需要注意的是，拉模式对推送流程的补充，这里的 version 是每个 sub 的 lastPushedVersion， 而 推模式的version 是 pub 的数据的 version。关于 lastPushedVersion 的获取可以参考 com.alipay.sofa.registry.server.session.store.SessionInterests#selectSubscribers

store.forEach((String dataInfoId, Map<String, Subscriber> subs) -> {
   // ...
  long maxVersion = 0;
  for (Subscriber sub : subs.values()) {
    // ...
    // 获取当前 sub 的 pushVersion
    final long pushVersion = sub.getPushedVersion(dataCenter);
    // 如果 pushVersion 比最大(最新)版本大，则将当前  pushVersion 作为最新版本推送版本
    if (maxVersion < pushVersion) {
      maxVersion = pushVersion;
    }
  }
  versions.put(dataInfoId, new DatumVersion(maxVersion));
});

DataServer 多副本之间的数据同步

主要是 slot对应的 data 的 follower 定期和 leader 进行数据同步，其同步逻辑与 SessionServer 和 DataServer 之间的数据同步逻辑差异不大；发起方式也是一样的；data 判断如果当前节点不是 leader，就会进行与 leader 之间的数据同步。

if (localIsLeader(slot)) {
   // 如果当前是 leader，则执行 session 同步或者 migrating
} else {
    // 如果当前不是 leader，则和 leader 同步数据
    syncLeader(slotState, syncLeaderIntervalMs, slotTableEpoch);
}

篇幅原因，这部分不展开讨论。

增量同步 diff 计算逻辑分析

不管是 SessionServer 和 DataServer 之间的同步，还是 DataServer 多副本之间的同步，都是基于增量 diff 同步的，不会一次性同步全量数据。本节对增量同步 diff 计算逻辑进行简单分析，核心代码在 com.alipay.sofa.registry.common.model.slot.DataSlotDiffUtils（建议阅读这部分代码时直接结合代码中的测试用例来看）；主要包括计算 digest 和 publishers 两个。

diffDigest

DataSlotDiffUtils#diffDigest 方法接收两个参数

• targetDigestMap 可以理解为目标数据
• sourceDigestMap 可以理解为基线数据

核心计算逻辑如下代码分析
�

// 遍历 sourceDigestMap 元素
for (Map.Entry<String, DatumDigest> e : sourceDigestMap.entrySet()) {
  // dataInfoId
  final String dataInfoId = e.getKey();
  // 从 目标数据 集中根据 dataInfoId 获取数据摘要
  DatumDigest targetDigest = targetDigestMap.get(dataInfoId);
  // 如果目标数据集中没有当前 dataInfoId 对应的数据摘要，
  // 则将当前 dataInfoId 作为新增项
  if (targetDigest == null) {
    adds.add(dataInfoId);
    continue;
  }
  // 如果目标数据集中有当前 dataInfoId 对应的数据摘要，
  // 但是数据摘要不同，则将当前 dataInfoId 作为待更新项
  if (!targetDigest.equals(e.getValue())) {
    updates.add(dataInfoId);
  }
}


// 如果目标数据集中的 dataInfoId 不再基线数据集中时，
// 则将当前 dataInfoId 作为待移除项。
List<String> removes = new ArrayList<>();
for (String dataInfoId : targetDigestMap.keySet()) {
  if (!sourceDigestMap.containsKey(dataInfoId)) {
    removes.add(dataInfoId);
  }
}

那么根据上述 diff 计算逻辑，这里有如下几种场景（假设基线数据集数据中 dataInfoId 为 a 和 b）

• 目标数据集为空：返回 dataInfoId 为 a 和 b 两项作为新增项
• 目标数据集与基线数据集相等，新增项、待更新项与待移除项均为空
• 目标数据集中包括 a,b,c 三个 dataInfoId，则返回 c 作为待移除项
• 目标数据集中包括 a 和 c 两个 dataInfoId，则返回 c 作为待移除项，b 作为新增项

diffPublisher

diffPublisher 与 �diffDigest 计算稍有不同，diffPublisher 接收三个参数，除了目标数据集和基线数据集之外，还有一个 publisherMaxNum（默认 400），用于限制每次处理的数据个数；这里同样给出核心代码解释：

// 遍历所有目标数据集
for (DatumSummary summary : targetDatumSummaries) {
      // 拿到 dataInfoId
      final String dataInfoId = summary.getDataInfoId();
      // 看基线数据集中是否包括当前 dataInfoId 对应的 Publisher 数据
      Map<String, Publisher> publisherMap = sourcePublishers.get(dataInfoId);
      // 这里表示 dataInfoId 移除被移除了，不需要做任何处理
      if (publisherMap == null) { continue; }
      
      Set<String> registerIds = summary.getPublisherVersions().keySet();
      // 遍历 registerIds
      for (String registerId : registerIds) {
        // 如果基线数据集中不包括此 registerId，则将当前 registerId 加入待移除列表中
        if (!publisherMap.containsKey(registerId)) {
          List<String> list = removedPublishers.computeIfAbsent(dataInfoId, k -> new ArrayList<>());
          list.add(registerId);
        }
      }
      List<Publisher> publishers = new ArrayList<>();
      Map<String, RegisterVersion> versions = summary.getPublisherVersions();
      // 遍历版本
      for (Map.Entry<String, Publisher> p : publisherMap.entrySet()) {
        final String registerId = p.getKey();
        // 如果目标数据集当前 dataInfoId 的 registerIds 集中不包括基线的
        // 则作为更新项
        if (!versions.containsKey(registerId)) {
          publishers.add(p.getValue());
          continue;
        }
        // 如果当前 registerId 版本相同，则不做处理
        if (p.getValue().registerVersion().equals(versions.get(registerId))) {
          // the same
          continue;
        }
        // 不相等，则作为更新项
        publishers.add(p.getValue());
      }
    }

这里同样分析几种场景（下面只的是更新 dataInfoId 对应的 publisher，registerId 与 publisher是 一一对应）：

• 目标数据集与基线数据集相同，且数据没有超过 publisherMaxNum，返回的待更新和待移除均为空，且没有剩余未处理数据
• 需要移除的情况：基线中不包括目标数据集 dataInfoId 的 registerId （移除的是 registerId，不是 dataInfoId）
• 需要更新的情况：
  • 目标数据集中存在基线数据集不存在的 registerId
  • 目标数据集和基线数据集存在的 registerId 的版本不同

总结

This paper mainly introduces the data synchronization module in SOFARegistry; firstly, the data synchronization problem between different roles is explained from the SOFARegistry role classification, and the data synchronization between SessionServer and DataServer and data synchronization between multiple copies of DataServer are analyzed; in SessionServer and DataServer In the data synchronization analysis, the overall process of data synchronization in the two scenarios of push and pull is analyzed. Finally, the diff calculation logic of data addition in SOFARegistry is introduced, and the specific scenarios are described in combination with the relevant core codes.

Overall, there are some points worth learning about the processing of SOFARegistry data synchronization:

• SOFARegistry is based on AP, which satisfies eventual consistency in consistency; in the actual synchronization logic processing, combined with the event mechanism, it is basically completed asynchronously, thus weakening the impact of data synchronization on the core process.
• In the two parts of pull mode and data change notification, a similar production-consumption model is used internally. On the one hand, the decoupling of production and consumption logic is more independent from the code; on the other hand, the speed of production and consumption is eliminated through cache or queue. different and mutually blocking problems.
• The pull mode complements the push mode; we know that the push mode is server -> client. When data changes, if some abnormality occurs, causing a server -> client link to fail to push, it will lead to data held by different clients. Inconsistent situation; the addition of the pull mode enables the client to actively complete the data consistency check.

Finally, thank you for reading. If there are any mistakes in the text, please point them out; you are also welcome to pay attention to the sofastack community. Original link: mp.weixin.qq.com/s/UqsFzSuxu…

• I am participating in the recruitment of the creator signing program of the Nuggets Technology Community, click the link to register and submit .