Apollo核心源码解析（一）：Portal发布配置、Admin Service发送ReleaseMessage、Config Service通知客户端

1、Apollo架构设计

• Config Service提供配置的读取、推送等功能，服务对象是Apollo客户端
• Admin Service提供配置的修改、发布等功能，服务对象是Apollo Portal（管理界面）
• Config Service和Admin Service都是多实例、无状态部署，所以需要将自己注册到Eureka中并保持心跳
• 在Eureka之上我们架了一层Meta Server用于封装Eureka的服务发现接口
• Client通过域名访问Meta Server获取Config Service服务列表（IP+Port），而后直接通过IP+Port访问服务，同时在Client侧会做load balance、错误重试
• Portal通过域名访问Meta Server获取Admin Service服务列表（IP+Port），而后直接通过IP+Port访问服务，同时在Portal侧会做load balance、错误重试
• 为了简化部署，实际上Config Service、Eureka和Meta Server三个逻辑角色部署在同一个JVM进程中

2、配置发布后的实时推送

上图简要描述了配置发布的大致过程：

1. 用户在Portal操作配置发布
2. Portal调用Admin Service的接口操作发布
3. Admin Service发布配置后，发送ReleaseMessage给各个Config Service
4. Config Service收到ReleaseMessage后，通知对应的客户端

下面我们就结合上述流程来解析对应源码：

1）、Portal发布配置

Apollo Portal模块：

在Apollo点击发布按钮，调用com.ctrip.framework.apollo.portal.controller.ReleaseController提供的API

@Validated
@RestController
public class ReleaseController {
    
    
  
  @PreAuthorize(value = "@permissionValidator.hasReleaseNamespacePermission(#appId, #namespaceName, #env)")
  @PostMapping(value = "/apps/{appId}/envs/{env}/clusters/{clusterName}/namespaces/{namespaceName}/releases")
  public ReleaseDTO createRelease(@PathVariable String appId,
                                  @PathVariable String env, @PathVariable String clusterName,
                                  @PathVariable String namespaceName, @RequestBody NamespaceReleaseModel model) {
    
    
    model.setAppId(appId);
    model.setEnv(env);
    model.setClusterName(clusterName);
    model.setNamespaceName(namespaceName);

    if (model.isEmergencyPublish() && !portalConfig.isEmergencyPublishAllowed(Env.valueOf(env))) {
    
    
      throw new BadRequestException(String.format("Env: %s is not supported emergency publish now", env));
    }

    //发布配置
    ReleaseDTO createdRelease = releaseService.publish(model);

    ConfigPublishEvent event = ConfigPublishEvent.instance();
    event.withAppId(appId)
        .withCluster(clusterName)
        .withNamespace(namespaceName)
        .withReleaseId(createdRelease.getId())
        .setNormalPublishEvent(true)
        .setEnv(Env.valueOf(env));

    publisher.publishEvent(event);

    return createdRelease;
  }  

这里核心代码是调用ReleaseService#publish(NamespaceReleaseModel)方法，调用Admin Service API，发布配置，NamespaceReleaseModel属性如下：

public class NamespaceReleaseModel implements Verifiable {
    
    

  /**
   * app编号
   */
  private String appId;

  /**
   * env名称
   */
  private String env;

  /**
   * cluster名称
   */
  private String clusterName;

  /**
   * namespace名称
   */
  private String namespaceName;

  /**
   * 发布标题
   */
  private String releaseTitle;

  /**
   * 发布描述
   */
  private String releaseComment;

  /**
   * 发布人
   */
  private String releasedBy;

  /**
   * 是否紧急发布
   */
  private boolean isEmergencyPublish;

接着来看ReleaseService#publish(NamespaceReleaseModel)方法：

@Service
public class ReleaseService {
    
    
  
  public ReleaseDTO publish(NamespaceReleaseModel model) {
    
    
    Env env = model.getEnv();
    boolean isEmergencyPublish = model.isEmergencyPublish();
    String appId = model.getAppId();
    String clusterName = model.getClusterName();
    String namespaceName = model.getNamespaceName();
    String releaseBy = StringUtils.isEmpty(model.getReleasedBy()) ?
                       userInfoHolder.getUser().getUserId() : model.getReleasedBy();

    //调用Admin Service API,发布Namespace的配置
    ReleaseDTO releaseDTO = releaseAPI.createRelease(appId, env, clusterName, namespaceName,
                                                     model.getReleaseTitle(), model.getReleaseComment(),
                                                     releaseBy, isEmergencyPublish);

    Tracer.logEvent(TracerEventType.RELEASE_NAMESPACE,
                    String.format("%s+%s+%s+%s", appId, env, clusterName, namespaceName));

    return releaseDTO;
  }  

ReleaseService#publish(NamespaceReleaseModel)中调用ReleaseAPI#createRelease(appId, env, clusterName, namespaceName, releaseTitle, releaseComment, releaseBy, isEmergencyPublish)方法，调用Admin Service API，发布Namespace的配置

com.ctrip.framework.apollo.portal.api.ReleaseAPI实现API抽象类，封装对Admin Service Release模块的API调用：

@Service
public class AdminServiceAPI {
    
    
  
	@Service
  public static class ReleaseAPI extends API {
    
    

    public ReleaseDTO createRelease(String appId, Env env, String clusterName, String namespace,
        String releaseName, String releaseComment, String operator,
        boolean isEmergencyPublish) {
    
    
      HttpHeaders headers = new HttpHeaders();
      headers.setContentType(MediaType.parseMediaType(MediaType.APPLICATION_FORM_URLENCODED_VALUE + ";charset=UTF-8"));
      MultiValueMap<String, String> parameters = new LinkedMultiValueMap<>();
      parameters.add("name", releaseName);
      parameters.add("comment", releaseComment);
      parameters.add("operator", operator);
      parameters.add("isEmergencyPublish", String.valueOf(isEmergencyPublish));
      HttpEntity<MultiValueMap<String, String>> entity =
          new HttpEntity<>(parameters, headers);
      ReleaseDTO response = restTemplate.post(
          env, "apps/{appId}/clusters/{clusterName}/namespaces/{namespaceName}/releases", entity,
          ReleaseDTO.class, appId, clusterName, namespace);
      return response;
    }

2）、Admin Service发送ReleaseMessage（异步）

Admin Service模块：

createRelease()中使用RestTemplate调用ReleaseController#publish(appId, env, clusterName, namespaceName, releaseTitle, releaseComment, releaseBy, isEmergencyPublish)方法提供的API，发布Namespace的配置

@RestController
public class ReleaseController {
    
      

	@Transactional
  @PostMapping("/apps/{appId}/clusters/{clusterName}/namespaces/{namespaceName}/releases")
  public ReleaseDTO publish(@PathVariable("appId") String appId,
                            @PathVariable("clusterName") String clusterName,
                            @PathVariable("namespaceName") String namespaceName,
                            @RequestParam("name") String releaseName,
                            @RequestParam(name = "comment", required = false) String releaseComment,
                            @RequestParam("operator") String operator,
                            @RequestParam(name = "isEmergencyPublish", defaultValue = "false") boolean isEmergencyPublish) {
    
    
    //校验对应的Namespace是否存在
    Namespace namespace = namespaceService.findOne(appId, clusterName, namespaceName);
    if (namespace == null) {
    
    
      throw new NotFoundException(String.format("Could not find namespace for %s %s %s", appId,
                                                clusterName, namespaceName));
    }
    Release release = releaseService.publish(namespace, releaseName, releaseComment, operator, isEmergencyPublish);

    Namespace parentNamespace = namespaceService.findParentNamespace(namespace);
    String messageCluster;
    if (parentNamespace != null) {
    
    
      messageCluster = parentNamespace.getClusterName();
    } else {
    
    
      messageCluster = clusterName;
    }
    //发送Release消息
    messageSender.sendMessage(ReleaseMessageKeyGenerator.generate(appId, messageCluster, namespaceName),
                              Topics.APOLLO_RELEASE_TOPIC);
    return BeanUtils.transform(ReleaseDTO.class, release);
  }

发送ReleaseMessage的实现方式：

Admin Service在配置发布后，需要通知所有的Config Service有配置发布，从而Config Service可以通知对应的客户端来拉取最新的配置

这是一个典型的消息使用场景，Admin Service作为producer发出消息，各个Config Service作为consumer消费消息。通过一个消息组件（Message Queue）就能很好的实现Admin Service和Config Service的解耦

Apollo没有采用外部的消息中间件，而是通过数据库实现了一个简单的消息队列，实现方式如下：

1. Admin Service在配置发布后会往ReleaseMessage表插入一条消息记录，消息内容就是配置发布的AppId+Cluster+Namespace，参见DatabaseMessageSender
2. Config Service有一个线程会每秒扫描一次ReleaseMessage表，看看是否有新的消息记录，参见ReleaseMessageScanner
3. Config Service如果发现有新的消息记录，那么就会通知到所有的消息监听器（ReleaseMessageListener），如NotificationControllerV2，消息监听器的注册过程参见ConfigServiceAutoConfiguration
4. NotificationControllerV2得到配置发布的AppId+Cluster+Namespace后，会通知对应的客户端

ReleaseMessage：

@Entity
@Table(name = "ReleaseMessage")
public class ReleaseMessage {
    
    
  /**
   * id自增主键
   */
  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  @Column(name = "Id")
  private long id;

  /**
   * 消息内容,通过ReleaseMessageKeyGenerator.generate()生成,appId+cluster+namespace
   */
  @Column(name = "Message", nullable = false)
  private String message;

  /**
   * 最后更新时间
   */
  @Column(name = "DataChange_LastTime")
  private Date dataChangeLastModifiedTime;

ReleaseMessageKeyGenerator：

public class ReleaseMessageKeyGenerator {
    
    

  private static final Joiner STRING_JOINER = Joiner.on(ConfigConsts.CLUSTER_NAMESPACE_SEPARATOR);

  public static String generate(String appId, String cluster, String namespace) {
    
    
    //将appId+cluster+namespace拼接,使用+作为间隔
    return STRING_JOINER.join(appId, cluster, namespace);
  }
}

将appId+cluster+namespace拼接，使用+作为间隔，因此对于同一个Namespace，生成的消息内容是相同的。通过这样的方式，可以使用最新的ReleaseMessage的id属性，作为Namespace是否发生变更的标识。而Apollo确实是通过这样的方式实现，Client通过不断使用获得到ReleaseMessage的id属性作为版本号，请求Config Service判断是否配置发生了变化。ReleaseMessage设计的意图是作为配置发生变化的通知，所以对于同一个Namespace，仅需要保留其最新的ReleaseMessage记录。所以，在DatabaseMessageSender中，有后台任务不断清理旧的ReleaseMessage记录。这里先留个印象后面会一一讲到

ReleaseController#publish(appId, env, clusterName, namespaceName, releaseTitle, releaseComment, releaseBy, isEmergencyPublish)中用ReleaseMessageKeyGenerator#(appId, cluster, namespace)来生成Release消息，然后调用MessageSender#sendMessage(message, channel)方法发送Release消息

public interface  {
    
    
  /**
   * 发送Message
   *
   * @param message 消息
   * @param channel 通道(主题)
   */
  void sendMessage(String message, String channel);
}

MessageSender的子类DatabaseMessageSender：

@Component
public class DatabaseMessageSender implements MessageSender {
    
    
  private static final Logger logger = LoggerFactory.getLogger(DatabaseMessageSender.class);
  
  /**
   * 清理Message队列最大容量
   */
  private static final int CLEAN_QUEUE_MAX_SIZE = 100;

  /**
   * 清理Message队列
   */
  private BlockingQueue<Long> toClean = Queues.newLinkedBlockingQueue(CLEAN_QUEUE_MAX_SIZE);

  /**
   * 清理Message ExecutorService
   */
  private final ExecutorService cleanExecutorService;

  /**
   * 是否停止清理Message标识
   */
  private final AtomicBoolean cleanStopped;

  private final ReleaseMessageRepository releaseMessageRepository;

  public DatabaseMessageSender(final ReleaseMessageRepository releaseMessageRepository) {
    
    
    //创建ExecutorService对象
    cleanExecutorService = Executors.newSingleThreadExecutor(ApolloThreadFactory.create("DatabaseMessageSender", true));
    //设置cleanStopped为false
    cleanStopped = new AtomicBoolean(false);
    this.releaseMessageRepository = releaseMessageRepository;
  }
  
  @Override
  @Transactional
  public void sendMessage(String message, String channel) {
    
    
    logger.info("Sending message {} to channel {}", message, channel);
    //仅允许发送APOLLO_RELEASE_TOPIC(apollo-release)
    if (!Objects.equals(channel, Topics.APOLLO_RELEASE_TOPIC)) {
    
    
      logger.warn("Channel {} not supported by DatabaseMessageSender!", channel);
      return;
    }

    Tracer.logEvent("Apollo.AdminService.ReleaseMessage", message);
    Transaction transaction = Tracer.newTransaction("Apollo.AdminService", "sendMessage");
    try {
    
    
      //保存ReleaseMessage对象
      ReleaseMessage newMessage = releaseMessageRepository.save(new ReleaseMessage(message));
      //添加到清理Message队列 若队列已满,添加失败,不阻塞等待
      toClean.offer(newMessage.getId());
      transaction.setStatus(Transaction.SUCCESS);
    } catch (Throwable ex) {
    
    
      logger.error("Sending message to database failed", ex);
      transaction.setStatus(ex);
      throw ex;
    } finally {
    
    
      transaction.complete();
    }
  }  

sendMessage()方法中主要保存ReleaseMessage对象，然后将消息ID添加到清理Message队列

清理ReleaseMessage任务：

@Component
public class DatabaseMessageSender implements MessageSender {
    
    
  
	@PostConstruct
  private void initialize() {
    
    
    cleanExecutorService.submit(() -> {
    
    
      while (!cleanStopped.get() && !Thread.currentThread().isInterrupted()) {
    
    
        try {
    
    
          //拉取
          Long rm = toClean.poll(1, TimeUnit.SECONDS);
          //队列非空,处理拉取到的消息
          if (rm != null) {
    
    
            cleanMessage(rm);
          }
          //队列为空,sleep,避免空跑,占用CPU
          else {
    
    
            TimeUnit.SECONDS.sleep(5);
          }
        } catch (Throwable ex) {
    
    
          Tracer.logError(ex);
        }
      }
    });
  }
  
  private void cleanMessage(Long id) {
    
    
    //查询对应的ReleaseMessage对象,避免已经删除
    ReleaseMessage releaseMessage = releaseMessageRepository.findById(id).orElse(null);
    if (releaseMessage == null) {
    
    
      return;
    }
    boolean hasMore = true;
    //循环删除相同消息内容(message)的老消息
    while (hasMore && !Thread.currentThread().isInterrupted()) {
    
    
      //拉取相同消息内容的100条老消息 按照id升序排序
      //老消息的定义:比当前消息编号小,即先发送的
      List<ReleaseMessage> messages = releaseMessageRepository.findFirst100ByMessageAndIdLessThanOrderByIdAsc(
          releaseMessage.getMessage(), releaseMessage.getId());

      //删除老消息
      releaseMessageRepository.deleteAll(messages);
      //若拉取不足100条,说明无老消息了
      hasMore = messages.size() == 100;

      messages.forEach(toRemove -> Tracer.logEvent(
          String.format("ReleaseMessage.Clean.%s", toRemove.getMessage()), String.valueOf(toRemove.getId())));
    }
  }

ReleaseMessageListener：

com.ctrip.framework.apollo.biz.message.ReleaseMessageListener监听器接口，代码如下：

public interface ReleaseMessageListener {
    
    
  /**
   * 处理ReleaseMessage
   *
   * @param message
   * @param channel 通道（主题）
   */
  void handleMessage(ReleaseMessage message, String channel);
}

ReleaseMessageListener实现子类如下图：

Config Service模块：

ReleaseMessageScanner：

com.ctrip.framework.apollo.biz.message.ReleaseMessageScanner实现org.springframework.beans.factory.InitializingBean接口，ReleaseMessage扫描器，被Config Service使用

public class ReleaseMessageScanner implements InitializingBean {
    
    
  private static final Logger logger = LoggerFactory.getLogger(ReleaseMessageScanner.class);
  @Autowired
  private BizConfig bizConfig;
  @Autowired
  private ReleaseMessageRepository releaseMessageRepository;

  /**
   * 从DB中扫描ReleaseMessage表的频率(毫秒)
   */
  private int databaseScanInterval;

  /**
   * 监听器数组
   */
  private List<ReleaseMessageListener> listeners;

  /**
   * 定时任务服务
   */
  private ScheduledExecutorService executorService;

  /**
   * 最后扫描到的ReleaseMessage的编号
   */
  private long maxIdScanned;

  public ReleaseMessageScanner() {
    
    
    //创建监听器数组
    listeners = Lists.newCopyOnWriteArrayList();
    //创建ScheduledExecutorService对象
    executorService = Executors.newScheduledThreadPool(1, ApolloThreadFactory
        .create("ReleaseMessageScanner", true));
  }

  /**
   * 通过Spring调用,初始化Scan任务
   * 
   * @throws Exception
   */
  @Override
  public void afterPropertiesSet() throws Exception {
    
    
    databaseScanInterval = bizConfig.releaseMessageScanIntervalInMilli();
    //获得最大的ReleaseMessage的编号
    maxIdScanned = loadLargestMessageId();
    //创建从DB中扫描ReleaseMessage表的定时任务
    executorService.scheduleWithFixedDelay(() -> {
    
    
      Transaction transaction = Tracer.newTransaction("Apollo.ReleaseMessageScanner", "scanMessage");
      try {
    
    
        //从DB中扫描ReleaseMessage
        scanMessages();
        transaction.setStatus(Transaction.SUCCESS);
      } catch (Throwable ex) {
    
    
        transaction.setStatus(ex);
        logger.error("Scan and send message failed", ex);
      } finally {
    
    
        transaction.complete();
      }
    }, databaseScanInterval, databaseScanInterval, TimeUnit.MILLISECONDS);

  }
  
  /**
   * 获得最大的ReleaseMessage的编号
   * 
   * @return
   */
  private long loadLargestMessageId() {
    
    
    ReleaseMessage releaseMessage = releaseMessageRepository.findTopByOrderByIdDesc();
    return releaseMessage == null ? 0 : releaseMessage.getId();
  } 

ReleaseMessageScanner中的监听器数组。通过addMessageListener()方法，注册ReleaseMessageListener。在MessageScannerConfiguration中，调用该方法，初始化ReleaseMessageScanner的监听器们

@Configuration
public class ConfigServiceAutoConfiguration {
    
    
  
	@Configuration
  static class MessageScannerConfiguration {
    
    
    private final NotificationController notificationController;
    private final ConfigFileController configFileController;
    private final NotificationControllerV2 notificationControllerV2;
    private final GrayReleaseRulesHolder grayReleaseRulesHolder;
    private final ReleaseMessageServiceWithCache releaseMessageServiceWithCache;
    private final ConfigService configService;

    public MessageScannerConfiguration(
        final NotificationController notificationController,
        final ConfigFileController configFileController,
        final NotificationControllerV2 notificationControllerV2,
        final GrayReleaseRulesHolder grayReleaseRulesHolder,
        final ReleaseMessageServiceWithCache releaseMessageServiceWithCache,
        final ConfigService configService) {
    
    
      this.notificationController = notificationController;
      this.configFileController = configFileController;
      this.notificationControllerV2 = notificationControllerV2;
      this.grayReleaseRulesHolder = grayReleaseRulesHolder;
      this.releaseMessageServiceWithCache = releaseMessageServiceWithCache;
      this.configService = configService;
    }

    @Bean
    public ReleaseMessageScanner releaseMessageScanner() {
    
    
      ReleaseMessageScanner releaseMessageScanner = new ReleaseMessageScanner();
      //0. handle release message cache
      releaseMessageScanner.addMessageListener(releaseMessageServiceWithCache);
      //1. handle gray release rule
      releaseMessageScanner.addMessageListener(grayReleaseRulesHolder);
      //2. handle server cache
      releaseMessageScanner.addMessageListener(configService);
      releaseMessageScanner.addMessageListener(configFileController);
      //3. notify clients
      releaseMessageScanner.addMessageListener(notificationControllerV2);
      releaseMessageScanner.addMessageListener(notificationController);
      return releaseMessageScanner;
    }
  }

scanMessages()从DB中扫描ReleaseMessage，代码如下：

public class ReleaseMessageScanner implements InitializingBean {
    
      

  /**
   * 从DB中扫描ReleaseMessage
   *
   */
  private void scanMessages() {
    
    
    boolean hasMoreMessages = true;
    //循环扫描消息,直到没有新的ReleaseMessage为止
    while (hasMoreMessages && !Thread.currentThread().isInterrupted()) {
    
    
      hasMoreMessages = scanAndSendMessages();
    }
  }
  
  private boolean scanAndSendMessages() {
    
    
    //current batch is 500
    //获得大于maxIdScanned的500条ReleaseMessage记录,按照id升序
    List<ReleaseMessage> releaseMessages =
        releaseMessageRepository.findFirst500ByIdGreaterThanOrderByIdAsc(maxIdScanned);
    if (CollectionUtils.isEmpty(releaseMessages)) {
    
    
      return false;
    }
    //触发监听器
    fireMessageScanned(releaseMessages);
    //获得新的maxIdScanned,取最后一条记录
    int messageScanned = releaseMessages.size();
    maxIdScanned = releaseMessages.get(messageScanned - 1).getId();
    //若拉取不足500条,说明无新消息了
    return messageScanned == 500;
  }
  
  /**
   * 触发监听器,处理ReleaseMessage
   *
   * @param messages
   */
  private void fireMessageScanned(List<ReleaseMessage> messages) {
    
    
    for (ReleaseMessage message : messages) {
    
    
      for (ReleaseMessageListener listener : listeners) {
    
    
        try {
    
    
          //触发监听器
          listener.handleMessage(message, Topics.APOLLO_RELEASE_TOPIC);
        } catch (Throwable ex) {
    
    
          Tracer.logError(ex);
          logger.error("Failed to invoke message listener {}", listener.getClass(), ex);
        }
      }
    }
  }  

fireMessageScanned()方法触发ReleaseMessageListener监听器，处理ReleaseMessage。ReleaseMessageListener的子类NotificationControllerV2得到配置发布的appId+cluster+namespace后，会通知对应的客户端

3）、Config Service通知客户端

Config Service通知客户端的实现方式：

1. 客户端会发起一个Http请求到Config Service的notifications/v2接口，也就是NotificationControllerV2，参见RemoteConfigLongPollService
2. NotificationControllerV2不会立即返回结果，而是通过Spring DeferredResult把请求挂起
3. 如果在60秒内没有该客户端关心的配置发布，那么会返回Http状态码304给客户端
4. 如果有该客户端关心的配置发布，NotificationControllerV2会调用DeferredResult的setResult()方法，传入有配置变化的namespace信息，同时该请求会立即返回。客户端从返回的结果中获取到配置变化的namespace后，会立即请求Config Service获取该namespace的最新配置

Admin Service发送ReleaseMessage（异步）中我们讲到NotificationControllerV2是ReleaseMessageListener的子类，ReleaseMessageScanner会扫描ReleaseMessage消息，如果有新消息会通知触发ReleaseMessageListener监听器，当然也包括NotificationControllerV2，NotificationControllerV2得到配置发布的appId+cluster+namespace后会通知对应的客户端，我们先从NotificationControllerV2看起：

在目前Apollo的实现里，如下的名词是等价的：

• 通知编号 = ReleaseMessage.id
• Watch Key = ReleaseMessage.message

NotificationControllerV2：

NotificationControllerV2使用DeferredResult进行异步请求处理

当一个请求到达API接口，如果该API接口的return返回值是DeferredResult，在没有超时或者DeferredResult对象设置setResult时，接口不会返回，但是Servlet容器线程会结束，DeferredResult另起线程来进行结果处理（即这种操作提升了服务短时间的吞吐能力）并setResult，如此以来这个请求不会占用服务连接池太久，如果超时或设置setResult，接口会立即返回

使用DeferredResult的流程：

1. 浏览器发起异步请求

2. 请求到达服务端被挂起

3. 向浏览器进行响应，分为两种情况：

   1）调用DeferredResult.setResult()，请求被唤醒，返回结果

   2）超时，返回一个你设定的结果

4. 浏览得到响应，处理此次响应结果

关于DeferredResult的知识推荐阅读：

DeferredResult使用方式和场景

SpringMVC DeferredResult的Long Polling的应用

com.ctrip.framework.apollo.configservice.controller.NotificationControllerV2实现ReleaseMessageListener接口，通知Controller，仅提供notifications/v2接口

@RestController
@RequestMapping("/notifications/v2")
public class NotificationControllerV2 implements ReleaseMessageListener {
    
    
  private static final Logger logger = LoggerFactory.getLogger(NotificationControllerV2.class);
  /**
   * Watch Key与DeferredResultWrapper的Multimap
   * Watch Key等价于ReleaseMessage的通知内容message字段,appId+cluster+namespace
   */
  private final Multimap<String, DeferredResultWrapper> deferredResults =
      Multimaps.synchronizedSetMultimap(TreeMultimap.create(String.CASE_INSENSITIVE_ORDER, Ordering.natural()));
  private static final Splitter STRING_SPLITTER =
      Splitter.on(ConfigConsts.CLUSTER_NAMESPACE_SEPARATOR).omitEmptyStrings();
  private static final Type notificationsTypeReference =
      new TypeToken<List<ApolloConfigNotification>>() {
    
    
      }.getType();

  private final ExecutorService largeNotificationBatchExecutorService;

  private final WatchKeysUtil watchKeysUtil;
  private final ReleaseMessageServiceWithCache releaseMessageService;
  private final EntityManagerUtil entityManagerUtil;
  private final NamespaceUtil namespaceUtil;
  private final Gson gson;
  private final BizConfig bizConfig;

  @Autowired
  public NotificationControllerV2(
      final WatchKeysUtil watchKeysUtil,
      final ReleaseMessageServiceWithCache releaseMessageService,
      final EntityManagerUtil entityManagerUtil,
      final NamespaceUtil namespaceUtil,
      final Gson gson,
      final BizConfig bizConfig) {
    
    
    largeNotificationBatchExecutorService = Executors.newSingleThreadExecutor(ApolloThreadFactory.create
        ("NotificationControllerV2", true));
    this.watchKeysUtil = watchKeysUtil;
    this.releaseMessageService = releaseMessageService;
    this.entityManagerUtil = entityManagerUtil;
    this.namespaceUtil = namespaceUtil;
    this.gson = gson;
    this.bizConfig = bizConfig;
  }

  @GetMapping
  public DeferredResult<ResponseEntity<List<ApolloConfigNotification>>> pollNotification(
      @RequestParam(value = "appId") String appId,
      @RequestParam(value = "cluster") String cluster,
      @RequestParam(value = "notifications") String notificationsAsString,
      @RequestParam(value = "dataCenter", required = false) String dataCenter,
      @RequestParam(value = "ip", required = false) String clientIp) {
    
    
    //1)解析notificationsAsString参数,创建ApolloConfigNotification数组
    List<ApolloConfigNotification> notifications = null;

    try {
    
    
      notifications =
          gson.fromJson(notificationsAsString, notificationsTypeReference);
    } catch (Throwable ex) {
    
    
      Tracer.logError(ex);
    }

    if (CollectionUtils.isEmpty(notifications)) {
    
    
      throw new BadRequestException("Invalid format of notifications: " + notificationsAsString);
    }

    //过滤并创建ApolloConfigNotification Map
    Map<String, ApolloConfigNotification> filteredNotifications = filterNotifications(appId, notifications);

    if (CollectionUtils.isEmpty(filteredNotifications)) {
    
    
      throw new BadRequestException("Invalid format of notifications: " + notificationsAsString);
    }

    //创建DeferredResultWrapper对象 超时时间60s
    DeferredResultWrapper deferredResultWrapper = new DeferredResultWrapper(bizConfig.longPollingTimeoutInMilli());
    //Namespace集合
    Set<String> namespaces = Sets.newHashSetWithExpectedSize(filteredNotifications.size());
    //客户端的通知Map key为namespace名,value为通知编号(ReleaseMessage.id)
    Map<String, Long> clientSideNotifications = Maps.newHashMapWithExpectedSize(filteredNotifications.size());
    
    for (Map.Entry<String, ApolloConfigNotification> notificationEntry : filteredNotifications.entrySet()) {
    
    
      String normalizedNamespace = notificationEntry.getKey();
      ApolloConfigNotification notification = notificationEntry.getValue();
      //添加到namespaces中
      namespaces.add(normalizedNamespace);
      //添加到clientSideNotifications中
      clientSideNotifications.put(normalizedNamespace, notification.getNotificationId());
      if (!Objects.equals(notification.getNamespaceName(), normalizedNamespace)) {
    
    
        deferredResultWrapper.recordNamespaceNameNormalizedResult(notification.getNamespaceName(), normalizedNamespace);
      }
    }

    //组装Watch Key Multimap
    Multimap<String, String> watchedKeysMap =
        watchKeysUtil.assembleAllWatchKeys(appId, cluster, namespaces, dataCenter);
    //生成Watch Key集合
    Set<String> watchedKeys = Sets.newHashSet(watchedKeysMap.values());

    /**
     * 1、set deferredResult before the check, for avoid more waiting
     * If the check before setting deferredResult,it may receive a notification the next time
     * when method handleMessage is executed between check and set deferredResult.
     */
    //注册超时事件
    deferredResultWrapper
          .onTimeout(() -> logWatchedKeys(watchedKeys, "Apollo.LongPoll.TimeOutKeys"));
    //注册结束事件
    deferredResultWrapper.onCompletion(() -> {
    
    
      //unregister all keys
      //移除Watch Key+DeferredResultWrapper出deferredResults
      for (String key : watchedKeys) {
    
    
        deferredResults.remove(key, deferredResultWrapper);
      }
      logWatchedKeys(watchedKeys, "Apollo.LongPoll.CompletedKeys");
    });

    //register all keys
    //注册Watch Key+DeferredResultWrapper到deferredResults中,等待配置发生变化后通知
    for (String key : watchedKeys) {
    
    
      this.deferredResults.put(key, deferredResultWrapper);
    }

    logWatchedKeys(watchedKeys, "Apollo.LongPoll.RegisteredKeys");
    logger.debug("Listening {} from appId: {}, cluster: {}, namespace: {}, datacenter: {}",
        watchedKeys, appId, cluster, namespaces, dataCenter);

    //2)获得Watch Key集合中,每个Watch Key对应的ReleaseMessage记录
    List<ReleaseMessage> latestReleaseMessages =
        releaseMessageService.findLatestReleaseMessagesGroupByMessages(watchedKeys);

    /**
     * 手动关闭EntityManager
     * 因为对于async请求,Spring在请求完成之前不会这样做
     * 这是不可接受的,因为我们正在做长轮询——意味着db连接将被保留很长时间
     * 实际上,下面的过程,我们已经不需要db连接,因此进行关闭
     */
    entityManagerUtil.closeEntityManager();
    //3)获得新的ApolloConfigNotification通知数组
    List<ApolloConfigNotification> newNotifications =
        getApolloConfigNotifications(namespaces, clientSideNotifications, watchedKeysMap,
            latestReleaseMessages);

    //若有新的通知,直接设置结果
    if (!CollectionUtils.isEmpty(newNotifications)) {
    
    
      deferredResultWrapper.setResult(newNotifications);
    }

    return deferredResultWrapper.getResult();
  }
  
  /**
   * 通过ReleaseMessage的消息内容,获得对应namespace的名字 拆分appId+cluster+namespace拿到namespace
   */
  private static final Function<String, String> retrieveNamespaceFromReleaseMessage =
      releaseMessage -> {
    
    
        if (Strings.isNullOrEmpty(releaseMessage)) {
    
    
          return null;
        }
        List<String> keys = STRING_SPLITTER.splitToList(releaseMessage);
        //message should be appId+cluster+namespace
        if (keys.size() != 3) {
    
    
          logger.error("message format invalid - {}", releaseMessage);
          return null;
        }
        return keys.get(2);
      };  

pollNotification()核心源码如下：

1. 代码1)解析notificationsAsString参数，解析成List<ApolloConfigNotification>

   因为一个客户端可以订阅多个Namespace，所以该参数是List。该接口返回的结果也是List<ApolloConfigNotification> ，仅返回配置发生变化的Namespace对应的ApolloConfigNotification。也就说，当有几个配置发生变化的 Namespace ，返回几个对应的 ApolloConfigNotification 。另外，客户端接收到返回后，会增量合并到本地的配置通知信息。客户端下次请求时，使用合并后的配置通知信息。这里先留个印象后面解析Apollo Client源码的时候会讲到

   ApolloConfigNotification源码如下：

   public class ApolloConfigNotification {
         
         
     /**
      * namespace名字
      */
     private String namespaceName;
     
     /**
      * 最新通知编号
      * 目前使用ReleaseMessage.id
      */
     private long notificationId;
     
     /**
      * 通知消息集合
      */
     private volatile ApolloNotificationMessages messages;
   

   public class ApolloNotificationMessages {
         
         
     /**
      * 明细Map
      * key:appId+clusterName+namespace,例如:100004458+default+application
      * value:通知编号
      */
     private Map<String, Long> details;
   

2. 代码1)和代码2)之间是有关于DeferredResultWrapper的操作：

   com.ctrip.framework.apollo.configservice.wrapper.DeferredResultWrapper是DeferredResult包装器，封装DeferredResult的公用方法

   public class DeferredResultWrapper implements Comparable<DeferredResultWrapper> {
         
         
     /**
      * 未修改时的ResponseEntity响应,使用302状态码
      */
     private static final ResponseEntity<List<ApolloConfigNotification>>
         NOT_MODIFIED_RESPONSE_LIST = new ResponseEntity<>(HttpStatus.NOT_MODIFIED);
   
     /**
      * 归一化和原始的Namespace的名字的Map
      */
     private Map<String, String> normalizedNamespaceNameToOriginalNamespaceName;
   
     /**
      * 响应的DeferredResult对象
      */
     private DeferredResult<ResponseEntity<List<ApolloConfigNotification>>> result;
   
   
     public DeferredResultWrapper(long timeoutInMilli) {
         
         
       result = new DeferredResult<>(timeoutInMilli, NOT_MODIFIED_RESPONSE_LIST);
     }
     
     public void onTimeout(Runnable timeoutCallback) {
         
         
       result.onTimeout(timeoutCallback);
     }
   
     public void onCompletion(Runnable completionCallback) {
         
         
       result.onCompletion(completionCallback);
     }
   
   
     public void setResult(ApolloConfigNotification notification) {
         
         
       setResult(Lists.newArrayList(notification));
     }
   
     /**
      * The namespace name is used as a key in client side, so we have to return the original one instead of the correct one
      */
     public void setResult(List<ApolloConfigNotification> notifications) {
         
         
       if (normalizedNamespaceNameToOriginalNamespaceName != null) {
         
         
         //恢复被归一化的Namespace的名字为原始的Namespace的名字
         notifications.stream().filter(notification -> normalizedNamespaceNameToOriginalNamespaceName.containsKey
             (notification.getNamespaceName())).forEach(notification -> notification.setNamespaceName(
                 normalizedNamespaceNameToOriginalNamespaceName.get(notification.getNamespaceName())));
       }
       //设置结果,并使用200状态码
       result.setResult(new ResponseEntity<>(notifications, HttpStatus.OK));
     }
   
     public DeferredResult<ResponseEntity<List<ApolloConfigNotification>>> getResult() {
         
         
       return result;
     }  
   

   主要是创建DeferredResultWrapper对象（超时时间60s）；拿到客户端的通知Map，key为namespace名，value为客户端通知编号ReleaseMessage.id，如果Config Service的ReleaseMessage.id大于客户端本地的ReleaseMessage.id，则对应Watch Key（Watch Key = ReleaseMessage.message = appId+cluster+namespace）有配置更新；拿到Watch Key的集合；注册Watch Key+DeferredResultWrapper到deferredResults中；等待配置发生变化后通知注册超时事件和结束事件（从deferredResults中移除Watch Key+DeferredResultWrapper）

3. 代码2)获得Watch Key集合中，每个Watch Key对应的ReleaseMessage记录，这里是从Config Service本地Cache中获取的

4. 代码3)调用getApolloConfigNotifications()方法获得新的ApolloConfigNotification通知数组，如果此时就有新的通知,直接设置结果，getApolloConfigNotifications()源码如下：

   @RestController
   @RequestMapping("/notifications/v2")
   public class NotificationControllerV2 implements ReleaseMessageListener {
         
           
   
   	/**
      * 获得新的ApolloConfigNotification通知数组
      *
      * @param namespaces
      * @param clientSideNotifications
      * @param watchedKeysMap
      * @param latestReleaseMessages
      * @return
      */
     private List<ApolloConfigNotification> getApolloConfigNotifications(Set<String> namespaces,
                                                                         Map<String, Long> clientSideNotifications,
                                                                         Multimap<String, String> watchedKeysMap,
                                                                         List<ReleaseMessage> latestReleaseMessages) {
         
         
       //创建ApolloConfigNotification数组
       List<ApolloConfigNotification> newNotifications = Lists.newArrayList();
       if (!CollectionUtils.isEmpty(latestReleaseMessages)) {
         
         
         //创建最新通知的Map 其中Key为Watch Key
         Map<String, Long> latestNotifications = Maps.newHashMap();
         for (ReleaseMessage releaseMessage : latestReleaseMessages) {
         
         
           latestNotifications.put(releaseMessage.getMessage(), releaseMessage.getId());
         }
   
         //循环Namespace的名字的集合,判断是否有配置更新
         for (String namespace : namespaces) {
         
         
           long clientSideId = clientSideNotifications.get(namespace);
           long latestId = ConfigConsts.NOTIFICATION_ID_PLACEHOLDER;
           //获得Namespace对应的Watch Key集合
           Collection<String> namespaceWatchedKeys = watchedKeysMap.get(namespace);
           //获得最大的通知编号
           for (String namespaceWatchedKey : namespaceWatchedKeys) {
         
         
             long namespaceNotificationId =
                 latestNotifications.getOrDefault(namespaceWatchedKey, ConfigConsts.NOTIFICATION_ID_PLACEHOLDER);
             if (namespaceNotificationId > latestId) {
         
         
               latestId = namespaceNotificationId;
             }
           }
           //若服务器的通知编号大于客户端的通知编号,意味着有配置更新
           if (latestId > clientSideId) {
         
         
             //创建ApolloConfigNotification对象
             ApolloConfigNotification notification = new ApolloConfigNotification(namespace, latestId);
             //循环添加通知编号到ApolloConfigNotification中
             namespaceWatchedKeys.stream().filter(latestNotifications::containsKey).forEach(namespaceWatchedKey ->
                 notification.addMessage(namespaceWatchedKey, latestNotifications.get(namespaceWatchedKey)));
             //添加ApolloConfigNotification对象到结果
             newNotifications.add(notification);
           }
         }
       }
       return newNotifications;
     }
   

   其实核心逻辑就是若服务器的通知编号大于客户端的通知编号，意味着有配置更新

5. Admin Service发送ReleaseMessage（异步）中我们说到，ReleaseMessageScanner中的fireMessageScanned()方法触发ReleaseMessageListener监听器，处理ReleaseMessage。ReleaseMessageListener的子类NotificationControllerV2得到配置发布的appId+cluster+namespace后，会通知对应的客户端，NotificationControllerV2对应实现源码如下：

@RestController
@RequestMapping("/notifications/v2")
public class NotificationControllerV2 implements ReleaseMessageListener {
    
      

  @Override
  public void handleMessage(ReleaseMessage message, String channel) {
    
    
    logger.info("message received - channel: {}, message: {}", channel, message);

    String content = message.getMessage();
    Tracer.logEvent("Apollo.LongPoll.Messages", content);
    //仅处理APOLLO_RELEASE_TOPIC
    if (!Topics.APOLLO_RELEASE_TOPIC.equals(channel) || Strings.isNullOrEmpty(content)) {
    
    
      return;
    }

    //获得对应的Namespace的名字
    String changedNamespace = retrieveNamespaceFromReleaseMessage.apply(content);

    if (Strings.isNullOrEmpty(changedNamespace)) {
    
    
      logger.error("message format invalid - {}", content);
      return;
    }

    //deferredResults存在对应的Watch Key
    if (!deferredResults.containsKey(content)) {
    
    
      return;
    }

    //create a new list to avoid ConcurrentModificationException
    //创建DeferredResultWrapper数组,避免并发问题
    List<DeferredResultWrapper> results = Lists.newArrayList(deferredResults.get(content));

    //创建ApolloConfigNotification对象
    ApolloConfigNotification configNotification = new ApolloConfigNotification(changedNamespace, message.getId());
    configNotification.addMessage(content, message.getId());

    //do async notification if too many clients
    //1)若需要通知的客户端过多,使用ExecutorService异步通知
    if (results.size() > bizConfig.releaseMessageNotificationBatch()) {
    
    
      largeNotificationBatchExecutorService.submit(() -> {
    
    
        logger.debug("Async notify {} clients for key {} with batch {}", results.size(), content,
            bizConfig.releaseMessageNotificationBatch());
        for (int i = 0; i < results.size(); i++) {
    
    
          //每N个客户端,sleep一段时间
          if (i > 0 && i % bizConfig.releaseMessageNotificationBatch() == 0) {
    
    
            try {
    
    
              TimeUnit.MILLISECONDS.sleep(bizConfig.releaseMessageNotificationBatchIntervalInMilli());
            } catch (InterruptedException e) {
    
    
              //ignore
            }
          }
          logger.debug("Async notify {}", results.get(i));
          //设置结果
          results.get(i).setResult(configNotification);
        }
      });
      return;
    }

    logger.debug("Notify {} clients for key {}", results.size(), content);

    for (DeferredResultWrapper result : results) {
    
    
      //2)设置结果
      result.setResult(configNotification);
    }
    logger.debug("Notification completed");
  }  

• 【异步】处理：

  当需要通知的客户端过多，使用ExecutorService异步通知，避免惊群效应

  假设一个公共Namespace有10W台机器使用，如果该公共Namespace发布时直接下发配置更新消息的话，就会导致这10W台机器一下子都来请求配置，这动静就有点大了，而且对Config Service的压力也会比较大

  默认超过100个客户端sleep 100毫秒

• 【同步】处理：

代码2)循环调用DeferredResultWrapper的setResult() 方法，设置DeferredResult的结果，从而结束长轮询

参考：

https://ctripcorp.github.io/apollo/#/zh/design/apollo-design

https://www.iocoder.cn/Apollo

https://time.geekbang.org/column/article/175164