Eureka self-protection mechanism source code analysis

By default, when EurekaServer does not receive a heartbeat from a client instance within a certain period of time (default 90 seconds), EurekaServer will log out of the instance. However, when a network partition failure occurs, the client and EurekaServer cannot communicate normally, and the client should not be logged out at this time. Eureka solves this problem through a "self-protection mechanism": when EurekaServer loses too many clients in a short period of time, the node will enter self-protection mode. In self-protection mode, EurekaServer will not cull any clients. When the network failure recovers, the node will automatically exit the self-protection mode

The implementation of the self-protection mechanism is maintained based on AbstractInstanceRegistry2 variables in the class that maintains the service registry

/**
* 期望最小每分钟续租次数
*/
protected volatile int numberOfRenewsPerMinThreshold;
/**
* 期望最大每分钟续租次数
*/
protected volatile int expectedNumberOfRenewsPerMin;

Server initialization

The server-side startup article can be found in this article: EurekaServer auto-assembly and startup process analysis After the server-side startup and synchronization of information from other clusters, a method is executed:openForTraffic

public void openForTraffic(ApplicationInfoManager applicationInfoManager, int count) {
   this.expectedNumberOfRenewsPerMin = count * 2;
   this.numberOfRenewsPerMinThreshold =
           (int) (this.expectedNumberOfRenewsPerMin * serverConfig.getRenewalPercentThreshold());
    }

The expected maximum number of lease renewals per minute is: the number of registered clients on the current server multiplied by 2, why, because the default Eureka renewal is 30 seconds. The expected minimum number of renewals per minute is: the maximum number of renewals multiplied by the renewal Percentage, the default renewal percentage is 0.85, which means that if more than 15% of the clients do not renew the lease within a certain time window, the self-protection mode will be turned on

Scheduled tasks in self-protection mode

DefaultEurekaServerContextThere is a initializemethod in the class, this method will start a scheduled task during the execution process

    @PostConstruct
    @Override
    public void initialize() {
        logger.info("Initializing ...");
        peerEurekaNodes.start();
        try {
            registry.init(peerEurekaNodes);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        logger.info("Initialized");
    }

public void init(PeerEurekaNodes peerEurekaNodes) throws Exception {
        this.numberOfReplicationsLastMin.start();
        this.peerEurekaNodes = peerEurekaNodes;
        initializedResponseCache();
        scheduleRenewalThresholdUpdateTask();
        initRemoteRegionRegistry();

        try {
            Monitors.registerObject(this);
        } catch (Throwable e) {
            logger.warn("Cannot register the JMX monitor for the InstanceRegistry :", e);
        }
    }

scheduleRenewalThresholdUpdateTaskThis timed task is related to the self-protection mode.

    private void scheduleRenewalThresholdUpdateTask() {
        timer.schedule(new TimerTask() {
                           @Override
                           public void run() {
                               updateRenewalThreshold();
                           }
                       }, serverConfig.getRenewalThresholdUpdateIntervalMs(),
                serverConfig.getRenewalThresholdUpdateIntervalMs());
    }

where the getRenewalThresholdUpdateIntervalMsdefault value is 15 minutes

private void updateRenewalThreshold() {
   try {
       // 1. 计算应用实例数
       Applications apps = eurekaClient.getApplications();
       int count = 0;
       for (Application app : apps.getRegisteredApplications()) {
           for (InstanceInfo instance : app.getInstances()) {
               if (this.isRegisterable(instance)) {
                   ++count;
               }
           }
       }
       // 2. 计算expectedNumberOfRenewsPerMin 、 numberOfRenewsPerMinThreshold 参数
       synchronized (lock) {
           if ((count * 2) > (serverConfig.getRenewalPercentThreshold() * numberOfRenewsPerMinThreshold)
                   || (!this.isSelfPreservationModeEnabled())) {
               this.expectedNumberOfRenewsPerMin = count * 2;
               this.numberOfRenewsPerMinThreshold = (int) ((count * 2) * serverConfig.getRenewalPercentThreshold());
           }
       }
       logger.info("Current renewal threshold is : {}", numberOfRenewsPerMinThreshold);
   } catch (Throwable e) {
       logger.error("Cannot update renewal threshold", e);
   }
}

Divided into 2 steps, the first step will not be mentioned, look at the second step. When the maximum number of lease renewals is greater than the minimum number of lease renewals or the self-protection mode is not turned on, the two values ​​can be recalculated, otherwise it cannot be recalculated.

Client registration

public void register(InstanceInfo registrant, int leaseDuration, boolean isReplication) {
    synchronized (lock) {
         if (this.expectedNumberOfRenewsPerMin > 0) {
             this.expectedNumberOfRenewsPerMin = this.expectedNumberOfRenewsPerMin + 2;
             this.numberOfRenewsPerMinThreshold =
                     (int) (this.expectedNumberOfRenewsPerMin * serverConfig.getRenewalPercentThreshold());
         }
     }

}

Whenever an instance is registered, the two parameters must be recalculated. The maximum expected number of lease renewals + 2 is also because the default lease renewal is 2 times per minute.

Client offline

public boolean cancel(final String appName, final String id,
                     final boolean isReplication) {
      
   synchronized (lock) {
        if (this.expectedNumberOfRenewsPerMin > 0) {
               this.expectedNumberOfRenewsPerMin = this.expectedNumberOfRenewsPerMin - 2;
               this.numberOfRenewsPerMinThreshold = (int) (this.expectedNumberOfRenewsPerMin * serverConfig.getRenewalPercentThreshold());
        }
   }
}

The processing logic for registration is just the opposite

Turn on self-protection mode

Previously, the judgment about self-protection mode was omitted in the process of lease expiration cleanup and analysis in the article on Eureka client renewal and server-side expired lease cleanup source code analysis . Now let's look at it again. This judgment is at the beginning of the lease expiration processing method:

    public void evict(long additionalLeaseMs) {
        logger.debug("Running the evict task");

        if (!isLeaseExpirationEnabled()) {
            logger.debug("DS: lease expiration is currently disabled.");
            return;
        }
        //....
   }

Details are isLeaseExpirationEnabledin

    public boolean isLeaseExpirationEnabled() {
        if (!isSelfPreservationModeEnabled()) {
            return true;
        }
        return numberOfRenewsPerMinThreshold > 0 && getNumOfRenewsInLastMin() > numberOfRenewsPerMinThreshold;
    }
    public boolean isSelfPreservationModeEnabled() {
        return serverConfig.shouldEnableSelfPreservation();
    }
    public long getNumOfRenewsInLastMin() {
        return renewsLastMin.getCount();
    }

The first if is to determine whether the self-protection mode is enabled. The last return is if the current minimum number of lease renewals is greater than 0, and the number of recently renewed instances is greater than the minimum expected number of lease renewals.

{{o.name}}

{{m.name}}