Ribbon load balancing strategy
It can be seen Ribbonthat a lot of selection strategies are implemented in. Let’s explain IRuleeach implementation of the interface in detail.
AbstractLoadBalancerRule
The abstract class of load balancing strategy, in which the load balancer object is defined ILoadBalancer. This object can obtain some information maintained in the load balancer as the basis for allocation when selecting a service strategy in specific implementation, and is designed accordingly. Some algorithms to implement efficient strategies for specific scenarios.
/**
* Class that provides a default implementation for setting and getting load balancer
* 该类提供用户设置和获取负载均衡器的默认实现
* @author stonse
*
*/
public abstract class AbstractLoadBalancerRule implements IRule, IClientConfigAware {
private ILoadBalancer lb;
@Override
public void setLoadBalancer(ILoadBalancer lb){
this.lb = lb;
}
@Override
public ILoadBalancer getLoadBalancer(){
return lb;
}
}
RandomRule
This strategy implements the function of randomly selecting a service instance from the service strength list.
@Override
public Server choose(Object key) {
return choose(getLoadBalancer(), key);
}
public Server choose(ILoadBalancer lb, Object key) {
if (lb == null) {
return null;
}
Server server = null;
while (server == null) {
if (Thread.interrupted()) {
return null;
}
//可用实例集合
List<Server> upList = lb.getReachableServers();
List<Server> allList = lb.getAllServers();
int serverCount = allList.size();
if (serverCount == 0) {
/*
* No servers. End regardless of pass, because subsequent passes
* only get more restrictive.
*/
return null;
}
int index = rand.nextInt(serverCount);
server = upList.get(index);
if (server == null) {
/*
* 服务器列表在被维护的情况下，可能会出现为null，释放CPU资源再重试
*/
Thread.yield();
continue;
}
if (server.isAlive()) {
return (server);
}
// Shouldn't actually happen.. but must be transient or a bug.
server = null;
Thread.yield();
}
return server;
}
You can see that the function implementation IRoleof the interface choose(Object key)is delegated to this class choose(ILoadBalancer lb, Object key). This method adds a parameter of the load balancer object. From the specific implementation point of view, he will use the incoming load balancer to obtain the available instance list upListand all instance lists allList, and rand.nextInt(serverCount)obtain a random number through the function, and use the random number as upListthe index to return the specific implementation. At the same time, the specific selection logic is within a while(server == null)loop. According to the implementation of the selection logic, under normal circumstances, a service instance should be selected for each selection. If an infinite loop occurs and the service instance cannot be obtained, there is a high possibility of concurrency. of Bug.
RoundRobinRule
This strategy implements the function of selecting each service instance at once in a linear polling manner.
Its detailed structure is RandomRulevery similar to except that the loop condition is different, which is the so-called logical difference obtained from the possible list. From the loop condition, we can see that a count variable has been added count, which will be accumulated after each loop value. That is to say, if the selection is less than server10 times, the attempt will end and a warning message will be printed. No up servers available from load balancer: .... The implementation of linear polling is AtomicInteger nextServerCyclicCounterimplemented through objects, and the increment is achieved by calling the function each time an instance is selected incrementAndGetModulo.
public Server choose(ILoadBalancer lb, Object key) {
if (lb == null) {
log.warn("no load balancer");
return null;
}
Server server = null;
int count = 0;
while (server == null && count++ < 10) {
List<Server> reachableServers = lb.getReachableServers();
List<Server> allServers = lb.getAllServers();
int upCount = reachableServers.size();
int serverCount = allServers.size();
if ((upCount == 0) || (serverCount == 0)) {
log.warn("No up servers available from load balancer: " + lb);
return null;
}
int nextServerIndex = incrementAndGetModulo(serverCount);
server = allServers.get(nextServerIndex);
if (server == null) {
/* Transient. */
Thread.yield();
continue;
}
if (server.isAlive() && (server.isReadyToServe())) {
return (server);
}
// Next.
server = null;
}
if (count >= 10) {
log.warn("No available alive servers after 10 tries from load balancer: "
+ lb);
}
return server;
}
RetryRule
This strategy implements an instance selection function with a retry mechanism. From the implementation below, we can see that an object is also defined internally IRule, and an instance is used by default RoundRobinRule. In choosethe method, the strategy of repeatedly trying the internally defined strategy is implemented. If a specific service instance can be selected during the period, it will be returned. If it cannot be selected, the set attempt end time will be the threshold ( ). When the threshold is maxRetryMillis 参数定义的值 + choose方法开始中的时间戳exceeded Then return null.
public class RetryRule extends AbstractLoadBalancerRule {
IRule subRule = new RoundRobinRule();
long maxRetryMillis = 500;
...
/*
* Loop if necessary. Note that the time CAN be exceeded depending on the
* subRule, because we're not spawning additional threads and returning
* early.
*/
public Server choose(ILoadBalancer lb, Object key) {
long requestTime = System.currentTimeMillis();
long deadline = requestTime + maxRetryMillis;
Server answer = null;
answer = subRule.choose(key);
if (((answer == null) || (!answer.isAlive()))
&& (System.currentTimeMillis() < deadline)) {
InterruptTask task = new InterruptTask(deadline
- System.currentTimeMillis());
while (!Thread.interrupted()) {
answer = subRule.choose(key);
if (((answer == null) || (!answer.isAlive()))
&& (System.currentTimeMillis() < deadline)) {
/* pause and retry hoping it's transient */
Thread.yield();
} else {
break;
}
}
task.cancel();
}
if ((answer == null) || (!answer.isAlive())) {
return null;
} else {
return answer;
}
}
...
}
WeightedResponseTimeRule
This strategy is an RoundRobinRuleextension of , adding the weight calculation based on the running status of the instance, and selecting instances based on the weight to achieve better allocation effects. Its implementation mainly has three core contents.
scheduled tasks
WeightedResponseTimeRuleWhen the policy is initialized serverWeightTimer.schedule(new DynamicServerWeightTask(), 0, serverWeightTaskTimerInterval), a scheduled task will be started to calculate the weight for each service instance. This task will be executed every 30 seconds by default.
class DynamicServerWeightTask extends TimerTask {
public void run() {
ServerWeight serverWeight = new ServerWeight();
try {
serverWeight.maintainWeights();
} catch (Throwable t) {
logger.error(
"Throwable caught while running DynamicServerWeightTask for "
+ name, t);
}
}
}
Weight calculation
We can easily find the object used to store weights in the source code List<Double> accumulatedWeights = new ArrayList<Double>(). The Listposition of each weight value in the corresponds to the position of all examples in the list of service instances maintained by the load balancer.
The calculation process of maintaining instance weight is through maintainWeightfunction matters, as shown in the following code:
public void maintainWeights() {
ILoadBalancer lb = getLoadBalancer();
if (lb == null) {
return;
}
if (serverWeightAssignmentInProgress.get()) {
return; // Ping in progress - nothing to do
} else {
serverWeightAssignmentInProgress.set(true);
}
try {
logger.info("Weight adjusting job started");
AbstractLoadBalancer nlb = (AbstractLoadBalancer) lb;
LoadBalancerStats stats = nlb.getLoadBalancerStats();
if (stats == null) {
// no statistics, nothing to do
return;
}
//所有示例的平均响应时间总和
double totalResponseTime = 0;
for (Server server : nlb.getAllServers()) {
// 如果服务实例的状态快照不在缓存汇总，这里会进行自动加载
ServerStats ss = stats.getSingleServerStat(server);
totalResponseTime += ss.getResponseTimeAvg();
}
// weight for each server is (sum of responseTime of all servers - responseTime)
// so that the longer the response time, the less the weight and the less likely to be chosen
Double weightSoFar = 0.0;
// create new list and hot swap the reference
List<Double> finalWeights = new ArrayList<Double>();
for (Server server : nlb.getAllServers()) {
ServerStats ss = stats.getSingleServerStat(server);
double weight = totalResponseTime - ss.getResponseTimeAvg();
weightSoFar += weight;
finalWeights.add(weightSoFar);
}
setWeights(finalWeights);
} catch (Throwable t) {
logger.error("Exception while dynamically calculating server weights", t);
} finally {
serverWeightAssignmentInProgress.set(false);
}
}
The implementation of this function is mainly divided into two steps:
- According to
LoadBalancerStatsthe statistics of each instance recorded in , the platform response time of all examples is accumulated to obtain the total average response timetotalResponseTime, which will be used in subsequent calculations. - Calculate the weight one by one (starting from the first one) for the instance list maintained in the load balancer. The calculation rule is
weightSoFar + totalResponseTime - 实例的平均响应时间, whereweightSoFaris initialized to zero, and each calculated weight needs to be accumulated toweightSoFarfor the next calculation.
Let's take a simple example to understand this calculation process. Suppose there are 4 instances A, B, C, and D. Their average response times are 10, 40, 60, 80, and 100, so the total average response time is, each
10 + 40 + 80 + 100 = 230instance The weight of is the cumulative difference between the total response time and the average response time of the instance itself, so the weights of instances A, B, C, and D are as follows.
- Example A:
230 - 10 = 220- Example B:
220 + (230 - 40) = 410- Example C:
410 + (230 - 80) = 560- Example D:
60 + (230 - 100) = 690It should be noted that the weight value here only represents the upper limit of the weight range of each instance, not the priority of a certain instance, so it does not mean that the larger the data, the greater the probability of being selected. So what is the weight interval? Taking the calculation above as an example, it actually constructs 4 different intervals for these 4 instances. The lower limit of the interval for each instance is the upper limit of the interval of the previous instance, and the upper limit of the interval for each instance is the upper limit of our interval.
List accumulatedWeightsThe weight value is calculated and stored in , where the lower bound for the first instance defaults to zero. Therefore, based on the weight calculation results of the above instances, we can get the weight range of each instance.
- Example A:
[0, 220]- Example B:
(220, 410]- Example C:
(410, 560]- Example D:
(560, 690)It is not difficult to find that the width of each interval on the instance is: total average response time - average response time of the instance, so the shorter the average response time of the instance, the greater the width of the weight interval, and the greater the width of the weight interval, the selected the higher the probability. How are the opening and closing of these interval boundaries determined? Why not so regular? This will be explained through the example selection algorithm below.
Instance selection
WeightedResponseTimeRuleThe implementation of the selected instance is similar to the algorithm structure introduced before. The following is the algorithm of its theme (the loop body and some judgments and other processing are omitted):
public Server choose(ILoadBalancer lb, Object key) {
if (lb == null) {
return null;
}
Server server = null;
while (server == null) {
// get hold of the current reference in case it is changed from the other thread
List<Double> currentWeights = accumulatedWeights;
if (Thread.interrupted()) {
return null;
}
List<Server> allList = lb.getAllServers();
int serverCount = allList.size();
if (serverCount == 0) {
return null;
}
int serverIndex = 0;
// 获取最后一个实例的权重
double maxTotalWeight = currentWeights.size() == 0 ? 0 : currentWeights.get(currentWeights.size() - 1);
// 如果最后一个实例的权重值小于0.001，则采用父类实现的线型轮询的策略
if (maxTotalWeight < 0.001d) {
server = super.choose(getLoadBalancer(), key);
if(server == null) {
return server;
}
} else {
// 如果最后一个实例的权重值大于等于0.001，就产生一个[0, maxTotalWeight)的随机数
double randomWeight = random.nextDouble() * maxTotalWeight;
int n = 0;
for (Double d : currentWeights) {
// 遍历维护的权重清单，若权重大于等于随机得到的数值，就选择这个实例
if (d >= randomWeight) {
serverIndex = n;
break;
} else {
n++;
}
}
server = allList.get(serverIndex);
}
if (server == null) {
/* Transient. */
Thread.yield();
continue;
}
if (server.isAlive()) {
return (server);
}
// Next.
server = null;
}
return server;
}
From the source code, we can see that the core process of selecting an instance is two steps:
-
Generate a
[0, 最大权重值)random number within a range. -
Traverse the weight list and compare the weight value with the random number. If the weight value is greater than or equal to the random number, use the index value of the current weight list to obtain the specific instance in the service instance list.
This is the principle that the service instances mentioned in the previous section will be selected based on the weight interval, and the opening and closing principle of the weight interval boundary is based on the algorithm. Normally each interval is in the form of, but why are the first instance and the last instance different
(x, y]? Woolen cloth? Since the minimum value of the random number can be 0, the lower limit of the first instance is a closed interval. At the same time, the maximum value of the random number cannot reach the maximum weight value, so the upper limit of the last instance is an open interval.
If you continue to use the above data as an example to select a service instance, this method will [0, 690)select a random number from the interval. For example, the selected random number is 230. Since this value is in the second interval, it will Select instance B to make the request.
ClientConfigEnabledRoundRobinRule
This strategy is quite special. We generally do not use it directly because it does not implement any special processing logic. As shown in the source code below, a strategy is defined inside it, and the implementation of the choose function also RoundRobinRuleuses RoundRobinRule’s linear polling mechanism, so the function it implements is actually the RoundRobinRulesame as .
chooseAlthough we will not use this strategy directly, by inheriting this strategy , the thread polling mechanism is implemented by default. When doing some advanced strategies in subclasses, there may often be some situations that cannot be implemented, so you can use the parent Class implementation as an alternative. In the following article, we will continue to introduce the advanced strategic military ClientConfigEnabledRoundRobinRuleexpansion based on .
public class ClientConfigEnabledRoundRobinRule extends AbstractLoadBalancerRule {
RoundRobinRule roundRobinRule = new RoundRobinRule();
...
@Override
public Server choose(Object key) {
if (roundRobinRule != null) {
return roundRobinRule.choose(key);
} else {
throw new IllegalArgumentException(
"This class has not been initialized with the RoundRobinRule class");
}
}
}
BestAvailableRule
This strategy is inherited from ClientConfigEnabledRoundRobinRule. In the implementation, it injects the statistical object of the load balancer , and uses the saved instance statistics LoadBalancerStatsin the specific choosealgorithm to select instances that meet the requirements. From the following source code, we can see that it traverses the load LoadBalancerStatsAll service instances maintained in the balancer will filter out failed instances and find the one with the smallest number of concurrent requests. Therefore, the characteristic of this strategy is to select the idlest instance.
public Server choose(Object key) {
if (loadBalancerStats == null) {
return super.choose(key);
}
List<Server> serverList = getLoadBalancer().getAllServers();
int minimalConcurrentConnections = Integer.MAX_VALUE;
long currentTime = System.currentTimeMillis();
Server chosen = null;
for (Server server: serverList) {
ServerStats serverStats = loadBalancerStats.getSingleServerStat(server);
if (!serverStats.isCircuitBreakerTripped(currentTime)) {
int concurrentConnections = serverStats.getActiveRequestsCount(currentTime);
if (concurrentConnections < minimalConcurrentConnections) {
minimalConcurrentConnections = concurrentConnections;
chosen = server;
}
}
}
if (chosen == null) {
return super.choose(key);
} else {
return chosen;
}
}
At the same time, since the core basis of the algorithm is the statistical object loadBalancerStats, when it is empty, the strategy cannot be executed. So we can see from the source code that when loadBalancerStatsis empty, it will adopt the linear polling strategy in the parent class. As we introduced ClientConfigEnabledRoundRobinRule, its subclasses can use it when they cannot meet the requirements for implementing advanced strategies. Characteristics of the linear polling strategy. The strategies that will be introduced later should all inherit from ClientConfigEnabledRoundRobinRule, so they will all have such characteristics.
PredicateBasedRule
This is an abstract strategy, which it also inherits ClientConfigEnabledRoundRobinRule. From its name, you can guess that this is a Predicatestrategy based on implementation, Predicatewhich is Google Guava Collectiona conditional interface for tools to filter collections.
As shown in the source code below, it defines an abstract function getPredicateto obtain AbstractServerPredicatethe implementation of the object, and in choosethe function, the specific service instance is selected through the function AbstractServerPredicate . chooseRoundRobinAfterFilteringFrom the naming of this function, we can roughly guess its basic logic: first filter some service instances through the logic implemented in the subclass Predicate, and then select one from the filtered instance list in a linear polling manner.
public abstract class PredicateBasedRule extends ClientConfigEnabledRoundRobinRule {
public abstract AbstractServerPredicate getPredicate();
@Override
public Server choose(Object key) {
ILoadBalancer lb = getLoadBalancer();
Optional<Server> server = getPredicate().chooseRoundRobinAfterFiltering(lb.getAllServers(), key);
if (server.isPresent()) {
return server.get();
} else {
return null;
}
}
}
The following AbstractServerPredicatesource code snippet can confirm the guess we made above. chooseThe method called in the above function first obtains the list of candidate instances (filtering is implemented) through the chooseRoundRobinAfterFilteringinternally defined function. If the returned list is empty, it is used to indicate that it does not exist. Otherwise, linear polling is used from Get an instance from the shortlist.getEligibleServersOptional.absent()
public abstract class AbstractServerPredicate implements Predicate<PredicateKey> {
...
public List<Server> getEligibleServers(List<Server> servers, Object loadBalancerKey) {
if (loadBalancerKey == null) {
return ImmutableList.copyOf(Iterables.filter(servers, this.getServerOnlyPredicate()));
} else {
List<Server> results = Lists.newArrayList();
for (Server server: servers) {
if (this.apply(new PredicateKey(loadBalancerKey, server))) {
results.add(server);
}
}
return results;
}
}
public Optional<Server> chooseRoundRobinAfterFiltering(List<Server> servers, Object loadBalancerKey) {
List<Server> eligible = getEligibleServers(servers, loadBalancerKey);
if (eligible.size() == 0) {
return Optional.absent();
}
return Optional.of(eligible.get(nextIndex.getAndIncrement() % eligible.size()));
}
...
}
After understanding the overall logic, let's take a detailed look at getEligibleServersthe functions that implement the filtering function. From the source code, its implementation structure is simple and clear. By traversing the service list, use this.applythe method to determine whether the instance needs to be retained, and if so, add it to the result list.
Friends who are not familiar with Google Guava Collectionscollection tools may be confused. This applydefinition cannot be found in AbstractServerPredicate, so how does it implement filtering? In fact, AbstractServerPredicate implements com.google.common.base.Predicatethe interface, and the apply method is defined in this interface. It is mainly used to implement the judgment logic of filter conditions. The parameters it inputs are some information (such as those in the source code) that the filter conditions need to use new PredicateKey(loadBalancerKey, server). Statistical information about the instance and the load balancer selection algorithm are passed in, so the function here chooseRoundRobinAfterFilteringjust defines a template strategy: "filter the list first, then poll for selection." As for how to filter, we need to AbstractServerPredicateimplement the method in the subclass applyto determine the specific filtering strategy.
The two strategies we will introduce later are implemented based on this abstract strategy, but they use different Predicateimplementations to complete the filtering logic to achieve different instance selection effects.
Google Guava Collectionsis anJava Collections Frameworkopen source project that enhances and extends . AlthoughJava Collections Frameworkit can meet our requirements for using collections in most cases, our code will be verbose and error-prone when encountering some special situations.Guava CollectionsIt can help us make the collection operation code shorter and more concise and greatly enhance the readability of the code.
AvailabilityFilteringRule
This strategy inherits from the abstract strategy introduced above PredicateBasedRule, so it also inherits the basic processing logic of "filter the list first, then poll for selection", in which the filter conditions are used AvailabilityPredicate:
public class AvailabilityPredicate extends AbstractServerPredicate {
...
@Override
public boolean apply(@Nullable PredicateKey input) {
LoadBalancerStats stats = getLBStats();
if (stats == null) {
return true;
}
return !shouldSkipServer(stats.getSingleServerStat(input.getServer()));
}
private boolean shouldSkipServer(ServerStats stats) {
if ((CIRCUIT_BREAKER_FILTERING.get() && stats.isCircuitBreakerTripped())
|| stats.getActiveRequestsCount() >= activeConnectionsLimit.get()) {
return true;
}
return false;
}
}
From the above source code, we can know that its main filtering logic is located shouldSkipServerin the method, and it mainly determines two contents of the service instance:
- It is a fault, that is, whether the circuit breaker is in effect and has been disconnected.
- The number of concurrent requests in the example is greater than the threshold. The default is
2的32次幂 - 1. This configuration can<clientName>.<nameSpace>.ActiveConnectionsLimitbe modified through parameters.
As long as one of these two items is satisfied, applyit will be returned false(indicating that the node may be faulty or the load is too high). If neither is satisfied, it will return true.
In this strategy, in addition to implementing the above filtering method, choosesome improvements and optimizations have also been made to the strategy, so the implementation of the parent class is only a backup option for it, and its specific implementation is as follows:
public Server choose(Object key) {
int count = 0;
Server server = roundRobinRule.choose(key);
while (count++ <= 10) {
if (predicate.apply(new PredicateKey(server))) {
return server;
}
server = roundRobinRule.choose(key);
}
return super.choose(key);
}
As you can see, it does not first traverse all nodes to filter, and then select instances in the filtered collection like in the parent class. Instead, select an instance linearly first. Then use the filtering conditions to determine whether the instance meets the requirements. If it is slowly grouped, use the instance directly. If it does not meet the requirements, select the next instance and check whether it meets the requirements. This cycle continues. When this process is repeated 10 times If an instance that meets the requirements is still not found, the implementation plan of the parent class is adopted.
Simply put, this strategy directly tries to find available and idle instances to use through linear abstraction, optimizing the overhead of the parent class traversing all examples every time.
ZoneAvoidanceRule
ZoneAvoidanceRuleWe have already mentioned this strategy when introducing the complex equalizer , and it is also PredicateBasedRulea specific implementation class of. The previous introduction mainly focused on some static functions used to select Zone regional ZoneAvoidanceRulestrategies, such as createSnapshot.getAvailableZones
Here we will take a closer look at ZoneAvoidanceRulehow this is implemented as a filter condition for service instances. As you can see from the source code judgment below ZoneAvoidanceRule, it uses CompositePredicateto filter the service instance list. This is a combined filter condition. In its constructor, it uses as ZoneAvoidancePredicatethe main filter condition and AvailabilityPredicateinitializes an instance of the combined filter condition for the secondary filter condition.
public class ZoneAvoidanceRule extends PredicateBasedRule {
...
private CompositePredicate compositePredicate;
public ZoneAvoidanceRule() {
super();
ZoneAvoidancePredicate zonePredicate = new ZoneAvoidancePredicate(this);
AvailabilityPredicate availabilityPredicate = new AvailabilityPredicate(this);
compositePredicate = createCompositePredicate(zonePredicate, availabilityPredicate);
}
...
}
When implemented, ZoneAvoidanceRule did not AvailabilityFilteringRulerewrite choosethe function for optimization like that, so it completely followed the main filtering logic of the parent class: "filter light first, then poll for selection." Among them, the condition for filtering lightness is the combined filtering condition we mentioned above, with is ZoneAvoidancePredicatethe main filtering condition and the secondary filtering condition . From the source code snippet, we can see that it defines a main filter condition and a set of secondary filter condition lists , so it can have multiple secondary filter lists, and because it uses storage, the secondary filter conditions are based on executed sequentially.AvailabilityPredicateCompositePredicateCompositePredicateAbstractServerPredicate delegateList<AbstractServerPredicate> fallbacksList
public class CompositePredicate extends AbstractServerPredicate {
private AbstractServerPredicate delegate;
private List<AbstractServerPredicate> fallbacks = Lists.newArrayList();
private int minimalFilteredServers = 1;
private float minimalFilteredPercentage = 0;
@Override
public boolean apply(@Nullable PredicateKey input) {
return delegate.apply(input);
}
...
/**
* 从主过滤条件获取筛选过的服务器，如果筛选过的服务器达不到指定要求，则用次过滤条件进行筛选
*/
@Override
public List<Server> getEligibleServers(List<Server> servers, Object loadBalancerKey) {
List<Server> result = super.getEligibleServers(servers, loadBalancerKey);
Iterator<AbstractServerPredicate> i = fallbacks.iterator();
while (!(result.size() >= minimalFilteredServers && result.size() > (int) (servers.size() * minimalFilteredPercentage))
&& i.hasNext()) {
AbstractServerPredicate predicate = i.next();
result = predicate.getEligibleServers(servers, loadBalancerKey);
}
return result;
}
}
In the implementation function for obtaining filtering results getEligibleServers, its processing logic is as follows:
- Filter all instances using the main filter and return the filtered instance list.
- Filter all instances using the filter conditions in the filter list in sequence.
- After each filtering (including primary filtering conditions and secondary filtering conditions), the following two conditions need to be judged. As long as one requirement is not met, continue to use the next filtering condition for filtering, and the final result that meets the two conditions will be returned to linear Polling algorithm selection:
- Total number of instances after filtering >= Minimum filtering is membership (
minimalFilteredSevers, default 1). - Filtered instance proportion > minimum filtered percentage (
minimalFilteredPercentage, default 0).
- Total number of instances after filtering >= Minimum filtering is membership (