coding ++: RateLimiter limiting leaky bucket algorithm, the token bucket algorithm

RateLimiter for restricting access frequency is under the concurrent bags Guava

 <dependency>
      <groupId>com.google.guava</groupId>
      <artifactId>guava</artifactId>
      <version>18.0</version>
</dependency>

Limiting:

Each API interface all have access to the upper limit, when the amount of concurrent access frequency or exceed its tolerance range, we have to consider limiting the availability or downgrade to ensure the availability of the interface. That interface is also need to install the fuse to prevent unintended request for the system caused by excessive pressure system failures.

A common strategy is to reject unwanted access, or allow excess access waiting service, or drainage.

If you want to accurately control the QPS, the simplest approach is to maintain Counter in a unit of time, such as determining the unit time has elapsed, then the Counter is reset to zero. This practice is considered no good boundary processing unit time, such as the former the last second of the first millisecond in a millisecond, and the next second triggers the greatest number of requests, the eyes move about, to see QPS happened twice within two milliseconds.

1, the leaky bucket algorithm

Bucket (Leaky Bucket) algorithm is very simple idea, water (request) to the drain into the first bucket, bucket water at a constant speed (rate responsive to the interface), the inflow rate when the water overflows through direct Assembly (access frequency response over the interface rate), then reject the request, the transmission rate can be seen that the leaky bucket algorithm can impose restrictions on the data in the following schematic: 

There are seen two variables is the size of a barrel, can support the number of stored water flow rate increases burst (Burst), and the other is the size of the bucket vulnerability (Rate), the following pseudo-code:

Double Rate; // Leak Rate in Calls / S 
Double Burst; // bucket size in Calls 
Long refreshTime; // Time for Water Last Refresh 
Double Water; // Water COUNT AT refreshTime 
refreshWater () {
 Long now = gettimeofday ();
 // water over time, continue to flow away, it is up to the dry stream to 0. 
water = max (0, water- (now - refreshTime) * Rate);
　　refreshTime = now;
}
bool permissionGranted() {
　　refreshWater();
　　IF (Water <Burst) { // bucket is not full, continue to increase. 1 
　　　　Water ++ ;
　　　　return  to true ;
　　} else {
　　　　return false;
　　}
}

 Because the leakage rate of leaky bucket parameters are fixed, so that, even if the network resource conflict does not exist (no congestion occurs), the token bucket algorithm does not make stream burst (Burst) to the port speed. Thus, the presence of the token bucket algorithm for projection flow characteristics of the hair is inefficient.

2, the token bucket algorithm

Token bucket algorithm (Token Bucket), and the same effect Leaky Bucket algorithm, but in the opposite direction, more readily understood. Over time, the system will be a constant 1 / QPS interval (if QPS = 100, the interval is 10ms) to the tub was added Token (imagine leaks and loopholes contrary, there is a constant increase in the tap water), if the bucket is full is not coupled with a new request comes, will each take a Token, if it took no Token can be blocked or denial of service.

Another advantage is the token bucket can easily change the speed. Once the need to increase the rate, the demand to increase the rate of tokens into the bucket. Generally timing (for example 100 milliseconds) increased the tub to a certain number of tokens , the number of variants of some of the real-time calculation algorithm should increase the token.

3, RateLimiter Profile

Google open source toolkit Guava provides current limiting tools RateLimiter, class token bucket algorithm (Token Bucket) based on the completion of current limiting, very easy to use .RateLimiter rate is often used to restrict access to some of the physical resources or logical resources. It supports two interfaces to obtain permits, if one is unable to get immediate returns false, one kind will block waiting for some time whether we can get.

Semaphore (java.util.concurrent.Semaphore) RateLimiter and Java, and the like, Semaphore commonly used to limit the amount of concurrency.

An example of source code comments, such as we have a lot of tasks to be performed, but we do not want more than two tasks are performed per second, then we can use RateLimiter:

final RateLimiter rateLimiter = RateLimiter.create(2.0);
void submitTasks(List<Runnable> tasks, Executor executor) {
    for (Runnable task : tasks) {
        rateLimiter.acquire(); // may wait
        executor.execute(task);
    }
}                

另外一个例子,假如我们会产生一个数据流,然后我们想以每秒5kb的速度发送出去.我们可以每获取一个令牌(permit)就发送一个byte的数据,这样我们就可以通过一个每秒5000个令牌的RateLimiter来实现:

final RateLimiter rateLimiter = RateLimiter.create(5000.0);
void submitPacket(byte[] packet) {
    rateLimiter.acquire(packet.length);
    networkService.send(packet);
}

另外,我们也可以使用非阻塞的形式达到降级运行的目的,即使用非阻塞的tryAcquire()方法:

if(limiter.tryAcquire()) { //未请求到limiter则立即返回false
    doSomething();
}else{
    doSomethingElse();
}

4、RateLimiter主要接口

RateLimiter其实是一个abstract类,但是它提供了几个static方法用于创建RateLimiter:

/**
* 创建一个稳定输出令牌的RateLimiter，保证了平均每秒不超过permitsPerSecond个请求
* 当请求到来的速度超过了permitsPerSecond，保证每秒只处理permitsPerSecond个请求
* 当这个RateLimiter使用不足(即请求到来速度小于permitsPerSecond)，会囤积最多permitsPerSecond个请求
*/
public static RateLimiter create(double permitsPerSecond);
/**
* 创建一个稳定输出令牌的RateLimiter，保证了平均每秒不超过permitsPerSecond个请求
* 还包含一个热身期(warmup period),热身期内，RateLimiter会平滑的将其释放令牌的速率加大，直到起达到最大速率
* 同样，如果RateLimiter在热身期没有足够的请求(unused),则起速率会逐渐降低到冷却状态
*
* 设计这个的意图是为了满足那种资源提供方需要热身时间，而不是每次访问都能提供稳定速率的服务的情况(比如带缓存服务，需要定期刷新缓存的)
* 参数warmupPeriod和unit决定了其从冷却状态到达最大速率的时间
*/
public static RateLimiter create(double permitsPerSecond, long warmupPeriod, TimeUnit unit);

提供了两个获取令牌的方法,不带参数表示获取一个令牌.如果没有令牌则一直等待,返回等待的时间(单位为秒),没有被限流则直接返回0.0:

public double acquire();
public double acquire(int permits);

尝试获取令牌,分为待超时时间和不带超时时间两种:

public boolean tryAcquire();
//尝试获取一个令牌,立即返回
public boolean tryAcquire(int permits);
public boolean tryAcquire(long timeout, TimeUnit unit);
//尝试获取permits个令牌,带超时时间
public boolean tryAcquire(int permits, long timeout, TimeUnit unit);

4.1：RateLimiter 方法摘要

5、RateLimiter设计

考虑一下RateLimiter是如何设计的,并且为什么要这样设计.

RateLimiter的主要功能就是提供一个稳定的速率,实现方式就是通过限制请求流入的速度,比如计算请求等待合适的时间阈值.

实现QPS速率的最简单的方式就是记住上一次请求的最后授权时间,然后保证1/QPS秒内不允许请求进入.比如QPS=5,如果我们保证最后一个被授权请求之后的200ms的时间内没有请求被授权,那么我们就达到了预期的速率.如果一个请求现在过来但是最后一个被授权请求是在100ms之前,那么我们就要求当前这个请求等待100ms.按照这个思路,请求15个新令牌(许可证)就需要3秒.

有一点很重要:上面这个设计思路的RateLimiter记忆非常的浅,它的脑容量非常的小,只记得上一次被授权的请求的时间.如果RateLimiter的一个被授权请求q之前很长一段时间没有被使用会怎么样?这个RateLimiter会立马忘记过去这一段时间的利用不足,而只记得刚刚的请求q.

过去一段时间的利用不足意味着有过剩的资源是可以利用的.这种情况下,RateLimiter应该加把劲(speed up for a while)将这些过剩的资源利用起来.比如在向网络中发生数据的场景(限流),过去一段时间的利用不足可能意味着网卡缓冲区是空的,这种场景下,我们是可以加速发送来将这些过程的资源利用起来.

另一方面,过去一段时间的利用不足可能意味着处理请求的服务器对即将到来的请求是准备不足的(less ready for future requests),比如因为很长一段时间没有请求当前服务器的cache是陈旧的,进而导致即将到来的请求会触发一个昂贵的操作(比如重新刷新全量的缓存).

为了处理这种情况,RateLimiter中增加了一个维度的信息,就是过去一段时间的利用不足(past underutilization),代码中使用storedPermits变量表示.当没有利用不足这个变量为0,最大能达到maxStoredPermits(maxStoredPermits表示完全没有利用).因此,请求的令牌可能从两个地方来:

1.过去剩余的令牌(stored permits, 可能没有)
2.现有的令牌(fresh permits,当前这段时间还没用完的令牌)

我们将通过一个例子来解释它是如何工作的:

对一个每秒产生一个令牌的RateLimiter,每有一个没有使用令牌的一秒,我们就将storedPermits加1,如果RateLimiter在10秒都没有使用,则storedPermits变成10.0.这个时候,一个请求到来并请求三个令牌(acquire(3)),我们将从storedPermits中的令牌为其服务,storedPermits变为7.0.这个请求之后立马又有一个请求到来并请求10个令牌,我们将从storedPermits剩余的7个令牌给这个请求,剩下还需要三个令牌,我们将从RateLimiter新产生的令牌中获取.我们已经知道,RateLimiter每秒新产生1个令牌,就是说上面这个请求还需要的3个请求就要求其等待3秒.

想象一个RateLimiter每秒产生一个令牌,现在完全没有使用(处于初始状态),限制一个昂贵的请求acquire(100)过来.如果我们选择让这个请求等待100秒再允许其执行,这显然很荒谬.我们为什么什么也不做而只是傻傻的等待100秒,一个更好的做法是允许这个请求立即执行(和acquire(1)没有区别),然后将随后到来的请求推迟到正确的时间点.这种策略,我们允许这个昂贵的任务立即执行,并将随后到来的请求推迟100秒.这种策略就是让任务的执行和等待同时进行.

一个重要的结论:RateLimiter不会记最后一个请求,而是即下一个请求允许执行的时间.这也可以很直白的告诉我们到达下一个调度时间点的时间间隔.然后定一个一段时间未使用的Ratelimiter也很简单:下一个调度时间点已经过去,这个时间点和现在时间的差就是Ratelimiter多久没有被使用,我们会将这一段时间翻译成storedPermits.所有,如果每秒钟产生一个令牌(rate==1),并且正好每秒来一个请求,那么storedPermits就不会增长

6、RateLimiter主要源码

RateLimiter定义了两个create函数用于构建不同形式的RateLimiter:

//用于创建SmoothBursty类型的RateLimiter
1.public static RateLimiter create(double permitsPerSecond)

2.public static RateLimiter create(double permitsPerSecond, long warmupPeriod, TimeUnit unit)

 源码下面以acquire为例子,分析一下RateLimiter如何实现限流:

public double acquire() {
　　return acquire(1);
}
public double acquire(int permits) {
　　long microsToWait = reserve(permits);
　　stopwatch.sleepMicrosUninterruptibly(microsToWait);
　　return 1.0 * microsToWait / SECONDS.toMicros(1L);
}
final long reserve(int permits) {
　　checkPermits(permits);
　　synchronized (mutex()) { //应对并发情况需要同步
　　return reserveAndGetWaitLength(permits, stopwatch.readMicros());
}
}
final long reserveAndGetWaitLength(int permits, long nowMicros) {
　　long momentAvailable = reserveEarliestAvailable(permits, nowMicros);
　　return max(momentAvailable - nowMicros, 0);
}

 下面方法来自RateLimiter的具体实现类SmoothRateLimiter:

final long reserveEarliestAvailable(int requiredPermits, long nowMicros) {
　　resync(nowMicros); //补充令牌
　　long returnValue = nextFreeTicketMicros;
　　//这次请求消耗的令牌数目
　　double storedPermitsToSpend = min(requiredPermits, this.storedPermits);
　　double freshPermits = requiredPermits - storedPermitsToSpend;
　　long waitMicros = storedPermitsToWaitTime(this.storedPermits, storedPermitsToSpend)
　　+ (long) (freshPermits * stableIntervalMicros);
　　this.nextFreeTicketMicros = nextFreeTicketMicros + waitMicros;
　　this.storedPermits -= storedPermitsToSpend;
　　return returnValue;
}
private void resync(long nowMicros) {
　　// if nextFreeTicket is in the past, resync to now
　　if (nowMicros > nextFreeTicketMicros) {
　　　　storedPermits = min(maxPermits,
　　　　storedPermits + (nowMicros - nextFreeTicketMicros) / stableIntervalMicros);
　　　　nextFreeTicketMicros = nowMicros;
　　}
}

 另外，对于storedPermits的使用，RateLimiter存在两种策略，二者区别主要体现在使用storedPermits时候需要等待的时间。这个逻辑由storedPermitsToWaitTime函数实现：

/**
* Translates a specified portion of our currently stored permits which we want to
* spend/acquire, into a throttling time. Conceptually, this evaluates the integral
* of the underlying function we use, for the range of
* [(storedPermits - permitsToTake), storedPermits].
*
* <p>This always holds: {@code 0 <= permitsToTake <= storedPermits}
*/
abstract long storedPermitsToWaitTime(double storedPermits, double permitsToTake);

存在两种策略就是为了应对我们上面讲到的，存在资源使用不足大致分为两种情况： (1).资源确实使用不足，这些剩余的资源我们私海可以使用的； (2).提供资源的服务过去还没准备好，比如服务刚启动等；

为此，RateLimiter实际上由两种实现策略，其实现分别见SmoothBursty和SmoothWarmingUp。二者主要的区别就是storedPermitsToWaitTime实现以及maxPermits数量的计算。

6.1 SmoothBursty

SmoothBursty使用storedPermits不需要额外等待时间。并且默认maxBurstSeconds未1，因此maxPermits为permitsPerSecond，即最多可以存储1秒的剩余令牌，比如QPS=5，则maxPermits=5.

下面这个RateLimiter的入口就是用来创建SmoothBursty类型的RateLimiter，

public static RateLimiter create(double permitsPerSecond)

/**
* This implements a "bursty" RateLimiter, where storedPermits are translated to
* zero throttling. The maximum number of permits that can be saved (when the RateLimiter is
* unused) is defined in terms of time, in this sense: if a RateLimiter is 2qps, and this
* time is specified as 10 seconds, we can save up to 2 * 10 = 20 permits.
*/
static final class SmoothBursty extends SmoothRateLimiter {
/** The work (permits) of how many seconds can be saved up if this RateLimiter is unused? */
　　final double maxBurstSeconds;
　　SmoothBursty(SleepingStopwatch stopwatch, double maxBurstSeconds) {
　　　　super(stopwatch);
　　　　this.maxBurstSeconds = maxBurstSeconds;
　　}
　　void doSetRate(double permitsPerSecond, double stableIntervalMicros) {
　　　　double oldMaxPermits = this.maxPermits;
　　　　maxPermits = maxBurstSeconds * permitsPerSecond;
　　　　System.out.println("maxPermits=" + maxPermits);
　　　　if (oldMaxPermits == Double.POSITIVE_INFINITY) {
　　　　　　// if we don't special-case this, we would get storedPermits == NaN, below
　　　　　　storedPermits = maxPermits;
　　　　} else {
　　　　　　storedPermits = (oldMaxPermits == 0.0)
　　　　　　? 0.0 // initial state
　　　　　　: storedPermits * maxPermits / oldMaxPermits;
　　　　}
　　}
　　　　long storedPermitsToWaitTime(double storedPermits, double permitsToTake) {
　　　　return 0L;
　　}
}

一个简单的使用示意图及解释，下面私海一个QPS=4的SmoothBursty:

(1).t=0,这时候storedPermits=0，请求1个令牌，等待时间=0；
(2).t=1,这时候storedPermits=3，请求3个令牌，等待时间=0；
(3).t=2,这时候storedPermits=4，请求10个令牌，等待时间=0，超前使用了2个令牌；
(4).t=3,这时候storedPermits=0，请求1个令牌，等待时间=0.5；

代码的输出：

maxPermits=4.0, storedPermits=7.2E-4, stableIntervalMicros=250000.0, nextFreeTicketMicros=1472
acquire(1), sleepSecond=0.0
maxPermits=4.0, storedPermits=3.012212, stableIntervalMicros=250000.0, nextFreeTicketMicros=1004345
acquire(3), sleepSecond=0.0
maxPermits=4.0, storedPermits=4.0, stableIntervalMicros=250000.0, nextFreeTicketMicros=2004668
acquire(10), sleepSecond=0.0
maxPermits=4.0, storedPermits=0.0, stableIntervalMicros=250000.0, nextFreeTicketMicros=3504668
acquire(1), sleepSecond=0.499591

6.2 SmoothWarmingUp

  static final class SmoothWarmingUp extends SmoothRateLimiter {
    private final long warmupPeriodMicros;
    /**
     * The slope of the line from the stable interval (when permits == 0), to the cold interval
     * (when permits == maxPermits)
     */
    private double slope;
    private double halfPermits;
  
    SmoothWarmingUp(SleepingStopwatch stopwatch, long warmupPeriod, TimeUnit timeUnit) {
      super(stopwatch);
      this.warmupPeriodMicros = timeUnit.toMicros(warmupPeriod);
    }
  
    @Override
    void doSetRate(double permitsPerSecond, double stableIntervalMicros) {
      double oldMaxPermits = maxPermits;
      maxPermits = warmupPeriodMicros / stableIntervalMicros;
      halfPermits = maxPermits / 2.0;
      // Stable interval is x, cold is 3x, so on average it's 2x. Double the time -> halve the rate
      double coldIntervalMicros = stableIntervalMicros * 3.0;
      slope = (coldIntervalMicros - stableIntervalMicros) / halfPermits;
      if (oldMaxPermits == Double.POSITIVE_INFINITY) {
        // if we don't special-case this, we would get storedPermits == NaN, below
        storedPermits = 0.0;
      } else {
        storedPermits = (oldMaxPermits == 0.0)
            ? maxPermits // initial state is cold
            : storedPermits * maxPermits / oldMaxPermits;
      }
    }
  
    @Override
    long storedPermitsToWaitTime(double storedPermits, double permitsToTake) {
      double availablePermitsAboveHalf = storedPermits - halfPermits;
      long micros = 0;
      // measuring the integral on the right part of the function (the climbing line)
      if (availablePermitsAboveHalf > 0.0) {
        double permitsAboveHalfToTake = min(availablePermitsAboveHalf, permitsToTake);
        micros = (long) (permitsAboveHalfToTake * (permitsToTime(availablePermitsAboveHalf)
            + permitsToTime(availablePermitsAboveHalf - permitsAboveHalfToTake)) / 2.0);
        permitsToTake -= permitsAboveHalfToTake;
      }
      // measuring the integral on the left part of the function (the horizontal line)
      micros += (stableIntervalMicros * permitsToTake);
      return micros;
    }
  
    private double permitsToTime(double permits) {
      return stableIntervalMicros + permits * slope;
    }
  }

maxPermits等于热身(warmup)期间能产生的令牌数，比如QPS=4，warmup为2秒，则maxPermits=8.halfPermits为maxPermits的一半.

参考注释中的神图：

* ^ throttling
* |
* 3*stable + /
* interval | /.
* (cold) | / .
* | / . <-- "warmup period" is the area of the trapezoid between
* 2*stable + / . halfPermits and maxPermits
* interval | / .
* | / .
* | / .
* stable +----------/ WARM . }
* interval | . UP . } <-- this rectangle (from 0 to maxPermits, and
* | . PERIOD. } height == stableInterval) defines the cooldown period,
* | . . } and we want cooldownPeriod == warmupPeriod
* |---------------------------------> storedPermits
* (halfPermits) (maxPermits)
*

 下面是我们QPS=4，warmup为2秒时候对应的图。 

 maxPermits=8，halfPermits=4，和SmoothBursty相同的请求序列：

(1).t=0,这时候storedPermits=8，请求1个令牌，使用1个storedPermits消耗时间=1×(0.75+0.625)/2=0.6875秒；
(2).t=1,这时候storedPermits=8，请求3个令牌，使用3个storedPermits消耗时间=3×(0.75+0.375)/2=1.6875秒(注意已经超过1秒了，意味着下次产生新Permit时间为2.6875)；
(3).t=2,这时候storedPermits=5，请求10个令牌，使用5个storedPermits消耗时间=1×(0.375+0.25)/2+4*0.25=1.3125秒，再加上额外请求的5个新产生的Permit需要消耗=5*0.25=1.25秒，即总共需要耗时2.5625秒，则下一次产生新的Permit时间为2.6875+2.5625=5.25，注意当前请求私海2.6875才返回的，之前一直阻塞；
(4).t=3,因为前一个请求阻塞到2.6875，实际这个请求3.6875才到达RateLimiter，请求1个令牌，storedPermits=0，下一次产生新Permit时间为5.25，因此总共需要等待5.25-3.6875=1.5625秒；

实际执行结果：

warmupPeriodMicros=2000000
stableIntervalMicros=250000.0, maxPermits=8.0, halfPermits=4.0, coldIntervalMicros=750000.0, slope=125000.0, storedPermits=8.0
maxPermits=8.0, storedPermits=8.0, stableIntervalMicros=250000.0, nextFreeTicketMicros=1524
acquire(1), sleepSecond=0.0
maxPermits=8.0, storedPermits=8.0, stableIntervalMicros=250000.0, nextFreeTicketMicros=1001946
acquire(3), sleepSecond=0.0
maxPermits=8.0, storedPermits=5.0, stableIntervalMicros=250000.0, nextFreeTicketMicros=2689446
acquire(10), sleepSecond=0.687186
maxPermits=8.0, storedPermits=0.0, stableIntervalMicros=250000.0, nextFreeTicketMicros=5251946
acquire(1), sleepSecond=1.559174