algorithm
Traffic restrictions
counter
- Count the number of requests per unit time. more than outright refusal.
- Features:
- Simple to implement.
- The unit time is exhausted at the beginning, and the remaining time will be rejected, that is, thrust consumption.
sliding window
- Refinement of the counter (differentiation + integration), counting and rejecting according to the time window.
- Features:
- To reduce the impact of Spike Flow:
- Double peak when switching counters.
- Reduced the area of effect of Spike consumption.
- Consume more memory CPU statistics and calculate throttling.
leaky bucket
- Buffer cache, the system processes requests at a constant rate.
- Features:
- Consumption at a uniform rate.
- Requests may have a certain waiting delay.
- Burst traffic will wait for a long time or be discarded.
token bucket
- Improved leaky bucket. A certain policy generates a token, and the system consumes the token to process the request
- Features:
- Control production rate.
- A certain burst of traffic can be consumed.
concurrency limit
- Limit the number of threads, connections, etc.
- Features:
- Restricted borders are stricter and the degree of isolation is high.
- It is difficult to tune the concurrency threshold.
Open source current limiting component
RateLimiter
- model:
- SmoothBursty mode: N tokens are issued every second, and a certain number of tokens are also allowed to be pre-borrowed.
- SmoothWarmingUp mode: When the system is just started, the issued tokens are gradually increased to the set maximum threshold.
- Token Bucket Algorithm : Burst traffic is allowed.
- Single machine current limit only.
core variable
- storedPermits: the current number of stored tokens
- stableIntervalMicros: Time interval for adding tokens . Gradually decreases to threshold when preheat mode starts.
- nextFreeTicketMicros: The next time the token can be obtained.
process
- Calculate the sleep time based on the number of tokens requested, and sleep threads.
public double acquire(int permits) {
long microsToWait = reserve(permits);
stopwatch.sleepMicrosUninterruptibly(microsToWait);
return 1.0 * microsToWait / SECONDS.toMicros(1L);
}
- reserve double lock serial calculation sleep time.
final long reserve(int permits) {
checkPermits(permits);
synchronized (mutex()) {
return reserveAndGetWaitLength(permits, stopwatch.readMicros());
}
}
final long reserveAndGetWaitLength(int permits, long nowMicros) {
long momentAvailable = reserveEarliestAvailable(permits, nowMicros);
return max(momentAvailable - nowMicros, 0);
}
- If nextFreeTicketMicros is in the past, refresh the token bucket:
- storedPermits, not to exceed the maximum value.
- nextFreeTicketMicros to current.
void resync(long nowMicros) {
if (nowMicros > nextFreeTicketMicros) {
double newPermits = (nowMicros - nextFreeTicketMicros) / coolDownIntervalMicros();
storedPermits = min(maxPermits, storedPermits + newPermits);
nextFreeTicketMicros = nowMicros;
}
}
- Calculate waiting time and consume tokens:
- This request waits until nextFreeTicketMicros
- If the token is not enough, it will be advanced and accumulated to nextFreeTicketMicros
- consume storedPermits in the bucket
final long reserveEarliestAvailable(int requiredPermits, long nowMicros) {
resync(nowMicros);
long returnValue = nextFreeTicketMicros;
double storedPermitsToSpend = min(requiredPermits, this.storedPermits);
double freshPermits = requiredPermits - storedPermitsToSpend;
long waitMicros =
storedPermitsToWaitTime(this.storedPermits, storedPermitsToSpend)
+ (long) (freshPermits * stableIntervalMicros);
this.nextFreeTicketMicros = LongMath.saturatedAdd(nextFreeTicketMicros, waitMicros);
this.storedPermits -= storedPermitsToSpend;
return returnValue;
}
Sentinel
- model:
- Wait in line . SmoothBursty mode similar to ratelimiter.
- preheating mode. Similar to the SmoothWarmingUp mode of ratelimiter.
- Fail fast. The circular sliding window counts qps, and if it exceeds, it will be rejected.
process
- Chain of responsibility mode, spi loading.
- Entry: SentinelResourceAspect
- Core node:
- StatisticSlot: traffic statistics.
- FlowSlot: flow control check.
Traffic Statistics
- ArrayMetric: sliding window .
- Second-level window: 2 intervals with a span of 1s.
public class StatisticNode implements Node {
private transient volatile Metric rollingCounterInSecond =
new ArrayMetric(SampleCountProperty.SAMPLE_COUNT, IntervalProperty.INTERVAL);
private transient Metric rollingCounterInMinute = new ArrayMetric(60, 60 * 1000, false);
}
public class ArrayMetric implements Metric {
private final LeapArray<MetricBucket> data;
public void addPass(int count) {
WindowWrap<MetricBucket> wrap = data.currentWindow();
wrap.value().addPass(count);
}
}
- LeapArray , calculate the start time of the current bucket based on the current time, and route to the bucket.
- If the bucket does not exist, it will be created and CASed.
- The current time is equal to the start time of the bucket that the bucket belongs to.
- The current time is greater than the start time of the bucket, lock and reset the bucket.
- The current time is less than the bucket's start time, theoretically not.
public LeapArray(int sampleCount, int intervalInMs) {
this.windowLengthInMs = intervalInMs / sampleCount;
this.intervalInMs = intervalInMs;
this.intervalInSecond = intervalInMs / 1000.0;
this.sampleCount = sampleCount;
this.array = new AtomicReferenceArray<>(sampleCount);
}
public WindowWrap<T> currentWindow(long timeMillis) {
if (timeMillis < 0) {
return null;
}
int idx = calculateTimeIdx(timeMillis);
long windowStart = calculateWindowStart(timeMillis);
while (true) {
WindowWrap<T> old = array.get(idx);
if (old == null) {
WindowWrap<T> window = new WindowWrap<T>(windowLengthInMs, windowStart, newEmptyBucket(timeMillis));
if (array.compareAndSet(idx, null, window)) {
return window;
} else {
Thread.yield();
}
} else if (windowStart == old.windowStart()) {
return old;
} else if (windowStart > old.windowStart()) {
if (updateLock.tryLock()) {
try {
return resetWindowTo(old, windowStart);
} finally {
updateLock.unlock();
}
} else {
Thread.yield();
}
} else if (windowStart < old.windowStart()) {
return new WindowWrap<T>(windowLengthInMs, windowStart, newEmptyBucket(timeMillis));
}
}
}
Flow Control Check
- 3 flow control methods: DefaultController (fast failure), RateLimiterController (waiting in line), WarmUpController (preheating)
- DefaultController compares the cumulative and set values of all windows within the time span, and limits the current if it exceeds.
public class DefaultController{
public boolean canPass(Node node, int acquireCount, boolean prioritized) {
int curCount = avgUsedTokens(node);
if (curCount + acquireCount > count) {
return false;
}
return true;
}
private int avgUsedTokens(Node node) {
if (node == null) {
return DEFAULT_AVG_USED_TOKENS;
}
return grade == RuleConstant.FLOW_GRADE_THREAD ? node.curThreadNum() : (int)(node.passQps());
}
}
public class StatisticNode implements Node {
private transient volatile Metric rollingCounterInSecond = new ArrayMetric(SampleCountProperty.SAMPLE_COUNT,
IntervalProperty.INTERVAL);
public double passQps() {
return rollingCounterInSecond.pass() / rollingCounterInSecond.getWindowIntervalInSec();
}
}
public class ArrayMetric{
public long pass() {
data.currentWindow();
long pass = 0;
List<MetricBucket> list = data.values();
for (MetricBucket window : list) {
pass += window.pass();
}
return pass;
}
}
Relevant information