1. Concept
In computer networks, rate limiting is used to control the rate of traffic sent or received by a network interface controller and is used to prevent DoS attacks
Translate it with my understanding: current limiting is to control the ingress and egress of the system to prevent large traffic from entering and leaving, resulting in insufficient resources and system instability.
The current limiting system is a control component for resource access. It controls two main functions: the current limiting strategy and the fusing strategy. For the fusing strategy, different systems have different requirements for the fusing strategy. Waiting in line, some systems expect service degradation, and some systems will customize their own fusing strategies, it is difficult to list them all, so this article only makes a detailed design for the function of current limiting strategy.
For the function of finding requests that exceed the rate threshold, there are two basic concepts in the current limiting system: resources and policies.
- Resources: or scarce resources, objects controlled by flow; such as write interfaces, external merchant interfaces, and read interfaces under high traffic
- Strategy: The current limiting strategy consists of a current limiting algorithm and adjustable parameters.
Circuit breaker strategy: The processing strategy for requests that exceed the rate threshold is a term that I understand, not the mainstream statement in the industry.
2. Current limiting algorithm
- Limit instantaneous concurrency
- Limit the maximum number of requests in the time window
- token bucket
2.1. Limit the number of instantaneous concurrency
Definition: Instantaneous concurrency, the number of requests/transactions processed by the system at the same time
Advantages: This algorithm can achieve the effect of controlling the number of concurrency
Disadvantages: The usage scenario is relatively simple, generally used to control incoming traffic
Java pseudo code implementation:
AtomicInteger atomic = new AtomicInteger(1)
try {
if(atomic.incrementAndGet() > 限流数) {
//熔断逻辑
} else {
//处理逻辑
}
} finally {
atomic.decrementAndGet();
}
2.2. Limit the maximum number of requests in the time window
Definition: The maximum number of requests in the time window, the maximum number of requests allowed within the specified time range
Advantages: This algorithm can meet the vast majority of flow control needs, and the maximum QPS can be directly converted through the maximum number of requests in the time window (QPS = number of requests/time window)
Disadvantages: In this way, the traffic may not be smooth, and a small amount of traffic in the time window accounts for a particularly large proportion.
Lua code implementation:
--- 资源唯一标识
local key = KEYS[1]
--- 时间窗最大并发数
local max_window_concurrency = tonumber(ARGV[1])
--- 时间窗
local window = tonumber(ARGV[2])
--- 时间窗内当前并发数
local curr_window_concurrency = tonumber(redis.call('get', key) or 0)
if current + 1 > limit then
return false
else
redis.call("INCRBY", key,1)
if window > -1 then
redis.call("expire", key,window)
end
return true
end2.3. Token Bucket
Algorithm Description
- If the average sending rate configured by the user is r, a token is added to the bucket every 1/r second
- It is assumed that at most b tokens can be stored in the bucket. If the token bucket is full when the token arrives, the token is discarded
- When the traffic enters at the rate v, the token is taken from the bucket at the rate v, the traffic that gets the token passes, and the traffic that does not get the token does not pass, and the circuit breaker logic is executed.
Attributes
- In the long run, the rate of conforming traffic is affected by the rate of token addition and is stabilized as: r
- Because the token bucket has a certain amount of storage, it can withstand certain traffic bursts
- M is the maximum possible transfer rate in bytes/sec. M>r
- T max = b/(Mr) time to withstand maximum transmission rate
- B max = T max * M The traffic transmitted within the time to bear the maximum transmission rate
Advantages: The traffic is relatively smooth, and it can withstand certain traffic bursts
Because the implementation of our current limiting system is based on the token bucket algorithm, please refer to the following for the specific code implementation.
3. Project realization
3.1. Technical selection
- mysql: stores metadata such as parameters of the current limiting policy
- redis+lua: token bucket algorithm implementation
3.2. Architecture diagram
3.3, data structure
| field | describe |
|---|---|
| name | The unique identifier of the token bucket |
| apps | List of apps that can use the token bucket |
| max_permits | The maximum number of tokens in the token bucket |
| rate | The rate at which tokens are added to the token bucket |
| created_by | founder |
| updated_by | updater |
The implementation of the current limiting system is based on redis, which could have nothing to do with the application. However, in order to do unified management of the current limiting metadata configuration, management and use by the application dimension, the field of apps is added to the data structure. If there is a problem, check it out. Also more convenient.
3.4, code implementation
3.4.1. Problems encountered in code implementation
Referring to the algorithm description of the token bucket, the general idea is to put a thread for repeated execution in the RateLimiter-client, and the thread adds tokens to the token bucket according to the configuration. This implementation has the following disadvantages:
- Need to add a repeating thread for each token bucket configuration
- The repetition interval is not precise enough: the thread needs to add a token to the bucket every 1/r second, and when r > 1000, the time interval between thread execution cannot be set at all (from the realization of the performance test later, RateLimiter-client is possible assumes a request rate of QPS > 5000)
3.4.2. Solutions
Based on the above shortcomings, referring to the implementation of RateLimiter in google's guava, we use the method of triggering token addition.
Algorithm Description
- Based on the above token bucket algorithm
- Change the adding token to a triggering method, and the token is taken to do the action of adding the token
- When removing the token, calculate the number of tokens that should be added during this period by calculating the time difference between the last token added and the current time, and then add it to the bucket
- curr_mill_second = current number of milliseconds
- last_mill_second = milliseconds since the token was last added
- r = rate of adding tokens
- reserve_permits = (curr_mill_second-last_mill_second)/1000 * r
- After adding the token, execute the token fetch logic
3.4.3, lua code implementation
--- 获取令牌
--- 返回码
--- 0 没有令牌桶配置
--- -1 表示取令牌失败，也就是桶里没有令牌
--- 1 表示取令牌成功
--- @param key 令牌（资源）的唯一标识
--- @param permits 请求令牌数量
--- @param curr_mill_second 当前毫秒数
--- @param context 使用令牌的应用标识
local function acquire(key, permits, curr_mill_second, context)
local rate_limit_info = redis.pcall("HMGET", key, "last_mill_second", "curr_permits", "max_permits", "rate", "apps")
local last_mill_second = rate_limit_info[1]
local curr_permits = tonumber(rate_limit_info[2])
local max_permits = tonumber(rate_limit_info[3])
local rate = rate_limit_info[4]
local apps = rate_limit_info[5]
--- 标识没有配置令牌桶
if type(apps) == 'boolean' or apps == nil or not contains(apps, context) then
return 0
end
local local_curr_permits = max_permits;
--- 令牌桶刚刚创建，上一次获取令牌的毫秒数为空
--- 根据和上一次向桶里添加令牌的时间和当前时间差，触发式往桶里添加令牌
--- 并且更新上一次向桶里添加令牌的时间
--- 如果向桶里添加的令牌数不足一个，则不更新上一次向桶里添加令牌的时间
if (type(last_mill_second) ~= 'boolean' and last_mill_second ~= false and last_mill_second ~= nil) then
local reverse_permits = math.floor(((curr_mill_second - last_mill_second) / 1000) * rate)
local expect_curr_permits = reverse_permits + curr_permits;
local_curr_permits = math.min(expect_curr_permits, max_permits);
--- 大于0表示不是第一次获取令牌，也没有向桶里添加令牌
if (reverse_permits > 0) then
redis.pcall("HSET", key, "last_mill_second", curr_mill_second)
end
else
redis.pcall("HSET", key, "last_mill_second", curr_mill_second)
end
local result = -1
if (local_curr_permits - permits >= 0) then
result = 1
redis.pcall("HSET", key, "curr_permits", local_curr_permits - permits)
else
redis.pcall("HSET", key, "curr_permits", local_curr_permits)
end
return result
end
3.4.4. Management interface
In the previous design, the current limiting configuration is associated with the application. In order to manage the configuration better, a unified management page is needed to manage and control the configuration:
- Manage current limiting configuration by application
- Different people are assigned different permissions; the relevant personnel have the permission to view the configuration, and the person in charge has the permission to modify and delete the configuration
3.5. Performance test
Configuration: aws-elasticcache-redis 2 cores 4g
Because the function of Ratelimiter-client is relatively simple, it basically discounts the performance of redis.
- Single-threaded token retrieval: Ratelimiter-client's QPS = 250/s
- 10 threads take tokens: QPS of Ratelimiter-client = 2000/s
- 100 threads take tokens: QPS of Ratelimiter-client = 5000/s
4. Summary
The current limiting system is relatively simple from design to implementation, but it is indeed very practical. It can be described in four words: short and powerful. The most important thing is to design a current limiting system that conforms to the company's specifications based on the company's authority system and system structure. system.
insufficient:
- Redis We use single-point redis, only master-slave, no redis high-availability cluster (may use redis high-availability cluster, which will bring new problems)
- The current limiting system is currently only implemented at the application level, not on the interface gateway.
- The circuit breaker strategy needs to be customized by yourself. If it is implemented better, you can give some commonly used circuit breaker strategy templates.