This article will implement a stand-alone version of the Http interface universal current limiting framework step by step from the background of requirements, requirements analysis, framework design, and framework implementation.
1. Analysis of Current Limiting Framework
1. Demand background
In the microservice system, the interface we develop may be provided to many different systems to call. If the caller does not handle it properly (such as a sudden increase in traffic in the spike scenario), the number of requests for the interface will increase suddenly. These requests will be compared with normal requests. Competing for the thread resources of the system may eventually lead to a large number of interface timeouts due to the failure to allocate thread resources for normal requests.
The solution to this problem is that as an interface provider, we need to limit the calling frequency of each interface caller to avoid the problem of exhaustion of thread resources due to the excessive frequency of a certain interface call frequency.
2. Demand analysis
(1) Functional requirements
In order to complete a general current limiting framework, the approximate execution steps are as follows:
- The current-limiting framework is started, and the current-limiting rules are read and loaded;
- Upon receiving the caller's request, determine whether the current limit will be restricted according to the current limit rules configured by the interface read;
(2) Non-functional requirements
As a general current-limiting framework, non-functional requirements usually need to consider the framework's ease of use, fault tolerance, performance, scalability, and flexibility .
- Ease of use: For users, third-party frameworks generally want to be as simple and easy to access as possible. Therefore, in terms of current-limiting rule configuration, we hope to support file configuration such as xml/yaml/properties, and also support distributed configuration sources; for current-limiting interfaces, we hope to be able to use them; for current-limiting algorithms, we hope It can provide a variety of current limiting algorithms, such as fixed time windows, sliding time windows, distributed current limiting algorithms, etc.; because most of the current frameworks are developed based on Spring, we hope that the current limiting framework can be easily integrated into Spring Go in the frame
- Scalability and flexibility: We need to consider the scalability of the framework and be able to flexibly support various current limiting algorithms and custom current limiting algorithms; for current limiting rules, we hope to support different formats (Json, Yaml, Xml, etc.) , Different data sources (local configuration or ZK and other configuration) current limiting rule configuration method;
- Performance: Each interface must be checked for current limiting before being called, which will increase the response time of the interface request. Therefore, we need to minimize the impact of the current limiting framework on the response time of the interface;
- Fault tolerance: The purpose of accessing the current-limiting framework is to improve the availability and stability of the system, so the availability of the service itself cannot be affected by the exception of the current-limiting framework.
2. Current limiting frame design
In the frame design, our main work is to divide and design the modules. The general current-limiting framework is mainly divided into current-limiting rules, current-limiting algorithms, current-limiting modes, and integrated design using four modules.
1. Current limit rules
The framework needs to define the grammatical format of the current-limiting rules, including the caller identification, the interface that requires current-limiting, the threshold of the current-limiting, the time granularity, the current-limiting algorithm, the current-limiting mode and other elements.
In order to make it simple, we do not consider current limiting algorithm configuration, current limiting mode configuration, etc. In this issue, we need to limit the caller app1 to call interface /v1/user not more than 100 times within one minute. The current limiting configuration example is as follows:
configs:
- appId: app1
limits:
- api: /v1/user
limit: 100
unit: 60
For file formats, we support YAML/XML/JSON and other formats;
for data sources, we support local configuration, as well as other configuration center data sources, such as Nacos.
2. Current limiting algorithm
Common current limiting algorithms include: fixed time window current limiting algorithm, sliding time window current limiting algorithm, token bucket current limiting algorithm, leaky bucket current limiting algorithm, etc.
The basic idea is: to count the number of times an application calls an interface within a certain period of time, and when the number of calls exceeds a threshold, the current is limited.
By default, we use a fixed time window current limiting algorithm. However, in order to facilitate expansion, we need to design and reserve expansion points in advance to facilitate the development of other current-limiting algorithms in the future.
3. Current limiting mode
The current-limiting modes are divided into single-machine current-limiting and cluster current-limiting.
Single-machine current limiting refers to limiting the number of accesses to a single instance of a certain service; cluster current limiting refers to limiting the total number of calls to multiple instances of a certain service.
The difference between single-machine current limiting and cluster current limiting lies in the implementation of interface access counters. Single-machine current limiting only needs to maintain its own interface request counter in a single instance, while cluster current limiting needs to manage all instance counters, which requires a third-party storage (such as Redis) to store the number of interface accesses for each instance.
4. Integrated use
Because most of the interface callers and providers are implemented based on the Spring framework, we can develop a class library similar to Mybatis-Spring to facilitate the integration and use of the current-limiting framework in projects that use the Spring framework.
In addition to the above three modules, we also need to consider the fault tolerance and performance of the framework. For cluster current limiting, in order to avoid affecting the response time of the interface due to the long response time of the current limiting framework, we can develop and implement the cluster current limiting framework based on Redis; for the exceptions that the current limiting framework may throw, we need to treat them differently. If the framework code is abnormal, we throw it directly without affecting the use of the interface.
3. Realization of current limiting framework
1. Minimal prototype (MVP) code
When developing the framework, we don't need to think about all the functions required by the framework, nor do we need to use design patterns and principles to implement an excellent framework. The first version of the code can complete the required functions without considering the code design and quality. The functions required by a basic current limiting framework are as follows:
- For the interface type, only the current limiting of the HTTP interface is supported, and other types of interface current limiting such as RPC are not currently supported;
- For current limiting rules, only local file configuration is supported, and configuration files only support YAML;
- For current limiting algorithms, only fixed time window algorithms are supported;
- For current limiting mode, only single-machine current limiting is supported.
The implementation of the overall class is as follows:
框架入口： RateLimiter，读取、加载、解析限流配置，并提供限流接口；
限流算法： RateLimitAlg， 默认采用固定时间窗口限流算法；
限流规则： ApiLimit、AppRuleConfig、RateLimitRule、RuleConfig
The specific code is as follows:
RateLimitAlg.java:
public class RateLimiter {
private RateLimitRule rule;
// 每个API内存中存储限流计数器, key为 api:url
private ConcurrentHashMap<String, RateLimitAlg> counters = new ConcurrentHashMap<>();
public RateLimiter() {
RuleConfig ruleConfig = loadFromYmlAsRuleConfig();
Assert.isTrue(ruleConfig != null, "Load from yaml file, RuleConfig is null");
this.rule = new RateLimitRule(ruleConfig);
}
/**
* 判断接口是否限流
*
* @param appId
* @param url
* @return true: 不限流; false: 限流
* @throws InterruptedException
*/
public boolean limit(String appId, String url) throws InterruptedException {
// 接口未配置限流, 直接返回
ApiLimit apiLimit = rule.getApiLimit(appId, url);
if (apiLimit == null) {
return true;
}
String counterKey = appId + ":" + url;
RateLimitAlg rateLimitAlg = counters.get(counterKey);
if (rateLimitAlg == null) {
// 没有计数器, 就构造一个
RateLimitAlg rateLimitCounterNew = new RateLimitAlg(apiLimit.getLimit());
RateLimitAlg rateLimitCounterOld = counters.putIfAbsent(counterKey, rateLimitCounterNew);
if (rateLimitCounterOld == null) {
rateLimitAlg = rateLimitCounterNew;
}
}
// 固定窗口统计, 判断是否超过限流阈值
return rateLimitAlg.tryAcquire();
}
private RuleConfig loadFromYmlAsRuleConfig() {
InputStream in = null;
RuleConfig ruleConfig = null;
try {
in = this.getClass().getResourceAsStream("/sentinel-rule.yml");
if (in != null) {
Yaml yaml = new Yaml();
ruleConfig = yaml.loadAs(in, RuleConfig.class);
return ruleConfig;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return null;
}
}
RateLimitAlg.java:
public class RateLimitAlg {
// ms
private static final long LOCK_EXPIRE_TIME = 200L;
private Stopwatch stopWatch;
// 限流计数器
private AtomicInteger counter = new AtomicInteger(0);
private final int limit;
private Lock lock = new ReentrantLock();
public RateLimitAlg(int limit) {
this(limit, Stopwatch.createStarted());
}
public RateLimitAlg(int limit, Stopwatch stopWatch) {
this.limit = limit;
this.stopWatch = stopWatch;
}
public boolean tryAcquire() throws InterruptedException {
int currentCount = counter.incrementAndGet();
// 未达到限流
if (currentCount < limit) {
return true;
}
// 使用固定时间窗口统计当前窗口请求数
// 请求到来时,加锁进行计数器统计工作
try {
if (lock.tryLock(LOCK_EXPIRE_TIME, TimeUnit.MILLISECONDS)) {
// 如果超过这个时间窗口, 则计数器counter归零, stopWatch, 窗口进入下一个窗口
if (stopWatch.elapsed(TimeUnit.MILLISECONDS) > TimeUnit.SECONDS.toMillis(1)) {
counter.set(0);
stopWatch.reset();
}
// 不超过, 则当前时间窗口内的计数器counter+1
currentCount = counter.incrementAndGet();
return currentCount < limit;
}
} catch (InterruptedException e) {
System.out.println("tryAcquire() wait lock too long:" + LOCK_EXPIRE_TIME + " ms");
throw new InterruptedException("tryAcquire() wait lock too long:" + LOCK_EXPIRE_TIME + " ms");
} finally {
lock.unlock();
}
// 出现异常 不能影响接口正常请求
return true;
}
}
Limiting rules: ApiLimit, AppRuleConfig, RuleConfig
public class ApiLimit {
private static final int DEFAULT_UNIT_SECONDS = 1;
private String api;
private int limit;
private int unit = DEFAULT_UNIT_SECONDS;
// ...
}
public class AppRuleConfig {
private String appId;
private List<ApiLimit> limits;
// ...
}
public class RuleConfig {
private List<AppRuleConfig> configs;
// ...
}
RateLimitRule provides a way to quickly query the current limit rules:
/**
* @author: wanggenshen
* @date: 2020/6/23 00:12.
* @description: 支持快速查询 ApiLimit
*
* TODO:
* (1) 精准匹配优化: 二分查找算法优化
* (2) 支持前缀匹配: 使用Trie树实现
* (3) 支持模糊匹配: 实现难度较高
*/
public class RateLimitRule {
/**
* key : appId + api, value: limit
*/
private HashMap<String, ApiLimit> map = new HashMap();
public RateLimitRule(RuleConfig ruleConfig) {
List<AppRuleConfig> configs = ruleConfig.getConfigs();
configs.stream().forEach(appRuleConfig -> {
String appId = appRuleConfig.getAppId();
List<ApiLimit> apiLimitList = appRuleConfig.getLimits();
apiLimitList.stream().forEach(apiLimit -> {
String key = appId + ":" + apiLimit.getApi();
map.put(key, apiLimit);
});
});
}
public ApiLimit getApiLimit(String appId, String api) {
String key = appId + ":" + api;
return map.get(key);
}
}
test:
(1) First configure current limiting rules:
configs:
- appId: app1
limits:
- api: /v1/user
limit: 5
unit: 60
- api: /v1/order
limit: 4
unit: 60
- appId: app2
limits:
- api: /v1/login
limit: 7
unit: 60
(2) Test:
public static void main(String[] args) {
RateLimiter rateLimiter = new RateLimiter();
try {
for (int i = 0; i < 10; i++) {
boolean b = rateLimiter.limit("app1", "/v1/user");
System.out.println("/v1/user接口限流结果: " + b);
}
System.out.println("=====");
for (int i = 0; i < 10; i++) {
boolean b = rateLimiter.limit("app1", "/v1/order");
System.out.println("/v1/order接口限流结果: " + b);
}
System.out.println("=====");
for (int i = 0; i < 10; i++) {
boolean b = rateLimiter.limit("app2", "/v1/login");
System.out.println("/v1/login接口限流结果:" + b);
}
} catch (Exception e) {
}
}
The test results are as follows, which are the same as the configured rules:
2. Optimize and refactor the V2 version
After implementing the functions of the framework, we need to analyze the design and implementation of the code in terms of readability from the perspective of Code Reviewer, combining SOLID, DRY, KISS, interface-based rather than implementation programming, high cohesion and loose coupling, and coding specifications. Is there any optimization in terms of, scalability, etc.?
(1) Code readability
For code readability, we need to focus on whether the directory design is reasonable, whether the module division is clear, whether the code structure is highly cohesive and low-coupling, and whether it complies with coding standards.
Since the code is less, the above points are relatively satisfied, and the readability is better.
(2) Code scalability
Extensibility is mainly to follow the programming ideas based on interfaces rather than implementation, and to have interface abstract consciousness.
The RateLimitAlg class only implements a fixed time window current limiting algorithm. If we need to use other current limiting algorithms, we need to rewrite the original code, so we need to provide a more abstract algorithm interface; the
RateLimitRule class only implements a simple query configuration rule interface , It is necessary to provide a more abstract interface to support optimized search algorithms such as binary search;
In addition, the entry class RateLimiter only provides regular loading and interface current limiting, and file reading needs to be removed.
See link for optimized code : Link to optimized code
to sum up
This article implements a stand-alone Http interface current-limiting framework step by step from the analysis, design, and implementation of the current-limiting framework. Of course, the implemented framework is very rough, and there is still a certain distance from the production environment, but it provides a general framework realization idea to help us better understand the idea of current limiting algorithm and the realization process of the general framework.