Design record of tens of billions of traffic API gateway

---

foreword

In this sharing, we start from the current status and problems of API Gateway (API Gateway) in the 7th MAD Technology Forum - "Record of Design and Implementation of API Gateway", explain the relationship between microservice architecture and API gateway, and straighten out traffic gateway and business The context of the gateway.
Original video address: The 7th MAD Technology Forum - "Design and Implementation of Ten Billion Traffic API Gateway"

---

1. Current situation and problems

1. Overview of API Gateway

Any problem in computer science can be solved by adding an intermediate layer of indirection. —David Wheeler

1.1 What is API Gateway (API Gateway) - History

API Gateway and integrated front-end, ESB
actually gateway has deep origin with service-oriented architecture (Service Oriented Architecture, SOA) and microservices architecture (MicroServices Architecture, MSA).

More than ten years ago, after banks and other financial institutions completed the national business system centralization, the decentralized systems became centralized, which also brought various problems: how to deal with the rapid business development, and how to integrate and manage too many docking systems. In order to solve these problems, the industry has implemented an intermediate gateway between channels and business systems, that is, a comprehensive front-end system, which adapts to various channels and services, and handles various protocol access, routing and message conversion, Synchronous asynchronous calls etc.

Based on the concept of SOA, people have further added functions such as metadata management, registration, intermediary, orchestration, and governance of services on the basis of comprehensive front-end, and gradually formed an enterprise service bus (ESB, EnterpriseService Bus).

PS: Of course, there are various problems in the centralized management SOA solution based on ESB, especially in the case of explosive development in the field of Internet technology.

1.2 Features of API usage and problems faced by the system

1.2.1 Features of use :

• very concentrated access
• Users distributed all over the world
• Fixed data format
• Divided into internal calls and external calls
• Small amount of message data
• 7x24 hours to provide service
• very frequent visits
• External users have certain technical capabilities

In the face of large-traffic access, the API gateway must be graded, distinguishing between internal access and external orientation, and the current limit of each system access.

1.2.2 Problems faced :

• 
  How to reasonably control the traffic and how
  to deal with the sudden traffic?
  Directly face the user, and any problem will be magnified a hundred times.
• The gateway system is too complex,
  a large number of services are embedded,
  multiple systems coexist,
  there are many hard-coded business rules, and the maintenance cost is relatively high
• Difficult to manage.
  Monitoring index measurement.
  The amount of collected data is too large
  to run 7x24 hours, and the maintenance cost is high.
• Push or pull
  Short connection or long connection
  REST API or websocket
  There are many channels (web, app, api, etc. more than a dozen), it is difficult to control
  

2. Research and Analysis (Open Source Gateway)

2.1 Gateway and Microservice

In recent years, with the rapid development of Internet technology, in order to solve various problems of the centralized management SOA solution represented by ESB, distributed service technology represented by Apache Dubbo (after open source in 2011) and Spring Cloud emerged , giving another option for SOA implementation: Decentralized Distributed Service Architecture (DSA). Distributed service architecture technology no longer depends on specific service center container technology (such as ESB), but completely separates service addressing and invocation, so that there is no need to use containers as service agents.

Later, on this basis, with the development of REST, Docker containerization, domain modeling, automated test operation and maintenance, etc., the microservice architecture (MSA) was gradually formed. In the microservice architecture, the granularity of services is further subdivided, and each business service can be independently designed, developed, tested, deployed, and managed. At this time, each independent deployment unit can be maintained by different development and testing teams, and can be designed using different programming languages ​​and technology platforms, but it is required to use a language- and platform-independent service protocol as the communication method between each unit.

In the microservice architecture, due to the subdivision of systems and services, the system structure becomes very complex. REST API is the best choice because of its simplicity, efficiency, cross-platform, easy development, easy testing, and easy integration. At this time, a system similar to the comprehensive front-end is produced, which is the API Gateway (API Gateway). The API gateway serves as the API aggregation point and unified access point scattered in the microservices of various business systems. External requests can access all internal REST API services by accessing this access point.

2.2 Gateway classification and functions

All the issues and functions that need to be considered can be divided into two categories:

• Global, completely irrelevant to specific back-end business systems and services, such as security policies, global traffic control policies, traffic distribution policies, etc.

• It is aimed at a specific back-end business system, or a part that has a certain relationship between the service and the business, and is generally deployed directly in front of the business service.

With the development of complex business systems on the Internet, these two types of function sets have gradually formed two common gateway systems: traffic gateways and business gateways

2.2.1 Traffic Gateway and WAF

We define a global policy gateway that has nothing to do with specific back-end business systems and services, which is called a traffic gateway. In this way, the traffic gateway focuses on the stability and security of global traffic, such as preventing various SQL injections, black and white list control, load balancing from access requests to business systems, etc., and usually has the following general specific functions:

• global flow control

• log statistics

• Prevent SQL injection

• Prevent web attacks

• Mask tool scan

• black and white list control

Through this function list, we can find that the function of the traffic gateway is very similar to that of the Web Application Firewall (WAF). WAF is generally a web application firewall developed based on the ngx_lua module of Nginx/OpenResty. Generally, the code of WAF is very simple, focusing on easy to use, high performance and lightweight. Simply put, in addition to the proxy capabilities of Nginx itself, security-related functions are added. To describe its principle in one sentence, it is to analyze HTTP requests (protocol analysis module), rule detection (rule module), perform different defense actions (action module), and record the defense process (log module).

A general WAF has the following functions :

• Prevent SQL injection, partial overflow, fuzzing test, XSS/SSRF and other web attacks

• Prevent attacks from stress testing tools such as Apache Bench

• Block common scanning hacking tools, such as scanners

• Prohibit the execution permission of the image attachment category directory and prevent webshell uploads

• Support IP whitelist and blacklist function, directly deny the IP access of blacklist

• Support URL whitelist, define URLs that do not need to be filtered

• Support User-Agent filtering, support CC attack protection, limit the number of visits of a single URL within a specified time

• Support Cookie filtering, URL and URL parameter filtering

• Support logging, record all rejected operations to the log

2.2.2 Service Gateway

We define a policy gateway for a specific back-end business system, or a policy gateway that has a certain relationship between services and businesses, that is, a business gateway. For example, the traffic control strategy for a certain system, a certain service or a certain user classification, the cache strategy for a certain type of service, the permission verification method for a specific system, the request filtering for certain user conditions, and the Specifically, the data aggregation package of several related APIs, and so on.
The business gateway is generally deployed after the traffic gateway and before the business system, closer to the business system than the traffic gateway. The API gateway we talk about in most cases refers to the business gateway in a narrow sense. If the scale of the system is small, we will also combine the two into one, and use a gateway to handle all the work.

2.3 Four major functions and three concerns of API gateway

Generally speaking, the API gateway has four major functions :

• Request access: As the access point for all API interface service requests, manage all access requests.

• Business aggregation: as the aggregation point of all back-end business services, all business services can be called here.

• Intermediary strategy: implement security, authentication, routing, filtering, flow control, caching and other strategies, and perform some necessary intermediary processing.

• Unified management: Provides configuration management tools for unified management of the call life cycle of all API services and corresponding intermediary strategies.
  
  Three concerns of the API gateway:
  The three most important concerns of the massively concurrent API gateway:
  (1) Maintain large-scale inbound request access capabilities (long and short connections), such as based on Netty implementation.
  (2) Maximize the reuse of outbound HTTP connection capabilities, such as the asynchronous HttpClient implementation based on HttpClient4.
  (3) It is convenient and flexible to implement various strategies such as security, verification, filtering, aggregation, current limiting, and monitoring.

2.4 Common open source gateways

At present, the common open source gateways are roughly classified into the following categories according to language:

• Java：Zuul/Zuul 2、Spring Cloud Gateway、Kaazing KWG、gravitee、Dromara soul等；

• Nginx+Lua：Open Resty、Kong、Orange、Abtesting Gateway etc；

• Go：Janus、fagongzi、Grpc-Gateway；

• .NET：Ocelot；

• Node.js：Express Gateway、MicroGateway。

According to the scope of use, maturity, etc., there are 4 mainstream ones: OpenResty, Kong, Zuul/Zuul 2, and Spring Cloud Gateway.

2.4.1 Comparative Analysis of Mainstream Open Source Gateways

2.4.1.1 SpringCloud Gateway

Spring Cloud Gateway is based on Java 8, Spring 5.0, Spring Boot 2.0, and Project Reactor. It developed earlier than Zuul 2 and is currently part of the Spring Cloud family bucket.
Spring Cloud Gateway can be seen as an upgraded version and replacement of Zuul 1.x. It uses Netty to implement asynchronous I/O earlier than Zuul 2, thus realizing a simple, more efficient than Zuul 1.x, and Spring Cloud Tightly-coordinated API Gateway.

Spring Cloud Gateway clearly distinguishes between Router and Filter, and has a lot of built-in out-of-the-box functions, all of which can be used through Spring Boot configuration or manual coding chain calls. For example, there are 10 built-in routers, and you can make routing according to Header, Path, Host or Query by direct configuration.
Its core features :

• Match routes by request parameters;
• Implement routing through assertions and filters;
• Integration with Hystrix fuse;
• Integrate with Spring Cloud DiscoveryClient;
• Very convenient to implement assertions and filters;
• Request current limit;
• Path rewriting.

Open source project address: https://github.com/spring-cloud/spring-cloud-gateway/

2.4.1.2 Zuul/Zul2

Zuul is Netflix's open source API gateway system. Its main design goals are dynamic routing, monitoring, elasticity and security. The internal principle of Zuul can be simply regarded as a collection of many different functional filters (in contrast, ESB can also be simply regarded as a collection of pipes and filters). These filters (filters) can be written using Groovy or other JVM-based scripts (of course, Java can also be written), placed in a specified location, and then polled by Zuul Server, and dynamically loaded after changes are found and take effect in real time. Zuul currently has two versions, 1.x and 2.x, which are very different.

Zuul 1.x is based on synchronous I/O and is also part of the Spring Cloud family bucket, which can be easily configured and used with Spring Boot/Spring Cloud. In Zuul 1.x, the types and processing flow of Filter are shown in Figure 7-7. The most important ones are pre, routing, and post filters, which are respectively used for request processing and direct response before calling the business service API. , Response processing after calling the business service API.

The biggest improvement of Zuul 2.x is to implement asynchronous I/O to access requests based on Netty Server, and to implement requests to back-end business service APIs based on Netty Client. This allows for higher performance and lower latency. In addition, the Filter type is also adjusted, and the original three core Filters are explicitly named Inbound Filter, Endpoint Filter, and Outbound Filter. The
core functions of Zuul 2.x : service discovery, load balancing, connection pool, status classification, retry, request Credentials, HTTP/2, TLS, proxy protocols, GZip, WebSocket.

Project address: https://github.com/Netflix/zuul

2.4.1.3 OpenResty

Based on Nginx, OpenResty integrates the Lua language, Lua's various tool libraries, and available third-party modules, so that we can obtain the dynamic expansion capability provided by Lua on the basis of Nginx's existing efficient HTTP processing. Therefore, we can make various Lua scripts that meet the gateway strategy we need, and implement the gateway system based on it.

Project address: http://openresty.org/

2.4.1.3 Kong

Based on OpenResty, Kong is a cloud-native, fast, scalable, and distributed microservice abstraction layer (MicroserviceAbstraction Layer), also called API Gateway (API Gateway), also called API Middleware (API Middleware) in Service Mesh. Kong was open sourced in 2015, and its core value lies in its high performance and scalability. According to the statistics of Fortune 5000 organizations, Kong is the most widely used gateway in production environments that is still being maintained.

Its core features :

• Scalable: Horizontal expansion can be easily achieved by adding nodes, which means that a large system load can be supported at very low latency.
• Modularity: Kong's capabilities can be extended by adding new plugins, which can be installed and configured through the RESTful Admin API.
• Runs on any infrastructure: Kong can run anywhere, such as deploying Kong in a cloud or hybrid environment, or a single/global data center.

Project address: http://openresty.org/

2.4.1.4 Comparative Analysis

Current limiting function :

• Spring Cloud Gateway currently provides a Redis-based Ratelimiter implementation. The algorithm used is the token bucket algorithm, which is configured through a YAML file;
• Zuul2 can configure cluster current limiting and single server current limiting through configuration files, and can also implement current limiting expansion through Filter;
• OpenResty can use resty.limit.count, resty.limit.conn, resty.limit.req to implement current limiting function, and can implement leaky bucket or token pass algorithm;
• Kong has a basic current limiting component, which can be used for Lua development based on the source code of the basic component.

OpenResty/Kong/Zuul 2/SpringCloudGateway performance test comparison:

the vertical axis in the figure is QPS, and the horizontal axis is the data of a Gateway. Each line is different gateway data in a scenario. The test conclusions are as follows:

• The measured situation is the performance SCG~Zuul 2 << OpenResty~< Kong << Direct (direct connection);
• The performance of Spring Cloud Gateway and Zuul 2 is similar, about 40% of direct connection;
• The performance of OpenResty and Kong is similar, about 60% to 70% of direct connection;
• Under large concurrency, for example, when simulating 200 concurrent users or 1000 concurrent users, Zuul 2 will return an error with a high probability.

2.4.1.5 Technical Summary of Open Source Gateway

Talking about performance out of the scene is all "hooliganism". Under the same benchmark conditions, different parameters and software, relatively speaking, horizontal comparison is valuable. For example, the same machine (such as 16GB memory/4 cores), the same Server (use Spring Boot, the configuration path is api/hello, and return a helloworld), the same pressure measurement method and tool (such as wrk, 10 threads, 20 concurrent connections). We tested the limit QPS (QPS-Direct, 29K) obtained by directly accessing the server; configured a Spring Cloud Gateway as the limit QPS for gateway access (QPS-SCG, 11K); configured a Zuul 2 in the same way to do the limit obtained by the gateway pressure test QPS (QPS-Zuul2, 13K); Limit QPS obtained by Kong (QPS-Kong, 21K); Limit QPS obtained by OpenResty (QPS-OR, 19K). This comparison makes sense.

In general, the performance of Kong is very good, it is very suitable for traffic gateway, and the abstraction of service, route, upstream, consumer, and plugins is also worth learning from the self-developed gateway.
However, it is not recommended to use Kong for complex business gateways. Spring Cloud Gateway/Zuul 2 is more convenient for the Java technology stack and can rely on some common class libraries of the business system. Lua is inconvenient, not only a language problem, but also a problem of reusing infrastructure. In addition, for the gateway system, the performance will not be an order of magnitude worse, the problem is not big, and it can be "fixed" by adding 2 more machines.
From the test results, if the latency of the backend API service is low (for example, 2ms level), if the direct connection throughput is 100QPS, Kong can reach 60QPS, OpenResty is 50QPS, Zuul 2 and Spring CloudGateway are about 35QPS, If the latency of the service itself is higher, then these gaps will gradually narrow.
At present, there are still many pitfalls in Zuul 2 :
(1) It has just been released, immature, has no documentation, and there are not many practical application cases.
(2) The error rate is high when the concurrency is high. When the concurrency is 1000, our test scenario has an error rate of nearly 50%.
For simple use or light customization of the business gateway system, it is currently recommended to use Spring CloudGateway as the basic framework.