Example Configuration
quote
http {
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
...
server {
...
location /search/ {
limit_req zone=one burst=5;
}
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
...
server {
...
location /search/ {
limit_req zone=one burst=5;
}
Refer to the Nginx official website: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone
Baidu spider crawls have increased sharply, resulting in high server load. Finally, the ngx_http_limit_req_module module of nginx is used to limit the crawling frequency of Baidu spider. Allow Baidu spider to crawl 200 times per minute, and return 503 for redundant crawl requests.
nginx configuration: #Global
configuration
limit_req_zone $anti_spider zone=anti_spider:60m rate=200r/m; #limit_req zone=anti_spider burst=5 nodelay
in a server ; if ($http_user_agent ~* "baiduspider") { set $anti_spider $ http_user_agent; } Parameter description: rate=200r/m in the instruction limit_req_zone means that only 200 requests can be processed per minute. The burst=5 in the instruction limit_req indicates that the maximum concurrency is 5. That is, only 5 requests can be processed at the same time. The nodelay in the instruction limit_req indicates that when the burst value has been reached, when a new request is made, the 503 IF part will be directly returned to determine whether it is the user agent of Baidu Spider. If so, assign a value to the variable $anti_spider. In this way, only Baidu spiders are restricted.
For detailed parameter descriptions, you can check the official documentation.
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone
This module uses the leaky bucket algorithm to limit requests.
For details on the leaky bucket algorithm, see http://baike.baidu.com/view/2054741.htm . For
related codes, please check the nginx source code file src/http/modules/ngx_http_limit_req_module.c
The core part of the code is the ngx_http_limit_req_lookup method.
Reference: http://www.bo56.com/%E4%BD%BF%E7%94%A8nginx%E9%99%90%E5%88%B6%E7%99%BE%E5%BA%A6%E8 %9C%98%E8%9B%9B%E7%9A%84%E9%A2%91%E7%B9%81%E6%8A%93%E5%8F%96/