Some common current limiting and anti-brush methods under high concurrency

Current limiting and anti-swiping
Internet projects are different from traditional projects. Internet projects are exposed to the Internet and are aimed at all netizens. At this time, the following two access forms may occur, requiring us to take some necessary measures to protect our services.
1. High-frequency access of a large number of normal users leads to server downtime
2. High-frequency access of malicious users leads to server downtime
3. Web crawler
In these cases, we need to limit current access to user access. Limit current.
Nginx is the layer with the largest granularity. We need to operate the frequency setting of this layer carefully. This will affect our entire website access. The frequency setting of the Nginx layer should be below the downtime threshold of our application server. Let's take a look at the details. How to set
Nginx current limit settings
to limit IP/domain name

http {
     limit_conn_zone $binary_remote_addr zone=perip :10m; # 保存IP的缓存为10M；16000个IP地址的状态信息约1MB
     limit_conn_zone $server_name zone=perserver:10m;
     ...
     server {
       limit_conn perserver 100;# 此域名下最多有100个连接
       limit_conn perip 10;# 一个IP最多有10个连接
         ...
     }
}
# $binary_remote_addr 要限流的IP地址
# $server_name 要限流的域名
location / {
        limit_req zone=perip burst=20 nodelay;# urst排队大小，nodelay不限制单个请求间的时间
        proxy_pass http://XXX.com;
}
# 限流白名单
geo $limit {
    default   1;
    192.168.2.0/24  0;# 192.168.2.1-192.168.2.254 且子网掩码是255.255.255.0 网段不限流
    # 24 表示子网掩码 255.255.255.0
    # 16 表示子网掩码 255.255.0.0
    # 8 表示子网掩码 255.0.0.0
}

map $limit $limit_key {
    1 $binary_remote_addr;
    0 "";
}

limit_req_zone $limit_key zone=mylimit:10m rate=1r/s;

location / {
        limit_req zone=perid burst=1 nodelay;
        proxy_pass http://XXXX.com;
}

The token algorithm current limiting (excerpted from https://blog.csdn.net/sunnyyoona/article/details/51228456 )
The token bucket algorithm originally originated from the computer network. When transmitting data on the network, in order to prevent network congestion, it is necessary to limit the flow out of the network, so that the flow is sent out at a relatively uniform speed. The token bucket algorithm implements this function, which can control the amount of data sent to the network and allow burst data to be sent.

Algorithm Description:

• If the average sending rate configured by the user is r, a token is added to the bucket every 1/r second (r tokens are put into the bucket every second);

• It is assumed that at most b tokens can be stored in the bucket. If the token bucket is full when the token arrives, the token will be discarded;

• When a packet of n bytes arrives, n tokens are removed from the token bucket (packets of different sizes, the number of tokens consumed is different), and the packet is sent to the network;

• If there are less than n tokens in the token bucket, the token is not removed and the packet is considered to be outside the traffic limit (n bytes, n tokens required. The packet will be cached or dropped) ;

• The algorithm allows bursts of up to b bytes, but from the long-run results, the packet rate is limited to a constant r. Packets outside the flow limit can be handled in different ways: (1) they can be dropped; (2) they can be queued for transmission when enough tokens have accumulated in the token bucket; ( 3) They can continue to be sent, but they need to be specially marked, and these specially marked packets will be discarded when the network is overloaded.

Note:
The token bucket algorithm should not be confused with another common algorithm, the leaky bucket algorithm. The main difference between the two algorithms is that the leaky bucket algorithm can forcibly limit the transmission rate of data, while the token bucket algorithm can limit the average transmission rate of data and also allow a certain degree of burst transmission. In the token bucket algorithm, as long as there are tokens in the token bucket, data is allowed to be transmitted in bursts until the user-configured threshold is reached, so it is suitable for traffic with burst characteristics.
Combined with Lua to limit the current count of different interfaces,
for example, the application limits 100 requests per second

http {
    local shared_data = ngx.shared.dict
    shared_data:set("draw", 0)

    content_by_lua_block {
        local request_uri = ngx.var.request_uri;
        if string.sub(request_uri,1,22) == "/activity/lottery/draw" then
              local val, err = ngx.shared.dict:incr("draw", 1); #进来一个请求就加1
              if val > 100 then #限流100
                  ngx.log(ngx.ERR,"draw limit val is:"..val)
                  ngx.exit(503)
              end
              ....业务处理
        end
    }
...
  log_by_lua_block{
        local request_uri = ngx.var.request_uri;
        if string.sub(request_uri,1,22) == "/activity/lottery/draw" then
              local val, err = ngx.shared.dict:incr("draw", -1); #出去一个请求就减1
              if val < 0 then 
                  ngx.shared.dict:set("draw", 0);
              end
        end

  }
}

Next, we are looking at an operation of redis using Lua, and the current limit is realized by setting the key count and validity period of redis.
The following code is configured in nginx.conf and can be used, for example:

server {
    listen  80;

    location / {
        access_by_lua_file /opt/lua/access/rateLimter.lua;
        proxy_pass http://www.xxx.com;
    }
}
# nginx常用lua模块还有
lua_code_cache
语法：lua_code_cache on | off
默认： on
适用上下文：http、server、location、location if
这个指令是指定是否开启lua的代码编译缓存，开发时可以设置为off，以便lua文件实时生效，如果是生产线上，为了性能，建议开启。
lua_package_path
语法：lua_package_path <lua-style-path-str>
默认：由lua的环境变量决定
适用上下文：http
设置lua代码的寻找目录。
例如：lua_package_path "/opt/nginx/conf/www/?.lua;;";
具体的路径设置要参考lua的模块机制
init_by_lua（_file）
语法：init_by_lua <lua-script-str>
适用上下文：http
 init_by_lua 'cjson = require "cjson"';

    server {
        location = /api {
            content_by_lua '
                ngx.say(cjson.encode({dog = 5, cat = 6}))
            ';
        }
    }
从这段配置代码，我们可以看出，其实这个指令就是初始化一些lua的全局变量，以便后续的代码使用。
注：有（_file）的选项代表可以直接引用外部的lua源代码文件，效果与直接写配置文件一样，不过可维护性当然是分开好点。
init_worker_by_lua(_file)
类似于上面的，不过是作用在work进程的，先于work进程启动而调用。
set_by_lua(_file)
语法：set_by_lua $res <lua-script-str> [$arg1 $arg2 ...]
适用上下文：server、location、location if
 location /foo {
        set $diff ''; # we have to predefine the $diff variable here

        set_by_lua $sum '
            local a = 32
            local b = 56

            ngx.var.diff = a - b;  -- write to $diff directly
            return a + b;          -- return the $sum value normally
        ';
        echo "sum = $sum, diff = $diff";
    }
这个指令是为了能够让nginx的变量与lua的变量相互作用赋值。
content_by_lua(_file)
语法：content_by_lua <lua-script-str>
适用上下文：location、location if
        location /nginx_var {
            # MIME type determined by default_type:
            default_type 'text/plain';

            # try access /nginx_var?a=hello,world
            content_by_lua "ngx.print(ngx.var['arg_a'], '\\n')";
        }
通过这个指令，可以由lua直接确定nginx响应页面的正文。
rewrite_by_lua(_file)
语法：rewrite_by_lua <lua-script-str>
适用上下文：location、location if
这个指令更多的是为了替代HttpRewriteModule的rewrite指令来使用的，优先级低于rewrite指令
比如
 location /foo {
          set $a 12; # create and initialize $a
          set $b ''; # create and initialize $b
          rewrite_by_lua 'ngx.var.b = tonumber(ngx.var.a) + 1';
          if ($b = '13') {
             rewrite ^ /bar redirect;
             break;
          }

         echo "res = $b";
    }
这个并不会像预期的那样子，因为我猜测，rewrite_by_lua是开启一个协程去工作的，可是下面却继续执行下去了，所以得不到预期的结果。
此时如果由lua代码来控制rewrite，那就没有问题了。
    location /foo {
        set $a 12; # create and initialize $a
        set $b ''; # create and initialize $b
        rewrite_by_lua '
            ngx.var.b = tonumber(ngx.var.a) + 1
            if tonumber(ngx.var.b) == 13 then
                return ngx.redirect("/bar");
            end
        ';

        echo "res = $b";
    }
access_by_lua(_file)
语法：access_by_lua <lua-script-str>
适用上下文：http, server, location, location if
 location / {
        deny    192.168.1.1;
        allow   192.168.1.0/24;
        allow   10.1.1.0/16;
        deny    all;

        access_by_lua '
            local res = ngx.location.capture("/mysql", { ... })
            ...
        ';

        # proxy_pass/fastcgi_pass/...
    }
    顾名思义，这个指令用在验证通过或者需要验证的时候。


header_filter_by_lua(_file)
语法：header_filter_by_lua <lua-script-str>
适用上下文：http, server, location, location if
    location / {
        proxy_pass http://mybackend;
        header_filter_by_lua 'ngx.header.Foo = "blah"';
    }
用lua的代码去指定http响应的 header一些内容。


body_filter_by_lua(_file)
语法：body_filter_by_lua <lua-script-str>
适用上下文：http, server, location, location if
   location /t {
        echo hello world;
        echo hiya globe;

        body_filter_by_lua '
            local chunk = ngx.arg[1]
            if string.match(chunk, "hello") then
                ngx.arg[2] = true  -- new eof
                return
            end

            -- just throw away any remaining chunk data
            ngx.arg[1] = nil
        ';
    }
这个指令可以用来篡改http的响应正文的。

Script storage directory: /opt/lua/access/rateLimter.lua

local function close_redis(red)  
    if not red then  
        return  
    end    
    local pool_max_idle_time = 10000  
    local pool_size = 100  
    local ok, err = red:set_keepalive(pool_max_idle_time, pool_size)  

    if not ok then  
        ngx_log(ngx_ERR, "set redis keepalive error : ", err)  
    end  
end  

local function wait()  
   ngx.sleep(1)  
end  

local redis = require "resty.redis"  
local red = redis:new()  
red:set_timeout(1000)  
local ip = "redis-ip"  
local port = redis-port  
local ok, err = red:connect(ip,port)  
if not ok then  
    return close_redis(red)  
end  

local uri = ngx.var.uri   
local uriKey = "req:uri:"..uri  
res, err = red:eval("local res, err = redis.call('incr',KEYS[1]) if res == 1 then local resexpire, err = redis.call('expire',KEYS[1],KEYS[2]) end return (res)",2,uriKey,1)  
while (res > 10)  
do   
   local twait, err = ngx.thread.spawn(wait)  
   ok, threadres = ngx.thread.wait(twait)  
   if not ok then  
      ngx_log(ngx_ERR, "wait sleep error: ", err)  
      break;  
   end  
   res, err = red:eval("local res, err = redis.call('incr',KEYS[1]) if res == 1 then local resexpire, err = redis.call('expire',KEYS[1],KEYS[2]) end return (res)",2,uriKey,1)  
end  
close_redis(red)  

Below we are looking at a brush-proof Lua code example

local function close_redis(red)  
    if not red then  
        return  
    end   
    local pool_max_idle_time = 10000 
    local pool_size = 100 
    local ok, err = red:set_keepalive(pool_max_idle_time, pool_size)  

    if not ok then  
        ngx_log(ngx_ERR, "set redis keepalive error : ", err)  
    end  
end  

local redis = require "resty.redis"  
local red = redis:new()  
red:set_timeout(1000)  
local ip = "redis-ip"  
local port = redis-port  
local ok, err = red:connect(ip,port)  
if not ok then  
    return close_redis(red)  
end  

local clientIP = ngx.req.get_headers()["X-Real-IP"]  
if clientIP == nil then  
   clientIP = ngx.req.get_headers()["x_forwarded_for"]  
end  
if clientIP == nil then  
   clientIP = ngx.var.remote_addr  
end  

local incrKey = "user:"..clientIP..":freq"  
local blockKey = "user:"..clientIP..":block"  

local is_block,err = red:get(blockKey)
if tonumber(is_block) == 1 then  
   ngx.exit(ngx.HTTP_FORBIDDEN)  
   return close_redis(red)  
end  

res, err = red:incr(incrKey)  

if res == 1 then  
   res, err = red:expire(incrKey,1)  
end  

if res > 200 then  
    res, err = red:set(blockKey,1)  
    res, err = red:expire(blockKey,600)  
end  

close_redis(red)  

Nginx blacklist
When we find some malicious IPs, we can put them in the blacklist, the configuration is as follows:

# 白名单设置，访问根目录
location / {
                allow 123.34.22.155;
                deny  all;# allow 优先级>deny
}

# 黑名单设置，访问根目录，这时候访问Nginx会出现403 forbidden字样
location / {
                deny 123.34.22.155;
}

# 特定目录访问限制
location /tree/list {
                allow 123.34.22.155;
                deny  all;
}

Or we can dynamically manage the blacklist IP through Luau+redis
First modify nginx.conf

lua_shared_dict ip_blacklist 1m;

server {
    listen  80;

    location / {
        access_by_lua_file lua/ip_blacklist.lua;
        proxy_pass http://real_server;
    }
}

local redis_host    = "192.168.1.132"
local redis_port    = 6379
local redis_pwd     = 123456
local redis_db = 2

-- connection timeout for redis in ms.
local redis_connection_timeout = 100

-- a set key for blacklist entries
local redis_key     = "ip_blacklist"

-- cache lookups for this many seconds
local cache_ttl     = 60

-- end configuration

local ip                = ngx.var.remote_addr
local ip_blacklist      = ngx.shared.ip_blacklist
local last_update_time  = ip_blacklist:get("last_update_time");

-- update ip_blacklist from Redis every cache_ttl seconds:
if last_update_time == nil or last_update_time < ( ngx.now() - cache_ttl ) then

  local redis = require "resty.redis";
  local red = redis:new();

  red:set_timeout(redis_connect_timeout);

  local ok, err = red:connect(redis_host, redis_port);
  if not ok then
    ngx.log(ngx.ERR, "Redis connection error while connect: " .. err);
  else
    local ok, err = red:auth(redis_pwd)
    if not ok then
      ngx.log(ngx.ERR, "Redis password error while auth: " .. err);
    else
        local new_ip_blacklist, err = red:smembers(redis_key);
        if err then
            ngx.log(ngx.ERR, "Redis read error while retrieving ip_blacklist: " .. err);
        else
        ngx.log(ngx.ERR, "Get data success:" .. new_ip_blacklist)
          -- replace the locally stored ip_blacklist with the updated values:
            ip_blacklist:flush_all();
          for index, banned_ip in ipairs(new_ip_blacklist) do
            ip_blacklist:set(banned_ip, true);
          end
          -- update time
          ip_blacklist:set("last_update_time", ngx.now());
      end
    end
  end
end

if ip_blacklist:get(ip) then
  ngx.log(ngx.ERR, "Banned IP detected and refused access: " .. ip);
  return ngx.exit(ngx.HTTP_FORBIDDEN);
end