Lua writes Nginx request data to Kafka - buried point log solution

origin

There is a buried point collection system, and the architecture is Nginx+Flume. Clients such as web, applet, and App send data to Nginx, Nginx writes the request to a local file, and then Flume reads the data in the log file and writes the log to Kafka. There is no problem with this architecture, but there is a problem when it is deployed in a K8s container. Currently, there are three Flumes behind one Nginx. Nginx writes logs into web.log, mp.log, and app.log according to channels, and the three log files correspond to each other. A Flume writes data to Kafka. The first problem encountered is the health check problem. A K8s container can only provide one health check address (the best practice), because there are three Flumes behind Nginx, so it is impossible to perceive these three health Status, at most one can be perceived. The second problem is dynamic expansion. For example, there is a lot of data submitted by the web in a certain period of time, but there are few other two channels. Expansion into two pods will waste resources. The third problem is graceful exit. Since Nginx is faster, writing It is also faster to import files, but Flume is slower to process, so if I want to close this container, I don't know if Flume has written all the logs to Kafka.
Optimization idea: Flume actually has multiple imports and multiple exports. For the customer's business, the only entrance is Nginx, and the exit is only Kafka. Therefore, it is decided to remove Flume and use Nginx+Lua scripts to directly write Nginx logs. Kafka, at the same time, write a file in case there is a problem to supplement the data, and it is also used as a bug location tracking.

Environmental preparation

openresty-1.15.8.2
download address: http://openresty.org/cn/download.html
lua-resty-kafka is the lua version of the kafka driver, built-in producer
download address: https://github.com/doujiang24/lua-resty-kafka
zlib's lua library is used by gzip. If you don't use gzip, you don't need to install it in the script.
Download link: https://github.com/madler/zlib
lua-zlib The library used by lua to call gzip
Download address: https://github.com/brimworks/lua-zlib
Nginx and dependencies for compiling the above components: (I use debian10.11)

apt-get update -y && apt-get install --fix-missing zlib1g zlib1g-dev libpcre3-dev libssl-dev perl make build-essential curl cmake -y

step

Unzip openresty and compile and install

cp openresty-1.15.8.2.tar.gz /opt/app/source/
tar -zxvf openresty-1.15.8.2.tar.gz

cd openresty-1.15.8.2 && ./configure --prefix=/opt/app/openresty && make && make install

To put the kafka driver into lualib is to unzip lua-resty-kafka-0.10.zip, and copy the resty folder under lua-resty-kafka-0.10/lib directly into /opt/app/openresty/site/lualib/

unzip lua-resty-kafka-0.10.zip
cd lua-resty-kafka-0.10/lib 
cp -r resty  /opt/app/openresty/site/lualib/

Install the zlib library and lua-zlib library written by lua

tar -zxvf zlib-master.tar.gz
cp -a zlib-master/* /opt/app/openresty/site/lualib/

tar -zxvf lua-zlib-master.tgz
cd lua-zlib-master \
    && cmake -DLUA_INCLUDE_DIR=/opt/app/openresty/luajit/include/luajit-2.1 -DLUA_LIBRARIES=/opt/app/openresty/luajit/lib -DUSE_LUAJIT=ON -DUSE_LUA=OFF \
    && make \
    && cp zlib.so /opt/app/openresty/lualib/zlib.so

Write kafkaconfig.lua and put it in /opt/app/openresty/site/lualib/ This step is to configure kafka's ip, port and topic.
The content is:

kafka_broker_list={
    
    
    {
    
    host="192.168.1.1",port=9092},
    {
    
    host="192.168.1.2",port=9092},
    {
    
    host="192.168.1.3",port=9092}
}

kafka_topic_mp="mp_topic"
kafka_topic_app="app_topic"
kafka_topic_web="web_topic"

Write the nginx configuration file nginx.conf. The content is as follows:

worker_processes  auto;
error_log /opt/app/logs/error.log;
events {
    worker_connections 10240;
}

http {
    gzip on;
    gzip_types application/javascript text/plain text/xml text/css application/x-javascript application/xml text/javascript application/x-httpd-php image/jpeg image/gif image/png;
    gzip_vary on;
    gzip_comp_level 9;
    
    server {
        listen 80;
        include /opt/app/openresty/nginx/conf/web.conf;
        }

    }
}

web.conf

location ^~ /health {
    default_type text/html;
    return 200 'ok';
}
# app 批量
location ^~ /app {
    if ($request_method != POST) {
        return 405;
    }
    default_type 'text/plain';
    expires off;
    add_header Last-Modified '';
    add_header Cache-Control 'no-cache';
    add_header Pragma "no-cache";
    empty_gif;
    echo_read_request_body;
    access_by_lua_file /opt/app/openresty/nginx/conf/app.lua;
    access_log   /opt/app/logs/app.log;
}

app.lua:

ngx.req.read_body()
local body = ngx.req.get_body_data()

#引入kafka的生产者
local producer = require "resty.kafka.producer"
#引入上面写的kafka配置
local kafka_config = require "kafkaconfig"

# 因为引入了kafkaconfig 里面的变量直接访问就好
# local p = producer:new(kafka_broker_list)
# --------> 2023-03-16 分界线 start<------------
# 请使用以下方式初始化producer 相比于上面的初始化，下面添加了broker刷新时间
# 可以在网络抖动获取不到boker list，防止后续无法继续使用的问题
# 详细见我另一篇文章 https://blog.csdn.net/codeblf2/article/details/129505283
local p = producer:new(kafka_broker_list, {
    
    producer_type = "async",refresh_interval=10000})
# --------> 2023-03-16 分界线 end <------------

# 中间为nil的参数是kafka将本条消息写入哪个分区所使用的key，为nil代表轮询写入
local offset, err = p:send(kafka_topic_app, nil, body)
if not offset then
    ngx.say("send err:", err)
    return
end

The above example only shows the app, web and mp are similar.