1. The problem of nginx reload
Problem Description
Nginx reload has a certain loss. If you are using a long connection, then when reloading nginx, all worker processes that are connected for a long time will exit gracefully, and when all the connections on the worker process are released, the process really exits.
Solution
There are currently three options for the community version of nginx:
The Dyups module of Tengine.
Upsync+Consul of Weibo realizes dynamic load balancing.
OpenResty's balancer_by_lua (Paiyun uses its open source slardar (Consul balancer_by_lua)).
This article uses the upsync module to solve the problem of performance degradation caused by the reload nginx process after the configuration file is modified.
Its function is to pull the list of consul's back-end servers and update Nginx routing information. This module does not depend on any third-party modules. As the db of Nginx, consul utilizes the KV service of consul. Each Nginx work process independently pulls the configuration of each upstream and updates its own route.
Two, actual combat
2.1 Patch nginx
You don’t need to do this step, if you don’t, delete this module when compiling
git clone https://github.com/xiaokai-wang/nginx_upstream_check_module
## 打补丁包
patch -p0 < /usr/local/src/nginx_upstream_check_module-master/check_1.9.2+.patch
Download nginx-upsync-module source code
git clone https://github.com/weibocom/nginx-upsync-module.git
下载nginx源码
wget 'http://nginx.org/download/nginx-1.10.1.tar.gz'
tar -xzvf nginx-1.10.1.tar.gz
cd nginx-1.10.1/
开始编译
./configure --prefix=/data/app/nginx-1.10.1 --user=nginx --group=nginx --with-http_ssl_module --with-http_stub_status_module --add-module=/usr/local/src/nginx-upsync-module-master/ --add-module=/usr/local/src/nginx_upstream_check_module-master/
make
make install
Three, start consul
wget https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_linux_amd64.zip
unzip consul_0.6.4_linux_amd64.zip
./consul agent -advertise=x.x.x.x -client=0.0.0.0 -dev
Create nginx configuration file
mkdir -p /usr/local/nginx/conf/servers
/usr/local/nginx/conf/nginx.conf
events {
worker_connections 4096; ## Default: 1024
}
http {
upstream test {
# fake server otherwise ngx_http_upstream will report error when startup
server 127.0.0.1:11111;
# all backend server will pull from consul when startup and will delete fake server
upsync 127.0.0.1:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;
}
upstream bar {
server 127.0.0.1:8090 weight=1 fail_timeout=10 max_fails=3;
}
server {
listen 8080;
location = /proxy_test {
proxy_pass http://test;
}
location = /bar {
proxy_pass http://bar;
}
location = /upstream_show {
upstream_show;
}
}
}
Four, test
for i in `seq 3`;do mkdir html$i/test -p && echo $i >html$i/test/test.html; done;
docker run -d -p 8001:80 -v /root/html1/:/usr/share/nginx/html nginx
docker run -d -p 8002:80 -v /root/html2/:/usr/share/nginx/html nginx
docker run -d -p 8003:80 -v /root/html3/:/usr/share/nginx/html nginx
Five, add services
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/192.168.56.12:8001
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/192.168.56.12:8002
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/192.168.56.12:8003
Check whether there is content in the conf/servers/servers_test.conf file
cat conf/servers/servers_test.conf
server 192.168.56.12:8003 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.56.12:8002 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.56.12:8001 weight=1 max_fails=2 fail_timeout=10s;
Or open the browser http://192.168.56.11:8080/upstream_show?test
The display content is as follows:
Upstream name: test; Backend server count: 3
server 192.168.56.12:8003 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.56.12:8002 weight=1 max_fails=2 fail_timeout=10s;
server 192.168.56.12:8001 weight=1 max_fails=2 fail_timeout=10s;
to sum up
This module only modifies the cache information in upstream, and cannot modify or add other configurations
Problems encountered in the test
The following error occurred when adding services, which caused the service addition to not be performed in real time, and it took about 3 minutes.
consul log:
2016/03/22 05:34:42 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (149.023µs) from=127.0.0.1:38853
2016/03/22 05:34:43 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (146.759µs) from=127.0.0.1:38854
2016/03/22 05:34:45 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (149.853µs) from=127.0.0.1:38855
2016/03/22 05:34:46 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (111.46µs) from=127.0.0.1:38856
2016/03/22 05:34:48 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (142.696µs) from=127.0.0.1:38857
2016/03/22 05:34:48 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (112.089µs) from=127.0.0.1:38858
2016/03/22 05:34:49 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (114.29µs) from=127.0.0.1:38859
2016/03/22 05:34:50 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (148.245µs) from=127.0.0.1:38860
nginx log
2016/03/22 05:35:09 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:09 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:10 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:10 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:11 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:11 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:13 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:13 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:13 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:13 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:14 [error] 18879#0: recv() failed (104: Connection reset by peer)
Problem phenomenon
When adding a service, this problem occurs, and the new service cannot be added to the load in time, and it does not affect the normal service. At this time, if you continue to add a service to consul, this error may be terminated, and the current two service records can be successfully added.
Six, help documentation
The official github address
nginx_upstream_check_module https://www.cnblogs.com/Carpe...
