nginx (multi-instance) installation and deployment operation reference (linux)

1. Installation environment preparation

1.1 Host environment preparation

1.1.1. Close selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

1.1.2. Deployment planning

Example 1 installation plan

Software installation path: /usr/local/nginx/

Software log path: /usr/local/nginx/logs/

Software binary path: /usr/local/nginx/sbin/

Software cache proxy and other paths: /usr/local/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}

Software main configuration file path: /usr/local/nginx/conf

Software sub-configuration file path: /usr/local/nginx/conf/conf.d/

Pidfile path: /usr/local/nginx/logs/nginx.pid

Lockfile path: /var/lock/nginx.lock

sbin-path路径：/usr/local/nginx/sbin/nginx

Port planning: 80

Example 2 installation planning

Software installation path: /usr/local/nginx2/

Software log path: /usr/local/nginx2/logs/

Software binary path: /usr/local/nginx2/sbin/

Software cache proxy and other paths: /usr/local/nginx2/{client_body,proxy,fastcgi,uwsgi,scgi}

Software main configuration file path: /usr/local/nginx2/conf

Software sub-configuration file path: /usr/local/nginx2/conf/conf.d/

Pidfile path: /usr/local/nginx2/logs/nginx2.pid

Lockfile path: /var/lock/nginx2.lock

sbin-path路径：/usr/local/nginx2/sbin/nginx2

Port planning: 8080

1.1.3. System host time, time zone, system language

l This section requires operations based on actual conditions

l  修改时区
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
l  修改系统语言环境
echo 'LANG="en_US.UTF-8"' >> /etc/profile && source /etc/profile
l  配置主机NTP时间同步
yum -y install ntp
systemctl enable ntpd && systemctl start ntpd
echo 'server ntp1.aliyun.com' >> /etc/ntp.conf
echo 'server ntp2.aliyun.com' >> /etc/ntp.conf

2. Nginx installation and deployment

Remarks: The dependent packages of chapters 2.2, 2.3 and 2.4 can be installed directly using yum. The installation instructions are as follows:

yum install -y pcre-devel zlib-devel openssl-devel

You can also use the source code to install according to this document. If you use yum to install, please note that the installation parameters of nginx should delete the following three lines

--with-pcre=../pcre-8.44 \
--with-zlib=../zlib-1.2.11 \
--with-openssl=../openssl-1.1.1g \  

2.1 Nginx dependency installation and environment preparation

l Add users and user groups (please define your own user name)

groupadd -r nginx && useradd -s /sbin/nologin -r -g nginx nginx

l CentOS platform installation dependencies

yum -y install gcc gcc-c++ automake autoconf libtool make wget net-tools
yum install -y libxslt* libxml2* gd-devel perl-devel perl-ExtUtils-Embed GeoIP GeoIP-devel GeoIP-data

l Download the nginx-1.19.1.tar.gz installation package and unzip it

cd /opt
wget http://nginx.org/download/nginx-1.19.1.tar.gz
tar -zxvf nginx-1.19.1.tar.gz

l Hide the nginx version number from the root cause

(1) Modify the following three lines of configuration information in the nginx.h file, for example as follows

vi /opt/nginx-1.19.1/src/core/nginx.h
修改前
#define nginx_version      1019001
#define NGINX_VERSION      "1.19.1"
#define NGINX_VER          "nginx/" NGINX_VERSION
修改后
#define nginx_version      1010001
#define NGINX_VERSION      "618"
#define NGINX_VER          "WEB/" NGINX_VERSION

(2) Modify the ngx_http_server_string display name of the ngx_http_header_filter_module.c file to be consistent with the NGINX_VER name in step 1.

vi /opt/nginx-1.19.1/src/http/ngx_http_header_filter_module.c
修改前
static u_char ngx_http_server_string[] = "Server: nginx" CRLF;
static u_char ngx_http_server_full_string[] = "Server: " NGINX_VER CRLF;
static u_char ngx_http_server_build_string[] = "Server: " NGINX_VER_BUILD CRLF;
修改后
static u_char ngx_http_server_string[] = "Server: WEB" CRLF;
static u_char ngx_http_server_full_string[] = "Server: " NGINX_VER CRLF;
static u_char ngx_http_server_build_string[] = "Server: " NGINX_VER_BUILD CRLF;

(2) Modify the ngx_http_error_tail display name of the ngx_http_special_response.c file to be consistent with the NGINX_VER name in step 1.

vi /opt/nginx-1.19.1/src/http/ngx_http_special_response.c
修改前
static u_char ngx_http_error_tail[] =
"<hr><center>nginx</center>" CRLF
"</body>" CRLF
"</html>" CRLF
;
修改后
static u_char ngx_http_error_tail[] =
"<hr><center>WEB</center>" CRLF
"</body>" CRLF
"</html>" CRLF
;

Note: Please save the configuration file after modification

l Nginx deployment environment preparation

mkdir -pv /usr/local/nginx/{logs,client_body,proxy,fastcgi,uwsgi,scgi}
chown -R nginx:nginx /usr/local/nginx

2.2 Pcre installation

cd /opt
wget https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz
tar -zxvf pcre-8.44.tar.gz
cd pcre-8.44/
./configure
make && make install

2.3 Zlib installation

cd /opt
wget http://www.zlib.net/fossils/zlib-1.2.11.tar.gz
tar -zxvf zlib-1.2.11.tar.gz
cd zlib-1.2.11
./configure
make && make install

2.4 Openssl installation

cd /opt
wget https://ftp.openssl.org/source/old/1.1.1/openssl-1.1.1g.tar.gz
tar -zxvf openssl-1.1.1g.tar.gz
cd openssl-1.1.1g
./config
make && make install
openssl version
OpenSSL 1.1.1g  31 Mar 2020

l Note: If the openssl version input does not correspond and needs to be reinstalled, the following methods are recommended

cd openssl-OpenSSL_1_1_1g
./config  --prefix=/usr/local/openssl
make
make install
ln -s /usr/local/openssl/lib/libssl.so.1.1 /usr/local/lib/
ln -s /usr/local/openssl/lib/libcrypto.so.1.1  /usr/local/lib/
ln -s /usr/local/lib/libssl.so.1.1  /usr/lib/
ln -s /usr/local/lib/libcrypto.so.1.1  /usr/lib/
ln -s /usr/local/lib/libssl.so.1.1  /usr/lib64/
ln -s /usr/local/lib/libcrypto.so.1.1  /usr/lib64/
/usr/local/openssl/bin/openssl version
ldd /usr/local/openssl/bin/openssl
ldconfig -v
mv /usr/bin/openssl /usr/bin/openssl.old
ln -s /usr/local/openssl/bin/openssl /usr/bin/openssl
openssl version
OpenSSL 1.1.1f  31 Mar 2020

l When installing nginx, you need to specify the openssl path

--with-openssl=../openssl-1.1.1g \

2.5 Nginx installation

cd /opt/nginx-1.19.1
./configure \
--prefix=/usr/local/nginx \
--pid-path=/usr/local/nginx/logs/nginx.pid  \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_xslt_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_random_index_module \
--with-http_degradation_module \
--with-http_secure_link_module \
--with-http_gzip_static_module \
--with-http_perl_module \
--with-pcre=../pcre-8.44 \
--with-zlib=../zlib-1.2.11 \
--with-openssl=../openssl-1.1.1g \
--with-debug \
--with-file-aio \
--with-mail \
--with-mail_ssl_module \
--http-client-body-temp-path=/usr/local/nginx/client_body \
--http-proxy-temp-path=/usr/local/nginx/proxy \
--http-fastcgi-temp-path=/usr/local/nginx/fastcgi \
--http-uwsgi-temp-path=/usr/local/nginx/uwsgi \
--http-scgi-temp-path=/usr/local/nginx/scgi \
--with-stream \
--with-ld-opt="-Wl,-E"
make && make install

2.6 Nginx2 installation

l Add users and user groups (please define your own user name)

groupadd -r nginx2 && useradd -s /sbin/nologin -r -g nginx2 nginx2

l Nginx2 deployment environment preparation

mkdir -pv /usr/local/nginx2/{logs,client_body,proxy,fastcgi,uwsgi,scgi}
chown -R nginx2:nginx2 /usr/local/nginx2

l Nginx2 installation

cd /opt/nginx-1.19.1
./configure \
--prefix=/usr/local/nginx2 \
--pid-path=/usr/local/nginx2/logs/nginx2.pid  \
--sbin-path=/usr/local/nginx2/sbin/nginx2 \
--conf-path=/usr/local/nginx2/conf/nginx.conf \
--error-log-path=/usr/local/nginx2/logs/error.log \
--http-log-path=/usr/local/nginx2/logs/access.log \
--lock-path=/var/run/nginx2.lock \
--user=nginx2 \
--group=nginx2 \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_xslt_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_random_index_module \
--with-http_degradation_module \
--with-http_secure_link_module \
--with-http_gzip_static_module \
--with-http_perl_module \
--with-pcre=../pcre-8.44 \
--with-zlib=../zlib-1.2.11 \
--with-openssl=../openssl-1.1.1g \
--with-debug \
--with-file-aio \
--with-mail \
--with-mail_ssl_module \
--http-client-body-temp-path=/usr/local/nginx2/client_body \
--http-proxy-temp-path=/usr/local/nginx2/proxy \
--http-fastcgi-temp-path=/usr/local/nginx2/fastcgi \
--http-uwsgi-temp-path=/usr/local/nginx2/uwsgi \
--http-scgi-temp-path=/usr/local/nginx2/scgi \
--with-stream \
--with-ld-opt="-Wl,-E"
make && make install
cd /usr/local/nginx2/conf
vim nginx.conf  --修改监听端口，避免启动冲突

2.7 Configure nginx system service

1. Add the nginx system service startup script

vi /etc/init.d/nginx
#!/bin/bash
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   2345 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
#
# processname: nginx
# config:      /usr/local/nginx/conf/nginx.conf
# pidfile:     /usr/local/nginx/logs/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/nginx.lock

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    echo -n "Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n "Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    status)
        rh_status
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart}"
        exit 2
        ;;
esac

2. Configure nginx system service and self-start

chmod +x /etc/init.d/nginx
chkconfig --add nginx && chkconfig nginx on
chkconfig --list nginx

3. Start and stop nginx service

service nginx start    或使用  systemctl start nginx
service nginx status    或使用  systemctl status nginx
ps -ef|grep nginx
service nginx stop    或使用  systemctl stop nginx

2.8 Configure nginx2 system service

1. Add the nginx system service startup script

vi /etc/init.d/nginx2
#!/bin/bash
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   2345 88 12
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
#
# processname: nginx
# config:      /usr/local/nginx2/conf/nginx.conf
# pidfile:     /usr/local/nginx2/logs/nginx2.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/local/nginx2/sbin/nginx2"
prog=$(basename $nginx)

NGINX_CONF_FILE="/usr/local/nginx2/conf/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/nginx2.lock

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    echo -n "Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n "Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    status)
        rh_status
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart}"
        exit 2
        ;;
esac

2. Configure nginx2 system service and self-start

chmod +x /etc/init.d/nginx2
chkconfig --add nginx2 && chkconfig nginx2 on
chkconfig --list nginx2

3. Start and stop the nginx service (note that the port cannot conflict when starting)

service nginx2 start    或使用  systemctl start nginx2
service nginx2 status    或使用  systemctl status nginx2
ps -ef|grep nginx2
service nginx2 stop    或使用  systemctl stop nginx2

2.9 Configure nginx environment variables

cat >>/etc/profile<<EOF
NGINX_HOME=/usr/local/nginx
PATH=\$NGINX_HOME/sbin:\$PATH

NGINX2_HOME=/usr/local/nginx2
PATH=\$NGINX2_HOME/sbin:\$PATH
EOF
source /etc/profile

2.10 Configure firewall

Configure the operating system firewall (the port number is added according to the actual situation)

For CentOS6:

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT

service iptables save

For CentOS7:

firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --permanent --zone=public --add-port=8080/tcp
firewall-cmd --reload

3. Nginx hardening

3.1 Security audit

l Enable error log

#错误日志
error_log  logs/error.log;

l Enable access log

#访问日志
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer"'
                      '"$http_user_agent" "$http_x_forwarded_for"'
                      '"$request_time" "$upstream_response_time"';
#日志缓存
 access_log  logs/access.log  main buffer=64k flush=60s;
  open_log_file_cache max=300 inactive=20s valid=1m min_uses=2;

3.2 Hide nginx version

l Add a parameter to hide the nginx version in the nginx.conf configuration file

# hide nginx version
server_tokens off;

l Add # comment to the fastcgi.conf configuration file as follows to hide the version information of nginx in PHP

fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

3.3 Data confidentiality

l Configure anti-theft chain, configure the following parameters in the server corresponding to nginx.conf (configure according to the actual environment)

location ~* ^.+\.(gif|jpg|png|swf|flv|rar|zip)$ {
valid_referers none blocked 域名;
if ($invalid_referer) {
return 403;
break;
}
access_log off;
}

3.4 Configuration error interface

l Put error.html under nginx/html. Configure the following parameters in http in nginx.conf

error_page  404 500 502 503 504 505 /error.html;

3.5 Web front-end security

l Prevent click hijacking, prevent IE content sniffing, prevent xss, only load resources from this domain name (external scripts cannot be executed), configure the following parameters in the nginx.conf server (configure according to the actual environment)

add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection 1;
#add_header Content-Security-Policy "default-src 'self'";

3.6 https security

l Do not use SSL and TLS1.1 or lower, use TLS1.2 or higher, configure the following parameters in the server of nginx.conf (configured in a scenario where https is enabled)

SSL_Protocols TLSv1.2;

3.7 Access control

l Restrict ip access (due to public network access to nginx, it is recommended not to set. Unless there is a malicious ip to try cc*** or brute force cracking and other illegal operations) (configure according to actual environment)

location / {
deny 192.168.1.1; #拒绝IP
allow 192.168.1.0/24; #允许IP
allow 10.1.1.0/16; #允许IP
deny all; #拒绝其他所有IP
}

3.8 Limit http request methods

l Configure the following parameters in the nginx.conf server, and only allow two http request methods, GET and POST

    location / {
        if ($request_method !~* GET|POST) {
        return 403;
        }
}

l nginx disables the option method, add the following statement to nginx.conf file or server module

if ($request_method ~* OPTIONS) {
return 404;
}

4. Nginx optimization

4.1 Number of Nginx worker processes

l Generally set the CPU core or the number of cores x2 (worker_processes can be enabled at most 8)

grep ^processor /proc/cpuinfo | wc -l    //获取cpu核心数
worker_processes  4;

4.2 Nginx running CPU affinity

l  比如2核配置

worker_processes 2;
worker_cpu_affinity 01 10;

l  比如4核配置

worker_processes  4;
worker_cpu_affinity 0001 0010 0100 1000;

l  比如8核配置

worker_processes 8;
worker_cpu_affinity 00000001 00000010 00000100 0000100000010000 00100000 01000000 10000000;

4.3 Optimize the kernel parameters and the number of connections

cat >>/etc/sysctl.conf<<EOF
fs.file-max = 6815744
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_max_tw_buckets = 6000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 40960
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_keepalive_time = 30
net.ipv4.ip_local_port_range = 1024 65000
EOF
sysctl -p
cat >>/etc/security/limits.conf<<EOF
*       soft    nofile      65535
*       hard    nofile      65535
*       soft    noproc      65535
*       hard    noproc      65535
EOF

4.4 Nginx event handling

l Enable epoll model to improve processing efficiency

events {
use epoll;
worker_connections 65535;
multi_accept on;
}

4.5 Enable efficient transmission mode

sendfile  on;
tcp_nopush  on;

4.6 Connection timeout

l Protect server resources, CPU, memory and control the number of connections

keepalive_timeout 60;
tcp_nodelay on;
client_header_buffer_size 4k;
open_file_cache max=102400 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 1;
client_header_timeout 60;
client_body_timeout 60;
reset_timedout_connection on;
send_timeout 20;
client_max_body_size 10m;

4.7 Portable configuration with dedicated path for configuration files

Separate conf files are configured for different services to improve the efficiency of operation and maintenance, and add include parameters to the nginx.conf configuration file

mkdir /usr/local/nginx/conf/conf.d
include /usr/local/nginx/conf/conf.d/*.conf;

4.8 Gzip tuning

Using gzip compression function may save us bandwidth and speed up transmission

gzip on;
gzip_min_length 2k;
gzip_buffers   4 32k;
gzip_http_version 1.1;
gzip_comp_level 6;
gzip_typestext/plain text/css text/javascriptapplication/json application/javascript application/x-javascriptapplication/xml;
gzip_vary on;
gzip_proxied any;

4.9 Expires cache tuning

Caching is mainly used when there are few chances of changing pictures, css, js and other elements, especially pictures, which take up a lot of bandwidth. You can set the picture to be cached locally in the browser 365d, css, js, html can be cached for 10 days .

location ~* \.(ico|jpe?g|gif|png|bmp|swf|flv)$ {
    expires 30d;
    #log_not_found off;
    access_log off;
}

location ~* \.(js|css)$ {
    expires 7d;
    log_not_found off;
    access_log off;
}

5. End