Linux system IO model summary Nginx compilation installation and basic configuration

Linux system IO model summary Nginx compilation installation and basic configuration

1. Linux system IO model

1. The concept of IO

Everything in the Linux world is a file, sockets, pipes, terminals, etc. are all files. Files in the Linux system can also be understood as a series of binary streams. In the process of information exchange, the operations of sending and receiving these binary streams are I/O operations (input and output), the most common of which are disk IO and network IO.

2. Kernel space and user space

Linux system divides the memory space into two parts: kernel space and user space. Kernel code and data are stored in the kernel space, which can execute privileged instructions and provide external service interfaces, such as accessing hardware. What is stored in the user space is the code and data of the user program, which can only execute non-privileged instructions.

All system resource management is done in the kernel space, such as reading and writing disk files, reading and writing data from the network interface, and so on. The user's application program cannot directly perform such operations, and can only complete such tasks through the interface provided by the kernel.

4. System IO model

Take the user program to read data as an example to understand the system IO model. Since the user does not have the authority to directly access the disk device, it can only be achieved through the kernel. When the kernel reads the data required by the user from the disk, it is restricted by the strict isolation between the kernel space and the user space. It is necessary to copy the memory data in the kernel space to the process memory of the user space. So in simple terms, an I/O is the whole process in which the kernel reads data from the disk to the kernel space, and then copies the memory data in the kernel space to the memory of the process in the user space. System IO has four models: synchronous, asynchronous, blocking, and non-blocking.

• Synchronous Asynchronous

  The focus is on the message communication mechanism of event processing, that is, whether the callee provides completion notification while waiting for the processing result of a thing. Synchronous: After the user process makes a request call, the kernel does not provide a notification mechanism, that is, the user process is not notified after the file IO processing is completed, and the user process needs to ask the kernel whether the processing is completed. Asynchronous: Asynchronous. After the user process makes a request call, the kernel will return the call result to the user process after the call is processed. Nginx is asynchronous.

• Blocking/non-blocking

  Pay attention to the state of the caller before waiting for the result to return. Blocking: Blocking means that the IO operation needs to be completely completed before returning to the user space. Before the call result is returned, the caller is suspended and can't do other things. Non-blocking: nonblocking means that the IO operation is called immediately to return a status value to the user. There is no need to wait until the IO operation is completely completed. Before the final call result is returned, the caller will not be suspended and can do other things.

Take me to eat as an example: I ordered 10 buns

• Synchronous and asynchronous:

  After I ordered the buns, did the chef tell me:

  • Synchronization: The chef will put the buns in the designated position after making the buns, but before making the buns, you need to go to see if the buns are done. The chef will not make the buns.

  Notify me when I'm done.

  • Asynchronous: After the chef makes the buns, he tells me where to put the buns.

• Blocking and non-blocking::

  The state after I ordered the buns:

  • Blockage: The chef has been waiting in front of the bun plate while he is making buns and can't do other things.

  • Non-blocking: After ordering the buns, you can do other things, such as going shopping or buying.

• IO model combination:

  • Synchronous blocking: I can’t do anything else after ordering the buns, and I don’t know if the buns are done. I need to wait and ask the chef again and again.

  No.

  • Synchronous non-blocking: After ordering the buns, you can do other things, but you can’t do other things for a long time, because I still don’t know if the buns are done well, so I have to

  I have been waiting and asking the chef again and again if he is done well, so I can only make time to do something else.

  • Asynchronous blocking: I can’t go and do other things after I order the buns, but the chef will tell me when the buns are made, that is, I don’t need to cook the buns again and again

  Is it done?

  • Asynchronous non-blocking: I can do other things after I order the buns, and I can always do other things, because the chef will tell me after the buns are made.

5. Five network IO models in Unix system and three working modes of Apache

There are five network IO models in the UNIX system:

• Synchronous blocking IO model (blocking IO): The blocking IO model is the simplest IO model. User threads are blocked when the kernel performs IO operations.
• Synchronous non-blocking I/O model (nonblocking IO): The application process has been waiting for the kernel to respond after sending an IO request to the kernel. If the kernel processing the requested IO operation cannot return the IO result immediately, the process will no longer wait and continue to process other Request, but it still needs the process to check whether the kernel IO is completed after a period of time.
• IO multiplexing: The system kernel buffers I/O data and allows a single process to monitor multiple file descriptors. Once a descriptor is ready, it can notify the program to perform corresponding read and write operations.
  • In the Linux operating system, everything is abstracted into files, so how do applications correspond to the various files in the system? File descriptor (File descriptor, referred to as fd) came into being. When an application requests the kernel to open/create a file, the kernel will return a file descriptor corresponding to the opened/created file.
  • select, poll, and epoll are the functional realization of IO multiplexing model in Linux system. Select, poll, and epoll are essentially synchronous I/O, because they all need to be responsible for reading and writing after the read and write event is ready, and the reading and writing process is blocked.
  • The Apache prefork mode is the main process + multi-process/single thread + select mode; the Apache work mode is the main process + multi-process/multi-thread + poll mode.
• Signal-driven IO (signal-driven IO): The user process can register a signal handler through the sigaction system call, and then the process can continue to execute downward. When an IO operation is ready, the kernel will notify and trigger a SIGIO signal handler to execute , And then copy the data needed by the user process from the kernel space to the user space.
  • The Apache event mode is the main process + multi-process/multi-thread + signal-driven mode.
• Asynchronous (non-blocking) IO (asynchronous IO): After the user process makes the aio_read system call, no matter whether the kernel data is ready, it will directly return to the user process, and then the user mode process can do other things until the socket data is ready , The kernel directly copies the data to the process, and then sends a notification from the kernel to the process. In the two phases of asynchronous non-blocking IO, the process is non-blocking.

Two, Nginx basics

1. Ubuntu 18.04.5 system optimization

# vim /etc/security/limits.conf   #在文件后补追加
#root账户的资源软限制和硬限制
root soft core unlimited
root hard core unlimited
root soft nproc 1000000
root hard nproc 1000000
root soft nofile 1000000
root hard nofile 1000000
root soft memlock 32000
root hard memlock 32000
root soft msgqueue 8192000
root hard msgqueue 8192000
#其他账户的资源软限制和硬限制
* soft core unlimited
* hard core unlimited
* soft nproc 1000000
* hard nproc 1000000
* soft nofile 1000000
* hard nofile 1000000
* soft memlock 32000
* hard memlock 32000
* soft msgqueue 8192000

# vim /etc/sysctl.conf      #在文件后补追加
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536

# # Controls the maximum size of a message, in bytes
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# # Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# TCP kernel paramater
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1

# socket buffer
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 20480
net.core.optmem_max = 81920

# TCP conn net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15

# tcp conn reuse
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_max_tw_buckets = 20000
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syncookies = 1

# keepalive conn
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.ip_local_port_range = 10001 65000

# swap
vm.overcommit_memory = 0
vm.swappiness = 10

#net.ipv4.conf.eth1.rp_filter = 0
#net.ipv4.conf.lo.arp_ignore = 1
#net.ipv4.conf.lo.arp_announce = 2
#net.ipv4.conf.all.arp_ignore = 1
#net.ipv4.conf.all.arp_announce = 2

# reboot

2. Compile and install Nginx

2.1 Operating system software version information

​ 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

2.2 Obtain nginx source code

​ Official website: http://nginx.org/ , there is a download entry on the right side of the page.

2.3 Compile and install environment preparation

• Create nginx account

# groupadd -g 2020 nginx
# useradd -u 2020 -g 2020 -r -s /bin/bash nginx

• Nginx directory permission settings

# mkdir -p /app/nginx
# chown -R nginx:nginx /app

• Edit and define installation parameters

# mkdir  /app
# tar xvf nginx-1.18.0.tar.gz
# cd nginx-1.18.0
# ./configure --help

  --help                             print this message

  --prefix=PATH                      set installation prefix
  --sbin-path=PATH                   set nginx binary pathname
  --modules-path=PATH                set modules path
  --conf-path=PATH                   set nginx.conf pathname
  --error-log-path=PATH              set error log pathname
  --pid-path=PATH                    set nginx.pid pathname
  --lock-path=PATH                   set nginx.lock pathname

  --user=USER                        set non-privileged user for
                                     worker processes
  --group=GROUP                      set non-privileged group for
                                     worker processes

  --build=NAME                       set build name
  --builddir=DIR                     set build directory

  --with-select_module               enable select module
  --without-select_module            disable select module
  --with-poll_module                 enable poll module
  --without-poll_module              disable poll module

  --with-threads                     enable thread pool support

  --with-file-aio                    enable file AIO support

  --with-http_ssl_module             enable ngx_http_ssl_module
  --with-http_v2_module              enable ngx_http_v2_module
  --with-http_realip_module          enable ngx_http_realip_module
  --with-http_addition_module        enable ngx_http_addition_module
  --with-http_xslt_module            enable ngx_http_xslt_module
  --with-http_xslt_module=dynamic    enable dynamic ngx_http_xslt_module
  --with-http_image_filter_module    enable ngx_http_image_filter_module
  --with-http_image_filter_module=dynamic
                                     enable dynamic ngx_http_image_filter_module
  --with-http_geoip_module           enable ngx_http_geoip_module
  --with-http_geoip_module=dynamic   enable dynamic ngx_http_geoip_module
  --with-http_sub_module             enable ngx_http_sub_module
  --with-http_dav_module             enable ngx_http_dav_module
  --with-http_flv_module             enable ngx_http_flv_module
  --with-http_mp4_module             enable ngx_http_mp4_module
  --with-http_gunzip_module          enable ngx_http_gunzip_module
  --with-http_gzip_static_module     enable ngx_http_gzip_static_module
  --with-http_auth_request_module    enable ngx_http_auth_request_module
  --with-http_random_index_module    enable ngx_http_random_index_module
  --with-http_secure_link_module     enable ngx_http_secure_link_module
  --with-http_degradation_module     enable ngx_http_degradation_module
  --with-http_slice_module           enable ngx_http_slice_module
  --with-http_stub_status_module     enable ngx_http_stub_status_module

  --without-http_charset_module      disable ngx_http_charset_module
  --without-http_gzip_module         disable ngx_http_gzip_module
  --without-http_ssi_module          disable ngx_http_ssi_module
  --without-http_userid_module       disable ngx_http_userid_module
  --without-http_access_module       disable ngx_http_access_module
  --without-http_auth_basic_module   disable ngx_http_auth_basic_module
  --without-http_mirror_module       disable ngx_http_mirror_module
  --without-http_autoindex_module    disable ngx_http_autoindex_module
  --without-http_geo_module          disable ngx_http_geo_module
  --without-http_map_module          disable ngx_http_map_module
  --without-http_split_clients_module disable ngx_http_split_clients_module
  --without-http_referer_module      disable ngx_http_referer_module
  --without-http_rewrite_module      disable ngx_http_rewrite_module
  --without-http_proxy_module        disable ngx_http_proxy_module
  --without-http_fastcgi_module      disable ngx_http_fastcgi_module
  --without-http_uwsgi_module        disable ngx_http_uwsgi_module
  --without-http_scgi_module         disable ngx_http_scgi_module
  --without-http_grpc_module         disable ngx_http_grpc_module
  --without-http_memcached_module    disable ngx_http_memcached_module
  --without-http_limit_conn_module   disable ngx_http_limit_conn_module
  --without-http_limit_req_module    disable ngx_http_limit_req_module
  --without-http_empty_gif_module    disable ngx_http_empty_gif_module
  --without-http_browser_module      disable ngx_http_browser_module
  --without-http_upstream_hash_module
                                     disable ngx_http_upstream_hash_module
  --without-http_upstream_ip_hash_module
                                     disable ngx_http_upstream_ip_hash_module
  --without-http_upstream_least_conn_module
                                     disable ngx_http_upstream_least_conn_module
  --without-http_upstream_random_module
                                     disable ngx_http_upstream_random_module
  --without-http_upstream_keepalive_module
                                     disable ngx_http_upstream_keepalive_module
  --without-http_upstream_zone_module
                                     disable ngx_http_upstream_zone_module

  --with-http_perl_module            enable ngx_http_perl_module
  --with-http_perl_module=dynamic    enable dynamic ngx_http_perl_module
  --with-perl_modules_path=PATH      set Perl modules path
  --with-perl=PATH                   set perl binary pathname

  --http-log-path=PATH               set http access log pathname
  --http-client-body-temp-path=PATH  set path to store
                                     http client request body temporary files
  --http-proxy-temp-path=PATH        set path to store
                                     http proxy temporary files
  --http-fastcgi-temp-path=PATH      set path to store
                                     http fastcgi temporary files
  --http-uwsgi-temp-path=PATH        set path to store
                                     http uwsgi temporary files
  --http-scgi-temp-path=PATH         set path to store
                                     http scgi temporary files

  --without-http                     disable HTTP server
  --without-http-cache               disable HTTP cache

  --with-mail                        enable POP3/IMAP4/SMTP proxy module
  --with-mail=dynamic                enable dynamic POP3/IMAP4/SMTP proxy module
  --with-mail_ssl_module             enable ngx_mail_ssl_module
  --without-mail_pop3_module         disable ngx_mail_pop3_module
  --without-mail_imap_module         disable ngx_mail_imap_module
  --without-mail_smtp_module         disable ngx_mail_smtp_module

  --with-stream                      enable TCP/UDP proxy module
  --with-stream=dynamic              enable dynamic TCP/UDP proxy module
  --with-stream_ssl_module           enable ngx_stream_ssl_module
  --with-stream_realip_module        enable ngx_stream_realip_module
  --with-stream_geoip_module         enable ngx_stream_geoip_module
  --with-stream_geoip_module=dynamic enable dynamic ngx_stream_geoip_module
  --with-stream_ssl_preread_module   enable ngx_stream_ssl_preread_module
  --without-stream_limit_conn_module disable ngx_stream_limit_conn_module
  --without-stream_access_module     disable ngx_stream_access_module
  --without-stream_geo_module        disable ngx_stream_geo_module
  --without-stream_map_module        disable ngx_stream_map_module
  --without-stream_split_clients_module
                                     disable ngx_stream_split_clients_module
  --without-stream_return_module     disable ngx_stream_return_module
  --without-stream_upstream_hash_module
                                     disable ngx_stream_upstream_hash_module
  --without-stream_upstream_least_conn_module
                                     disable ngx_stream_upstream_least_conn_module
  --without-stream_upstream_random_module
                                     disable ngx_stream_upstream_random_module
  --without-stream_upstream_zone_module
                                     disable ngx_stream_upstream_zone_module

  --with-google_perftools_module     enable ngx_google_perftools_module
  --with-cpp_test_module             enable ngx_cpp_test_module

  --add-module=PATH                  enable external module
  --add-dynamic-module=PATH          enable dynamic external module

  --with-compat                      dynamic modules compatibility

  --with-cc=PATH                     set C compiler pathname
  --with-cpp=PATH                    set C preprocessor pathname
  --with-cc-opt=OPTIONS              set additional C compiler options
  --with-ld-opt=OPTIONS              set additional linker options
  --with-cpu-opt=CPU                 build for the specified CPU, valid values:
                                     pentium, pentiumpro, pentium3, pentium4,
                                     athlon, opteron, sparc32, sparc64, ppc64

  --without-pcre                     disable PCRE library usage
  --with-pcre                        force PCRE library usage
  --with-pcre=DIR                    set path to PCRE library sources
  --with-pcre-opt=OPTIONS            set additional build options for PCRE
  --with-pcre-jit                    build PCRE with JIT compilation support

  --with-zlib=DIR                    set path to zlib library sources
  --with-zlib-opt=OPTIONS            set additional build options for zlib
  --with-zlib-asm=CPU                use zlib assembler sources optimized
                                     for the specified CPU, valid values:
                                     pentium, pentiumpro

  --with-libatomic                   force libatomic_ops library usage
  --with-libatomic=DIR               set path to libatomic_ops library sources

  --with-openssl=DIR                 set path to OpenSSL library sources
  --with-openssl-opt=OPTIONS         set additional build options for OpenSSL

  --with-debug                       enable debug logging

You can check the function description of the nginx module through the official website document

• Compile and install software environment preparation

  • Upgrade apt-get # apt update
  • Install gcc # apt install -y gcc
  • Install C++ dependency library # apt install build-essential # apt install libtool
  • Install pcre dependency library # apt install libpcre3 libpcre3-dev
  • Install zlib dependency library # apt install zlib1g-dev
  • Install ssl dependency library #apt install libssl-dev

• Determine custom installation parameters

  On the official website http://nginx.org/en/docs/configure.html, check the meaning of the custom parameters, and the final installation option parameters:

  $ sudo ./configure --prefix=/app/nginx --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream --with-stream_ssl_module --with-stream_realip_module

2.4 Start to compile and install

• configure

  #  ./configure --prefix=/app/nginx --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream --with-stream_ssl_module --with-stream_realip_module
  

• make

  # make

• make install

  # make install

• Start nginx after compiling and installing

  # /app/nginx/sbin/nginx

• Access the compiled and installed nginx web interface:

  

3. Common configuration and optimization of Nginx

3.1 Add nginx path to environment parameters

root@dl-homework:~# pwd
/root
root@dl-homework:~# ll
total 64
drwx------  5 root root  4096 Nov 19 10:03 ./
drwxr-xr-x 24 root root  4096 Nov 18 21:05 ../
-rw-------  1 root root 10257 Nov 19 09:46 .bash_history
-rw-r--r--  1 root root  3140 Nov 19 09:44 .bashrc
drwx------  2 root root  4096 Nov 14 11:28 .cache/
drwx------  3 root root  4096 Nov 14 11:28 .gnupg/
-rw-r--r--  1 root root   148 Aug 17  2015 .profile
drwxr-xr-x  2 root root  4096 Nov 14 11:55 .vim/
-rw-------  1 root root 12836 Nov 19 09:45 .viminfo
-rw-r--r--  1 root root    17 Nov 19 09:45 .vimrc
-rw-------  1 root root   134 Nov 19 10:03 .Xauthority
root@dl-homework:~# vim .bashrc

100 export PATH=$PATH:/app/nginx/sbin   #在文件最后一行添加

# . .bashrc

3.2 Use startup scripts to realize self-starting nginx service

# cd /lib/systemd/system
# vim nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking   
PIDFile=/run/nginx.pid   #要与nginx.conf里面的配置保持一致
ExecStart=/app/nginx/sbin/nginx -c /app/nginx/conf/nginx.conf  #更改为nginx的实际路径
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

# systemctl daemon-reload

# systemctl start nginx.service
## systemctl enable nginx.service
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.

3.3 Nginx basic configuration

All parameters can be found in the documentation on the nginx.com website. The default parameters of nginx.conf:

# cd /app/nginx/conf
# grep -v "#"  nginx.conf |grep -v "^$"
worker_processes  1;
pid        /run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

# vim /app/nginx/conf/nginx.conf

 worker_processes  auto;       #进程数量
 worker_cpu_affinity  auto;    #进程与CPU绑定

 pid        /run/nginx.pid;    #文件位置要与nginx.service里的配置保持一致

 worker_rlimit_nofile   65535;   #允许一个工作进程打开的文件数

 events {
      worker_connections  102400;  #单个进程最大并发连接数
      accept_mutex on;   #避免群惊
      multi_accept on;   #允许接受多个新连接
  }

  http {
    include       mime.types;   #支持的文件类型
    default_type  application/octet-stream;  #无法识别就下载文件

     sendfile        on;  #实现文件零拷贝MMAP
     tcp_nopush      on;  #合并请求后统一发送给客户端,降低服务器端负载，配合sendfile使用
     gzip  on;  #开启文件压缩
     keepalive_timeout  65;  #长连接时间65秒
     server {
        listen       80;
        server_name  localhost;
        charset utf-8;   #更改中文字符集
        location / {
            root   html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

  }

# nginx -t
nginx: the configuration file /app/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /app/nginx/conf/nginx.conf test is successful
# systemctl reload nginx.service

3.4 Create PC Site and Mobile Site

# mkdir /app/nginx/conf/conf.d
# vim /app/nginx/conf/conf.d/pc.conf
server {
    listen 80;
    server_name pc.home.net;

    location / {
        root /app/nginx/html/pc;
        }
    }

# vim /app/nginx/conf/conf.d/mobile.conf 
server {
    listen 80;
    server_name mobile.home.net;

    location / {
        root /app/nginx/html/mobile;
        }
    }

# mkdir /app/nginx/html/{pc,mobile}
# echo "mobile web" > /app/nginx/html/mobile/index.html
# echo "pc web" > /app/nginx/html/pc/index.html

# nginx -t
nginx: the configuration file /app/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /app/nginx/conf/nginx.conf test is successful

# systemctl reload nginx.service

Modify the C:\Windows\System32\drivers\etc\hosts file under the windows system, add 172.20.200.138 pc.home.net mobile.home.net, visit the site: