nginx source code analysis - multi-process socket processing

这篇文章主要分析的是linux及windows的socket处理，如何避免惊群及进程间负载均衡的探讨，
这里的惊群主要是指多进程对于新建的连接如何避免同时争用accept现象的处理。

process creation

• linux

  The method of process creation is mainly to create child processes through fork

  // src/os/unix/ngx_process.c
  ngx_pid_t ngx_spawn_process(ngx_cycle_t *cycle, ngx_spawn_proc_pt proc, void *data,
  char *name, ngx_int_t respawn) {
      ...
      pid = fork();
      ...
  }
  

• windows

  The method of process creation is mainly to create child processes through CreateProcess, and to create child processes through non-inheritance (that is, child processes do not share the file handle of the parent process).

  // src/os/win32/ngx_process.c
  ngx_pid_t ngx_execute(ngx_cycle_t *cycle, ngx_exec_ctx_t *ctx) {
      ...
      if (CreateProcess(ctx->path, ctx->args,
                    NULL, NULL, 0, //此变量为0表示句柄不继承
                    CREATE_NO_WINDOW, NULL, NULL, &si, &pi)
          == 0)
      {
          ngx_log_error(NGX_LOG_CRIT, cycle->log, ngx_errno,
                      "CreateProcess(\"%s\") failed", ngx_argv[0]);
          return 0;
      }
      ...
  }
  

The establishment of ListenSocket

• linux

  The main process first listens to the port, and after listening, the new child process fork shares the socket handle of the parent process, so in Linux, the same address will only be monitored once. When reload is executed, the new monitor will be checked, or the old monitor will be removed (but if it is the same port, assuming 127.0.0.1:80, it will not take effect if it is changed to 0.0.0.0:80), and then start a new process , while sending an exit status to the old process, at which point the old process no longer accepts new connections. (starts three linux processes, but only one of them is listening)

• windows

  Since the handle of the parent process is not shared, each child process is a relatively independent entity, and each process monitors independently (using the setting SO_REUSEADDR to achieve the effect of multiple bindings to the same address). However, in the actual measurement on Windows, the use of SO_REUSEADDR to monitor the same address can only successfully call the Accept function in the first process. Only after the first process is closed, the second monitor can successfully accept. (8 processes are started, and each program repeatedly listens to the port) (this shows the situation of the initial operation) (The stress test performed with ab test shows that only one process is serving the outside world, but it is in an idle state)

How to control accept

• linux

  Mainly through shared locks, only the process that gets the lock will try to call the accept event

  // src/event/ngx_event.c
  void
  ngx_process_events_and_timers(ngx_cycle_t *cycle)
  {
      //是否启用共享锁控制，linux默认启动
      if (ngx_use_accept_mutex) {
          //每次accept成功后都会重新赋该值，如果负载高，这值为正
          //从而减少负载高的进程得到锁的概率
          if (ngx_accept_disabled > 0) {
              ngx_accept_disabled--;
          } else {
              //尝试获取共享锁，该函数立即返回不等待
              //如果成功获取该锁，则进行accept事件的投递
              if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) {
                  return;
              }
          }
      }
  }
  

  When the number of connections in a process exceeds 7/8 of the total worker_connections, pressure control begins

  // src/event/ngx_event_accept.c
  void ngx_event_accept(ngx_event_t *ev)
  {
      ...
      //ngx_cycle->connection_n表示当前配置总的work_connections
      //ngx_cycle->free_connection_n表示剩余可接受的连接数
      //当可用连接数越少时，ngx_accept_disabled值越大，也就是获取锁的难度越高
      ngx_accept_disabled = ngx_cycle->connection_n / 8
                          - ngx_cycle->free_connection_n;
      ...
  }
  

• windows

  Each process of windows is independently controlled to accept and receive, there is no lock control, because the pressure of no process is measured on a single process (windows10 test).

Discussion on other feasible solutions

• linux

  By enabling SO_REUSEADDR and SO_REUSEPORT in Linux, the same address can be monitored multiple times in multiple processes, and the system will assign the socket to whom to accept. Advantages: Avoid using locks and unify system allocation Disadvantages: Process load distribution is not as precise as manual control. If there are other programs on the system, you can steal data by listening to the same port. Lower versions of linux do not support this option Example Reference: Linux ReusePort, ReuseAddr

• windows

  Windows passes CreateProcess and sets the child process inheritance, passes the handle value to the child process through the command line, and closes the handle of the main process after starting the process, so that the child processes have their own independent accept permissions. Control who accepts through locks or timing. Example participation: Rust version of windows CreateProcess control (running screenshot, where 584 processes sleep for 10 seconds each time they accept a new socket)