There thundering herd phenomenon to accept socket of the prior Linux2.6 version. After the release has been to get rid of this problem.
Thundering herd means multiple processes / threads while waiting for the same resources, whenever resources are available, all processes / threads have to compete for resources phenomenon.
Nginx uses a multi-process model. Assuming that the Linux system is version 2.6 ago, when a client wants to connect to the Nginx server, Nginx the N process will go listens accept socket is, if all of the N processes have on the client socket connections were listening, competition for resources will cause confusion and even data. We want to ensure that a link on a process Nginx handling, including accept and read / write events.
Nginx and remedy thundering herd and load balancing treatment process
- Nginx process will compete for the N file lock, when only got the file lock process, to process accept the event .
- Did not get the file locked in the process, can only deal with the current connection object read event
- When the total of a single process when the number of connection connection to reach 7/8 of the total number, will no longer receive new accpet event.
- If the lock process can quickly get processed accpet, but did not get the lock has been waiting (waiting time: ngx_accept_mutex_delay), a process likely to cause busy busy, very empty empty
Specific implementation
ngx_process_events_and_timers process event dispatcher
This method is the core function of the process of implementation. Main functions: distribution of events; thundering herd processing; simple load balancing.
Load Balancing:
- When the event configuration initialization time, will set a global variable: ngx_accept_disabled ngx_cycle- => connection_n / 8 - ngx_cycle-> free_connection_n;
2. When is the time ngx_accept_disabled positive number, connection to reach 7/8 of the total connection time, no longer accept new connection event processing, event processing only read the current connection
Shock treatment group:
- By ngx_trylock_accept_mutex competition for file locking, file locks to get before they can accept processing events.
- ngx_accept_mutex_held is a sign to get the lock, when the lock get, flags will be set to NGX_POST_EVENTS, all events this flag (accept and read) into the corresponding ngx_posted_accept_events ngx_posted_events queue and will be cast in the event handler ngx_process_events post-processing.
- When did not get the lock, the event handler is invoked when ngx_process_events, you can clearly read all the events, so you can directly call the event ev-> handler method callback handler.
- Got locked in the process, the next priority will accept event on the queue ngx_posted_accept_events, handler: ngx_event_process_posted
- Processed after accept event, it will release the file locks
6. The next event on the queue processing read ngx_posted_events, handler: ngx_event_process_posted
/ ** *进程事件分发器 * / void ngx_process_events_and_timers (ngx_cycle_t * Cycle) { ngx_uint_t flags; ngx_msec_t timer, Delta; if (ngx_timer_resolution) { timer = NGX_TIMER_INFINITE; flags = 0 ; } Else { timer = ngx_event_find_timer (); flags = NGX_UPDATE_TIME; # If (NGX_WIN32) / * handle signals from master in case of network inactivity * / if (timer == NGX_TIMER_INFINITE || timer> 500 ) { Timer = 500 ; } #endif } / ** * ngx_use_accept_mutex variable represents whether accept mutex * default is to use, by accept_mutex off; Close command; * accept mutex action is to avoid shock group, while load balancing * / iF (ngx_use_accept_mutex) { / ** * = ngx_cycle- ngx_accept_disabled> connection_n /. 8 - ngx_cycle-> free_connection_n; * when the connection reaches 7/8 of the total number of connections, the connection will no longer accept new event processing, only the processing of the current connection the read event * this is a relatively simple method of load balancing * / IF (ngx_accept_disabled> 0 ) { ngx_accept_disabled - ; } the else { / * Get the lock fails * / IF (ngx_trylock_accept_mutex (Cycle) == NGX_ERROR) { return ; } / * get locked * / IF (ngx_accept_mutex_held) { / ** * to increase flag flags NGX_POST_EVENTS, this flag as a function of the processing time of the core ngx_process_events of a parameter, this function will delay the processing of all events. * Accept ngx_posted_accept_events events into a linked list, * epollin | epollout normal event in the list are put ngx_posted_events * * / the flags | = NGX_POST_EVENTS; } the else { / ** * 1. Obtain the lock fails, the current worker process means that neither can make frequent attempts to grab the lock, it can not be too long after the event and then try to steal the lock * 2. Turn the timer_resolution time accuracy, no need to ngx_process_change method in new incident wait at least when ngx_accept_mutex_delay milliseconds after trying to go grab a lock * 3 the absence of on-time accuracy, if a timer timeout recent events than ngx_accept_mutex_delay milliseconds from now, we should also set the timer to ngx_accept_mutex_delay ms * 4. can not let ngx_process_change method when there is no new event waiting longer than ngx_accept_mutex_delay, this affects the entire load balancing mechanism * 5. If the lock process can quickly get processed accpet, but did not get the lock has been waiting for, a process likely to cause busy busy, very empty empty * / IF (Timer == NGX_TIMER_INFINITE || Timer> ngx_accept_mutex_delay) { Timer = ngx_accept_mutex_delay; } } } } Delta = ngx_current_msec; / ** * event scheduler function * get the lock 1. When, flags = NGX_POST_EVENTS, and does not directly handle the event, * accept the event into ngx_posted_accept_events, read into the event queue ngx_posted_events * 2. when did not get the lock, then the process is read all events, the callback function directly handle * parameters: timer-epoll_wait timeout (ngx_accept_mutex_delay- delay lock to take the event NGX_TIMER_INFINITE- normal epollwait wait event) * / ( void ) ngx_process_events (Cycle , Timer, the flags); Delta = ngx_current_msec - Delta; ngx_log_debug1 (NGX_LOG_DEBUG_EVENT, Cycle -> log, 0 , "Timer Delta:% M" , Delta); / ** * 1. ngx_posted_accept_events an event queue from the temporary epoll listen socket to accept the event wait * 2. This loop process is queued in the event accpet accpet event * / ngx_event_process_posted (Cycle, & ngx_posted_accept_events); / ** * If you get the lock, accept processed event, the lock is released * / IF (ngx_accept_mutex_held) { ngx_shmtx_unlock ( & ngx_accept_mutex); } IF (Delta) { ngx_event_expire_timers (); } / ** . *. 1 are located in a common event ngx_posted_events queue . * 2 this method is read event loop processing on the event queue read * / ngx_event_process_posted (cycle, & ngx_posted_events); }
ngx_trylock_accept_mutex acquisition accept lock
- ngx_accept_mutex_held is a lock to get global variables uniquely identified.
- When to get the lock, the calling ngx_enable_accept_events, the new connection will join the event event
- If you do not get the lock, the calling ngx_disable_accept_events.
/ ** * Get lock accept * / ngx_int_t ngx_trylock_accept_mutex (ngx_cycle_t * Cycle) { / * * get locked * / IF (ngx_shmtx_trylock (& ngx_accept_mutex)) { ngx_log_debug0 (NGX_LOG_DEBUG_EVENT, Cycle -> log, 0 , "Locked accept the mutex " ); / * repeatedly come in, to determine whether've got the lock * / iF (ngx_accept_mutex_held && ngx_accept_events == 0 ) { return NGX_OK; } / * call ngx_enable_accept_events, open listening accpet event * / iF (ngx_enable_accept_events (Cycle) == NGX_ERROR) { ngx_shmtx_unlock ( & ngx_accept_mutex); return NGX_ERROR; } ngx_accept_events = 0 ; ngx_accept_mutex_held =. 1 ; return NGX_OK; } ngx_log_debug1 (NGX_LOG_DEBUG_EVENT, Cycle -> log, 0 , "Accept the mutex Lock failed:% UI" , ngx_accept_mutex_held); / ** * did not get the lock, but 1 = ngx_accept_mutex_held * / IF (ngx_accept_mutex_held) { / * do not get the lock, call ngx_disable_accept_events, accpet event will delete * / IF(ngx_disable_accept_events (Cycle, 0) == NGX_ERROR) { return NGX_ERROR; Ngx_accept_mutex_held = 0 ; } Return NGX_OK; }
ngx_enable_accept_events 和ngx_disable_accept_events
/** * 开启accept事件的监听 * 并将accept事件加入到event上 */ static ngx_int_t ngx_enable_accept_events(ngx_cycle_t *cycle) { ngx_uint_t i; ngx_listening_t *ls; ngx_connection_t *c; ls = cycle->listening.elts; for (i = 0; i < cycle->listening.nelts; i++) { c = ls[i].connection; if (c == NULL || c->read->active) { continue; } if (ngx_add_event(c->read, NGX_READ_EVENT, 0) == NGX_ERROR) { return NGX_ERROR; } } return NGX_OK; } / ** * Close accept event listener * accept event and remove it from the Event * / static ngx_int_t ngx_disable_accept_events (ngx_cycle_t * Cycle, ngx_uint_t All) { ngx_uint_t I; ngx_listening_t * LS; ngx_connection_t * C; LS = cycle-> listening.elts; for (I = 0; I <cycle-> listening.nelts; I ++ ) { C = LS [I] .connection; / * if c-> read-> active, then representation is active connections, has been in use * / IF (c == NULL || !c->read->active) { continue; } #if (NGX_HAVE_REUSEPORT) /* * do not disable accept on worker's own sockets * when disabling accept events due to accept mutex */ if (ls[i].reuseport && !all) { continue; } #endif /* 删除事件 */ if (ngx_del_event(c->read, NGX_READ_EVENT, NGX_DISABLE_EVENT) == NGX_ERROR) { return NGX_ERROR; } } Return NGX_OK; }
ngx_event_process_posted event queue processing
To accept / read events on ngx_posted_accept_events or ngx_posted_events callback queue for processing.
/ ** *处理事件队列 * * / void ngx_event_process_posted (ngx_cycle_t * Cycle, ngx_queue_t * posted) { ngx_queue_t * q; ngx_event_t * eV; while (! ngx_queue_empty (posted)) { q = ngx_queue_head (posted); Ev = ngx_queue_data (q, ngx_event_t, queue); ngx_log_debug1 (NGX_LOG_DEBUG_EVENT, Cycle -> log, 0 , "posted event% p" , eV); ngx_delete_posted_event (EV); / * 事件回调函数* / EV ->handler(ev); } }
The core of the event handler ngx_process_events
This way, we mainly see ngx_epoll_process_events method under epoll model (ngx_epoll_module.c)
- If you grab the lock, it will be accpet / read events onto the queue to delay treatment.
2. No grab locked in the process is handled read current events connected so directly for processing.
/ * Read event EPOLLIN * / IF ((& EPOLLIN the revents) && Rev-> Active) { # IF (NGX_HAVE_EPOLLRDHUP) IF (the revents & EPOLLRDHUP) { Rev -> = pending_eof. 1 ; } Rev -> Available =. 1 ; # endif Rev -> READY = 1 ; / * If the process is to grab the lock is placed in the event queue * / IF (flags & NGX_POST_EVENTS) { queue = Rev-> the Accept &? ngx_posted_accept_events : &ngx_posted_events; ngx_post_event (Rev, Queue); } the else { / * not grab the lock, the process directly read event * / Rev -> Handler (Rev); } }
https://mp.weixin.qq.com/s/vkvYJnKfQyuUeD_BDQy_1g
For more learning materials, can be added to the group: 473 984 645 or under the Fanger Wei code scanning
