WAF (NGINX) the difference between 502 and 504

0x00 Foreword

WAF Huawei cloud server and use the best reverse proxy bandwidth customers and security services detection, but during use, the individual user's request 502 or 504 appeared. So our team are often subject to user feedback, and everyone here can be broadly clarify the relevant knowledge points.

0x01 definitions

Nginx by reading the source code, notes: I see here is openresty in nginx-1.11.2 source code, we find that the definition of 502 and 504.
130 lines of code have the following ngx_http_request.h

#define NGX_HTTP_INTERNAL_SERVER_ERROR     500
#define NGX_HTTP_NOT_IMPLEMENTED           501
#define NGX_HTTP_BAD_GATEWAY               502
#define NGX_HTTP_SERVICE_UNAVAILABLE       503
#define NGX_HTTP_GATEWAY_TIME_OUT          504
#define NGX_HTTP_INSUFFICIENT_STORAGE      507

Literally face Translation:
1.502 Bad Gateway
2.504 Gateway Timeout

Both had a lot of false solutions, and indeed they say the method can solve these two errors on the network, behind I will summarize solutions. But the source did not say understand why the NGINX will throw this error code, there is, under what circumstances would throw this error code.

0x02 Tracking Code

In order to understand these two circumstances under which the error code is generated to produce, we continue to view the source code:
the definition of a ngx_http_upstream_next in 3935 in line ngx_http_upstream.c

static voidngx_http_upstream_next(ngx_http_request_t *r, ngx_http_upstream_t *u,
    ngx_uint_t ft_type){  //-------------省略代码
  switch (ft_type) {  case NGX_HTTP_UPSTREAM_FT_TIMEOUT:
      status = NGX_HTTP_GATEWAY_TIME_OUT;      break;  case NGX_HTTP_UPSTREAM_FT_HTTP_500:
      status = NGX_HTTP_INTERNAL_SERVER_ERROR;      break;  case NGX_HTTP_UPSTREAM_FT_HTTP_403:
      status = NGX_HTTP_FORBIDDEN;      break;  case NGX_HTTP_UPSTREAM_FT_HTTP_404:
      status = NGX_HTTP_NOT_FOUND;      break;  default:
      status = NGX_HTTP_BAD_GATEWAY;
  }  //-------------省略代码}

Effect of this code is: be set according to status ft_type, it is then determined according to the relevant property passed to the next or upstream end connected ngx_http_upstream_finalize_request.

We can see that in addition to the specified types ft_type outside, such as 504 corresponding NGX_HTTP_UPSTREAM_FT_TIMEOUT, 502 are corresponding NGX_HTTP_BAD_GATEWAY.

Seen from the above code, we only need to track what calls ngx_http_upstream_next the same time, it is determined that the third parameter assignment ft_type can be.

But from the entire code file, we can see virtually all of the functions to call it, such as ngx_http_upstream_connect and so on.
Specific various parts of the calling code as follows:

ngx_http_upstream_connect

Role: connection and sends a request to the upstream, where if the SSL settings, then the call will be SSL connection

rc = ngx_event_connect_peer(&u->peer);
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http upstream connect: %i", rc);if (rc == NGX_ERROR) {
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_INTERNAL_SERVER_ERROR); //500
    return;
}

u->state->peer = u->peer.name;if (rc == NGX_BUSY) {
    ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "no live upstreams");
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_NOLIVE); //502
    return;
}if (rc == NGX_DECLINED) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); //502
    return;
}

ngx_http_upstream_ssl_init_connection

Action: initialize a connection to the upstream of ssl, including SSL handshake. Code calls is relatively small:

if (ngx_http_upstream_test_connect(c) != NGX_OK) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); //502
    return;
}

ngx_http_upstream_ssl_handshake

Action: in the SSL handshake process, the certificate verification operation

failed:
    c = r->connection;
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); //502
    ngx_http_run_posted_requests(c);

ngx_http_upstream_send_request

Action: send a request packet to the upstream

if (!u->request_sent && ngx_http_upstream_test_connect(c) != NGX_OK) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); //502
    return;
}

ngx_http_upstream_send_request_handler

Action: write_event and the upstream connection handler

if (c->write->timedout) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_TIMEOUT); //504
    return;
}

ngx_http_upstream_process_header

Action: read_event and the upstream connection handler

if (c->read->timedout) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_TIMEOUT); //504
    return;
}if (!u->request_sent && ngx_http_upstream_test_connect(c) != NGX_OK) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); //502
    return;
}

Unable to recv from the connection to the data, and return 502 error

n = c->recv(c, u->buffer.last, u->buffer.end - u->buffer.last);
........if (n == 0) {
    ngx_log_error(NGX_LOG_ERR, c->log, 0, "upstream prematurely closed connection");
}if (n == NGX_ERROR || n == 0) {
    ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR); //502
    return;
}

Received content for HTTP header parsing, invalid header structure returned 502, 500 other error return

rc = u->process_header(r);if (rc == NGX_AGAIN) {    if (u->buffer.last == u->buffer.end) {
        ngx_log_error(NGX_LOG_ERR, c->log, 0, "upstream sent too big header");
        ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_INVALID_HEADER); //502
        return;
    }    continue;
}break;
}if (rc == NGX_HTTP_UPSTREAM_INVALID_HEADER) {
  ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_INVALID_HEADER); //502
  return;
}if (rc == NGX_ERROR) {
  ngx_http_upstream_finalize_request(r, u, NGX_HTTP_INTERNAL_SERVER_ERROR); //500
  return;
}

ngx_http_upstream_process_body_in_memory

Effect: the content processing response packet body upstream of
reading the content timeout error code is returned 504, and there is no call next_upstream

c = u->peer.connection;
rev = c->read;
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, c->log, 0, "http upstream process body on memory");if (rev->timedout) {
    ngx_connection_error(c, NGX_ETIMEDOUT, "upstream timed out");
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_GATEWAY_TIME_OUT); //504
    return;
}

ngx_http_upstream_process_upgraded

Role: upstream of the upgrade, the specific call here also need to find out.

if (upstream->read->timedout || upstream->write->timedout) {
    ngx_connection_error(c, NGX_ETIMEDOUT, "upstream timed out");
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_GATEWAY_TIME_OUT); //504
    return;
}

ngx_http_upstream_process_non_buffered_upstream

Action: receiving a non buffered upstream data

if (c->read->timedout) {
    ngx_connection_error(c, NGX_ETIMEDOUT, "upstream timed out");
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_GATEWAY_TIME_OUT); //504
    return;
}

ngx_http_upstream_process_non_buffered_request

Action: The above processing request buffered upstream of the non

if (upstream->read->eof) {
    ngx_log_error(NGX_LOG_ERR, upstream->log, 0, "upstream prematurely closed connection");
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_BAD_GATEWAY); //502
    return;
}if (upstream->read->error) {
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_BAD_GATEWAY); //502
    return;
}

ngx_http_upstream_process_request

Action: upstream processing request.

if (p->upstream_done || p->upstream_eof || p->upstream_error) {
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0, "http upstream exit: %p", p->out);    if (p->upstream_done || (p->upstream_eof && p->length == -1))
    {
        ngx_http_upstream_finalize_request(r, u, 0);        return;
    }    if (p->upstream_eof) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0, "upstream prematurely closed connection");
    }
    ngx_http_upstream_finalize_request(r, u, NGX_HTTP_BAD_GATEWAY); //502
    return;
}

0x03 summary

504

The number 504 appears relatively small, I tried to get it right.
1. The write request data from the upstream timeout
2. read response data from upstream timeout

NGINX error log will appear the following words:
1. upstream the Timed OUT

In asynchronous programming socket inside, as if the socket connected and closed events are triggered in the event read, of course, I have for a long time did not write this piece of code, so these are some of the memories I have previously written event_select of windows programming, there may be some deviation.

I just think connect_timeout should also trigger the 504, of course, be verified here, I had reservations.

1. In write_timeout, then the data is not transmitted upstream to the server, so the rear end of the service is not implemented.

Read_timeout case 2, represents a data transmission to the upstream, but upstream not return any byte within a predetermined period of time. So here back-end business may not perform, TCP server request queue is still there; there may have been in the implementation, but have not yet executed. If the back-end business processing speed faster than the average, then the greater the likelihood that the request still in the queue.

There may be reasons for systematic TCP queue is set too large, while the back-end WebServer the backlog is not enough.

The common solution:

1. If you are proxy_pass, then increase the profile proxy_timeout, proxy_connect_timeout, proxy_buffer_size
2. If fastcgi_pass, then increase the profile fastcgi_buffer_size, fastcgi_connect_timeout, fastcgi_read_timeout, fastcgi_write_timeout

So turn up the parameters will have some effect, but if a large volume of requests has continued to fill the relevant TCP queue, then the entire upstream might avalanche, so the best solution is to optimize the business logic.

502

502 The reason is very large, broadly divided into what I have here:

1. upstream not connected, there is no back-end services such as opening
2. SSL handshake initialized or failure, such as a certificate does
3. When sending a request, and the upstream connection has been broken
4. recv upstream data from the failed or length of 0 or EOF
5. the upstream data is too large or the recv not valid HTTP header

NGINX error log will appear in the following words:

1. no live upstreams
2. upstream prematurely closed connection
3. upstream sent too big header
4. Connection reset by peer

In the case of 2, 3 are connected or send data error occurred during, upstream data is not received, then the back-end business is not carried out, the specific cause of the failure of the server NGINX need to look at it.

In contrast, 4 and 5 are the case when the data received in response, closes the connection or the active upstream data transmission disorder caused, and therefore need to find the reason for the failure in upstream above the rear stage service has already been implemented. Common causes are backend WebServer sets the maximum execution time, but business has not been completed.

The common solution:

1. Increase the back-end service server execution time for a single request, such as in php.ini max_execution_timout and request_terminate_timeout

Of course, this is a solution to the surface, for such a scenario the best solution is to back-end and front-end expansion limiting.

0x04 References

Source: Huawei cloud community Author: HuangJacky