Detailed proxy_pass, upstream and resolver

Reprinted from   Detailed proxy_pass, upstream and resolver

Scenarios

Here are a few scenarios, the following scenarios will be combined for several code analysis.

（1）proxy_pass + upstream

    upstream foo.example.com {
        server 127.0.0.1:8001;
    }

    server {
        listen       80;
        server_name  localhost;

        location /foo {
            proxy_pass http://foo.example.com;
        }
    }

Access HTTP: // localhost / foo , Proxy module forwards the request to the port 8001 of 127.0.0.1.

(2) only proxy_pass, no upstream and resolver

    server {
        listen       80;
        server_name  localhost;

        location /foo {
            proxy_pass http://foo.example.com;
        }
    }

In fact implicitly created upstream, upstream name is foo.example.com . upstream module uses the DNS server provided the machine (or / etc / hosts), the foo.example.com resolves IP, access http: // localhost / foo forwarding, proxy IP module passes the request on to the parsed.

If the unit is not set DNS server, or DNS server can not resolve domain names, will report similar to the following error when you start nginx:

nginx: [emerg] host not found in upstream "foo.example.com" in /path/nginx/conf/nginx.conf:110

(3) proxy_pass + resolver (variable set domain)

    server {
        listen       80;
        server_name  localhost;

        resolver 114.114.114.114;
        location /foo {
            set $foo foo.example.com;
            proxy_pass http://$foo;
        }
    }

Access HTTP: // localhost / foo , the DNS server (DNS server machine settings or / etc / hosts invalid) by using the dynamic set nginx resolver will, resolve a domain name into the IP, forwarded to parse the IP proxy module will request on.

(4) proxy_pass + upstream (explicit) + resolver (variable set domain)

    upstream foo.example.com {
        server 127.0.0.1:8001;
    }

    server {
        listen       80;
        server_name  localhost;

        resolver 114.114.114.114;
        location /foo {
            set $foo foo.example.com;
            proxy_pass http://$foo;
        }
    }

Visit http: // localhost / foo time, upstream module will give priority to find whether there is defined upstream back-end server, if the definition of the direct use, no longer walk DNS resolution. Therefore proxy module forwards the request to the port 8001 of 127.0.0.1.

(5) proxy_pass + upstream (implicit) + resolver (variable set domain)

    server {
        listen       80;
        server_name  localhost;

        resolver 114.114.114.114;
        location /foo {
            set $foo foo.example.com;
            proxy_pass http://$foo;
        }

        location /foo2 {
            proxy_pass http://foo.example.com;
        }
    }

location / foo2 actually upstream implicitly defined foo.example.com , carried out by the local DNS server DNS, visit http: // localhost / foo time, upstream module will give priority to look upstream, that is implicitly defined foo .example.com , Proxy module forwards the request to the IP parsed.

(6) proxy_pass + resolver (variable not set the domain name)

    server {
        listen       80;
        server_name  localhost;

        resolver 114.114.114.114;
        location /foo {
            proxy_pass http://foo.example.com;
        }
    }

Not using a variable set domain name, the resolver is set not active and corresponds to the scene 2, scene proxy_pass only.

(7) proxy_pass + upstream + resolver (variable not set the domain name)

    upstream foo.example.com {
        server 127.0.0.1:8001;
    }

    server {
        listen       80;
        server_name  localhost;

        resolver 114.114.114.114;
        location /foo {
            proxy_pass http://foo.example.com;
        }
    }

Not using a variable set domain name, the resolver is set not active and equivalent to the scene 1 proxy_pass + upstream.

(8) proxy_pass directly specifying IP plus port number

    server {
        listen       80;
        server_name  localhost;

        location /foo {
            proxy_pass http://127.0.0.1:8001/;
        }
    }

In fact implicitly created upstream, the request will be forwarded to the 8001 proxy_pass port of 127.0.0.1.

The main code

Analytical proxy_pass instruction code:

static char *
ngx_http_proxy_pass(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_proxy_loc_conf_t *plcf = conf;

    size_t                      add;
    u_short                     port;
    ngx_str_t                  *value, *url;
    ngx_url_t                   u;
    ngx_uint_t                  n;
    ngx_http_core_loc_conf_t   *clcf;
    ngx_http_script_compile_t   sc;

    if (plcf->upstream.upstream || plcf->proxy_lengths) {
        return "is duplicate";
    }

    clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);

    clcf->handler = ngx_http_proxy_handler;

    if (clcf->name.data[clcf->name.len - 1] == '/') {
        clcf->auto_redirect = 1;
    }

    value = cf->args->elts;

    url = &value[1];

    /* 查找指令中$符号的位置，判断是否使用了变量 */
    n = ngx_http_script_variables_count(url);

    if (n) {
        /* 使用变量设置域名 */
        ngx_memzero(&sc, sizeof(ngx_http_script_compile_t));

        sc.cf = cf;
        sc.source = url;
        sc.lengths = &plcf->proxy_lengths;
        sc.values = &plcf->proxy_values;
        sc.variables = n;
        sc.complete_lengths = 1;
        sc.complete_values = 1;

        if (ngx_http_script_compile(&sc) != NGX_OK) {
            return NGX_CONF_ERROR;
        }

#if (NGX_HTTP_SSL)
        plcf->ssl = 1;
#endif

        return NGX_CONF_OK;
    }

    if (ngx_strncasecmp(url->data, (u_char *) "http://", 7) == 0) {
        add = 7;
        port = 80;

    } else if (ngx_strncasecmp(url->data, (u_char *) "https://", 8) == 0) {

#if (NGX_HTTP_SSL)
        plcf->ssl = 1;

        add = 8;
        port = 443;
#else
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                           "https protocol requires SSL support");
        return NGX_CONF_ERROR;
#endif

    } else {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "invalid URL prefix");
        return NGX_CONF_ERROR;
    }

    ngx_memzero(&u, sizeof(ngx_url_t));

    u.url.len = url->len - add;
    u.url.data = url->data + add;
    u.default_port = port;
    u.uri_part = 1;
    u.no_resolve = 1;

    plcf->upstream.upstream = ngx_http_upstream_add(cf, &u, 0);
    if (plcf->upstream.upstream == NULL) {
        return NGX_CONF_ERROR;
    }

    plcf->vars.schema.len = add;
    plcf->vars.schema.data = url->data;
    plcf->vars.key_start = plcf->vars.schema;

    ngx_http_proxy_set_vars(&u, &plcf->vars);

    plcf->location = clcf->name;

    if (clcf->named
#if (NGX_PCRE)
        || clcf->regex
#endif
        || clcf->noname)
    {
        if (plcf->vars.uri.len) {
            ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                               "\"proxy_pass\" cannot have URI part in "
                               "location given by regular expression, "
                               "or inside named location, "
                               "or inside \"if\" statement, "
                               "or inside \"limit_except\" block");
            return NGX_CONF_ERROR;
        }

        plcf->location.len = 0;
    }

    plcf->url = *url;

    return NGX_CONF_OK;
}

upstream processing logic backend server definition, including explicitly defined and implicitly defined. Implicitly defined, i.e. proxy_pass specified address back-end server is not explicitly defined by upstream, defines an internal nginx.

ngx_int_t
ngx_http_upstream_init_round_robin(ngx_conf_t *cf,
    ngx_http_upstream_srv_conf_t *us)
{
    ngx_url_t                      u;
    ngx_uint_t                     i, j, n, w;
    ngx_http_upstream_server_t    *server;
    ngx_http_upstream_rr_peer_t   *peer, **peerp;
    ngx_http_upstream_rr_peers_t  *peers, *backup;

    us->peer.init = ngx_http_upstream_init_round_robin_peer;

    /* 
     * 使用upstream指令显式定义upstream 
     * 或者proxy_pass直接指定IP的场景
     */
    if (us->servers) {
        server = us->servers->elts;

        n = 0;
        w = 0;

        for (i = 0; i < us->servers->nelts; i++) {
            if (server[i].backup) {
                continue;
            }

            n += server[i].naddrs;
            w += server[i].naddrs * server[i].weight;
        }

        if (n == 0) {
            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                          "no servers in upstream \"%V\" in %s:%ui",
                          &us->host, us->file_name, us->line);
            return NGX_ERROR;
        }

        peers = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peers_t));
        if (peers == NULL) {
            return NGX_ERROR;
        }

        peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) * n);
        if (peer == NULL) {
            return NGX_ERROR;
        }

        peers->single = (n == 1);
        peers->number = n;
        peers->weighted = (w != n);
        peers->total_weight = w;
        peers->name = &us->host;

        n = 0;
        peerp = &peers->peer;

        for (i = 0; i < us->servers->nelts; i++) {
            /* 设置sockaddr、weight、max_fails、fail_timeout等属性 */
        }

        us->peer.data = peers;

        /* 处理backup servers相关逻辑 */

        return NGX_OK;
    }

    /* 未使用upstream指令，使用proxy_pass隐式定义upstream */
    /* an upstream implicitly defined by proxy_pass, etc. */

    if (us->port == 0) {
        ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                      "no port in upstream \"%V\" in %s:%ui",
                      &us->host, us->file_name, us->line);
        return NGX_ERROR;
    }

    ngx_memzero(&u, sizeof(ngx_url_t));

    u.host = us->host;
    u.port = us->port;

    if (ngx_inet_resolve_host(cf->pool, &u) != NGX_OK) {
        if (u.err) {
            ngx_log_error(NGX_LOG_EMERG, cf->log, 0,
                          "%s in upstream \"%V\" in %s:%ui",
                          u.err, &us->host, us->file_name, us->line);
        }

        return NGX_ERROR;
    }

    n = u.naddrs;

    peers = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peers_t));
    if (peers == NULL) {
        return NGX_ERROR;
    }

    peer = ngx_pcalloc(cf->pool, sizeof(ngx_http_upstream_rr_peer_t) * n);
    if (peer == NULL) {
        return NGX_ERROR;
    }

    peers->single = (n == 1);
    peers->number = n;
    peers->weighted = 0;
    peers->total_weight = n;
    peers->name = &us->host;

    peerp = &peers->peer;

    for (i = 0; i < u.naddrs; i++) {
        /* 设置sockaddr、weight、max_fails、fail_timeout等属性 */
    }

    us->peer.data = peers;

    /* implicitly defined upstream has no backup servers */

    return NGX_OK;
}

The upstream logic module initialization request:

static void
ngx_http_upstream_init_request(ngx_http_request_t *r)
{
    ngx_str_t                      *host;
    ngx_uint_t                      i;
    ngx_resolver_ctx_t             *ctx, temp;
    ngx_http_cleanup_t             *cln;
    ngx_http_upstream_t            *u;
    ngx_http_core_loc_conf_t       *clcf;
    ngx_http_upstream_srv_conf_t   *uscf, **uscfp;
    ngx_http_upstream_main_conf_t  *umcf;

    if (r->aio) {
        return;
    }

    u = r->upstream;

    /* NGX_HTTP_CACHE 等其他处理 */

    cln->handler = ngx_http_upstream_cleanup;
    cln->data = r;
    u->cleanup = &cln->handler;

    if (u->resolved == NULL) {
        /* 如果没有使用resolver设置DNS，直接取upstream的设置 */
        uscf = u->conf->upstream;

    } else {

#if (NGX_HTTP_SSL)
        u->ssl_name = u->resolved->host;
#endif

        host = &u->resolved->host;

        if (u->resolved->sockaddr) {

            if (u->resolved->port == 0
                && u->resolved->sockaddr->sa_family != AF_UNIX)
            {
                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                              "no port in upstream \"%V\"", host);
                ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
                return;
            }

            if (ngx_http_upstream_create_round_robin_peer(r, u->resolved)
                != NGX_OK)
            {
                ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
                return;
            }

            ngx_http_upstream_connect(r, u);

            return;
        }

        umcf = ngx_http_get_module_main_conf(r, ngx_http_upstream_module);

        uscfp = umcf->upstreams.elts;

        /* 在显式/隐式定义的upstream中查找 */
        for (i = 0; i < umcf->upstreams.nelts; i++) {

            uscf = uscfp[i];

            if (uscf->host.len == host->len
                && ((uscf->port == 0 && u->resolved->no_port)
                     || uscf->port == u->resolved->port)
                && ngx_strncasecmp(uscf->host.data, host->data, host->len) == 0)
            {
                goto found;
            }
        }

        if (u->resolved->port == 0) {
            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                          "no port in upstream \"%V\"", host);
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }

        temp.name = *host;

        ctx = ngx_resolve_start(clcf->resolver, &temp);
        if (ctx == NULL) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }

        if (ctx == NGX_NO_RESOLVER) {
            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                          "no resolver defined to resolve %V", host);

            ngx_http_upstream_finalize_request(r, u, NGX_HTTP_BAD_GATEWAY);
            return;
        }

        ctx->name = *host;
        ctx->handler = ngx_http_upstream_resolve_handler;
        ctx->data = r;
        ctx->timeout = clcf->resolver_timeout;

        u->resolved->ctx = ctx;

        if (ngx_resolve_name(ctx) != NGX_OK) {
            u->resolved->ctx = NULL;
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }

        return;
    }

found:

    if (uscf == NULL) {
        ngx_log_error(NGX_LOG_ALERT, r->connection->log, 0,
                      "no upstream configuration");
        ngx_http_upstream_finalize_request(r, u,
                                           NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
    }

#if (NGX_HTTP_SSL)
    u->ssl_name = uscf->host;
#endif

    if (uscf->peer.init(r, uscf) != NGX_OK) {
        ngx_http_upstream_finalize_request(r, u,
                                           NGX_HTTP_INTERNAL_SERVER_ERROR);
        return;
    }

    u->peer.start_time = ngx_current_msec;

    if (u->conf->next_upstream_tries
        && u->peer.tries > u->conf->next_upstream_tries)
    {
        u->peer.tries = u->conf->next_upstream_tries;
    }

    ngx_http_upstream_connect(r, u);
}

Detailed analysis

scene 1

Ngx_http_proxy_pass analytic function in proxy_pass did not find the $ symbol (ie, the variable is set domain name), take the second half of the processing logic ngx_http_proxy_pass. When ngx_http_upstream_init_round_robin upstream initialization, go upstream to explicitly define the logic. When the transfer request proxy_pass initialization, ngx_http_upstream_init_request establish a connection directly upstream backend server.

Scene 2

When ngx_http_upstream_init_round_robin initialization upstream, take the logic implicitly defined upstream of calls ngx_inet_resolve_host of proxy_pass the domain name resolution, set upstream. When the transfer request proxy_pass initialization, ngx_http_upstream_init_request used directly disposed upstream of, i.e. the use of a local DNS server set in the IP parsed, to establish a connection.

Scenario 3

Proxy_pass parsing command, found the $ symbol set ngx_http_script_compile_t, and use ngx_http_script_compile compile, do not take the latter part of logic. Profile does not explicitly / implicitly defined upstream, it does not call ngx_http_upstream_init_round_robin method. When proxy_pass forward the request to initialize, ngx_http_upstream_init_request found not explicitly nor implicitly defined upstream, subsequent calls ngx_resolve_start, parsing the domain name, and then forwards the request in the past.

Scene 4

Proxy_pass parsing command, found the $ symbol set ngx_http_script_compile_t, and use ngx_http_script_compile compile, do not take the latter part of logic. The upstream explicit call, so the call processing logic ngx_http_upstream_init_round_robin method of explicit upstream. When proxy_pass forward the request to initialize, ngx_http_upstream_init_request the first search upstream, if found, the request will be forwarded directly to the upstream of the back-end server. If the upstream is not found, then the domain name resolution, and then forwards the request to the IP parsed.

Scene 5

4 is substantially the same scene, except that the method is invoked ngx_http_upstream_init_round_robin, processing logic implicit go upstream portion.

Scene 6

2 the same scene.

Scene 7

1 the same scene.

Scene 8

In fact implicitly created upstream, but because proxy_pass specified IP and port number, so when ngx_http_upstream_init_round_robin initialization upstream, us-> servers is not empty, so take the upper part of the logic of the function. 1 is somewhat similar to the scene.