I saw this article online and reprinted it here to record it. Pitfall memo---server_name and ip of Nginx reverse proxy | CQ's notes
background
Have you ever encountered 502 Bad Gateway when using Nginx to reverse proxy a website? It seems that there is no problem with normal reverse proxy, but 502 Bad Gateway appears when reverse proxy. Check the error log and it will show:
*6565 SSL_do_handshake() failed (SSL: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream
Preliminary research on the problem found that because the website has SNI enabled, Nginx does not add SNI proxy_ssl_server_name on; by default , and Nginx cannot successfully handshake the upstream SSL, resulting in 502 Bad Gateway.
proxy_ssl_server_name
I was writing a management backend recently. When I was referring to Alibaba Cloud CDNinteraction, one of the parameter items called back-to-origin SNI caught my attention.
Although I know about SNI and Nginx upstream also uses HTTPS at work, I have not paid attention to the need for special configuration. I feel that it may trigger a knowledge blind spot. After some inquiries, I found that Nginx has a parameter similar to proxy_ssl_server_namethis.
Syntax: proxy_ssl_server_name on | off;
Default:
proxy_ssl_server_name off;
Context: http, server, location
This directive appeared in version 1.7.0.
Enables or disables passing of the server name through TLS Server Name Indication extension (SNI, RFC 6066) when establishing a connection with the proxied HTTPS server.
When the end server is HTTPS, SNIthe default value is whether to enable reverse proxy off. It is very common for one IP to bind multiple domain names. Why is it not enabled by default?
I felt that just reading the documentation was not enough, so I also looked at the source code. SNI processing must call SSL_set_tlsext_host_namea function, and searching for SSL_set_tlsext_host_namethe function can quickly locate the relevant logic.
The parameter is turned off by default. When turned on, it will be passed to the upstream server during the SSL handshake HostNameso that the upstream server knows which certificate to use.
Now that you know how to use it, let’s test it.
verify
I will use my own blog as an upstream server for testing. The topology is as follows:
-> www.dianduidian.com -> blog.dianduidian.com
Use the simplest configuration first
server {
listen 80;
server_name www.dianduidian.com;
location / {
proxy_pass https://blog.dianduidian.com;
}
}
No problem, it can be opened normally. Let's capture the packet to see how Nginx establishes a connection with the upstream server.
Client HelloIt can be seen that during the TLS handshake phase, Nginx does not include SNI information when sending messages to the upstream server . This confirms that when Nginx reverse proxy is used to request the upstream server using HTTPS, it is not enabled by defaultSNI .
Let’s continue to look at the certificate information returned by the upstream server
You can see that the server returns not the certificate of the domain name blog.dianduidian.com but a default certificate configured by nginx. This is because SNI is not enabled , so during the TLS handshake, the upstream server does not know which domain name certificate to use and uses the default certificate. Certificate returned.
Although the certificate returned is incorrect, the request is not affected. We speculate that when Nginx reverse proxy is used to request the upstream server via HTTPS, the certificate returned by the upstream server is not verified by default.
By consulting the documentation, I learned that the upstream certificate is not verified by default.
Syntax: proxy_ssl_verify on | off;
Default:
proxy_ssl_verify off;
Context: http, server, location
This directive appeared in version 1.7.0.
Enables or disables verification of the proxied HTTPS server certificate.
Certificate status is not verified by default.
So let's enable it and see what happens?
Get the file first CA,
curl https://curl.se/ca/cacert.pem -o /etc/nginx/conf.d/cacert.pem
Modify the configuration file as follows:
server {
listen 80;
server_name www.dianduidian.com;
location / {
proxy_pass https://blog.dianduidian.com;
proxy_ssl_verify on;
proxy_ssl_trusted_certificate /etc/nginx/conf.d/cacert.pem;
}
}
502, check the log and report upstream SSL certificate verify erroran error.
2022/05/18 16:52:14 [error] 20325#1337445: *23 upstream SSL certificate verify error: (18:self signed certificate) while SSL handshaking to upstream, client: 127.0.0.1, server: sni.dianduidian.com, request: "GET / HTTP/1.1", upstream: "https://47.100.x.x:443/", host: "sni.dianduidian.com"
You can see that after enabling upstream certificate verification, Nginx will indeed certify the certificate returned by the upstream. However, through the packet capture above, we can see that the upstream returned a default certificate configured by Nginx. This certificate was self-signed by us, and the CA verification passed naturally. No, but if the CA is legitimate, CommonNamewhat will happen if the certificate is inconsistent? Will it not be verified? The test here requires multiple legal certificates, which is troublesome. We will find the answer directly from the source code.
static void
ngx_http_upstream_ssl_handshake(ngx_http_request_t *r, ngx_http_upstream_t *u,
ngx_connection_t *c)
{
long rc;
if (c->ssl->handshaked) {
if (u->conf->ssl_verify) {
rc = SSL_get_verify_result(c->ssl->connection);
if (rc != X509_V_OK) {
ngx_log_error(NGX_LOG_ERR, c->log, 0,
"upstream SSL certificate verify error: (%l:%s)",
rc, X509_verify_cert_error_string(rc));
goto failed;
}
if (ngx_ssl_check_host(c, &u->ssl_name) != NGX_OK) {
ngx_log_error(NGX_LOG_ERR, c->log, 0,
"upstream SSL certificate does not match \"%V\"",
&u->ssl_name);
goto failed;
}
}
c->write->handler = ngx_http_upstream_handler;
c->read->handler = ngx_http_upstream_handler;
ngx_http_upstream_send_request(r, u, 1);
return;
}
if (c->write->timedout) {
ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_TIMEOUT);
return;
}
failed:
ngx_http_upstream_next(r, u, NGX_HTTP_UPSTREAM_FT_ERROR);
}
It can be seen from the source code that after turning on the upstream certificate verification, it will not only verify the legitimacy of the certificate issuing authority, but also compare the certificates , that is, the consistency of commonNamethe verificationcommonName .
Open it belowSNI
server {
listen 80;
server_name www.dianduidian.com;
location / {
proxy_pass https://blog.dianduidian.com;
proxy_ssl_verify on;
proxy_ssl_trusted_certificate /etc/nginx/conf.d/cacert.pem;
proxy_ssl_server_name on;
}
}
The website opens normally, this will capture the packets and take a look
You can see that during the TLS handshake phase, Nginx Client Hellohas an extra extension in the information sent to the upstream server server_name, telling the upstream server which certificate information should be used to respond. Server NameYou can see that what is passed here is blog.dianduidian.com, but in the above configuration we did not specify which domain name to pass. What is the logic of this piece?
After consulting the documentation, I learned that there is actually another parameter proxy_ssl_namecontrol
Syntax: proxy_ssl_name name;
Default:
proxy_ssl_name $proxy_host;
Context: http, server, location
This directive appeared in version 1.7.0.
Allows overriding the server name used to verify the certificate of the proxied HTTPS server and to be passed through SNI when establishing a connection with the proxied HTTPS server.
By default, the host part of the proxy_pass URL is used.
You can see that the default is $proxy_host, let’s try to modify it.
server {
listen 80;
server_name www.dianduidian.com;
location / {
proxy_pass https://blog.dianduidian.com;
proxy_ssl_verify on;
proxy_ssl_trusted_certificate /etc/nginx/conf.d/cacert.pem;
proxy_ssl_server_name on;
proxy_ssl_name www.baidu.com;
}
}
Grab the packet and take a look
You can see Server Namethat it has been changed to www.baidu.com.
Example
Example 1
http {
upstream bff-app {
server my-bff.azurewebsites.net:443;
}
server {
listen 80 default_server;
location /api {
proxy_pass https:/bff-app;
proxy_ssl_server_name on;
# Manually set Host header to "my-bff.azurewebsites.net",
# otherwise it will default to "bff-app".
proxy_set_header Host my-bff.azurewebsites.net;
}
}
}Example 2
upstream abtest_management_api_backend {
server 域名:443;
}
location ^~ /modules/abm/ {
proxy_ssl_server_name on;
proxy_ssl_name 域名;
proxy_set_header Host 域名;
proxy_pass https://abtest_management_api_backend/modules/abm/;
proxy_read_timeout 1800s;
proxy_set_header Origanization-Id qiancheng;
proxy_set_header X-Real-IP $clientRealIp;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header X-Accel-Buffering;
}
Summarize
When Nginx acts as a reverse proxy and establishes a connection with the upstream server using HTTPS,
- Not enabled by default
SNI,proxy_ssl_server_name on;enable with parameters; - By default, the certificate returned by the upstream server is not verified, use
proxy_ssl_verify on; - After upstream certificate verification is enabled, Nginx will use the CA specified in the configuration file to verify the legitimacy of the certificate returned by the upstream server, and will also compare the CommonName information in the certificate.
reference
Nginx reverse proxy https site 502 troubleshooting ideas | One small step
c - How to implement Server Name Indication (SNI) - Stack Overflow
ssl - How to update cURL CA bundle on RedHat? - Server Fault