DNS Forwarder in-depth study

I recently dealt with a Windows DNS Forwarder failure in a production environment. The failure is probably like this:
As a global company A, company A has deployed a set of DNS systems in Chengdu and Taiwan in the Asia-Pacific region at the same time, and refer to each other's DNS Forwarder. In order to play the role of DNS redundancy, that is, when the internal DNS server of one place is down, the backup DNS of another place can be used to provide normal domain name resolution for internal users.

Trouble phenomenon:

Due to geographical factors, when the DNS of the two places resolves public domain names, such as www.baidu.com, the IP in Taiwan will be resolved to Hong Kong, and the IP in Beijing will be resolved in Chengdu, and there will be two DNSs. When the server is alive at the same time, there will be parsing errors, resulting in very slow access to www.baidu.com (accessing Baidu Hong Kong site from Chengdu, it is strange that it is not slow).

Cause Analysis:

In the DNS service update of Windows Server 2012 R2, there is a feature update called Dynamic DNS Forwarders. The official description is as follows:

在Windows Server 2012 R2的DNS服务器的设置中添加多个转发器时，
DNS服务会根据列表中每个服务器的响应时间对转发器列表中的服务器列表进行重新排序。
默认情况下，Windows Server 2012 R2中启用了重新排序和响应检查操作。
如果希望禁用此功能，则需要将下列注册表DWORD值更改为0：
HKLM\System\CurrentControlSet\Services\DNS\Parameters\EnableForwarderReordering

Obviously, while the two places refer to each other as Forwarder, the two DNS servers have re-prioritized due to the fast or slow response time.

To verify this, we created a DNS Server in the test environment and added three DNS Forwarder records,

In the first DNS Server 10.16.75.52, the local DNS Zone of 163.com is added, and two A records of www.163.com and ffff.163.com are added. Other domain names cannot be resolved.
The other two can resolve external domain names normally. (Remember the current Forwarder priority)
Then we go to the client to perform a domain name resolution test:

After analyzing 4 records in a row, go back to the DNS Server to see if the Forwarder priority has changed:

Solution:

Disable Dynamic DNS Forwarders, change the value of HKLM\System\CurrentControlSet\Services\DNS\Parameters\EnableForwarderReordering
to 0 (if there is no such value, create it)
or execute the command directly:
Set-DnsServerForwarder -EnableReordering $false

further research:

1. Function of Number of seconds befor forward queries time out option in DNS Forwarder
   

In order to facilitate packet capture, we change the value to 7s, and then request the www.baidu.com domain name on the client side:

It can be seen that the client 10.16.75.55 initiates a DNS resolution request to DNS Server 10.16.75.50, requesting to resolve the www.baidu.com domain name, 10.16.75.50 forwards the request to the first Forwarder server, and waits for a reply within 7s, and after 7s Immediately forward the request to the second forwarder server 10.16.42.8, and then 10.16.42.8 quickly responded to the analysis result. This change in response time will cause the DNS Forwarder to reorder.
Look at the results:

How appropriate is the time set for this value?
The default value is 3s. Generally, you can keep the default value. If the network condition is not good, you can increase it appropriately. The better the network status, the lower it can be to optimize the access speed.
Reference link:
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn305898(v=ws.11 )
https:/ /docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc772774(v=ws.10)?redirectedfrom=MSDN

2. What is the situation of DNS Forwarder under Linux?
   Linux configuration DNS is configured in the resolv.conf file. The description of the nameserver part in the file is as follows, that is, it is resolved in order. The default timeout time is 5s, and the longest can be set up to 30s:

nameserver Name server IP address
              Internet address of a name server that the resolver should
              query, either an IPv4 address (in dot notation), or an
              IPv6 address in colon (and possibly dot) notation as per
              RFC 2373.  Up to MAXNS (currently 3, see <resolv.h>) name
              servers may be listed, one per keyword.  If there are
              multiple servers, the resolver library queries them in the
              order listed.  If no nameserver entries are present, the
              default is to use the name server on the local machine.
              (The algorithm used is to try a name server, and if the
              query times out, try the next, until out of name servers,
              then repeat trying all the name servers until a maximum
              number of retries are made.)

options
              Options allows certain internal resolver variables to be
              modified.  The syntax is
                     options option

              where option is one of the following:

timeout:n
                     Sets the amount of time the resolver will wait for
                     a response from a remote name server before
                     retrying the query via a different name server.
                     This may not be the total time taken by any
                     resolver API call and there is no guarantee that a
                     single resolver API call maps to a single timeout.
                     Measured in seconds, the default is RES_TIMEOUT
                     (currently 5, see <resolv.h>).  The value for this
                     option is silently capped to 30.

In order to verify this experiment, we also captured packets under Linux for analysis and proved that the switching time is indeed 5s.

Pure technical research, please correct me if there are any errors.