Do not open simultaneously tcp_timestamps scene description and tcp_tw_recycle
Under FULL NAT
When FULL NAT VIP client request, not only to replace the dst ip package, but also the replacement of src ip package; VIP but also back to the client replaced src ip
lvs for the back-end web server.
If the web server opens tcp of tcp_timestamps and tcp_tw_recycle these two parameters. Then there is the following situation
RFC1323 is described in the following paragraph:
An additional mechanism could be added to the TCP, a per-hostcache of the last timestamp received from any connection.This value could then be used in the PAWS mechanism to rejectold duplicate segments from earlier incarnations of theconnection, if the timestamp clock can be guaranteed to haveticked at least once since the old connection was open. Thiswould require that the TIME-WAIT delay plus the RTT togethermust be at least one tick of the sender’s timestamp clock.Such an extension is not part of the proposal of this RFC.
Probably means that there is a TCP behavior, can cache the latest timestamp for each connection, the subsequent request if the time stamp is less than the cached timestamp, i.e. as invalid, the corresponding data packet is dropped. Means that the same source is connected to the same IP packet destination port must be a timestamp increment
Linux enable this behavior depends on whether tcp_timestamps and tcp_tw_recycle, because tcp_timestamps is turned on by default, so when tcp_tw_recycle is turned on, in fact, this behavior is activated.
Many companies are now doing with LVS load balancing, usually in front of a LVS, behind multiple back-end servers, which is actually NAT, LVS when the request arrives, it is forwarded to the back-end server to modify data after the address,
but will not modification timestamp data for the back-end server, the source address of the request is LVS address, plus the web port will be multiplexed, so from the perspective of looking back-end server, the client's original request a different forwarded by LVS, and
it is possible it will be considered the same connection, in addition to time different clients may be inconsistent, so there will be a time stamp disorder phenomenon, so the back of the packet is discarded, the
specific performance usually is obviously sent by a client SYN, but the server is not responding ACK, acknowledgment packets may be dropped continuously phenomenon following command
If LVS address is 222.222.222.222, the name is L
web server address is 111.111.111.111, the name is W
Address of a client such as the client C1 to the address 100.100.100.101, 100.100.100.102 address of the client C2
If this happens
13:23:02 this point in time
TCP packets sent by the source IP C1 and source port 100.100.100.101:6332, destination address and port are 222.222.222.222:80
13:23:05 this point in time
TCP packets sent by the source IP C2 and source port 100.100.100.102:52223, destination address and port are 222.222.222.222:80
After a full nat LVS.
If C2 LVS received packets at time 13:23:06
C2 is converted into data packets
222.222.222.222:52223 ---->111.111.111.111:80
If C1 LVS received packets in time 13:23:07
C1 is converted into data packets
222.222.222.222:6332 ---->111.111.111.111:80
If the web server opens tcp of tcp_timestamps and tcp_tw_recycle these two parameters. The web server time stamp packet
Since the timestamp of the packet C1 is less than the current system time stamp registered with this source IP connection, the data is considered to be retransmitted, the data packet is C1 discarded
(Not only the full nat lvs cases occur source address is replaced, and some Layer 3 switches as load balancing will replace the source address)
This parameter is the default 2.4 kernel is disabled. We can not open, do not open
important point
1. tw_reuse, tw_recycle must be turned on when the tube (by default) on the client and server timestamps, in fact, means that if the server and client sides have not turned aside timestamps. tw_reuse and tw_recycle are nothing role
2. tw_reuse only customer reach for the role, after opening the client recovered within 1s. reuse is the reuse time_wait socket connection. Server with a port to be connected is theoretically no limit.
3. tw_recycle client and the server operate simultaneously, after opening recovered in 3.5 * RTO, RTO 200ms ~ 120s specific time depending on network conditions.
Slightly faster than the network status tw_reuse, most of the public mobile network in particular slower than tw_reuse, advantage is capable of recovering the number of server TIME_WAIT
For the client
1 as a client because of the port 65535 question, TIME_OUT too much direct impact on processing power, open tw_reuse can be solved, not recommended to open at the same tw_recycle, not very helpful.
2. tw_reuse help recovery client 1s complete the connection, can achieve a single base 6w / s request, then another job requires it to increase the number of IP.
3. If the internal pressure measurement scenario, the client does not receive a connection and, at the same time there will be little advantage tw_recycle.
For server
1. Open tw_reuse invalid because the client is connected to the web server, the server will certainly not take the initiative to reuse the socket connecting the client. This parameter is generally less than the server, unless the web server and as a client to connect to the back-end database was used.
But when the web server as a client to connect to the database limit is reached 60,000 early port of your database can not withstand the pressure paralyzed. 5000 general database connections has been very high.
tw_resue this parameter is only used to get clients. Reuse means that in time_wait socket connection.
2. Do not open the online environment tw_recycle
after the server is behind a NAT load, or the client is in NAT (which is a certain thing, basic home networking companies have gone NAT);
public service may cause some to open the connection fails, then the network depending upon the situation to be open;
some load balancing device will timestamp gave empty, open the back-end web server does not open tw_recycle does not matter.
3. The server how to do high TIME_WAIT
Server time_wait do not worry, because I am a server, client IP and port number of a port to connect my initiative, such as my connection port 80. It may be a case that although there are on my machine 100 000 time_wait connection. But I only used a 80-port port.
Unlike the client has a port restrictions, handling a large number of TIME_WAIT Linux has been optimized very good, very few connections in each memory consumption in TIME_WAIT state,
but also through tcp_max_tw_buckets = 262144 configure the maximum limit, modern machines generally do not lack this memory .
In short, the production, with or without server behind nat device.
tcp_tw_recycle not open on the line. The default is not turned on, a value of 0
tcp_timestamps remain enabled by default on the line, a value of 1
tcp_tw_reuse. The client is best to open. When connected to a web server load balancing equipment, auxiliary equipment also try to open the equalizer
About a large number of server-side time_wait, some people will ask, I am a web server, why is there a client that time_wait.
In fact, on time_wait, it appears that the initiative to request for a closed connection. Http server closes the connection. It transitions to the client.
Disconnect initiated this action, does not mean that it must be initiated by the client disconnected. Most of the time the server to initiate a disconnect operation. For example, many http server, short connection. Many times server initiative calls.
Service tcp connection problems can occur under the following first check to see if there are many, many times, opened a lead of tcp_tw_recycle
|
1
2
3
|
[game@localhost ~]$
netstat
-s |
grep
timestamp
351 packets rejects
in
established connections because of timestamp
[game@localhost ~]$
|
Client-related optimization of place as follows. (Server can not open tcp_tw_recycle)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[root@B ~]
# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_timestamps
1
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_tw_recycle
0
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_tw_reuse
0
[root@B ~]
# echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_tw_reuse
1
[root@B ~]
# echo "10240 62000" > /proc/sys/net/ipv4/ip_local_port_range
[root@B ~]
# cat /proc/sys/net/ipv4/ip_local_port_range
10240 62000
[root@B ~]
#
|
Do not open simultaneously tcp_timestamps scene description and tcp_tw_recycle
Under FULL NAT
When FULL NAT VIP client request, not only to replace the dst ip package, but also the replacement of src ip package; VIP but also back to the client replaced src ip
lvs for the back-end web server.
If the web server opens tcp of tcp_timestamps and tcp_tw_recycle these two parameters. Then there is the following situation
RFC1323 is described in the following paragraph:
An additional mechanism could be added to the TCP, a per-hostcache of the last timestamp received from any connection.This value could then be used in the PAWS mechanism to rejectold duplicate segments from earlier incarnations of theconnection, if the timestamp clock can be guaranteed to haveticked at least once since the old connection was open. Thiswould require that the TIME-WAIT delay plus the RTT togethermust be at least one tick of the sender’s timestamp clock.Such an extension is not part of the proposal of this RFC.
Probably means that there is a TCP behavior, can cache the latest timestamp for each connection, the subsequent request if the time stamp is less than the cached timestamp, i.e. as invalid, the corresponding data packet is dropped. Means that the same source is connected to the same IP packet destination port must be a timestamp increment
Linux enable this behavior depends on whether tcp_timestamps and tcp_tw_recycle, because tcp_timestamps is turned on by default, so when tcp_tw_recycle is turned on, in fact, this behavior is activated.
Many companies are now doing with LVS load balancing, usually in front of a LVS, behind multiple back-end servers, which is actually NAT, LVS when the request arrives, it is forwarded to the back-end server to modify data after the address,
but will not modification timestamp data for the back-end server, the source address of the request is LVS address, plus the web port will be multiplexed, so from the perspective of looking back-end server, the client's original request a different forwarded by LVS, and
it is possible it will be considered the same connection, in addition to time different clients may be inconsistent, so there will be a time stamp disorder phenomenon, so the back of the packet is discarded, the
specific performance usually is obviously sent by a client SYN, but the server is not responding ACK, acknowledgment packets may be dropped continuously phenomenon following command
If LVS address is 222.222.222.222, the name is L
web server address is 111.111.111.111, the name is W
Address of a client such as the client C1 to the address 100.100.100.101, 100.100.100.102 address of the client C2
If this happens
13:23:02 this point in time
TCP packets sent by the source IP C1 and source port 100.100.100.101:6332, destination address and port are 222.222.222.222:80
13:23:05 this point in time
TCP packets sent by the source IP C2 and source port 100.100.100.102:52223, destination address and port are 222.222.222.222:80
After a full nat LVS.
If C2 LVS received packets at time 13:23:06
C2 is converted into data packets
222.222.222.222:52223 ---->111.111.111.111:80
If C1 LVS received packets in time 13:23:07
C1 is converted into data packets
222.222.222.222:6332 ---->111.111.111.111:80
If the web server opens tcp of tcp_timestamps and tcp_tw_recycle these two parameters. The web server time stamp packet
Since the timestamp of the packet C1 is less than the current system time stamp registered with this source IP connection, the data is considered to be retransmitted, the data packet is C1 discarded
(Not only the full nat lvs cases occur source address is replaced, and some Layer 3 switches as load balancing will replace the source address)
This parameter is the default 2.4 kernel is disabled. We can not open, do not open
important point
1. tw_reuse, tw_recycle must be turned on when the tube (by default) on the client and server timestamps, in fact, means that if the server and client sides have not turned aside timestamps. tw_reuse and tw_recycle are nothing role
2. tw_reuse only customer reach for the role, after opening the client recovered within 1s. reuse is the reuse time_wait socket connection. Server with a port to be connected is theoretically no limit.
3. tw_recycle client and the server operate simultaneously, after opening recovered in 3.5 * RTO, RTO 200ms ~ 120s specific time depending on network conditions.
Slightly faster than the network status tw_reuse, most of the public mobile network in particular slower than tw_reuse, advantage is capable of recovering the number of server TIME_WAIT
For the client
1 as a client because of the port 65535 question, TIME_OUT too much direct impact on processing power, open tw_reuse can be solved, not recommended to open at the same tw_recycle, not very helpful.
2. tw_reuse help recovery client 1s complete the connection, can achieve a single base 6w / s request, then another job requires it to increase the number of IP.
3. If the internal pressure measurement scenario, the client does not receive a connection and, at the same time there will be little advantage tw_recycle.
For server
1. Open tw_reuse invalid because the client is connected to the web server, the server will certainly not take the initiative to reuse the socket connecting the client. This parameter is generally less than the server, unless the web server and as a client to connect to the back-end database was used.
But when the web server as a client to connect to the database limit is reached 60,000 early port of your database can not withstand the pressure paralyzed. 5000 general database connections has been very high.
tw_resue this parameter is only used to get clients. Reuse means that in time_wait socket connection.
2. Do not open the online environment tw_recycle
after the server is behind a NAT load, or the client is in NAT (which is a certain thing, basic home networking companies have gone NAT);
public service may cause some to open the connection fails, then the network depending upon the situation to be open;
some load balancing device will timestamp gave empty, open the back-end web server does not open tw_recycle does not matter.
3. The server how to do high TIME_WAIT
Server time_wait do not worry, because I am a server, client IP and port number of a port to connect my initiative, such as my connection port 80. It may be a case that although there are on my machine 100 000 time_wait connection. But I only used a 80-port port.
Unlike the client has a port restrictions, handling a large number of TIME_WAIT Linux has been optimized very good, very few connections in each memory consumption in TIME_WAIT state,
but also through tcp_max_tw_buckets = 262144 configure the maximum limit, modern machines generally do not lack this memory .
In short, the production, with or without server behind nat device.
tcp_tw_recycle not open on the line. The default is not turned on, a value of 0
tcp_timestamps remain enabled by default on the line, a value of 1
tcp_tw_reuse. The client is best to open. When connected to a web server load balancing equipment, auxiliary equipment also try to open the equalizer
About a large number of server-side time_wait, some people will ask, I am a web server, why is there a client that time_wait.
In fact, on time_wait, it appears that the initiative to request for a closed connection. Http server closes the connection. It transitions to the client.
Disconnect initiated this action, does not mean that it must be initiated by the client disconnected. Most of the time the server to initiate a disconnect operation. For example, many http server, short connection. Many times server initiative calls.
Service tcp connection problems can occur under the following first check to see if there are many, many times, opened a lead of tcp_tw_recycle
|
1
2
3
|
[game@localhost ~]$
netstat
-s |
grep
timestamp
351 packets rejects
in
established connections because of timestamp
[game@localhost ~]$
|
Client-related optimization of place as follows. (Server can not open tcp_tw_recycle)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[root@B ~]
# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_timestamps
1
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_tw_recycle
0
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_tw_reuse
0
[root@B ~]
# echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
[root@B ~]
# cat /proc/sys/net/ipv4/tcp_tw_reuse
1
[root@B ~]
# echo "10240 62000" > /proc/sys/net/ipv4/ip_local_port_range
[root@B ~]
# cat /proc/sys/net/ipv4/ip_local_port_range
10240 62000
[root@B ~]
#
|